16:00:22 <mkonecny> #startmeeting Infrastructure (2021-01-21)
16:00:22 <zodbot> Meeting started Thu Jan 21 16:00:22 2021 UTC.
16:00:22 <zodbot> This meeting is logged and archived in a public location.
16:00:22 <zodbot> The chair is mkonecny. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:22 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:00:22 <zodbot> The meeting name has been set to 'infrastructure_(2021-01-21)'
16:00:22 <mkonecny> #meetingname infrastructure
16:00:22 <mkonecny> #chair nirik pingou smooge cverna mizdebsk mkonecny abompard siddharthvipul mobrien
16:00:22 <mkonecny> #info Agenda is at: https://board.net/p/fedora-infra
16:00:22 <mkonecny> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
16:00:22 <zodbot> The meeting name has been set to 'infrastructure'
16:00:22 <zodbot> Current chairs: abompard cverna mizdebsk mkonecny mobrien nirik pingou siddharthvipul smooge
16:00:23 <mkonecny> #topic aloha
16:00:45 <mkonecny> Welcome everyone to our weekly show
16:00:47 <siddharthvipul> good morning :D
16:00:51 <nirik> morning everyone.
16:01:02 <seddik> hi everone :)
16:01:03 <pingou> ó/
16:01:04 <bodanel> Hi
16:01:11 <mkonecny> Today we will look at the best of the best in Fedora Infrastructure
16:01:36 <mkonecny> So let's start with welcoming the new faces
16:01:42 <mkonecny> #topic New folks introductions
16:01:42 <mkonecny> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
16:01:42 <mkonecny> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
16:01:45 <mobrien> .hi
16:01:46 <zodbot> mobrien: mobrien 'Mark O'Brien' <markobri@redhat.com>
16:01:52 <siddharthvipul> .hello siddharthvipul1
16:01:53 <zodbot> siddharthvipul: siddharthvipul1 'Vipul Siddharth' <siddharthvipul1@gmail.com>
16:01:55 <mkonecny> Do we have anybody new around?
16:02:00 <bodanel> HI, yes me
16:02:07 <mobrien> welcome bodanel
16:02:39 <bodanel> I'm Bogdan Benea, I've sent the meeting agenda item for last week but I didnt managed to enter the meeting last week
16:02:51 <bodanel> on the wiki page the meeting is at 14:00 UTC
16:02:58 <austinpowered> .hello2
16:02:59 <zodbot> austinpowered: austinpowered 'T.C. Williams' <fedoraproject@wootenwilliams.com>
16:03:06 <bodanel> so perhap we should update that
16:03:09 <mkonecny> bodanel: Welcome to Fedora Infrastructure
16:03:14 <bodanel> thks
16:03:41 <mkonecny> It's true that we are ignoring the wiki most of the time
16:03:51 <smooge> .time
16:03:51 <zodbot> smooge: 04:03 PM, January 21, 2021
16:04:00 <nirik> Where Infromation Kills Itself
16:04:11 <mobrien> haha
16:04:15 <bodanel> If you wish I can take the task of updating the wiki
16:04:38 <bodanel> Since I'm just starting it will be a good point to check and update the info
16:05:00 <nirik> sure! althought you will need to be in our apprentice group or be added to wikiedit or some other group to have access. :)
16:05:03 <smooge> fixed
16:05:21 <mkonecny> smooge: You are fast
16:05:31 <bodanel> about aprentice, that's the first step if I understood corectly
16:05:33 <smooge> oh sorry and this is where i hurt our process of getting people involved. my apologies bodanel
16:05:42 <smooge> I will add you and you can edit that page
16:05:44 <bodanel> dont worry
16:06:23 <smooge> what is your fas account?
16:06:27 <bodanel> bodanel
16:07:40 <mkonecny> Anyone else new here?
16:08:57 <mkonecny> It seems not, so let us go to next item
16:09:16 <mkonecny> Who will be the winner of next chair?!
16:09:26 <mkonecny> #topic Next chair
16:09:26 <mkonecny> #info magic eight ball says:
16:09:26 <mkonecny> #info chair 2021-01-14 - mobrien
16:09:26 <mkonecny> #info chair 2021-01-21 - mkonecny
16:09:26 <mkonecny> #info chair 2021-01-28 - pingou
16:09:27 <mkonecny> #info chair 2021-02-04 - ???
16:09:38 <siddharthvipul> you can mark me
16:09:51 <mkonecny> It seems that next chair is actually taken, congrats pingou
16:09:52 <siddharthvipul> if no one else is interested of course
16:10:10 <mkonecny> siddharthvipul: You just one the chair for 2021-02-04
16:10:19 <siddharthvipul> mkonecny, I will take it :)
16:10:19 <mkonecny> s/one/won
16:10:39 <mkonecny> #info chair 2021-02-04 - siddhartvipul
16:10:43 <siddharthvipul> just make sure nothing major happens in that week
16:10:53 <siddharthvipul> no updates, no deployments
16:11:20 <siddharthvipul> mobrien, capisce?
16:11:24 <siddharthvipul> :P
16:11:31 <mobrien> no dice
16:11:34 <mkonecny> I can't guarantee anything
16:11:40 <siddharthvipul> darn it
16:11:47 <mobrien> its just chair not oncall
16:11:53 <siddharthvipul> oooo
16:11:54 <siddharthvipul> lol
16:11:57 <siddharthvipul> oh I love that
16:11:59 <mkonecny> Let's look at the news
16:12:00 <siddharthvipul> sorry I was confused
16:12:01 <mkonecny> #topic announcements and information
16:12:01 <mkonecny> #info CPE Infra&Releng EU-hours team has a Monday through Friday 30 minute meeting going through tickets at 1030 Europe/paris in #centos-meeting
16:12:01 <mkonecny> #info CPE Infra&Releng NA-hours team has a Monday through Friday 30 minute meeting going through tickets at 1800 UTC in #fedora-admin
16:12:01 <mkonecny> #info Datacenter move is over, but some items still need to be done: see https://fedoraproject.org/wiki/Infrastructure/2020-post-datacenter-move-known-issues
16:12:16 <nirik> #info mass rebuild likely pushed to next monday
16:12:36 <nirik> #info all builders are now f33, but armv7 builders are slow/have some issues left to investigate
16:13:05 <nirik> #info some progress in stg for ssh/sudo with noggin (new account system)
16:13:21 <mkonecny> #info Anitya (https://release-monitoring.org/) 1.0.0 was released, currently running in staging (https://stg.release-monitoring.org/), plan to deploy on production on Friday
16:14:07 <nirik> mkonecny: sorry that caching was so hard to figure... I wonder if we want to look at our other apps or document that somewhere (but not sure where)
16:14:12 <mkonecny> Hm, the agenda is currently down :-/
16:14:59 <mobrien> mkonecny, that happened me last week too, went down for a couple of mins
16:15:05 <mkonecny> nirik: I want to send e-mail tomorrow, I have Captain log for the issue
16:15:30 <mkonecny> I started taking them when I troubleshoot something
16:15:35 <mobrien> mkonecny, if it doesn't reload I can take over the meeting
16:15:50 <mkonecny> It reloaded now
16:16:05 <mkonecny> Do we have anything else we want to announce?
16:16:56 <mkonecny> If not, I will take us to the wonderful land of Oncall
16:16:58 <mkonecny> #topic Oncall
16:16:58 <mkonecny> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
16:16:58 <mkonecny> #info mobrien is oncall for 2021-01-07 to 2021-01-14
16:16:58 <mkonecny> #info siddharthvipul1 is oncall for 2021-01-14 to 2021-01-21
16:16:58 <mkonecny> #info pingou is oncall for 2021-01-21 to 2021-01-28
16:16:59 <mkonecny> #info ??? is oncall for 2021-01-28 to 2021-02-04
16:17:23 <mkonecny> Who wants to be on a call for a whole week?
16:17:53 <smooge> I will take it
16:18:14 <mkonecny> smooge first
16:18:16 <smooge> I will take the ring to Mt Doom
16:18:20 <mkonecny> smooge second
16:18:23 <mkonecny> smooge third
16:18:27 <mkonecny> and sold!
16:18:45 <mkonecny> smooge: You are now the proud owner of oncall for one week :-)
16:19:06 <smooge> wait I was looking for a ring of power
16:19:15 <mkonecny> #info smooge is oncall for 2021-01-28 to 2021-02-04
16:19:38 <mkonecny> #info Summary of last week: (from current oncall )
16:19:40 <nirik> you have my AXE. wait, my neighbors axe!
16:19:50 <siddharthvipul> hey
16:19:54 <mkonecny> siddharthvipul: The floor is yours
16:20:05 <siddharthvipul> so there wasn't much.. I got 2 pings (unless I missed any over the weekends)
16:20:22 <siddharthvipul> one was about koji where mobrien, mboddu and I were on call (I was shadowing)
16:20:42 <siddharthvipul> kojira was not active.. that fixed koji issue where it was not picking builds
16:21:08 <siddharthvipul> and one today about nagios showing wrong notification about copr being down
16:21:15 <siddharthvipul> I requested a ticket as there was nothing urgent
16:21:21 * siddharthvipul fetches the link
16:21:24 <nirik> siddharthvipul++
16:21:37 <mkonecny> siddharthvipul: Nicely done :-)
16:21:52 <siddharthvipul> https://pagure.io/fedora-infrastructure/issue/9593
16:21:56 <siddharthvipul> this is already fixed :)
16:22:00 <mobrien> siddharthvipul++
16:22:02 <siddharthvipul> mkonecny, nirik thank you :D
16:22:08 <siddharthvipul> mobrien, ^-^
16:23:07 <mkonecny> So let's look what is happening in the world of Fedora Infra
16:23:11 <smooge> thanks for getting that ticket in on copr. I was able to focus on it this morning
16:23:21 <mkonecny> #topic Monitoring discussion [nirik]
16:23:22 <mkonecny> #info https://nagios.fedoraproject.org/nagios
16:23:22 <mkonecny> #info Go over existing out items and fix
16:23:41 <nirik> lets see
16:24:10 <nirik> we have 2 down machines... one aarch64 emag (lost a disk) and one power8 box (lost it's mind)
16:24:55 <nirik> sorry, 3 machines... and a aarch64 mustang that dropped off and we need to power cycle but can't yet.
16:25:20 <nirik> other than those... 5 alerts:
16:25:40 <nirik> * oci-registry01 is low on disk space. I think we need to run the manual playbook that cleans up old containers on it?
16:26:10 <nirik> * busgateway hasn't see a faf message in forever. we need to talk to the retrace team and see if this is even something they are sending anymore.
16:26:47 <nirik> * bvmhost-x86-08.iad2.fedoraproject.org was miscreated and has no swap, we could try and remove that check, or try and reinstall it with some swap
16:27:09 <nirik> * retrace-stg.aws.fedoraproject.org vpn is down - someone needs to debug / fix it.
16:27:31 <nirik> * sign-bridge01.stg.iad2.fedoraproject.org - signing server is down, needs someone to setup signing in stg again after the move.
16:27:43 <nirik> thats it. ;) So, if anyone wants to look into those things great.
16:28:06 <mkonecny> Plenty of things happening in our infra
16:28:12 <mkonecny> wow
16:28:29 <bodanel> I'm available for shadow any of the ta
16:28:35 <bodanel> those tasks
16:29:16 <nirik> Those have all been there a while... we just keep not getting to them. ;)
16:29:44 <bodanel> I'll have a look, perhaps I can help with any of them
16:30:07 <mkonecny> #topic Learning topic
16:30:22 <mkonecny> We don't have anything prepared, do we want to talk about something?
16:30:24 <nirik> yeah, or if anyone wants to start on them, please announce it so others can watch too
16:30:47 <nirik> mkonecny: I guess we should ask people to sign up like the oncall and meeting chair.
16:31:20 <mkonecny> Good idea
16:31:49 <mkonecny> So who wants to prepare learning topic for next week?
16:32:20 <mkonecny> It could be about anything infra related
16:32:49 <mkonecny> Could be some issue you struggled to solve, some system, something that could help infra
16:32:57 * nirik is usually happy to speak on anything if people say what they would like.
16:33:28 <mobrien> I wouldn't mind giving one but don't really know what to speak on
16:33:35 <bodanel> I would propose about the ansible setup you have
16:33:58 <mkonecny> Not bad topic
16:33:58 <austinpowered> +1 for ansible
16:34:01 <bodanel> I think it would be an easy start for someone new
16:34:29 <bodanel> I have some Ansible experience, I could pick up some Ansible related stuff, I suppose I'm not the only one
16:34:42 <mkonecny> Should we do it now or just add it to agenda for next week?
16:34:59 <bodanel> however you see fit
16:35:09 <mkonecny> nirik: Do you feel you can talk about it right now?
16:35:26 <nirik> I could, but next week might be better... lots of things going on this week.
16:35:36 <bodanel> next week it is
16:36:04 <mobrien> If you want I can do some/most of it next week and you can fill in the blanks nirik
16:36:19 <nirik> +1 sounds great to me!
16:36:35 <mkonecny> nirik: Ok
16:37:17 <mkonecny> #info ansible setup [nirik] on 2020-01-28
16:37:28 <nirik> and mobrien! :)
16:37:42 <mkonecny> Ok
16:37:55 <mobrien> :)
16:37:57 <mkonecny> #info ansible setup [nirik/mobrien] on 2020-01-28
16:38:23 <mkonecny> We could actually prepare more topics in advance if people here are interested
16:38:43 <mkonecny> For one I would be interested in kerberos deployment in Fedora Infra
16:38:54 <nirik> yeah, might be good to line up some more.
16:38:56 <mobrien> I think getting the topics is the hard part, people are generally willing to speak
16:39:10 <nirik> mkonecny: yeah, ipa setup ?
16:39:25 <mobrien> +1000
16:39:56 <bodanel> Since I'm new I will propose a topic each week, if you think the topic is good we can scheduled it for the next week
16:40:08 <mkonecny> nirik: If it means the SSO than yes, I'm actually not sure how the kerberos is used in Fedora, just interested
16:40:11 <pingou> bodanel++
16:40:11 <zodbot> pingou: Karma for bodanel changed to 1 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
16:40:16 <nirik> bodanel: great to have input from newcomers on topics!
16:40:25 <mobrien> something openshift related might be good
16:40:31 <mkonecny> bodanel++
16:40:31 <zodbot> mkonecny: Karma for bodanel changed to 2 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
16:40:34 <siddharthvipul> bodanel++
16:40:34 <zodbot> siddharthvipul: Karma for bodanel changed to 3 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
16:40:39 <nirik> mkonecny: sure, I can talk about that. I need to learn up myself and should be more up to speed in a few weeks.
16:40:43 <siddharthvipul> bodanel, that's a good idea
16:40:56 <mobrien> bodanel++
16:40:56 <zodbot> mobrien: Karma for bodanel changed to 4 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
16:41:18 <austinpowered> +1 openshift
16:41:39 <mkonecny> #info IPA [nirik] on 2020-02-04
16:41:47 <siddharthvipul> IPA++
16:42:15 <siddharthvipul> could someone also talk about new monitoring initiates planned? or too early?
16:42:17 <bodanel> Openshift it's a good ideea, I have some experience with that also so I can help in that part
16:42:20 <siddharthvipul> I mean teenager
16:42:22 <mkonecny> We already have openshift setup topic on 2020-09-17
16:42:33 <bodanel> ok, no worries
16:42:42 <mkonecny> Do we want to talk about something else openshift related?
16:43:01 <mkonecny> Like updating to OpenShift 4?
16:43:20 <nirik> well, we could, but it depends on us getting hardware and a bunch of things. ;)
16:43:25 <bodanel> mkonecny, meeting is stilla available archived ?
16:43:40 <mkonecny> bodanel: Yes they are archived
16:43:41 <siddharthvipul> bodanel, yess
16:43:43 <siddharthvipul> let me get the link
16:43:55 <nirik> also as a side note, people should sign up for our internal CPE tech talks if you are a member of that team. ;)
16:44:24 <mkonecny> I will add the topics to the agenda for CPE Tech talks
16:44:47 <austinpowered> Openshift was a learningtopic on 202-09-17
16:44:48 <siddharthvipul> bodanel, https://meetbot.fedoraproject.org/sresults/?group_id=infrastructure&type=team
16:45:05 <siddharthvipul> ^ link to get all infrastructure meeting logs
16:45:42 <bodanel> got it, thks
16:46:05 <siddharthvipul> nirik, about internal CPE talk, it's at 11-11:30 pm my time.. I am getting old and usually very sleepy by then
16:46:06 <siddharthvipul> :(
16:46:30 <nirik> siddharthvipul: ah yeah. perhaps we should alternate times or something.
16:46:31 <mkonecny> The same will be said here, so don't worry
16:46:32 <bodanel> :)
16:46:42 <mkonecny> siddharthvipul: And we should record them
16:46:54 <mobrien> I think they are recorded
16:46:54 <mkonecny> Maybe make them public later
16:46:55 <siddharthvipul> ++
16:47:04 * linuxmodder coming in late sorry
16:47:12 <siddharthvipul> linuxmodder, \o :D
16:47:19 <nirik> mkonecny: some of them we have been... but not sure how to make them public. we should look out of meeting for that
16:47:25 <mkonecny> mobrien: Not every one of them, sometimes we forget to push the REC button
16:47:49 <mkonecny> nirik: We could probably talk about this in mailing list
16:49:25 <mkonecny> I think this is everything we have related to learning topics, we now have topics for next two meetings
16:49:32 <mkonecny> \o/
16:49:50 <mkonecny> So let's dance
16:49:51 <mkonecny> #topic Open Floor
16:50:21 * mkonecny headbanging
16:50:29 <nirik> 💃💃
16:51:07 <mkonecny> Doom(2016) running in background here :-)
16:51:20 <mkonecny> * soundtrack
16:51:40 <mkonecny> So anything else people want to talk about?
16:51:50 <smooge> Sorry I can't hear you over the Imp screams.
16:51:58 <smooge> can you turn it down a bit?
16:52:15 <smooge> I have nothing to add beyond a flippant comment
16:52:29 * mkonecny turning volume down
16:52:35 <mkonecny> smooge: better?
16:52:55 <smooge> yes.. my eyes have stopped bleeding
16:53:06 <mkonecny> I want to thank everyone for joining today, this meeting was really productive :-)
16:53:13 <smooge> yes it was
16:53:27 <nirik> thanks for running it mkonecny!
16:53:36 <siddharthvipul> mkonecny++
16:53:41 <mobrien> thanks mkonecny
16:53:46 <mobrien> mkonecny++
16:54:08 <austinpowered> thanks all
16:54:09 <smooge> mkonecny++
16:54:15 <Southern_Gentlem> mkonecny++
16:54:15 <zodbot> Southern_Gentlem: Karma for zlopez changed to 7 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
16:54:43 <bodanel> thks everyone for your warm welcome
16:54:56 <mkonecny> Bye everyone, see you next meeting :-)
16:55:03 <mkonecny> #endmeeting