15:00:38 <pingou> #startmeeting Infrastructure (2020-09-03)
15:00:38 <zodbot> Meeting started Thu Sep  3 15:00:38 2020 UTC.
15:00:38 <zodbot> This meeting is logged and archived in a public location.
15:00:38 <zodbot> The chair is pingou. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:38 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:00:38 <zodbot> The meeting name has been set to 'infrastructure_(2020-09-03)'
15:00:39 <pingou> #meetingname infrastructure
15:00:39 <zodbot> The meeting name has been set to 'infrastructure'
15:00:41 <pingou> #chair nirik pingou smooge cverna mizdebsk mkonecny abompard siddharthvipul mobrien
15:00:41 <zodbot> Current chairs: abompard cverna mizdebsk mkonecny mobrien nirik pingou siddharthvipul smooge
15:00:43 <pingou> #info Agenda is at: https://board.net/p/fedora-infra
15:00:44 <mobrien> .hello2
15:00:45 <pingou> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
15:00:45 <zodbot> mobrien: mobrien 'Mark O'Brien' <markobri@redhat.com>
15:00:47 <nirik> morning
15:00:47 <pingou> #topic aloha
15:00:58 <TomAPC> hey folks
15:01:11 <siddharthvipul> .hello siddharthvipul1
15:01:14 <zodbot> siddharthvipul: siddharthvipul1 'Vipul Siddharth' <siddharthvipul1@gmail.com>
15:01:15 <siddharthvipul> TomAPC, heyy :D
15:01:23 <TomAPC> .hello tomapc
15:01:24 <zodbot> TomAPC: tomapc 'Tom Collins' <tcollins@teratux.com>
15:01:35 <TomAPC> Nice :D hey siddharthvipul!
15:01:49 <darknao> .hello2
15:01:50 <zodbot> darknao: darknao 'Francois Andrieu' <naolwen@gmail.com>
15:02:00 <pingou> ó/ everyone!
15:02:04 <miausX> .hello2
15:02:05 <zodbot> miausX: Sorry, but you don't exist
15:02:10 <miausX> ;(
15:02:16 <pingou> "computer says no"
15:02:19 <miausX> lol
15:02:35 <mboddu> Hello
15:02:55 <mboddu> .hello mohanboddu
15:02:56 <zodbot> mboddu: mohanboddu 'Mohan Boddu' <mboddu@bhujji.com>
15:02:59 <mobrien> miausX, try .hello <fas_name>
15:02:59 <nirik> (hello2 works only if your irc nick and fas account name are the same. Otherwise you need .hello fasaccountname)
15:03:08 <miausX> .hello manher
15:03:09 <zodbot> miausX: manher 'None' <manu@littlemousetrap.com>
15:03:36 <mboddu> miausX: .hello2 only works if you fas username and your irc nick are the same
15:03:48 <miausX> Thanks, nirik, mobrien and mboddu :)
15:04:11 * mboddu didn't see nirik's message :D
15:04:37 * pingou hands on some glasses to mboddu
15:05:07 <mkonecny> .hello zlopez
15:05:08 <zodbot> mkonecny: zlopez 'Michal Konečný' <michal.konecny@psmail.xyz>
15:05:08 <mboddu> pingou: I denfinitely need to get my eyes checked, esp after the rabbit queue thing :D
15:06:07 <pingou> alright, let's continue
15:06:11 * nirik pictures mboddu moving around rabbits in various queues
15:06:17 <pingou> #topic New folks introductions
15:06:18 <pingou> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
15:06:20 <pingou> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
15:06:39 * pingou wonders how nirik pictures the queues
15:06:56 <nirik> long lines like to get into a theme park. ;)
15:07:12 * nirik has more coffee
15:07:12 <pingou> alright, now I see the picture :D
15:07:32 <mboddu> lol
15:07:48 <pingou> mboddu: soon: i.dailymail.co.uk/i/pix/2015/07/22/09/2ABE4EB600000578-3170464-image-a-45_1437555508499.jpg
15:07:53 * mboddu now wants some bunnies as pets :D
15:08:18 <mboddu> Thats the life... :)
15:08:29 <pingou> talk to nils  :)
15:08:30 <nirik> nice
15:08:40 <pingou> (no new comers today?)
15:09:23 <pingou> okido
15:09:24 <miausX> pingou: I did a small intro in the stand up a few days ago
15:09:25 <pingou> #topic Next chair
15:09:27 <pingou> #info magic eight ball says:
15:09:29 <pingou> #info 2020-09-03 - pingou
15:09:34 <nils> mboddu, that'll hold on as long as you have food for them :P
15:09:34 <pingou> miausX: cool :)
15:09:53 <pingou> miausX: feel free to jump in at any time :)
15:10:01 <miausX> Thanks :)
15:10:23 <mboddu> miausX: I think you are now part of apprentice group, please request it if you are not already
15:10:36 <mboddu> .fasinfo manher
15:10:37 <zodbot> mboddu: User: manher, Name: None, email: manu@littlemousetrap.com, Creation: 2015-06-14, IRC Nick: None, Timezone: None, Locale: None, GPG key ID: None, Status: active
15:10:40 <zodbot> mboddu: Approved Groups: fi-apprentice cla_fpca cla_done
15:10:50 <pingou> I believe mobrien  was to take the chair next week, but the dark shadows maybe have decided something else
15:10:52 <pingou> mobrien:?
15:11:05 <mobrien> I am actually ok to do it next week
15:11:11 <siddharthvipul> if there is a slot, count me in as well
15:11:13 <pingou> cool
15:11:15 <pingou> #info 2020-09-10 - mobrien
15:11:16 <siddharthvipul> maybe the one after that
15:11:24 <pingou> siddharthvipul: there is a slot for sept 17th?
15:11:25 <mobrien> The training was in CET time rather than UTC so I will be just finished
15:11:26 <pingou> want it?
15:11:34 <siddharthvipul> pingou, sure thing :)
15:11:38 <pingou> siddharthvipul: sold!
15:11:43 <pingou> #info 2020-09-17 - siddharthvipul
15:11:59 <pingou> moving on to announcements
15:12:02 <pingou> #topic announcements and information
15:12:03 <pingou> #info CPE Sustaining EU-hours team has a Monday through Friday 30 minute meeting going through tickets at 0830 UTC in #centos-meeting
15:12:05 <pingou> #info CPE Sustaining NA-hours team has a Monday through Friday 30 minute meeting going through tickets at 1800 UTC in #fedora-admin
15:12:07 <pingou> #info Datacenter move is over, but some items still need to be done: see https://fedoraproject.org/wiki/Infrastructure/2020-post-datacenter-move-known-issues
15:12:09 <pingou> #info New blog post about release-monitoring.org is out https://communityblog.fedoraproject.org/stories-from-the-amazing-world-of-release-monitoring-org-10/
15:12:11 <pingou> ```
15:12:16 <pingou> off by one :)
15:12:25 <siddharthvipul> I have one, the gitlab ama thingy
15:12:27 <siddharthvipul> let me find some links
15:13:00 <pingou> #info gitlab AMA scheduled for Sept 10th 2020 at 13:00 UTC
15:13:05 <nirik> #info staging bringup is continuing. openshift, ipa, proxies, sundries, rabbitmq cluster are all up. Next koji and build sys
15:13:09 <pingou> more information to come
15:13:12 <mobrien> #info infra releng doc hackfest tomorrow 13:00UTC agenda at https://board.net/p/infra-doc-hackfest
15:13:19 * smooge is split between meetings
15:13:39 <pingou> nirik: let me know if I can help in any way
15:13:43 <smooge> #info smooge is on PTO for 2020-09-05->2020-09-20
15:13:59 <pingou> smooge: enjoy! :)
15:14:04 <nirik> pingou: sure thing...
15:14:05 <siddharthvipul> pingou, thank you
15:14:07 <siddharthvipul> smooge, super glad for this.. have a great time off
15:14:18 <pingou> mobrien: good catch :)
15:14:37 <pingou> anything else we want to bring forward?
15:14:48 <mobrien> Enjoy smooge, I'll try not to put everything on fire while you are away
15:15:06 <pingou> (*try* being the keyword)
15:16:21 <pingou> let's talk about oncalls :)
15:16:23 <pingou> #topic Oncall
15:16:25 <pingou> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
15:16:27 <pingou> #info smooge is oncall for 2020-08-27 to 2020-09-03
15:16:29 <pingou> #info mboddu  is oncall for 2020-09-03 to 2020-09-10
15:16:41 <pingou> anyone up to take oncall next week?
15:16:47 <pingou> from the 10th to the 17th?
15:16:49 <mboddu> .oncalltakeus
15:16:49 <zodbot> mboddu: Error: You don't have the alias.add capability. If you think that you should have this capability, be sure that you are identified before trying again. The 'whoami' command can tell you if you're identified.
15:16:51 <mkonecny> pingou: We don't talk about oncalls
15:17:01 <pingou> mkonecny: see rule #2
15:17:02 <nirik> I can take it... if no one else wants it.
15:17:05 * mboddu logging in
15:17:37 <pingou> nirik: I'll take it, you have enough on your plate
15:17:44 <pingou> #info pingou  is oncall for 2020-09-10 to 2020-09-17
15:17:46 <nirik> ok, sure.
15:18:02 <mboddu> .oncalltakeus
15:18:02 <zodbot> mboddu: Kneel before zod!
15:18:05 <pingou> #info Summary of last week: (from current oncall )
15:18:42 <pingou> smooge: you're split b/w meetings but anything specific to report from the last week?
15:19:13 <smooge> we had email problems due to dmarc
15:19:28 <smooge> we had an outage on fedorapeople.org because of an expired ssl
15:19:32 <smooge> and that was it
15:19:39 * pingou found an old ticket about dmarc on our tracker
15:19:57 <pingou> I've pinged you on it in case you fixed it while looking at this
15:20:40 <miausX> smooge: is there a shared calendar for SSL renewals? Can it be automated (Let's Encrypt?)
15:21:29 <smooge> sorry I am getting pinged by upper mgmt in a different meeting
15:21:53 <miausX> no prob :)
15:22:01 <smooge> so that is all I have.
15:22:03 <pingou> let's move on to monitoring and we can discuss this outside of the meeting
15:22:05 <nirik> we have put them in nagios before
15:22:05 <pingou> #topic Monitoring discussion [nirik]
15:22:07 <pingou> #info https://nagios.fedoraproject.org/nagios
15:22:08 <smooge> thanks mboddu for taking it over
15:22:09 <pingou> #info Go over existing out items and fix
15:22:14 * nirik looks
15:22:35 <nirik> looking much better this week
15:22:37 <pingou> nirik: for the certs, I was wondering if a toddlers would help (but that's likely duplicate to nagios)
15:23:10 <mboddu> smooge: :)
15:23:28 <nirik> Some arm machines down still.
15:24:17 <nirik> some low swap
15:24:25 <pingou> pagure01 seems to have a low disk
15:24:52 <nirik> yep. has for a while.
15:25:03 <nirik> bugzilla fedmsgs aren't coming thru still... ;(
15:25:11 <pingou> :(
15:25:19 <pingou> is there a ticket for this?
15:25:40 <nirik> I filed one and they closed it... I need to try and do a new one
15:26:06 <nirik> it's not bugzilla's fault, it's the message bus...
15:26:25 <pingou> the umb?
15:26:39 <nirik> yeah
15:26:45 <pingou> :s
15:26:51 <nirik> I can try and do that... it's been on my list.
15:27:00 <nirik> There's also a number of 'unknown' ones we should try and fix.
15:27:22 <nirik> "Check proxies for oversubscription"
15:27:27 <nirik> on a number of proxies.
15:27:33 <mobrien> nirik I have a question on nagios, I mentioned it to smooge earlier. I added new hosts to ansible on Tuesday but they haven't been added to nagios. The noc playbook ran succesfully this morning. Is this related to the freeze?
15:27:58 <nirik> no, the playbook should update it.
15:28:21 <smooge> what were the hosts
15:28:33 <mobrien> Ok, I'll do more investigation. proxy37/38
15:28:52 <nirik> sometimes there's an issue with the config and nagios won't restart.
15:28:56 <nirik> but that should fail the playbook.
15:29:19 <nirik> thats all for nagios I think.
15:29:28 <smooge> no it looks like it wasn't added
15:29:50 <mobrien> I just checked the log and saw it passed on the automated run. I'll do a manual one later and see.
15:30:08 <mobrien> We can move the meeting on. I'll figure it out
15:30:24 <pingou> #topic Open Floor
15:30:42 <pingou> there is a topic, but I don't know if it was covered last week: Any estimate when "abrt" and "package index" services will come back
15:30:58 <siddharthvipul> pingou, it's new
15:31:11 <siddharthvipul> I was asked this in a channel and I had no answer, so thought of asking it here
15:31:11 <pingou> so someone was interested by this question :)
15:31:13 <smooge> hmmm I wonder if need to add proxy32-38 into roles/httpd/proxy/templates/00-namevirtualhost.conf and roles/mediawiki/templates/LocalSettings.php.fp.j2 anymore
15:31:34 <nirik> abrt / retrace: machine is up and installed, it's waiting on config from the team that does that... so, I guess we should ask them for ETA? or ask the reporter to ask them
15:31:45 <siddharthvipul> nirik, gotcha, thank you :)
15:31:52 <pingou> iirc they said on email on devel that they hoped to get it down this week
15:31:56 <smooge> msuchy said he ran into some issues and is working on it
15:31:56 <nirik> packages: we need to decide what the replacement is and deploy it.
15:32:03 <pingou> and thus, hopefully back online for next week
15:32:38 <siddharthvipul> great, I shall point them to this discussion
15:32:43 <nirik> if we don't deploy a new packages soon, we should repoint it's url at a wiki page or something that explains whats going on.
15:34:47 <pingou> anything else we want to cover?
15:34:50 <nirik> pingou: also, we may have another toddler to add to the playpen at some point... a archiving thing for coreos/silverblue.
15:35:02 <pingou> cool
15:35:16 <TomAPC> I guess the docs hackfest is still on for tomorrow?
15:35:23 <pingou> yup
15:35:26 <nirik> it needs a fuse mount tho, so that has to be sorted before it can run in openshift
15:36:06 <TomAPC> sweet, will be there!
15:36:08 * nirik probibly won't make the start of that, but hopes to make some of it
15:36:25 <miausX> Count me in too!
15:37:03 <pingou> if nothing else, I propose to give you back about 23 minutes of time
15:37:27 <mobrien> nirik the initial part will be a demo from pbokoc on the video link and then a quick outline of what we want to go through so you won't miss too much hopefully
15:38:34 <pingou> alright folks!
15:38:57 <nirik> mobrien: ok.
15:39:02 <pingou> #endmeeting