15:00:04 <mkonecny> #startmeeting Infrastructure (2020-06-04)
15:00:04 <zodbot> Meeting started Thu Jun  4 15:00:04 2020 UTC.
15:00:04 <zodbot> This meeting is logged and archived in a public location.
15:00:04 <zodbot> The chair is mkonecny. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:04 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:00:04 <zodbot> The meeting name has been set to 'infrastructure_(2020-06-04)'
15:00:04 <mkonecny> #meetingname infrastructure
15:00:04 <zodbot> The meeting name has been set to 'infrastructure'
15:00:04 <mkonecny> #chair nirik pingou smooge cverna mizdebsk mkonecny abompard siddharthvipul
15:00:04 <mkonecny> #info Agenda is at: https://board.net/p/fedora-infra
15:00:04 <zodbot> Current chairs: abompard cverna mizdebsk mkonecny nirik pingou siddharthvipul smooge
15:00:04 <mkonecny> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
15:00:05 <mkonecny> #topic aloha
15:00:10 <mkonecny> .hello zlopez
15:00:11 <zodbot> mkonecny: zlopez 'Michal Konečný' <michal.konecny@packetseekers.eu>
15:00:14 <nils> .hello nphilipp
15:00:14 <zodbot> nils: nphilipp 'Nils Philippsen' <nphilipp@redhat.com>
15:00:17 <nirik> morning
15:00:23 <siddharthvipul> .hello2 siddharthvipul1
15:00:24 <zodbot> siddharthvipul: siddharthvipul 'Vipul Siddharth' <siddharthsurfing1@gmail.com>
15:00:41 <amrmzaki> .hello amrmzaki
15:00:44 <zodbot> amrmzaki: Sorry, but you don't exist
15:00:44 <mkonecny> Hi everyone, welcome to our weekly "everything is on fire" meeting
15:01:07 <nirik> FYI, smooge and I will have to leave in 45min... we will be moving batcave01.phx2 to the new batcave01.iad2 in an hour or so
15:01:18 <amrmzaki> .hello amrmostafazaki
15:01:19 <zodbot> amrmzaki: amrmostafazaki 'Amr Mostafa Zaki' <amr.mostafa.zaki@gmail.com>
15:01:25 <mkonecny> nirik: No problem
15:01:35 <mkonecny> Have fun both of you
15:01:43 <nirik> you bet. it will be an adventure
15:02:16 <amrmzaki> @nirik, @smooge enjoy
15:02:40 <mkonecny> Let's start
15:02:50 <mkonecny> #topic New folks introductions
15:02:50 <mkonecny> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
15:02:50 <mkonecny> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
15:03:01 <mkonecny> Anyone new, who wants to say hello?
15:05:03 <mkonecny> Probably no
15:05:07 <mkonecny> #topic Next chair
15:05:08 <mkonecny> #info magic eight ball says:
15:05:08 <mkonecny> #info 2020-06-04 - mkonecny
15:05:08 <mkonecny> #info 2020-06-11 - siddharthvipul
15:05:08 <mkonecny> #info 2020-06-18 - ???
15:05:08 <mkonecny> #info 2020-06-25 - ???
15:05:44 <mkonecny> Anybody wants to chair weeks 2020-06-18 and 2020-06-25?
15:06:10 <siddharthvipul> it's fun
15:06:17 <siddharthvipul> someone should try it :)
15:06:21 <mkonecny> yeah, it is
15:06:22 <cverna> I can do the 18th if needed
15:06:33 <siddharthvipul> I mean other than those who already do it haha
15:06:42 <mkonecny> cverna: Ok it's yours
15:06:51 <siddharthvipul> oh great, and I can take the next (if there is no one who raises hand)
15:06:55 <siddharthvipul> but lets see
15:06:57 <mkonecny> #info 2020-06-18 - cverna
15:07:24 <mkonecny> siddharthvipul: It's yours
15:07:33 <cverna> oh I can't do 18th sorry
15:07:42 <cverna> but I can do 25th :P
15:07:43 <mkonecny> #undo
15:07:43 <zodbot> Removing item from minutes: INFO by mkonecny at 15:06:57 : 2020-06-18 - cverna
15:07:57 <siddharthvipul> let's swap then
15:08:01 <cverna> +1
15:08:05 <mkonecny> Ok
15:08:09 <siddharthvipul> I can take 18th, you do 25th
15:08:51 <cverna> works for me
15:08:58 <mkonecny> #info 2020-06-18 - siddharhtvipul
15:08:58 <mkonecny> #info 2020-06-25 - cverna
15:09:26 <mkonecny> Ok, so the chairs are taken, let's move on
15:09:27 <mkonecny> #topic announcements and information
15:09:28 <mkonecny> #info CPE Sustaining EU-hours team has standups on Tuesday and Thursday at 1400 UTC in #fedora-admin - please join
15:09:28 <mkonecny> #info CPE Sustaining NA-hours team has a Monday through Friday 30 minute meeting going through tickets at 1800 UTC in #fedora-admin
15:09:28 <mkonecny> #info Fedora Infrastructure will be moving in 2020-06 from its Phoenix Az datacenter to one near Herndon Va. A lot of planning will be involved on this. Please watch out for announcements on changes.
15:09:30 <mkonecny> #info Fedora Communishift move has started but will take longer than expected. Current estimate for bringing back into production is TBD
15:10:03 <mkonecny> Any other announcements?
15:10:33 <nirik> the dc move starts today...
15:10:39 <nirik> with batcave and wiki moving today.
15:10:45 <nirik> zodbot and meeting minutes tomorrow
15:10:52 <mkonecny> #info EU sustaining team thinks about changing the channel for standups
15:10:54 <nirik> then next week is... everything else. ;)
15:11:20 <mkonecny> nirik: Plenty of fun
15:11:36 <nirik> oh? why? I find it kinda nice to look back and see it in logs...
15:11:54 <nirik> but I guess sometimes there's other people trying to discuss things.
15:12:31 <mkonecny> nirik: We are thinking about some fedora-meeting-X channel
15:13:14 <mkonecny> It will be much less disruptive for any discussions going on
15:13:17 <nirik> yeah, that could work. we could move the later US one too.
15:13:56 <mkonecny> Let's move to oncall
15:13:57 <mkonecny> #topic Oncall
15:13:58 <mkonecny> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
15:13:58 <mkonecny> #info cverna is oncall for 2020-06-04 -> 2020-06-11
15:13:58 <mkonecny> #info ??? is oncall for 2020-06-11 -> 2020-06-18
15:13:58 <mkonecny> #info ??? is oncall for 2020-06-18 -> 2020-06-25
15:14:47 <mkonecny> Do we have any volunteers for the weeks 2020-06-11 -> 2020-06-18 and 2020-06-18 -> 2020-06-25
15:15:13 <nirik> I can take 11-18th... I will be around anyhow finishing up the move and trying to fix things.
15:15:46 <mkonecny> #info nirik is oncall for 2020-06-11 -> 2020-06-18
15:16:25 <mkonecny> I'm thinking about taking the next one, but I'm not sure if I need some special permission
15:16:46 <nirik> nope... just a bit to add to the bot.
15:17:13 <nirik> anyone can do it. It's just talking to people and getting them to file tickets or deciding if something is important enough to bother people for
15:17:13 <mkonecny> Ok than
15:17:42 <mkonecny> So it's just a filter
15:17:53 <mkonecny> for incoming pings
15:18:03 <nirik> yep.
15:18:07 <mkonecny> #info mkonecny is oncall for 2020-06-18 -> 2020-06-25
15:18:38 <mkonecny> #info Summary of last week: (from current oncall )
15:18:46 <mkonecny> cverna: The mic is yours
15:19:18 <mkonecny> My bad, cverna has this week
15:19:33 <mkonecny> Who had the previous one
15:20:00 <mkonecny> ?
15:21:32 <nirik> I'm not sure...
15:21:35 * nirik looks back
15:21:44 <mkonecny> I see siddharthvipul in the admin channel
15:22:10 <mkonecny> I accidentally deleted this from the agenda :-(
15:22:12 <smooge> i think cverna has been listed as oncall
15:22:12 <siddharthvipul> oh oh
15:22:19 <siddharthvipul> I was in another channel
15:22:26 <siddharthvipul> what did i miss, /me reads the log
15:22:56 <siddharthvipul> mkonecny: what's up?
15:22:59 <mkonecny> siddharthvipul: Did you had oncall this week?
15:23:10 <siddharthvipul> mkonecny: no, Mohan had, right?
15:23:15 <siddharthvipul> let me check minutes
15:23:17 <nirik> ah yeah, it was mboddu
15:23:53 <mkonecny> So we solved the mystery of missing oncall
15:23:56 <siddharthvipul> oh no, it was cverna lol
15:24:02 <siddharthvipul> as per the minutes :P
15:24:11 <siddharthvipul> verna is oncall for 2020-05-28 -> 2020-06-04 (siddharthvipul, 15:15:04)
15:24:29 <mkonecny> So I was right :-D
15:24:47 <mkonecny> It looks like he is not here anyway, so let's move on
15:25:16 <mkonecny> #topic Monitoring discussion [nirik]
15:25:17 <mkonecny> #info https://nagios.fedoraproject.org/nagios
15:25:17 <mkonecny> #info Go over existing out items and fix
15:25:21 <nirik> lets see
15:25:52 <nirik> typical low swap on some machnies...
15:26:18 <nirik> the Check queue mirror_pagure_ansible_iad2 one we need to fix...
15:26:29 <nirik> but we can do that after we move batcave
15:27:14 <nirik> some misc disk space issues...
15:27:27 <nirik> openqa-stg01, pagure01, and proxy05
15:27:54 <nirik> some rabbitmq queues with a ton of things... centos-ci and openqa_retvalstanging...
15:28:26 <nirik> also a websites build happened recently alert
15:28:47 <nirik> so a lot of random things, but pretty low pri
15:29:19 <nirik> so, we can move on...
15:29:29 <amrmzaki> nirik, do you know what's root cause of the problems ?
15:30:21 <nirik> various things.
15:30:28 <nirik> the low swap is caused by a kernel bug...
15:31:05 <nirik> the disk space is common on openqa-stg01. On pagure01 it's due to some backup copies pingou made I think? proxy05 is due to docs building taking up too much space... there's a ticket on that
15:31:36 <nirik> the rabbitmq queues are centos-ci not consuming from that queue all the time and the openqa one is because it can't login to the staging wiki.
15:31:44 <nirik> not sure on the websites build.
15:33:08 <mkonecny> Thanks nirik
15:33:12 <mkonecny> Let's move on
15:33:14 <mkonecny> #topic Data-Center Move update
15:33:22 <mkonecny> Who wants to take this?
15:33:27 <nirik> I can...
15:33:40 <nirik> so as mentioned we are moving batcave starting in about 15min or so. ;)
15:33:48 <nirik> I'll mail the list info on it after we are done.
15:33:55 <nirik> and later today we will be moving the wiki.
15:34:12 <nirik> We still have all the ppc64le builders to setup. ;(
15:34:50 <nirik> so, mostly on track, but a not as far ahead as I would like
15:35:48 <nirik> so everyone follow along in the migration doc and be patient with us as we move. ;)
15:36:24 <mkonecny> We wish you best luck
15:36:46 <nirik> thanks, we will need it. :)
15:37:08 <mkonecny> #topic Open Floor
15:37:26 <mkonecny> Let's play floor is a lava :-)
15:37:36 <nirik> ouch. :)
15:37:44 <Saffronique|w> jump!
15:37:47 <mkonecny> Who wants to go on the open floor?
15:38:38 * mkonecny puts fireproof suit
15:38:49 <mkonecny> Let's test the waters
15:38:53 <siddharthvipul> heet
15:38:55 <mkonecny> Oh I mean lava
15:39:26 <mkonecny> Oh, it's hot
15:39:31 <mkonecny> Ouch!
15:39:43 <amrmzaki> i wanna update some of missing CSI of servers .. so i need a info about owners and some sort of that to complete
15:40:08 <amrmzaki> where can i get this info ?
15:40:55 <mkonecny> amrmzaki: Which servers?
15:40:57 <nirik> possibly in the infra-docs repo? at least for anything that has a sop in there it should mention owners and such...
15:41:05 <nirik> but otherwise, just asking I guess.
15:42:27 <amrmzaki> mkonecny, i don't remember them now but by grep on ansible i can mention them here
15:43:08 <amrmzaki> nirik +1
15:43:09 <mkonecny> amrmzaki: Look at the infra-docs repo, if there are mentioned there first
15:43:21 <amrmzaki> mkonecny, +1
15:43:45 * mboddu can be oncall for this week if no one has taken it
15:43:47 <mkonecny> amrmzaki: https://pagure.io/infra-docs
15:44:17 <mkonecny> mboddu: cverna is, we just wanted to have some summary
15:44:26 <mkonecny> of last one
15:44:55 <siddharthvipul> yes, and mboddu, you were already marked for oncall this week :P
15:44:57 <siddharthvipul> in last meeting
15:45:03 <mboddu> Okay
15:45:05 <siddharthvipul> so go ahead and take it :D
15:45:06 <mboddu> .oncallus
15:45:12 <siddharthvipul> takeoncallus
15:45:19 <mboddu> .takeoncallus
15:45:29 * mboddu forgot the command :D
15:45:32 <mkonecny> Ok, the lava is not safe, so it will be better to close the floor
15:45:33 <mkonecny> #endmeeting