15:00:59 <mkonecny> #startmeeting Infrastructure (2020-07-02)
15:00:59 <zodbot> Meeting started Thu Jul  2 15:00:59 2020 UTC.
15:00:59 <zodbot> This meeting is logged and archived in a public location.
15:00:59 <zodbot> The chair is mkonecny. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:59 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:00:59 <zodbot> The meeting name has been set to 'infrastructure_(2020-07-02)'
15:00:59 <mkonecny> #meetingname infrastructure
15:00:59 <zodbot> The meeting name has been set to 'infrastructure'
15:00:59 <mkonecny> #chair nirik pingou smooge cverna mizdebsk mkonecny abompard siddharthvipul mobrien
15:00:59 <mkonecny> #info Agenda is at: https://board.net/p/fedora-infra
15:00:59 <zodbot> Current chairs: abompard cverna mizdebsk mkonecny mobrien nirik pingou siddharthvipul smooge
15:00:59 <mkonecny> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
15:01:00 <mkonecny> #topic aloha
15:01:04 <mkonecny> .hello zlopez
15:01:05 <zodbot> mkonecny: zlopez 'Michal Konečný' <michal.konecny@packetseekers.eu>
15:01:10 <nirik> morning
15:01:15 <siddharthvipul> .hello siddharthvipul1
15:01:19 <zodbot> siddharthvipul: siddharthvipul1 'Vipul Siddharth' <siddharthvipul1@gmail.com>
15:01:21 <smooge> morning
15:01:51 <mkonecny> Welcome everyone on today's show
15:02:07 <nao> hey o/
15:02:25 <mkonecny> We will look at the various topics today and maybe we will surprise you with what you see
15:02:50 <mkonecny> So let's start
15:02:57 <mkonecny> #topic New folks introductions
15:02:57 <mkonecny> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
15:02:57 <mkonecny> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
15:03:07 <mkonecny> Anybody new on this show?
15:03:15 <nao> I am :)
15:03:43 <siddharthvipul> nao: \o
15:04:06 <mkonecny> nao: Welcome, do you want to say something?
15:04:26 <nao> sure
15:04:34 <nao> a little introduction
15:04:41 * pingou in a meeting at the same time :(
15:05:14 <nao> i'm a sysadmin from France, for several years now (about 10)
15:05:27 <mkonecny> Don't worry pingou, we have you covered in this meeting ;-)
15:05:42 <smooge> welcome nao
15:05:43 <nao> specialized in Ansible, python dev, and few RH tech
15:05:48 <pingou> tx!
15:05:50 <nirik> welcome nao
15:06:16 <pingou> nao: sounds like you'll fit right in!
15:06:19 <mkonecny> nao: Nice set of skills, exactly the skills we need in our infrastructure
15:06:44 <Nikolay> Hello, My name is Nikolay. I am new as well. I am network specialist.
15:07:14 <siddharthvipul> Nikolay: \o :D
15:07:14 <Nikolay> :)
15:07:21 <mboddu> Hello
15:07:23 <nirik> welcome Nikolay also. ;)
15:07:25 <Nikolay> I almost RHCE and want to contribute
15:07:26 <nao> mkonecny: yeah i think this is the right place ^^
15:07:37 <nao> welcome Nikolay
15:08:00 <mkonecny> Nikolay: Welcome
15:14:15 <mkonecny> We have a plenty of new folks today
15:14:15 <siddharthvipul> \o/ nao++ Nikolay++
15:14:15 <mboddu> Welcome nao and Nikolay
15:14:15 <mkonecny> nao and Nikolay: Do you have any questions?
15:14:15 <pingou> welcome Nikolay  and nao :)
15:14:15 <Nikolay> Thanks, Not yet
15:14:15 <nao> mkonecny: not yet, lot of reading for now
15:14:15 <mkonecny> Ok, just don't be afraid to ask
15:14:16 <Nikolay> sure, thanks
15:14:16 <mkonecny> So let's move to next part of our show
15:14:16 <mkonecny> #topic Next chair
15:14:16 <mkonecny> #info magic eight ball says:
15:14:16 <mkonecny> #info 2020-07-09 - nirik
15:14:16 <mkonecny> #info 2020-07-16 - mkonecny
15:14:16 <mkonecny> #info 2020-07-23 - ???
15:14:16 * siddharthvipul can take it
15:14:16 <mkonecny> Do we lost zodbot
15:14:16 <mkonecny> ?
15:14:16 <nirik> zodbot: ping
15:14:16 <nirik> seems like
15:14:16 <mkonecny> Hm
15:14:16 <mkonecny> siddharthvipul: Thanks for volunteering
15:14:16 <smooge> yeah.. the meetbot part seems to have crashed a couple of times
15:14:16 <mkonecny> #info 2020-07-23 - siddharthvipul
15:14:17 <nirik> it's running, and not saying it got disconnected. not sure whats going on.
15:14:18 <mkonecny> Maybe it's overwhelmed by the number of requests
15:14:19 <mkonecny> Let's continue
15:14:26 <zodbot> pong
15:14:35 <mkonecny> #topic announcements and information
15:14:35 <mkonecny> #info CPE Sustaining EU-hours team has standups on Tuesday and Thursday at 1400 UTC in #fedora-meeting-2 - please join
15:14:35 <mkonecny> #info CPE Sustaining NA-hours team has a Monday through Friday 30 minute meeting going through tickets at 1800 UTC in #fedora-admin
15:14:35 <mkonecny> #info Fedora Infrastructure will be moving in 2020-06 from its Phoenix Az datacenter to one near Herndon Va. A lot of planning will be involved on this. Please watch out for announcements on changes.
15:14:36 <mkonecny> #info Fedora Communishift move has started but will take longer than expected. Current estimate for bringing back into production is TBD
15:15:58 <mkonecny> Anything else we want to announce to our audience?
15:16:19 <bcotton> do we have an estimate for when we'll have an estimate for communishift being back?
15:16:28 <nirik> #info tomorrow is a holiday in the US, and monday is a holiday in some parts of EU
15:17:40 <nirik> bcotton: we should know more in the next week or two... basically we need to figure out a bunch of networking and logistics for it... then do that.
15:17:50 <bcotton> nirik: ack, thanks
15:18:10 <nirik> smooge might have more up to date info, but thats the last I know...
15:18:40 <nirik> oh, reminds me...
15:18:41 <smooge> bcotton, there are 2 issues
15:19:00 <smooge> we have to get the networking layout worked out and then put in the time in the dc to do the work.
15:19:25 <smooge> the second is we have to work out still what can be run in it once we get it up
15:19:29 <nirik> #info the only items that are still down from the datacenter move are: communishift, retrace and packages. packages is due to be replaced sometime so the old one will not be coming back.
15:19:53 <smooge> i will be working on getting retrace back up next week
15:20:53 <smooge> i can't promise it will happen as i need to get access to the DC and other items but it is on my top list
15:21:37 <smooge> all from me
15:22:22 <mkonecny> Next topic is on the way
15:22:28 <mkonecny> Fresh from the oven
15:22:30 <mkonecny> #topic Oncall
15:22:30 <mkonecny> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
15:22:30 <mkonecny> #info siddharthvipul is oncall for 2020-06-25 -> 2020-07-02
15:22:30 <mkonecny> #info nirik is oncall for 2020-07-02 -> 2020-07-09
15:22:31 <mkonecny> #info ??? is oncall for 2020-07-09 -> 2020-07-16
15:23:14 <mkonecny> If no one is interested in oncall 2020-07-09 -> 2020-07-16, I will take it
15:23:30 <nirik> .oncalltakeus
15:23:30 <zodbot> nirik: Kneel before zod!
15:23:58 <mkonecny> #info mkonecny is oncall for 2020-07-09 -> 2020-07-16
15:24:36 <mkonecny> nirik: I always read it as Oncall take us and this reminds me of one metal album Till the light take us :-D
15:24:52 <nirik> \o/
15:25:13 <mkonecny> #info Summary of last week: (from current oncall )
15:25:27 <mkonecny> siddharthvipul: The mic is yours
15:25:39 <siddharthvipul> right! actually nothing.. pretty silent. In total just 1 ping and that was referred to ticket :)
15:26:17 <mkonecny> It looks like the tickets are more used than the ping
15:26:52 <mkonecny> #topic Monitoring discussion [nirik]
15:26:52 <mkonecny> #info https://nagios.fedoraproject.org/nagios
15:26:52 <mkonecny> #info Go over existing out items and fix
15:27:40 <nirik> we do have nagios back now! :)
15:27:49 <nirik> and there's a number of things to fix. ;(
15:27:59 <mkonecny> nirik: The show must go on
15:28:17 <nirik> there's 2 'outages' which are firewall changes we need to get done... they are in process.
15:29:10 <nirik> there's a check_raid script... thats python2...
15:29:17 <nirik> and is failing on python3 using hosts.
15:29:48 <nirik> otherwise things are not too bad... just need to work on cleaning up things and getting firewall rules updated.
15:30:36 <nirik> we can move on...
15:31:33 <mkonecny> Thanks for update nirik
15:31:35 <mkonecny> #topic Data-Center Move update
15:32:54 <nirik> so, lets see...
15:33:46 <nirik> We have a few more build machines to bring back online: mustangs and some emags and some power8's... 2 power9s to reinstall...
15:34:19 <nirik> then we have all the rest of the virthosts to bring back up.
15:34:45 <nirik> Then we need to re-add 02's of some things... ie, second frontend vm's so they are more highly available.
15:35:02 <nirik> then we can look at staging bringup.
15:35:15 <smooge> I think we have about 60 systems left to reinstall
15:35:21 <smooge> hardware..
15:35:33 <smooge> we have gobs of vms to put on those systems
15:35:33 <nirik> wow... that seems high... but ok
15:35:53 <smooge> I am trying to get a finished number
15:36:05 <smooge> so yeah it could be only 40
15:36:09 <nirik> I'd really like to get 2 emags up as buildhw or at least 2 more mustangs... that would help rawhide composes a lot
15:36:30 <nirik> oh, and we have the openqa worker hardware to bring up still too.
15:37:11 <nirik> on the other DC as noted earlier, retrace hopefully next week, then networking and other changes and communishift after
15:37:53 <smooge> retrace will depend on whether I can get networking people which isn't looking good.. or if bringing up emags/power/qa is higher priority
15:39:03 <nirik> I'd put it: emags/mustangs for buildhw, retrace, qa, rest of hardware, staging...
15:39:43 <nirik> anyone have any questions? :) Probibly confusing if you haven't looked at this before...
15:40:30 <mkonecny> Thanks for the great work
15:41:24 <mkonecny> #topic Open Floor
15:41:54 <mkonecny> Floor is now open and everybody the microphone is free to use
15:43:24 <mkonecny> How could anybody help with the move right now?
15:44:01 <nirik> well, we have some python script porting that needs doing...
15:44:20 <nirik> I think the next big time we will need people is when we get staging started.
15:45:47 <mkonecny> thanks nirik
15:47:07 <mkonecny> I will left this open for few minutes, if anybody want to say something on Open Floor, now is your time
15:48:11 <smooge> i have nothing more
15:48:52 <mkonecny> Goodbye everyone, this show is approaching it's end
15:48:53 <smooge> plus my singing is a war crime in two continents so the karaoke machine is off limits
15:49:20 <mkonecny> We are looking forward for next episode
15:49:28 <nirik> ditto
15:49:31 <nao> nirik: If you need some extra hands for script porting, i'll be happy to help
15:49:43 <mkonecny> smooge will accompany us with his singing :-D
15:50:35 <nirik> sure! https://pagure.io/fedora-infra/ansible/blob/master/f/scripts/logview is one...
15:51:11 <nirik> https://pagure.io/fedora-infra/ansible/blob/master/f/roles/nagios_client/files/scripts/check_raid.py is another one
15:51:21 <nirik> PR's welcome.
15:51:26 <nirik> happy to provide more info too
15:51:54 <nao> great, i'll take a look
15:52:26 <nirik> although... the 2nd one might require some thought. As it needs to work on both python2 and python3 hosts... or we need 2 versions.
15:55:46 <mkonecny> Ending the show now
15:55:53 <mkonecny> That's all, folks!
15:55:55 <mkonecny> #endmeeting