15:00:59 #startmeeting Infrastructure (2020-07-02) 15:00:59 Meeting started Thu Jul 2 15:00:59 2020 UTC. 15:00:59 This meeting is logged and archived in a public location. 15:00:59 The chair is mkonecny. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:59 Useful Commands: #action #agreed #halp #info #idea #link #topic. 15:00:59 The meeting name has been set to 'infrastructure_(2020-07-02)' 15:00:59 #meetingname infrastructure 15:00:59 The meeting name has been set to 'infrastructure' 15:00:59 #chair nirik pingou smooge cverna mizdebsk mkonecny abompard siddharthvipul mobrien 15:00:59 #info Agenda is at: https://board.net/p/fedora-infra 15:00:59 Current chairs: abompard cverna mizdebsk mkonecny mobrien nirik pingou siddharthvipul smooge 15:00:59 #info About our team: https://docs.fedoraproject.org/en-US/cpe/ 15:01:00 #topic aloha 15:01:04 .hello zlopez 15:01:05 mkonecny: zlopez 'Michal Konečný' 15:01:10 morning 15:01:15 .hello siddharthvipul1 15:01:19 siddharthvipul: siddharthvipul1 'Vipul Siddharth' 15:01:21 morning 15:01:51 Welcome everyone on today's show 15:02:07 hey o/ 15:02:25 We will look at the various topics today and maybe we will surprise you with what you see 15:02:50 So let's start 15:02:57 #topic New folks introductions 15:02:57 #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves 15:02:57 #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted 15:03:07 Anybody new on this show? 15:03:15 I am :) 15:03:43 nao: \o 15:04:06 nao: Welcome, do you want to say something? 15:04:26 sure 15:04:34 a little introduction 15:04:41 * pingou in a meeting at the same time :( 15:05:14 i'm a sysadmin from France, for several years now (about 10) 15:05:27 Don't worry pingou, we have you covered in this meeting ;-) 15:05:42 welcome nao 15:05:43 specialized in Ansible, python dev, and few RH tech 15:05:48 tx! 15:05:50 welcome nao 15:06:16 nao: sounds like you'll fit right in! 15:06:19 nao: Nice set of skills, exactly the skills we need in our infrastructure 15:06:44 Hello, My name is Nikolay. I am new as well. I am network specialist. 15:07:14 Nikolay: \o :D 15:07:14 :) 15:07:21 Hello 15:07:23 welcome Nikolay also. ;) 15:07:25 I almost RHCE and want to contribute 15:07:26 mkonecny: yeah i think this is the right place ^^ 15:07:37 welcome Nikolay 15:08:00 Nikolay: Welcome 15:14:15 We have a plenty of new folks today 15:14:15 \o/ nao++ Nikolay++ 15:14:15 Welcome nao and Nikolay 15:14:15 nao and Nikolay: Do you have any questions? 15:14:15 welcome Nikolay and nao :) 15:14:15 Thanks, Not yet 15:14:15 mkonecny: not yet, lot of reading for now 15:14:15 Ok, just don't be afraid to ask 15:14:16 sure, thanks 15:14:16 So let's move to next part of our show 15:14:16 #topic Next chair 15:14:16 #info magic eight ball says: 15:14:16 #info 2020-07-09 - nirik 15:14:16 #info 2020-07-16 - mkonecny 15:14:16 #info 2020-07-23 - ??? 15:14:16 * siddharthvipul can take it 15:14:16 Do we lost zodbot 15:14:16 ? 15:14:16 zodbot: ping 15:14:16 seems like 15:14:16 Hm 15:14:16 siddharthvipul: Thanks for volunteering 15:14:16 yeah.. the meetbot part seems to have crashed a couple of times 15:14:16 #info 2020-07-23 - siddharthvipul 15:14:17 it's running, and not saying it got disconnected. not sure whats going on. 15:14:18 Maybe it's overwhelmed by the number of requests 15:14:19 Let's continue 15:14:26 pong 15:14:35 #topic announcements and information 15:14:35 #info CPE Sustaining EU-hours team has standups on Tuesday and Thursday at 1400 UTC in #fedora-meeting-2 - please join 15:14:35 #info CPE Sustaining NA-hours team has a Monday through Friday 30 minute meeting going through tickets at 1800 UTC in #fedora-admin 15:14:35 #info Fedora Infrastructure will be moving in 2020-06 from its Phoenix Az datacenter to one near Herndon Va. A lot of planning will be involved on this. Please watch out for announcements on changes. 15:14:36 #info Fedora Communishift move has started but will take longer than expected. Current estimate for bringing back into production is TBD 15:15:58 Anything else we want to announce to our audience? 15:16:19 do we have an estimate for when we'll have an estimate for communishift being back? 15:16:28 #info tomorrow is a holiday in the US, and monday is a holiday in some parts of EU 15:17:40 bcotton: we should know more in the next week or two... basically we need to figure out a bunch of networking and logistics for it... then do that. 15:17:50 nirik: ack, thanks 15:18:10 smooge might have more up to date info, but thats the last I know... 15:18:40 oh, reminds me... 15:18:41 bcotton, there are 2 issues 15:19:00 we have to get the networking layout worked out and then put in the time in the dc to do the work. 15:19:25 the second is we have to work out still what can be run in it once we get it up 15:19:29 #info the only items that are still down from the datacenter move are: communishift, retrace and packages. packages is due to be replaced sometime so the old one will not be coming back. 15:19:53 i will be working on getting retrace back up next week 15:20:53 i can't promise it will happen as i need to get access to the DC and other items but it is on my top list 15:21:37 all from me 15:22:22 Next topic is on the way 15:22:28 Fresh from the oven 15:22:30 #topic Oncall 15:22:30 #info https://fedoraproject.org/wiki/Infrastructure/Oncall 15:22:30 #info siddharthvipul is oncall for 2020-06-25 -> 2020-07-02 15:22:30 #info nirik is oncall for 2020-07-02 -> 2020-07-09 15:22:31 #info ??? is oncall for 2020-07-09 -> 2020-07-16 15:23:14 If no one is interested in oncall 2020-07-09 -> 2020-07-16, I will take it 15:23:30 .oncalltakeus 15:23:30 nirik: Kneel before zod! 15:23:58 #info mkonecny is oncall for 2020-07-09 -> 2020-07-16 15:24:36 nirik: I always read it as Oncall take us and this reminds me of one metal album Till the light take us :-D 15:24:52 \o/ 15:25:13 #info Summary of last week: (from current oncall ) 15:25:27 siddharthvipul: The mic is yours 15:25:39 right! actually nothing.. pretty silent. In total just 1 ping and that was referred to ticket :) 15:26:17 It looks like the tickets are more used than the ping 15:26:52 #topic Monitoring discussion [nirik] 15:26:52 #info https://nagios.fedoraproject.org/nagios 15:26:52 #info Go over existing out items and fix 15:27:40 we do have nagios back now! :) 15:27:49 and there's a number of things to fix. ;( 15:27:59 nirik: The show must go on 15:28:17 there's 2 'outages' which are firewall changes we need to get done... they are in process. 15:29:10 there's a check_raid script... thats python2... 15:29:17 and is failing on python3 using hosts. 15:29:48 otherwise things are not too bad... just need to work on cleaning up things and getting firewall rules updated. 15:30:36 we can move on... 15:31:33 Thanks for update nirik 15:31:35 #topic Data-Center Move update 15:32:54 so, lets see... 15:33:46 We have a few more build machines to bring back online: mustangs and some emags and some power8's... 2 power9s to reinstall... 15:34:19 then we have all the rest of the virthosts to bring back up. 15:34:45 Then we need to re-add 02's of some things... ie, second frontend vm's so they are more highly available. 15:35:02 then we can look at staging bringup. 15:35:15 I think we have about 60 systems left to reinstall 15:35:21 hardware.. 15:35:33 we have gobs of vms to put on those systems 15:35:33 wow... that seems high... but ok 15:35:53 I am trying to get a finished number 15:36:05 so yeah it could be only 40 15:36:09 I'd really like to get 2 emags up as buildhw or at least 2 more mustangs... that would help rawhide composes a lot 15:36:30 oh, and we have the openqa worker hardware to bring up still too. 15:37:11 on the other DC as noted earlier, retrace hopefully next week, then networking and other changes and communishift after 15:37:53 retrace will depend on whether I can get networking people which isn't looking good.. or if bringing up emags/power/qa is higher priority 15:39:03 I'd put it: emags/mustangs for buildhw, retrace, qa, rest of hardware, staging... 15:39:43 anyone have any questions? :) Probibly confusing if you haven't looked at this before... 15:40:30 Thanks for the great work 15:41:24 #topic Open Floor 15:41:54 Floor is now open and everybody the microphone is free to use 15:43:24 How could anybody help with the move right now? 15:44:01 well, we have some python script porting that needs doing... 15:44:20 I think the next big time we will need people is when we get staging started. 15:45:47 thanks nirik 15:47:07 I will left this open for few minutes, if anybody want to say something on Open Floor, now is your time 15:48:11 i have nothing more 15:48:52 Goodbye everyone, this show is approaching it's end 15:48:53 plus my singing is a war crime in two continents so the karaoke machine is off limits 15:49:20 We are looking forward for next episode 15:49:28 ditto 15:49:31 nirik: If you need some extra hands for script porting, i'll be happy to help 15:49:43 smooge will accompany us with his singing :-D 15:50:35 sure! https://pagure.io/fedora-infra/ansible/blob/master/f/scripts/logview is one... 15:51:11 https://pagure.io/fedora-infra/ansible/blob/master/f/roles/nagios_client/files/scripts/check_raid.py is another one 15:51:21 PR's welcome. 15:51:26 happy to provide more info too 15:51:54 great, i'll take a look 15:52:26 although... the 2nd one might require some thought. As it needs to work on both python2 and python3 hosts... or we need 2 versions. 15:55:46 Ending the show now 15:55:53 That's all, folks! 15:55:55 #endmeeting