15:00:15 #startmeeting Infrastructure (2020-03-26) 15:00:15 Meeting started Thu Mar 26 15:00:15 2020 UTC. 15:00:15 This meeting is logged and archived in a public location. 15:00:15 The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:15 Useful Commands: #action #agreed #halp #info #idea #link #topic. 15:00:15 The meeting name has been set to 'infrastructure_(2020-03-26)' 15:00:16 #meetingname infrastructure 15:00:16 #chair nirik pingou smooge cverna mizdebsk mkonecny abompard 15:00:16 The meeting name has been set to 'infrastructure' 15:00:16 Current chairs: abompard cverna mizdebsk mkonecny nirik pingou smooge 15:00:16 #info Agenda is at: https://board.net/p/fedora-infra 15:00:16 #info About our team: https://docs.fedoraproject.org/en-US/cpe/ 15:00:17 #topic aloha 15:00:39 .hello zlopez 15:00:40 mkonecny: zlopez 'Michal Konečný' 15:00:51 hello 15:01:04 #topic New folks introductions 15:01:04 #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves 15:01:04 #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted 15:01:09 any new folks around today? 15:01:12 .hello 15:01:14 austinpowered: (hello ) -- Alias for "hellomynameis $1". 15:01:33 * cyberpear tunes in 15:01:40 Hello 15:01:43 hi 15:01:45 I am new :-) 15:01:48 .hello austinpowered 15:01:49 austinpowered: austinpowered 'None' 15:01:55 .hello2 15:01:56 cyberpear: cyberpear 'James Cassell' 15:02:06 * pingou around but in something else as well 15:02:11 * smooge is fighting zombies and will be elsewhere 15:02:14 welcome osa1989. :) Care to give a short one line intro? and do see the getting started guide... 15:02:56 .hello jbwillia 15:02:57 Southern_Gentlem: jbwillia 'Ben Williams' 15:03:15 I am 30 years old, joined red hat as a consultant few months ago, as I miss my sysadmin work, I decided to join here and do some sysadmin work with the community :-) 15:03:40 cool. Welcome again... 15:03:58 ### Determine who the next chair is 15:03:58 #topic Next chair 15:03:58 #info magic eight ball says: 15:03:59 #info 2020-03-26 - nirik 15:03:59 #info 2020-04-02 - cverna 15:04:00 #info 2020-04-09 - ???? 15:04:11 anyone want to take the 04-09 meeting? :) 15:04:26 Before Red hat I was a sysadmin in a German company "Drillisch AG", worked with a lot of tools/software such as ceph, openstack, haproxy, keepalived, ldirectord 15:04:39 osa welcome 15:04:55 nice. we use some of those for sure. 15:04:56 Rhh osa1989 welcome ☺️ 15:05:11 Thanks everyone :-) 15:05:17 I guess we can punt and figure out the 04-09 meeting next week. 15:05:28 #topic announcements and information 15:05:29 #info CPE Sustaining team has daily standups (Monday-Thursday) at 1400 and 1900 UTC in #fedora-admin - please join 15:05:29 #info bodhi 5.2 released and deployed still looking at fixing couple of issues - cverna 15:05:29 #info Fedora Infrastructure will be moving in 2020-06 from its Phoenix Az datacenter to one near Herndon Va. A lot of planning will be involved on this. Please watch out for announcements on changes. 15:05:29 #info Fedora Communishift will be moving to new datacentre in April. Current downtime is expected to be from 2020-04-10 -> 2020-05-01. Please watch out for announcements on changes. 15:05:32 #info Taskotron will EOL in 2020-05 15:05:37 any other announcements or infos? 15:07:00 ok then, moving on 15:07:03 #topic Oncall 15:07:03 #info https://fedoraproject.org/wiki/Infrastructure/Oncall 15:07:03 #info nirik is oncall 2020-03-19 -> 2020-03-26 15:07:03 #info cverna is oncall 2020-03-26 -> 2020-04-02 15:07:04 #info ???? is oncall 2020-04-02 -> 2020-04-09 15:07:25 oncall was pretty quiet... lots of tickets tho... so I think people are getting the idea that they should just file a ticket. 15:07:37 cverna: you want to take over now? and anyone want the following week? 15:08:17 .takeoncalleu 15:08:45 it's oncalltakeeu :) 15:08:54 we should just alias those together 15:09:30 Ah ah trying again 15:09:45 .oncalltakeeu 15:09:47 cverna: Error: You don't have the alias.add capability. If you think that you should have this capability, be sure that you are identified before trying again. The 'whoami' command can tell you if you're identified. 15:10:18 I always thought is oncall take us, like you literally saying oncall to take us :-D 15:10:21 zodbot: capability add cverna +Alias 15:10:24 nirik: Error: The command "capability add" is available in the Admin and Channel plugins. Please specify the plugin whose command you wish to call by using its name as a command before "capability add". 15:10:29 Until I saw the eu one 15:10:32 zodbot: admin capability add cverna +Alias 15:10:32 nirik: Kneel before zod! 15:10:46 mkonecny: :) 15:11:07 #topic Monitoring discussion [nirik] 15:11:07 #info https://nagios.fedoraproject.org/nagios 15:11:07 #info Go over existing out items and fix 15:11:29 .oncalltakeeu 15:11:29 cverna: Error: You don't have the alias.add capability. If you think that you should have this capability, be sure that you are identified before trying again. The 'whoami' command can tell you if you're identified. 15:11:42 so it's a bit red looking right now because mboddu is rebooting some buildvmhosts... 15:11:50 I think I have to identify 15:11:55 cverna: I can sort it after the meeting... 15:12:23 Yeah 15:12:30 FYI, it should be all good in 5 min 15:12:38 so, datagrepper has been crashing... 15:13:24 it seemed like it was long queries for anitya messages? 15:14:08 I guess we should investigate more. 15:14:33 this is another pair of services I think we should rewrite/port to fedora-messaging... but I guess we will see. 15:14:56 You mean datagrepper/datanommer? 15:15:02 yeah. 15:16:12 there's a number of things from proxy31 that smooge is working to put in service, otherwise some low-swap and the normal fedmsg ones we have had for a while. 15:17:22 thats about it. 15:17:32 #topic Datacenter Move 15:17:47 So I wanted to talk a bit about the upcoming datacenter move... 15:18:05 supposedly the playbook on 31 has run completely but nagios complains about the non-existant socket file still 15:18:24 we are still writing up things for planning... but there's definitely things we will want help with 15:19:15 right now after 2020-04-06 (we hope) we will have some network access to the new datacenter and can start setting up new machines that are racked and waiting there. 15:20:06 once we have the base infra setup, we will need to start installing applications, etc... at that point we will want help on those, and/or help testing that things are working, etc 15:21:05 for koji and such we will likely set it up initially like staging... so we can confirm things work... and then redo it for prod before the switchover 15:21:08 Has all of the equipment been racked and cabled? 15:21:19 the things we are getting new yes. 15:21:31 Excellent 15:21:34 but we have to ship some things from phx2 over to the new dc 15:21:51 that will happen at some point (dates are fluid due to pandemic planning) 15:22:16 but the goal is that we switch over to the new dc the last week of may (or perhaps before for some things) 15:22:34 so it's going to be a lot of work and testing and fixing things... 15:23:30 so we will be sending out more detailed plans and asking for help as we go. :) 15:24:01 For those in the apprentice program (like me) can we help? 15:24:04 after we have net to the new datacenter, we may want to start parking incoming tickets too. 15:24:19 ie, 'sorry, we are busy, will get to your ticket later' 15:24:38 osa1989: absolutely. we are going to need a lot of testing from different places on apps as we stand them up. 15:24:58 adjustments to ansible playbooks, etc. 15:25:16 +1 for parking incoming tickets 15:25:34 also, week after next is final f32 freeze... 15:26:30 Anyhow, thats all, just wanted to bring it up and share some info... any questions or comments? 15:27:11 I don't have a RH badge, so I doubt this has much value. But I live in Maryland and can be in Herndon, VA in about an hour. 15:27:52 austinpowered: cool, thanks for the offer. We are actually trying to do as much remote as we can (to avoid anyone having to travel ) 15:28:29 and of course everything could get changed/delayed/canceled by the pandemic 15:29:00 No worries. Just wanted to let you the resource was available if needed. 15:29:03 but for right now at least we are moving forward. :) 15:29:34 Thanks. 15:29:59 #topic Open Floor 15:30:08 anyone have anything for open floor? 15:31:15 ok, if nothing else will let everyone go back to their isolation in a minute... 15:32:42 Thanks for coming everyone. 15:32:45 #endmeeting