15:00:15 <nirik> #startmeeting Infrastructure (2020-03-26)
15:00:15 <zodbot> Meeting started Thu Mar 26 15:00:15 2020 UTC.
15:00:15 <zodbot> This meeting is logged and archived in a public location.
15:00:15 <zodbot> The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:15 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:00:15 <zodbot> The meeting name has been set to 'infrastructure_(2020-03-26)'
15:00:16 <nirik> #meetingname infrastructure
15:00:16 <nirik> #chair nirik pingou smooge cverna mizdebsk mkonecny abompard
15:00:16 <zodbot> The meeting name has been set to 'infrastructure'
15:00:16 <zodbot> Current chairs: abompard cverna mizdebsk mkonecny nirik pingou smooge
15:00:16 <nirik> #info Agenda is at: https://board.net/p/fedora-infra
15:00:16 <nirik> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
15:00:17 <nirik> #topic aloha
15:00:39 <mkonecny> .hello zlopez
15:00:40 <zodbot> mkonecny: zlopez 'Michal Konečný' <michal.konecny@packetseekers.eu>
15:00:51 <cverna> hello
15:01:04 <nirik> #topic New folks introductions
15:01:04 <nirik> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
15:01:04 <nirik> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
15:01:09 <nirik> any new folks around today?
15:01:12 <austinpowered> .hello
15:01:14 <zodbot> austinpowered: (hello <an alias, 1 argument>) -- Alias for "hellomynameis $1".
15:01:33 * cyberpear tunes in
15:01:40 <osa1989> Hello
15:01:43 <mymindstorm> hi
15:01:45 <osa1989> I am new :-)
15:01:48 <austinpowered> .hello austinpowered
15:01:49 <zodbot> austinpowered: austinpowered 'None' <fedoraproject@wootenwilliams.com>
15:01:55 <cyberpear> .hello2
15:01:56 <zodbot> cyberpear: cyberpear 'James Cassell' <fedoraproject@cyberpear.com>
15:02:06 * pingou around but in something else as well
15:02:11 * smooge is fighting zombies and will be elsewhere
15:02:14 <nirik> welcome osa1989. :) Care to give a short one line intro? and do see the getting started guide...
15:02:56 <Southern_Gentlem> .hello jbwillia
15:02:57 <zodbot> Southern_Gentlem: jbwillia 'Ben Williams' <vaioof@gmail.com>
15:03:15 <osa1989> I am 30 years old, joined red hat as a consultant few months ago, as I miss my sysadmin work, I decided to join here and do some sysadmin work with the community :-)
15:03:40 <nirik> cool. Welcome again...
15:03:58 <nirik> ### Determine who the next chair is
15:03:58 <nirik> #topic Next chair
15:03:58 <nirik> #info magic eight ball says:
15:03:59 <nirik> #info 2020-03-26 - nirik
15:03:59 <nirik> #info 2020-04-02 - cverna
15:04:00 <nirik> #info 2020-04-09 - ????
15:04:11 <nirik> anyone want to take the 04-09 meeting? :)
15:04:26 <osa1989> Before Red hat I was a sysadmin in a German company "Drillisch AG", worked with a lot of tools/software such as ceph, openstack, haproxy, keepalived, ldirectord
15:04:39 <cverna> osa welcome
15:04:55 <nirik> nice. we use some of those for sure.
15:04:56 <cverna> Rhh osa1989 welcome ☺️
15:05:11 <osa1989> Thanks everyone :-)
15:05:17 <nirik> I guess we can punt and figure out the 04-09 meeting next week.
15:05:28 <nirik> #topic announcements and information
15:05:29 <nirik> #info CPE Sustaining team has daily standups (Monday-Thursday) at 1400 and 1900 UTC in #fedora-admin - please join
15:05:29 <nirik> #info bodhi 5.2 released and deployed still looking at fixing couple of issues - cverna
15:05:29 <nirik> #info Fedora Infrastructure will be moving in 2020-06 from its Phoenix Az datacenter to one near Herndon Va. A lot of planning will be involved on this. Please watch out for announcements on changes.
15:05:29 <nirik> #info Fedora Communishift will be moving to new datacentre in April. Current downtime is expected to be from 2020-04-10 -> 2020-05-01. Please watch out for announcements on changes.
15:05:32 <nirik> #info Taskotron will EOL in 2020-05
15:05:37 <nirik> any other announcements or infos?
15:07:00 <nirik> ok then, moving on
15:07:03 <nirik> #topic Oncall
15:07:03 <nirik> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
15:07:03 <nirik> #info nirik is oncall 2020-03-19 -> 2020-03-26
15:07:03 <nirik> #info cverna is oncall 2020-03-26 -> 2020-04-02
15:07:04 <nirik> #info ???? is oncall 2020-04-02 -> 2020-04-09
15:07:25 <nirik> oncall was pretty quiet... lots of tickets tho... so I think people are getting the idea that they should just file a ticket.
15:07:37 <nirik> cverna: you want to take over now? and anyone want the following week?
15:08:17 <cverna> .takeoncalleu
15:08:45 <nirik> it's oncalltakeeu :)
15:08:54 <nirik> we should just alias those together
15:09:30 <cverna> Ah ah trying again
15:09:45 <cverna> .oncalltakeeu
15:09:47 <zodbot> cverna: Error: You don't have the alias.add capability. If you think that you should have this capability, be sure that you are identified before trying again. The 'whoami' command can tell you if you're identified.
15:10:18 <mkonecny> I always thought is oncall take us, like you literally saying oncall to take us :-D
15:10:21 <nirik> zodbot: capability add cverna +Alias
15:10:24 <zodbot> nirik: Error: The command "capability add" is available in the Admin and Channel plugins.  Please specify the plugin whose command you wish to call by using its name as a command before "capability add".
15:10:29 <mkonecny> Until I saw the eu one
15:10:32 <nirik> zodbot: admin capability add cverna +Alias
15:10:32 <zodbot> nirik: Kneel before zod!
15:10:46 <nirik> mkonecny: :)
15:11:07 <nirik> #topic Monitoring discussion [nirik]
15:11:07 <nirik> #info https://nagios.fedoraproject.org/nagios
15:11:07 <nirik> #info Go over existing out items and fix
15:11:29 <cverna> .oncalltakeeu
15:11:29 <zodbot> cverna: Error: You don't have the alias.add capability. If you think that you should have this capability, be sure that you are identified before trying again. The 'whoami' command can tell you if you're identified.
15:11:42 <nirik> so it's a bit red looking right now because mboddu is rebooting some buildvmhosts...
15:11:50 <cverna> I think I have to identify
15:11:55 <nirik> cverna: I can sort it after the meeting...
15:12:23 <cverna> Yeah
15:12:30 <mboddu> FYI, it should be all good in 5 min
15:12:38 <nirik> so, datagrepper has been crashing...
15:13:24 <nirik> it seemed like it was long queries for anitya messages?
15:14:08 <nirik> I guess we should investigate more.
15:14:33 <nirik> this is another pair of services I think we should rewrite/port to fedora-messaging... but I guess we will see.
15:14:56 <mkonecny> You mean datagrepper/datanommer?
15:15:02 <nirik> yeah.
15:16:12 <nirik> there's a number of things from proxy31 that smooge is working to put in service, otherwise some low-swap and the normal fedmsg ones we have had for a while.
15:17:22 <nirik> thats about it.
15:17:32 <nirik> #topic Datacenter Move
15:17:47 <nirik> So I wanted to talk a bit about the upcoming datacenter move...
15:18:05 <smooge> supposedly the playbook on 31 has run completely but nagios complains about the non-existant socket file still
15:18:24 <nirik> we are still writing up things for planning... but there's definitely things we will want help with
15:19:15 <nirik> right now after 2020-04-06 (we hope) we will have some network access to the new datacenter and can start setting up new machines that are racked and waiting there.
15:20:06 <nirik> once we have the base infra setup, we will need to start installing applications, etc... at that point we will want help on those, and/or help testing that things are working, etc
15:21:05 <nirik> for koji and such we will likely set it up initially like staging... so we can confirm things work... and then redo it for prod before the switchover
15:21:08 <austinpowered> Has all of the equipment been racked and cabled?
15:21:19 <nirik> the things we are getting new yes.
15:21:31 <austinpowered> Excellent
15:21:34 <nirik> but we have to ship some things from phx2 over to the new dc
15:21:51 <nirik> that will happen at some point (dates are fluid due to pandemic planning)
15:22:16 <nirik> but the goal is that we switch over to the new dc the last week of may (or perhaps before for some things)
15:22:34 <nirik> so it's going to be a lot of work and testing and fixing things...
15:23:30 <nirik> so we will be sending out more detailed plans and asking for help as we go. :)
15:24:01 <osa1989> For those in the apprentice program (like me) can we help?
15:24:04 <nirik> after we have net to the new datacenter, we may want to start parking incoming tickets too.
15:24:19 <nirik> ie, 'sorry, we are busy, will get to your ticket later'
15:24:38 <nirik> osa1989: absolutely. we are going to need a lot of testing from different places on apps as we stand them up.
15:24:58 <nirik> adjustments to ansible playbooks, etc.
15:25:16 <cverna> +1 for parking incoming tickets
15:25:34 <nirik> also, week after next is final f32 freeze...
15:26:30 <nirik> Anyhow, thats all, just wanted to bring it up and share some info... any questions or comments?
15:27:11 <austinpowered> I don't have a RH badge, so I doubt this has much value. But I live in Maryland and can be in Herndon, VA in about an hour.
15:27:52 <nirik> austinpowered: cool, thanks for the offer. We are actually trying to do as much remote as we can (to avoid anyone having to travel )
15:28:29 <nirik> and of course everything could get changed/delayed/canceled by the pandemic
15:29:00 <austinpowered> No worries. Just wanted to let you the resource was available if needed.
15:29:03 <nirik> but for right now at least we are moving forward. :)
15:29:34 <nirik> Thanks.
15:29:59 <nirik> #topic Open Floor
15:30:08 <nirik> anyone have anything for open floor?
15:31:15 <nirik> ok, if nothing else will let everyone go back to their isolation in a minute...
15:32:42 <nirik> Thanks for coming everyone.
15:32:45 <nirik> #endmeeting