15:00:34 <pingou> #startmeeting Infrastructure (2020-04-16)
15:00:34 <zodbot> Meeting started Thu Apr 16 15:00:34 2020 UTC.
15:00:34 <zodbot> This meeting is logged and archived in a public location.
15:00:34 <zodbot> The chair is pingou. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:34 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:00:34 <zodbot> The meeting name has been set to 'infrastructure_(2020-04-16)'
15:00:35 <pingou> #meetingname infrastructure
15:00:35 <zodbot> The meeting name has been set to 'infrastructure'
15:00:37 <pingou> #chair nirik pingou smooge cverna mizdebsk mkonecny abompard
15:00:37 <zodbot> Current chairs: abompard cverna mizdebsk mkonecny nirik pingou smooge
15:00:37 <clime> hi
15:00:39 <pingou> #info Agenda is at: https://board.net/p/fedora-infra
15:00:41 <pingou> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
15:00:43 <pingou> #topic aloha
15:01:03 <pingou> ó/ everyone
15:01:05 <austinpowered> good morning all
15:01:17 <nirik> morning
15:01:20 <Saffronique> o/
15:01:49 <mattiaV> morning/evening
15:02:47 <mkonecny> .hello zlopez
15:02:49 <zodbot> mkonecny: zlopez 'Michal Konečný' <michal.konecny@packetseekers.eu>
15:03:24 <pingou> we're 3 minutes past the hour, let's get started :)
15:03:26 <pingou> #topic New folks introductions
15:03:28 <pingou> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
15:03:30 <pingou> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
15:03:36 <pingou> any new folks who would like to introduce themselves?
15:03:44 <cyberworm54> morning
15:04:26 <cverna> hello
15:04:35 <smooge> bonjour
15:04:40 * cverna waves at mattiaV
15:05:22 <pingou> I'll take this as nonew folks :)
15:05:24 <pingou> #topic Next chair
15:05:26 <pingou> #info magic eight ball says:
15:05:28 <pingou> #info 2020-04-16 - pingou
15:05:36 <pingou> so who wants to chair a meeting ?
15:05:47 <mkonecny> pingou: I will take it :-)
15:05:47 <pingou> we have a seat available next week, April 23rd
15:05:55 <pingou> #info 2020-04-23 - mkonecny
15:05:57 <pingou> #info 2020-04-30 - smooge
15:05:59 <pingou> thank you mkonecny :)
15:06:06 <cverna> I can take the one after that
15:06:16 <pingou> the first one of May?
15:06:22 <pingou> it's all yours cverna
15:06:37 <cverna> yep :)
15:06:39 <pingou> #info 2020-05-07 - cverna
15:06:52 <pingou> moving on to announcements
15:06:54 <pingou> #topic announcements and information
15:06:56 <pingou> #info CPE Sustaining team has standups on Tuesday and Thursday at 1400 UTC in #fedora-admin - please join
15:06:58 <pingou> #info Fedora Infrastructure will be moving in 2020-06 from its Phoenix Az datacenter to one near Herndon Va. A lot of planning will be involved on this. Please watch out for announcements on changes.
15:07:00 <pingou> #info Fedora Communishift will be moving to new datacentre in April. Current downtime is expected to be from 2020-04-10 -> 2020-05-01. Please watch out for announcements on changes.
15:07:02 <pingou> #info F32 final freeze is in effect!
15:07:04 <pingou> #info Taskotron will EOL in 2020-05
15:07:06 <pingou> anything else people would like to announce?
15:07:07 <King_InuYasha> .hello ngompa
15:07:08 <zodbot> King_InuYasha: ngompa 'Neal Gompa' <ngompa13@gmail.com>
15:07:17 <nils> .hello nphilipp
15:07:18 <zodbot> nils: nphilipp 'Nils Philippsen' <nphilipp@redhat.com>
15:07:41 <King_InuYasha> so, somewhat tangentially related to fedora infra...
15:08:11 <King_InuYasha> myself and lcp in openSUSE have basically ~2 weeks to deploy new openSUSE Accounts system, so it looks like the Fedora AAA solution is going to break ground in openSUSE first
15:08:26 <King_InuYasha> this actually means that noggin needs to get packaged for Fedora 31
15:08:38 <pingou> abompard: ^
15:08:41 <King_InuYasha> hopefully the work done here can also help for Fedora infrastructure
15:08:51 * nirik has no idea if it's to that point yet.
15:08:58 <King_InuYasha> I've filed a dependency check ticket for noggin already: https://github.com/fedora-infra/noggin/issues/227
15:09:13 <pingou> King_InuYasha: this is a really good news, I do share nirik's ponder though
15:09:34 <King_InuYasha> pingou: it may not be ready for Fedora *now*, but it's better than the situation we have in openSUSE
15:09:41 <King_InuYasha> which is we're losing our account system in 4 weeks
15:09:53 <King_InuYasha> MicroFocus owns it, and they're killing it on the first week of May
15:09:59 <pingou> aouch :s
15:10:01 <nirik> bummer. ;(
15:10:04 <King_InuYasha> SUSE and openSUSE are splitting off into their own account systems for this
15:10:19 <King_InuYasha> and we're likely to get SUSE Bugzilla replaced with an instance based on RHBZ, because again MF is pulling the plug
15:10:25 * mboddu is here
15:10:46 <King_InuYasha> nirik: so yeah, I don't really have a choice anymore
15:10:56 <pingou> you may have to look at how to easily upgrade then, so you can easily get the improvements abompard and his team are making as they are making them
15:11:05 <King_InuYasha> yep
15:11:08 <nirik> well, I hope it can all work out. ;( pretty short runway tho...
15:11:19 <King_InuYasha> they don't call us openSUSE Heroes for nothing, apparently
15:11:33 <pingou> I hope you'll all get a cape at the end
15:11:38 * King_InuYasha hopes so too
15:11:54 <King_InuYasha> anyway, I was bringing this up because as part of timeline crunch, we're deploying everything using CentOS 8 and Fedora 31
15:12:00 <pingou> King_InuYasha: iirc there is a #fedora-aaa channel
15:12:02 <King_InuYasha> so this will be directly beneficial for Fedora Infra
15:12:10 <King_InuYasha> pingou: yup, I'm in it :)
15:12:15 <pingou> cool :)
15:12:46 <King_InuYasha> we don't have our data dump yet though, so... :o
15:12:52 <cverna> are we going to use s2i for noggin ?
15:12:57 <pingou> this is probably fine
15:13:32 * x3mboy wishes best luck to King_InuYasha
15:13:37 <King_InuYasha> cverna: I'd hope not... s2i would be a bit scary for prod stuff since it would mean we'd get weird versions of openssl and such
15:13:52 <King_InuYasha> and not all deps are in pypi
15:14:00 <pingou> cverna: looks like our AAA team isn't here, so we may have to ask again in another venue
15:14:02 <King_InuYasha> afaik
15:14:15 <cverna> pingou: +1
15:14:29 <King_InuYasha> anyway, I just wanted to let everyone know just incase anyone wanted to help :)
15:14:35 <pingou> g'd luck to you and the rest of the team King_InuYasha !
15:14:39 <King_InuYasha> thanks!
15:14:56 <pingou> any other announcement?
15:15:06 <King_InuYasha> ipsilon has been rebased in Fedora
15:15:22 <King_InuYasha> don't know what prod is running on, but since no python3-lasso in EL8, it's only available in Fedora
15:15:32 <nirik> we are on f31
15:15:34 <King_InuYasha> including bs4 based FAS theme contributed by lcp
15:15:57 <King_InuYasha> https://bodhi.fedoraproject.org/updates/FEDORA-2020-8ce489a068
15:16:12 <pingou> might be something to look into in stg
15:16:13 <King_InuYasha> ipsilon-theme-Fedora package includes it
15:16:33 <pingou> should we log this in a ticket?
15:16:42 <King_InuYasha> probably a good idea
15:16:51 <nirik> I'm not sure...
15:17:01 <nirik> I'm happy to update stg...
15:17:04 <King_InuYasha> my plan for ipsilon is to release version 3.0.0 once both Fedora and openSUSE ipsilon instances have been proven out
15:17:50 <King_InuYasha> I'd like to see staging updated to this code so we can be sure it works :)
15:18:12 <smooge> I hate to call for another meeting, but I think this is starting to go over any timebox we have for announcements and more of needing a specific meeting
15:18:30 <King_InuYasha> oh sorry
15:18:34 <King_InuYasha> anyway, that's it for me infra-wise
15:18:35 <pingou> nirik: I'm confused on what you are not  sure?
15:18:47 <pingou> the ticket or the upgrade? :)
15:18:55 <nirik> I am confused what a ticket would track? just updating staging? or ?
15:19:06 <pingou> updating stg yes
15:19:41 <smooge> King_InuYasha, no problem. I just want to make sure that we get the people who need to help you who aren't in this meeting to be with you on it
15:19:47 <pingou> anyway, we can see this after the meeting
15:20:00 <pingou> let's move to oncall!
15:20:02 <pingou> #topic Oncall
15:20:04 <pingou> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
15:20:06 <pingou> #info pingou is oncall 2020-04-09 -> 2020-04-16
15:20:12 <pingou> so who wants to be oncall? :)
15:21:19 <pingou> come on folks, it's not that hard :)
15:21:31 <cverna> I can take it if nobody wants :)
15:21:39 <siddharthvipul> I can try
15:21:42 <nirik> or I can... whats the schedule upcoming?
15:21:45 <siddharthvipul> not sure how much helpful I can be though
15:21:57 <cverna> siddharthvipul++
15:21:58 <pingou> nirik: there is none :)
15:22:12 <pingou> so siddharthvipul this week, cverna the next one?
15:22:19 <siddharthvipul> if it's about pinging pingou or cverna when something seems major problem, else requesting ticket.. I am in
15:22:27 <nirik> yep. thats exactly it...
15:22:30 <siddharthvipul> would help me understand some of the services for sure
15:22:45 <nirik> answering people, telling them to file tickets
15:22:55 <siddharthvipul> nirik: did I undersell the responsibility :P
15:22:58 <siddharthvipul> ahh, great
15:23:00 <cverna> works for me (ie vipul this week and me next week)
15:23:05 <pingou> trying to do a first triage of the tickets coming in
15:23:10 <pingou> #info siddharthvipul is oncall 2020-04-16 -> 2020-04-23
15:23:12 <pingou> #info cverna is oncall 2020-04-23-> 2020-04-30
15:23:14 <pingou> thank you siddharthvipul and cverna :)
15:23:21 <Saffronique> Do you folks do handovers between people who were oncall previously?
15:23:21 <siddharthvipul> :)
15:23:21 <pingou> #info Summary of last week: (from current oncall )
15:23:35 <Saffronique> aha question answered just as I asked :D
15:23:40 <pingou> #info fairly quiet week overall
15:23:49 <cverna> we need to add siddharthvipul  the capability to get the .oncall alias thing
15:24:01 <pingou> #info I saw cverna answered some of the by-passing questions
15:24:28 <pingou> #info I also answered some of them, but considering I don't remember much I'm inclined to think there was nothing major
15:24:32 <cverna> yeah I probably should not have :P but I was around
15:25:12 <pingou> it happens to all of us, we should just not make it a habbit :)
15:25:18 <pingou> #topic Monitoring discussion [nirik]
15:25:20 <pingou> #info https://nagios.fedoraproject.org/nagios
15:25:22 <pingou> #info Go over existing out items and fix
15:25:24 <nirik> lets see
15:25:54 <nirik> we have our 'swap is low' machines (rhel 8.1 bug)
15:26:16 <nirik> openqa-stg disk low. should ping adamw about that...
15:26:21 <adamw> again?!
15:26:29 <nirik> yeah. ;(
15:26:30 <adamw> didn't we just bump it?
15:26:41 <nirik> DISK WARNING - free space: / 84227 MiB (8.80% inode=100%):
15:26:44 <nirik> this is /
15:26:50 <King_InuYasha> oh dear :(
15:26:51 <adamw> oh
15:26:53 <nirik> on openqa-stg01
15:27:01 <adamw> let's talk in -admin ?
15:27:04 <nirik> sure. ;)
15:27:19 <nirik> then the only things left are the no fas fedmsgs and no planet fedmsgs
15:27:49 <pingou> pagure01 alerted about a low swap as well
15:28:10 <pingou> which seems to be still ongoing though there is some ram free
15:29:26 <nirik> yeah... it's done that before
15:30:29 <pingou> anything else?
15:30:36 <pingou> otherwise I'll open the open floor :)
15:31:05 <King_InuYasha> I got nothing :)
15:31:09 <pingou> #topic Open Floor
15:31:28 <pingou> the floor is yours if you would like it :)
15:31:46 <mkonecny> I will take it with me :-D
15:32:04 <nirik> it really brings the room together. :)
15:32:09 <pingou> alright, mkonecny has the floor
15:32:15 <pingou> who wants the ceiling?
15:32:18 <nirik> I have a short datacenter move update...
15:32:25 <pingou> oh cool :)
15:32:29 <pingou> nirik: you have the ceiling
15:33:01 <cverna> datacenter move is more like the underground than the ceiling :)
15:33:14 <nirik> first shipmet of things was deracked and shipped earlier this week. (communishift and some other stuff for rdu2 and some misc stuff for iad2). It is planned to get there next monday
15:33:30 <nirik> we are waiting for initial access to the new datacenter (iad2).
15:33:51 <nirik> as soon as we get that we are going to install a bootstrap box to install other boxes with and start building out the infra there.
15:34:04 <King_InuYasha> nice!
15:34:08 <pingou> should we have the initial access before or after the first shipmet arrives in iad2?
15:34:31 <pingou> (in theory)
15:34:32 <King_InuYasha> wait, what's iad2?
15:34:33 <nirik> late may is still the target to start moving over to it.... then first week of june will be shutting down phx2 and moving all of it (we hope)
15:34:40 <nirik> before.
15:34:49 <pingou> fingers crossed then
15:34:53 <nirik> we have a bunch of new hardware there thats been racked and cabled, but isn't on yet.
15:35:04 <nirik> King_InuYasha: our new datacenter in virginia
15:35:06 <cverna> King_InuYasha: the new data centre
15:35:07 <pingou> that's fustrating
15:35:18 <pingou> King_InuYasha: the new 127.0.0.1
15:35:56 <nirik> thats all I had I think, unless there's more questions
15:36:30 <King_InuYasha> so after 15 years, phx2 will burn into ashes, never to rise again?
15:36:44 <smooge> no it will fly off to a new head wizard
15:36:49 <King_InuYasha> haha :D
15:37:02 <mkonecny> minute of silence for phx2
15:37:08 <smooge> not yet please
15:37:09 <pingou> long live phx3
15:37:10 <Saffronique> f
15:37:11 <smooge> we need it til june
15:37:18 <cverna> lol yeah :)
15:37:28 <King_InuYasha> we can press 'f' in july ;)
15:37:33 <smooge> you can have your minute of silence in June while I am trying to get stuff working in IAD
15:37:55 <pingou> smooge: we'll have another minute for you and nirik then :)
15:38:03 <smooge> that is in July
15:38:05 * nirik considers adding a dc move status section to the meeting template
15:38:16 <smooge> we already planned our funeral pyres
15:38:27 <pingou> nirik: +1
15:38:27 <smooge> yeah sounds like a good idea
15:38:46 <pingou> alright, if nothing else, I'll give you ~20 minutes of your life back :)
15:38:53 <pingou> 5
15:38:56 <pingou> 4
15:38:58 <pingou> 3
15:39:00 <pingou> 2
15:39:00 <mkonecny> thanks pingou
15:39:02 <pingou> 1
15:39:04 <pingou> #endmeeting