15:00:19 <smooge> #startmeeting Infrastructure (2020-03-19)
15:00:19 <smooge> #meetingname infrastructure
15:00:19 <zodbot> Meeting started Thu Mar 19 15:00:19 2020 UTC.
15:00:19 <zodbot> This meeting is logged and archived in a public location.
15:00:19 <zodbot> The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:19 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:00:19 <zodbot> The meeting name has been set to 'infrastructure_(2020-03-19)'
15:00:19 <zodbot> The meeting name has been set to 'infrastructure'
15:00:19 <smooge> #chair nirik pingou smooge cverna mizdebsk mkonecny abompard
15:00:19 <zodbot> Current chairs: abompard cverna mizdebsk mkonecny nirik pingou smooge
15:00:20 <smooge> #info Agenda is at: https://board.net/p/fedora-infra
15:00:21 <smooge> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
15:00:23 <smooge> #topic aloha
15:00:31 <mizdebsk> hello
15:00:32 <nils> .hello nphilipp
15:00:37 * pingou aloha
15:00:39 <zodbot> nils: nphilipp 'Nils Philippsen' <nphilipp@redhat.com>
15:00:45 <nils> hi :)
15:00:48 <Southern_Gentlem> .hello jbwillia
15:00:49 <zodbot> Southern_Gentlem: jbwillia 'Ben Williams' <vaioof@gmail.com>
15:01:06 <nirik> morning
15:01:15 <austinpowered> .hello austinpowered
15:01:17 <zodbot> austinpowered: austinpowered 'None' <fedoraproject@wootenwilliams.com>
15:01:20 <tflink> morning
15:01:31 <cverna> hello
15:02:03 <smooge> #topic Next chair
15:02:03 <smooge> #info magic eight ball says:
15:02:03 <smooge> #info 2020-03-19 - smooge
15:02:03 <smooge> #info 2020-03-26 - nirik
15:02:03 <smooge> #info 2020-04-02 - ????
15:02:04 <smooge> #info 2020-04-09 - ????
15:02:30 <smooge> any takers for the April Not Fools Day meeting?
15:02:38 * nirik hopes he remembers next week. ;)
15:02:51 <cverna> can do
15:03:19 <cverna> the 04-02 is for me :)
15:04:04 <smooge> nirik, I found the easy way to remember you have the next meeting is to take oncall
15:04:19 <nirik> ha, good thought
15:04:22 <smooge> #topic announcements and information
15:04:22 <smooge> #info ops folks are doing a 30min ticket triage every day at 19UTC in #fedora-admin - please join
15:04:22 <smooge> #info CPE Sustaining team has daily standup (Monday-Thursday) at 15UTC in #fedora-admin - please join
15:04:22 <smooge> #info bodhi 5.2 released, plan is to deploy in production Mon 03.23 - cverna
15:04:23 <smooge> #info Fedora Infrastructure will be moving in 2020-06 from its Phoenix Az datacenter to one near Herndon Va. A lot of planning will be involved on this. Please watch out for announcements on changes.
15:04:25 <smooge> #info Fedora Communishift will be moving to new datacentre in April. Current downtime is expected to be from 2020-04-10 -> 2020-05-01. Please watch out for announcements on changes.
15:04:30 <smooge> #info Taskotron will EOL in 2020-05
15:04:33 <pingou> I think we shold couple them (oncall and meeting)
15:04:45 <pingou> but that didn't prevent me from forgetting it last time :(
15:04:48 <nirik> #info Fedora 32 beta released!
15:04:56 <nirik> #info Fedora 32 beta freeze over
15:05:05 <pingou> \ó/
15:05:12 <smooge> cverna, is that time for the 15UTC meeting correct? Since it is 1500 UTC right now?
15:05:16 <smooge> .time
15:05:16 <zodbot> smooge: 03:05 PM, March 19, 2020
15:05:40 <cverna> hey I guess it is 1400 UTC :)
15:05:57 <smooge> fuxed
15:06:13 <cverna> thanks :)
15:06:15 <smooge> congrats to the people who got F32 Beta out the door
15:06:26 <smooge> #topic Oncall
15:06:26 <smooge> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
15:06:26 <smooge> #info smooge is oncall 2020-03-12 -> 2020-03-19
15:06:26 <smooge> #info cverna is oncall 2020-03-19 -> 2020-03-26
15:06:27 <smooge> #info ???? is oncall 2020-03-26 -> 2020-04-02
15:06:44 <smooge> so to make this work.. nirik and cverna should swap taking over meetings and oncall :P
15:06:55 <nirik> sure, if you like.
15:07:08 <cverna> works for me
15:07:15 <smooge> ok fixing document
15:07:34 <nirik> we should also add the 'contacting us' doc to the oncall links. I can do that after the meeting if everyone is ok with it.
15:07:41 <smooge> #info magic eight ball says:
15:07:41 <smooge> #info 2020-03-26 - cverna
15:07:41 <smooge> #info 2020-04-02 - nirik
15:07:41 <smooge> #info 2020-04-09 - ????
15:07:57 <smooge> #info cverna is oncall 2020-03-19 -> 2020-03-26
15:07:57 <smooge> #info nirk is oncall 2020-03-26 -> 2020-04-02
15:08:11 <nirik> other way?
15:08:18 <smooge> ok
15:08:53 <smooge> #info 2020-03-26 - nirik
15:08:53 <smooge> #info 2020-04-02 - cverna
15:08:53 <smooge> #info nirik is oncall 2020-03-19 -> 2020-03-26
15:08:53 <smooge> #info cverna is oncall 2020-03-26 -> 2020-04-02
15:08:57 <smooge> there fixed
15:09:02 <smooge> final.. not changing it again
15:09:10 <pingou> sure?
15:09:17 <pingou> even if we say pretty please? :)
15:09:25 <nirik> final answer! oh wait, want to phone a friend! :)
15:09:40 <pingou> ^^
15:09:41 <smooge> #info ???? is oncall 2020-04-02 -> 2020-04-09
15:09:48 <nirik> .takeoncallus
15:09:54 <nirik> .oncalltakeus
15:09:55 <zodbot> nirik: Kneel before zod!
15:10:21 * cverna is confused now :P
15:10:35 <nirik> cverna: ?
15:10:48 <cverna> ok :)
15:11:11 <nirik> I thought we wanted oncall to match when you are running the meeting so you remember you are running the meeting...
15:11:29 <smooge> I think cverna thought he was agreeing to what we wrote that he would take oncall next week and meeting and you would do the next next meeting
15:11:39 <smooge> uses the royal we
15:12:04 <cverna> yeah I really don't mind, I can do this week or next, nirik up to you
15:12:08 <nirik> ah, ok.
15:12:08 <smooge> anyway..
15:12:12 <nirik> I don't care either. ;)
15:12:20 <smooge> #info Summary of last week:
15:12:38 <smooge> #info Mostly dealing with people trying to create accounts and having problems
15:13:11 <smooge> #info Outages were self-induced due to smooge 'fixing' DNS
15:13:21 <smooge> that is all
15:13:26 <cverna> lol ok so nirik you do this week and I ll do next :)
15:13:34 <cverna> +1
15:13:53 <nirik> ok, ack
15:13:54 <smooge> anyway.. nirik does this week, cverna does next week, pingou thinks about doing the week afterwords
15:14:16 * pingou hoping not to land this on smooge the week afterword
15:14:23 <pingou> .members sysadmin-main
15:14:24 <zodbot> pingou: Members of sysadmin-main: codeblock cverna jstanley @kevin mizdebsk mohanboddu pbrobinson pingou puiterwijk @smooge tflink
15:14:27 <smooge> he will be running a wire to his boat which he will have flipped over as his man cave
15:14:39 <pingou> I do wonder if we shouldn't cycle oncall through all sysadmin-main
15:14:51 <smooge> .members sysadmin-main
15:14:52 <zodbot> smooge: Members of sysadmin-main: codeblock cverna jstanley @kevin mizdebsk mohanboddu pbrobinson pingou puiterwijk @smooge tflink
15:15:09 <pingou> (you get super-power, but you get to be oncall once in a while)
15:15:10 <nirik> well, some are not as active anymore. ;)
15:15:20 <pingou> spring cleaning time? :)
15:15:26 <pingou> oh wait, we have to wait 2 days for this
15:15:27 <smooge> maybe they should be in sysadmin-emeritius
15:15:33 <tflink> I don't anticipate being sysadmin-main after taskotron goes EOL next month
15:15:42 <nirik> well, perhaps. But last I asked, most of them wanted to remain.
15:15:53 <cverna> pingou: I think it is not limited to sysadmin-main, anyone can do it this is mostly a filter for requests
15:15:58 <pingou> nirik: oncall may change their mind :D
15:16:04 <pingou> cverna: agreed
15:16:10 <mizdebsk> i can participate in oncall if needed
15:16:20 <mizdebsk> but people seem to eagerly take it before i do :)
15:16:24 <pbrobinson> I don't use it much but it's useful for me because it means I don't need to put load on others to get IoT things done
15:16:47 <smooge> anyway.. sorry derailing my own meeting
15:16:56 <smooge> #topic Monitoring discussion [nirik]
15:16:56 <smooge> #info https://nagios.fedoraproject.org/nagios
15:16:56 <smooge> #info Go over existing out items and fix
15:17:01 <pbrobinson> I physically don't have the bandwidth to do on call though, so if that becomes a requirement I'll just remove it and file tickets for the things I do now
15:17:36 <tflink> yeah, same here. I'm game for leaving sysadmin-main but that'll mean more work for you all in terms of deprecating and turning off taskotron
15:17:43 <nirik> ok, there's an aarch64 builder down, we can try re-power cycling it.
15:18:28 <nirik> well, I don't want us to have more work. :) but perhaps we should consider this and discuss it as another topic down the road?
15:18:52 <pingou> it was an idea, it's fine turning it down
15:19:15 <nirik> there's a bunch of virthosts low on swap, which is a rhel8 kernel bug... I wonder if we shouldn't do a update/reboot cycle next week?
15:19:39 <smooge> i don't think the kernel bug has been pushed out yet
15:19:47 <smooge> kernel bug fix
15:19:53 <nirik> there's a whatcanidoforfedora.org cert issue. That should be fixed by running the playbook, which I can do.
15:20:12 <smooge> nirik, ok thanks on that.
15:20:32 <smooge> nirik, the power cycle problem is still with the pdu not working. I need to put in a ticket for that
15:20:45 <nirik> qa-stg01.qa.fedoraproject.org - Disk_Space_/
15:20:54 <nirik> tflink: ^ perhaps you could look at that one?
15:21:10 <nirik> proxy31 is the other one.
15:21:55 <nirik> note: to get registry working on proxy31 you MUST run the oci-registries playbook. It updates the firewall there based on the proxy group... so if you add something there you have to re-run it there to update the iptables setup
15:21:56 <tflink> nirik: yeah, I need to delete that host. I was waiting for beta freeze to end
15:22:31 <nirik> ok, anytime now then. ;)
15:22:52 <tflink> yeah, it's on my todo list for today
15:23:25 <smooge> nirik, ok I didn't know that. I will run the playbook before I add proxy31 to being an active proxy
15:23:57 <tflink> resultsdb01.qa is also having swap issues but we're not sure why. the swap just gradually goes to 0 until httpd is restarted
15:24:19 <nirik> some kind of leak in httpd?
15:24:20 <x3mboy> .hello2
15:24:21 <zodbot> x3mboy: x3mboy 'Eduard Lucena' <eduardlucena@gmail.com>
15:24:52 <tflink> that could make sense. the resultsdb code hasn't changed in a while
15:25:24 <smooge> #topic Tickets discussion [nirik]
15:25:24 <smooge> #info https://pagure.io/fedora-infrastructure/report/Meetings%20ticket
15:25:44 <smooge> nirik, since we do this daily now.. do you want me to drop this from the weekly meetings?
15:25:47 <nirik> any tickets people want to discuss/hilight?
15:25:54 <nirik> yeah, we could.
15:26:21 <smooge> #topic backlog discussion
15:26:21 <smooge> #info go over our backlog and discuss and determine priority
15:26:46 <smooge> cverna, I don't know if the tickets in the board.net are for this week or last weeks?
15:27:04 <cverna> ha yeah this is last week
15:27:32 <cverna> I forgot to update that, but we can use the mailing list for it
15:27:33 <smooge> I am going to put you as the owner of this section in future meetings
15:27:45 <cverna> +1
15:27:59 <nirik> do we think thats helping out? I like it so far... would be good to get more input... but oh well.
15:28:38 <nirik> we even have one in the best state...
15:28:42 <smooge> i think at this point it is too soon to know if it is helping or not. I don't see a large pool of people having any time to help out so our input is what it is
15:28:45 <cverna> I find value in it, that allow to share more info about these tickets
15:28:46 <nirik> .ticket 8167
15:28:48 <pingou> as an outsider, I have sometime issues evaluating the amount of work
15:28:49 <zodbot> nirik: Issue #8167: Adding topic authorization to our RabbitMQ instances - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8167
15:29:10 <nirik> low-trouble, high-gain.
15:29:19 <pingou> +1 on the high gain
15:29:19 <nirik> yeah, we are all bad at estimating. ;)
15:29:54 <cverna> yeah now tat freeze is over we could try to work on that
15:30:22 <nirik> stg should be the new version (we still need to update prod)
15:31:21 <cverna> is it a matter of adapting the playbook and running it against prod ?
15:31:45 <pingou> we had a few issues in stg where users/queues were not correctly re-created
15:31:53 <pingou> we may need to see how we can fix this
15:32:27 <nirik> cverna: the upgrade for prod? no. Thats: take the cluster down, reinstall as rhel8, resetup cluster
15:32:28 <pingou> maybe add the tag in the rabbitmq/... roles so if some app role don't have the corresponding tag, we can still re-create?
15:32:52 <cverna> nirik: ha ok a bit more work then :)
15:32:54 <nirik> yeah, I think tags were incomplete...
15:33:10 <nirik> cverna: the actual setting up of the auth still needs doing tho.
15:33:19 <nirik> (in stg)
15:33:27 <smooge> tflink, I see an item in the agenda for moving stuff to openshift.. is that for this meeting or last?
15:33:44 <tflink> last meeting, I think
15:33:52 * tflink looks in case he forgot adding something
15:34:02 * relrod waves and checks into the meeting super late.
15:34:23 <smooge> relrod you have oncall in 3 weeks :P
15:34:53 <tflink> smooge: yeah, that was last week
15:34:59 <pingou> hey relrod
15:35:13 <relrod> smooge: Let's see what my new workload looks like for a bit first ;)
15:35:28 <relrod> I'd not be opposed to occasionally taking a shift to stay in the loop though.
15:35:48 <smooge> #topic Open Floor
15:36:01 <smooge> ok anything from the floor
15:36:12 <cverna> nothing here
15:36:16 <nirik> stay safe out there everyone!
15:36:28 <relrod> yes that ^
15:36:47 <nils> thirded
15:37:51 <pingou> fourthed
15:38:09 <smooge> <<goes back into his plastic bubble>>
15:38:32 <smooge> carbonite freezing was my second choice but expensive
15:38:32 <pingou> smooge: https://duckduckgo.com/?t=ffab&q=plastic+bertrand&iax=images&ia=images ?
15:39:27 <smooge> https://duckduckgo.com/?q=plastic+human+bubble&t=ffab&iar=images&iax=images&ia=images
15:39:30 <smooge> #endmeeting