15:00:19 #startmeeting Infrastructure (2020-03-19) 15:00:19 #meetingname infrastructure 15:00:19 Meeting started Thu Mar 19 15:00:19 2020 UTC. 15:00:19 This meeting is logged and archived in a public location. 15:00:19 The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:19 Useful Commands: #action #agreed #halp #info #idea #link #topic. 15:00:19 The meeting name has been set to 'infrastructure_(2020-03-19)' 15:00:19 The meeting name has been set to 'infrastructure' 15:00:19 #chair nirik pingou smooge cverna mizdebsk mkonecny abompard 15:00:19 Current chairs: abompard cverna mizdebsk mkonecny nirik pingou smooge 15:00:20 #info Agenda is at: https://board.net/p/fedora-infra 15:00:21 #info About our team: https://docs.fedoraproject.org/en-US/cpe/ 15:00:23 #topic aloha 15:00:31 hello 15:00:32 .hello nphilipp 15:00:37 * pingou aloha 15:00:39 nils: nphilipp 'Nils Philippsen' 15:00:45 hi :) 15:00:48 .hello jbwillia 15:00:49 Southern_Gentlem: jbwillia 'Ben Williams' 15:01:06 morning 15:01:15 .hello austinpowered 15:01:17 austinpowered: austinpowered 'None' 15:01:20 morning 15:01:31 hello 15:02:03 #topic Next chair 15:02:03 #info magic eight ball says: 15:02:03 #info 2020-03-19 - smooge 15:02:03 #info 2020-03-26 - nirik 15:02:03 #info 2020-04-02 - ???? 15:02:04 #info 2020-04-09 - ???? 15:02:30 any takers for the April Not Fools Day meeting? 15:02:38 * nirik hopes he remembers next week. ;) 15:02:51 can do 15:03:19 the 04-02 is for me :) 15:04:04 nirik, I found the easy way to remember you have the next meeting is to take oncall 15:04:19 ha, good thought 15:04:22 #topic announcements and information 15:04:22 #info ops folks are doing a 30min ticket triage every day at 19UTC in #fedora-admin - please join 15:04:22 #info CPE Sustaining team has daily standup (Monday-Thursday) at 15UTC in #fedora-admin - please join 15:04:22 #info bodhi 5.2 released, plan is to deploy in production Mon 03.23 - cverna 15:04:23 #info Fedora Infrastructure will be moving in 2020-06 from its Phoenix Az datacenter to one near Herndon Va. A lot of planning will be involved on this. Please watch out for announcements on changes. 15:04:25 #info Fedora Communishift will be moving to new datacentre in April. Current downtime is expected to be from 2020-04-10 -> 2020-05-01. Please watch out for announcements on changes. 15:04:30 #info Taskotron will EOL in 2020-05 15:04:33 I think we shold couple them (oncall and meeting) 15:04:45 but that didn't prevent me from forgetting it last time :( 15:04:48 #info Fedora 32 beta released! 15:04:56 #info Fedora 32 beta freeze over 15:05:05 \รณ/ 15:05:12 cverna, is that time for the 15UTC meeting correct? Since it is 1500 UTC right now? 15:05:16 .time 15:05:16 smooge: 03:05 PM, March 19, 2020 15:05:40 hey I guess it is 1400 UTC :) 15:05:57 fuxed 15:06:13 thanks :) 15:06:15 congrats to the people who got F32 Beta out the door 15:06:26 #topic Oncall 15:06:26 #info https://fedoraproject.org/wiki/Infrastructure/Oncall 15:06:26 #info smooge is oncall 2020-03-12 -> 2020-03-19 15:06:26 #info cverna is oncall 2020-03-19 -> 2020-03-26 15:06:27 #info ???? is oncall 2020-03-26 -> 2020-04-02 15:06:44 so to make this work.. nirik and cverna should swap taking over meetings and oncall :P 15:06:55 sure, if you like. 15:07:08 works for me 15:07:15 ok fixing document 15:07:34 we should also add the 'contacting us' doc to the oncall links. I can do that after the meeting if everyone is ok with it. 15:07:41 #info magic eight ball says: 15:07:41 #info 2020-03-26 - cverna 15:07:41 #info 2020-04-02 - nirik 15:07:41 #info 2020-04-09 - ???? 15:07:57 #info cverna is oncall 2020-03-19 -> 2020-03-26 15:07:57 #info nirk is oncall 2020-03-26 -> 2020-04-02 15:08:11 other way? 15:08:18 ok 15:08:53 #info 2020-03-26 - nirik 15:08:53 #info 2020-04-02 - cverna 15:08:53 #info nirik is oncall 2020-03-19 -> 2020-03-26 15:08:53 #info cverna is oncall 2020-03-26 -> 2020-04-02 15:08:57 there fixed 15:09:02 final.. not changing it again 15:09:10 sure? 15:09:17 even if we say pretty please? :) 15:09:25 final answer! oh wait, want to phone a friend! :) 15:09:40 ^^ 15:09:41 #info ???? is oncall 2020-04-02 -> 2020-04-09 15:09:48 .takeoncallus 15:09:54 .oncalltakeus 15:09:55 nirik: Kneel before zod! 15:10:21 * cverna is confused now :P 15:10:35 cverna: ? 15:10:48 ok :) 15:11:11 I thought we wanted oncall to match when you are running the meeting so you remember you are running the meeting... 15:11:29 I think cverna thought he was agreeing to what we wrote that he would take oncall next week and meeting and you would do the next next meeting 15:11:39 uses the royal we 15:12:04 yeah I really don't mind, I can do this week or next, nirik up to you 15:12:08 ah, ok. 15:12:08 anyway.. 15:12:12 I don't care either. ;) 15:12:20 #info Summary of last week: 15:12:38 #info Mostly dealing with people trying to create accounts and having problems 15:13:11 #info Outages were self-induced due to smooge 'fixing' DNS 15:13:21 that is all 15:13:26 lol ok so nirik you do this week and I ll do next :) 15:13:34 +1 15:13:53 ok, ack 15:13:54 anyway.. nirik does this week, cverna does next week, pingou thinks about doing the week afterwords 15:14:16 * pingou hoping not to land this on smooge the week afterword 15:14:23 .members sysadmin-main 15:14:24 pingou: Members of sysadmin-main: codeblock cverna jstanley @kevin mizdebsk mohanboddu pbrobinson pingou puiterwijk @smooge tflink 15:14:27 he will be running a wire to his boat which he will have flipped over as his man cave 15:14:39 I do wonder if we shouldn't cycle oncall through all sysadmin-main 15:14:51 .members sysadmin-main 15:14:52 smooge: Members of sysadmin-main: codeblock cverna jstanley @kevin mizdebsk mohanboddu pbrobinson pingou puiterwijk @smooge tflink 15:15:09 (you get super-power, but you get to be oncall once in a while) 15:15:10 well, some are not as active anymore. ;) 15:15:20 spring cleaning time? :) 15:15:26 oh wait, we have to wait 2 days for this 15:15:27 maybe they should be in sysadmin-emeritius 15:15:33 I don't anticipate being sysadmin-main after taskotron goes EOL next month 15:15:42 well, perhaps. But last I asked, most of them wanted to remain. 15:15:53 pingou: I think it is not limited to sysadmin-main, anyone can do it this is mostly a filter for requests 15:15:58 nirik: oncall may change their mind :D 15:16:04 cverna: agreed 15:16:10 i can participate in oncall if needed 15:16:20 but people seem to eagerly take it before i do :) 15:16:24 I don't use it much but it's useful for me because it means I don't need to put load on others to get IoT things done 15:16:47 anyway.. sorry derailing my own meeting 15:16:56 #topic Monitoring discussion [nirik] 15:16:56 #info https://nagios.fedoraproject.org/nagios 15:16:56 #info Go over existing out items and fix 15:17:01 I physically don't have the bandwidth to do on call though, so if that becomes a requirement I'll just remove it and file tickets for the things I do now 15:17:36 yeah, same here. I'm game for leaving sysadmin-main but that'll mean more work for you all in terms of deprecating and turning off taskotron 15:17:43 ok, there's an aarch64 builder down, we can try re-power cycling it. 15:18:28 well, I don't want us to have more work. :) but perhaps we should consider this and discuss it as another topic down the road? 15:18:52 it was an idea, it's fine turning it down 15:19:15 there's a bunch of virthosts low on swap, which is a rhel8 kernel bug... I wonder if we shouldn't do a update/reboot cycle next week? 15:19:39 i don't think the kernel bug has been pushed out yet 15:19:47 kernel bug fix 15:19:53 there's a whatcanidoforfedora.org cert issue. That should be fixed by running the playbook, which I can do. 15:20:12 nirik, ok thanks on that. 15:20:32 nirik, the power cycle problem is still with the pdu not working. I need to put in a ticket for that 15:20:45 qa-stg01.qa.fedoraproject.org - Disk_Space_/ 15:20:54 tflink: ^ perhaps you could look at that one? 15:21:10 proxy31 is the other one. 15:21:55 note: to get registry working on proxy31 you MUST run the oci-registries playbook. It updates the firewall there based on the proxy group... so if you add something there you have to re-run it there to update the iptables setup 15:21:56 nirik: yeah, I need to delete that host. I was waiting for beta freeze to end 15:22:31 ok, anytime now then. ;) 15:22:52 yeah, it's on my todo list for today 15:23:25 nirik, ok I didn't know that. I will run the playbook before I add proxy31 to being an active proxy 15:23:57 resultsdb01.qa is also having swap issues but we're not sure why. the swap just gradually goes to 0 until httpd is restarted 15:24:19 some kind of leak in httpd? 15:24:20 .hello2 15:24:21 x3mboy: x3mboy 'Eduard Lucena' 15:24:52 that could make sense. the resultsdb code hasn't changed in a while 15:25:24 #topic Tickets discussion [nirik] 15:25:24 #info https://pagure.io/fedora-infrastructure/report/Meetings%20ticket 15:25:44 nirik, since we do this daily now.. do you want me to drop this from the weekly meetings? 15:25:47 any tickets people want to discuss/hilight? 15:25:54 yeah, we could. 15:26:21 #topic backlog discussion 15:26:21 #info go over our backlog and discuss and determine priority 15:26:46 cverna, I don't know if the tickets in the board.net are for this week or last weeks? 15:27:04 ha yeah this is last week 15:27:32 I forgot to update that, but we can use the mailing list for it 15:27:33 I am going to put you as the owner of this section in future meetings 15:27:45 +1 15:27:59 do we think thats helping out? I like it so far... would be good to get more input... but oh well. 15:28:38 we even have one in the best state... 15:28:42 i think at this point it is too soon to know if it is helping or not. I don't see a large pool of people having any time to help out so our input is what it is 15:28:45 I find value in it, that allow to share more info about these tickets 15:28:46 .ticket 8167 15:28:48 as an outsider, I have sometime issues evaluating the amount of work 15:28:49 nirik: Issue #8167: Adding topic authorization to our RabbitMQ instances - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8167 15:29:10 low-trouble, high-gain. 15:29:19 +1 on the high gain 15:29:19 yeah, we are all bad at estimating. ;) 15:29:54 yeah now tat freeze is over we could try to work on that 15:30:22 stg should be the new version (we still need to update prod) 15:31:21 is it a matter of adapting the playbook and running it against prod ? 15:31:45 we had a few issues in stg where users/queues were not correctly re-created 15:31:53 we may need to see how we can fix this 15:32:27 cverna: the upgrade for prod? no. Thats: take the cluster down, reinstall as rhel8, resetup cluster 15:32:28 maybe add the tag in the rabbitmq/... roles so if some app role don't have the corresponding tag, we can still re-create? 15:32:52 nirik: ha ok a bit more work then :) 15:32:54 yeah, I think tags were incomplete... 15:33:10 cverna: the actual setting up of the auth still needs doing tho. 15:33:19 (in stg) 15:33:27 tflink, I see an item in the agenda for moving stuff to openshift.. is that for this meeting or last? 15:33:44 last meeting, I think 15:33:52 * tflink looks in case he forgot adding something 15:34:02 * relrod waves and checks into the meeting super late. 15:34:23 relrod you have oncall in 3 weeks :P 15:34:53 smooge: yeah, that was last week 15:34:59 hey relrod 15:35:13 smooge: Let's see what my new workload looks like for a bit first ;) 15:35:28 I'd not be opposed to occasionally taking a shift to stay in the loop though. 15:35:48 #topic Open Floor 15:36:01 ok anything from the floor 15:36:12 nothing here 15:36:16 stay safe out there everyone! 15:36:28 yes that ^ 15:36:47 thirded 15:37:51 fourthed 15:38:09 <> 15:38:32 carbonite freezing was my second choice but expensive 15:38:32 smooge: https://duckduckgo.com/?t=ffab&q=plastic+bertrand&iax=images&ia=images ? 15:39:27 https://duckduckgo.com/?q=plastic+human+bubble&t=ffab&iar=images&iax=images&ia=images 15:39:30 #endmeeting