14:59:49 #startmeeting Infrastructure (2018-12-13) 14:59:49 Meeting started Thu Dec 13 14:59:49 2018 UTC. 14:59:49 This meeting is logged and archived in a public location. 14:59:49 The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:59:49 Useful Commands: #action #agreed #halp #info #idea #link #topic. 14:59:49 The meeting name has been set to 'infrastructure_(2018-12-13)' 14:59:49 #meetingname infrastructure 14:59:49 The meeting name has been set to 'infrastructure' 14:59:49 #topic aloha 14:59:49 #chair nirik pingou puiterwijk relrod smooge tflink threebean cverna mkonecny 14:59:50 Current chairs: cverna mkonecny nirik pingou puiterwijk relrod smooge tflink threebean 15:00:05 hello 15:00:30 hello o/ 15:00:37 Hello 15:00:41 morning 15:00:41 hola 15:01:31 .hello2 15:01:34 bowlofeggs: bowlofeggs 'Randy Barlow' 15:02:11 #topic New folks introductions 15:02:11 #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves 15:02:11 #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted 15:05:08 #topic announcements and information 15:05:08 #info tflink is on extended PTO 15:05:08 #info nirik in PTO 2018-12-19 -> 2019-01-14 15:05:08 #info smooge is on PTO 2018-12-27 -> 2019-01-07 15:05:08 #info mkonecny is on PTO 2018-12-18 -> 2019-01-07 15:05:09 #info puiterwijk is on PTO 2018-12-14 -> 2019-01-07 15:05:10 #info there will be no infrastructure meeting 2018-12-20/2018-12-27 15:05:12 #info devconf is happening from 2019-01-25 -> 2019-01-27 15:05:14 #info minor outage: phx2 2018-12-13 15:05:16 #info major outage: s390 2019-01-11 -> 2019-01-14 15:05:20 #info Bodhi stakeholders' meeting next Tuesday https://apps.fedoraproject.org/calendar/meeting/9429/ 15:05:23 #info Proposed backwards incompatible changes in upcoming Bodhi 4.0.0: https://github.com/fedora-infra/bodhi/projects/6 15:05:26 so a lot of upcoming outages 15:05:37 yep 15:05:40 and ptos 15:05:56 I think I need to amend something 15:06:07 #info there will be no infrastructure meeting 2018-12-20/2018-12-27/2019-01-03 15:06:35 so this is our last meeting of 2018 then. :) 15:07:05 I think that pretty much any problems between 2018-12-28->2019-01-07 are going to take a while to fix 15:07:38 I should be around for any outages/issues... but hopefully it's nice and quiet. 15:07:53 i'm going to deploy bodhi 4.0.0 on the 21st 15:07:57 at 5 pm 15:08:06 then sign off for the year 15:08:19 bowlofeggs, I know where I can get a can of bear bait 15:08:23 hahaha 15:08:28 * wa1em late 15:08:33 .hello2 15:08:34 wa1em: Sorry, but you don't exist 15:08:34 and a bear 15:08:38 .hello2 linuxmodder 15:08:40 wa1em: Sorry, but you don't exist 15:08:58 .hello linuxmodder 15:08:59 mkonecny: linuxmodder 'Corey W Sheldon' 15:09:15 well I think you have a bug to look for and fix wa1em :) 15:09:27 bowlofeggs: meant to ask... rawhide gating would be 4.1? or 5.0? since I didn't see it in the 4.0.0 stuff? 15:09:43 no I'm using my ham call and it's not linking to my fasid 15:09:53 .hello linuxmodder 15:09:54 wa1em: linuxmodder 'Corey W Sheldon' 15:10:20 nirik: rawhide gating doesn't need a backwards incompatible change as far as i know, so it can happen anytime (even in the 3.x series) 15:10:36 it'll probably be in the 4.x series 15:10:43 but not necessarily 4.0 15:10:46 sounds like something to turn on the 21st then 15:10:49 ok, so that list was just the incompatible changes? 15:10:52 4.0 is about removing tons of code that is slwoing me down 15:11:10 alright 15:11:13 yeah it's just a bunch of removals 15:11:18 removing code ++ 15:11:26 hopefully it'll speed me up 15:11:47 ok so normally during a large 'outage' we are moving to a slush in infrastructure 15:12:29 yeah, please see my email to the list about holidays. Try and not cause outages or alerts or problems... :) 15:12:38 find a peer and get them to review/sign off for any major changes 15:12:53 ack 15:13:06 I will put it quite clearly.. if you put in a change that breaks things, I WILL turn off that service until you are back 15:13:24 or are dragged back from where-ever you are camping by my trained bears 15:13:42 * aikidouke wonders if he breaks the internet, would smooge just turn it off? 15:15:15 #topic Oncall 15:15:15 #info puiterwijk is on call from 2018-12-06->2018-12-13 (smooge has 2018-12-07) 15:15:15 #info bowlofeggs is on call from 2018-12-13->2018-12-20 15:15:15 #info smooge is on call from 2018-12-20->2018-12-27 15:15:16 #info nirik is on call from 2018-12-27 -> 2019-01-03 15:15:17 #info relrod is on call from 2019-01-03 -> 2019-01-10 15:15:19 aikidouke: The smooge will sent his trained bears to do this for him :-) 15:15:20 indeed 15:15:20 #info smooge is on call from 2019-01-10 -> 2019-01-17 15:15:22 #info ??? is on call from 2019-01-17 -> 2019-01-24 15:15:24 #info ??? is on call from 2019-01-24 -> 2019-01-31 15:15:27 #info Summary of last week: (from puiterwijk ) 15:15:35 hah :) 15:15:58 looks like puiterwijk isn't around right now... 15:16:35 ok this week has been pretty calm 15:16:50 bowlofeggs, you are on call starting this meeting 15:17:10 here is the bear bait 15:17:14 here i go! 15:17:31 we need a ceremony for changing of the oncall 15:17:52 yeah, needs twirling of rifles and salutes 15:17:57 lol 15:18:17 * bcotton was on the JROTC color guard in high school... 15:18:22 nirik: and a hand off of a coffee mug 15:18:40 * smooge takes the ceremonial red coffee mug from puiterwijk's clone and hands it to bowlofeggs 15:18:40 mug? more like a IV drip 15:18:46 💂 🥆 ☕ 15:18:52 haha 15:19:04 the clone has green tea in it 15:19:10 no swingline stapler? 15:19:13 no 15:19:29 * aikidouke sighs 15:19:47 that is the ceremony of the accountant 15:20:10 ok back to the meeting 15:20:27 #topic Monitoring discussion 15:20:27 #info https://nagios.fedoraproject.org/nagios 15:20:27 #info Go over existing out items and fix 15:20:38 This is a new entry. 15:20:44 Sorry, back now.. 15:20:47 oh yeah, I was hoping for a clean monitoring slate before the holidays 15:21:17 not going to be easy tho. ;) 15:21:33 osuosl03 / ibiblio04 are down and need reinstalls 15:22:04 busgateway is alerting on several fedmsgs it's not seen in a while... greenwave, bugzilla and askbot 15:22:33 osuosl03/ibiblio04 will be rhel-8 test machines. I will remove them from ansible for the time being while working on that 15:22:34 the zodbot fedmsg plugin is alerting... I have no idea why there. 15:22:49 smooge: I don't think we should make them rhel8b 15:23:07 there's no security updates for that and they are kinda exposed... 15:23:40 nirik: i think greenwave has not been sending messages, isn't there an infra ticket about that? 15:23:59 yep 15:24:03 i was seeing that as a bonus for testing.. but I can reinstall them as normal boxes 15:25:09 yeah, I think 7 is best... we can do testing on a cloud box or something. 15:25:33 theres also various machines that have lost a drive from raid... 15:25:44 some or all of those might just need reboots to add the drives back in? 15:26:09 they will need reboots and someone on console to deal with the raid controller 15:26:11 there is also a weird osbs alert: 15:26:21 15:26:21 Check OSBS build listing 15:26:22 I can do that early next week 15:26:58 cool. I can do some of them today/tomorrow too if you like. 15:27:22 sure if you want. I was thinking you were focusing on shambolic cloud 15:27:42 I can help with the OSBS alerts 15:27:59 I've got many things going on... hard to focus on any one thing these days. ;) 15:28:21 cverna: cool. Can provide details after meeting? 15:28:30 nirik: sure 15:29:13 understood. focusing lately has been the pits 15:29:55 so my hope is that we will be down to few red lines by next friday 15:30:08 #topic Tickets discussion 15:30:08 #info https://pagure.io/fedora-infrastructure/report/Meetings%20ticket 15:31:07 there are currently 111 open tickets 15:31:30 yeah, they are piling up. I haven't been able to do much since I have been working on cloud and other things. ;( 15:31:49 I would implore people who are assigned on tickets to see if they can resolve some. ;) 15:33:01 i was wondering if in the new year we also start assigning tickets 15:33:08 because most of the tickets don't seem assigned 15:33:19 well, I am not sure that would help any 15:33:24 but perhaps. 15:34:18 with outages over those will go away. ;) 15:34:28 would we consider putting some kind of time limit on responses from the original requestor? 15:34:47 i know that doesn't help with incoming 15:34:55 we could... how many do we have waiting on reporter? 15:35:03 umm 15:35:14 mostly we are waiting on assignee 15:35:16 6 it looks like 15:36:02 on bodhi upstream, i give pull request people 1 month to respond then close 15:36:03 yeah 15:36:25 soemtimes people make a pull request and then don't respond to requests to change it 15:36:29 kinda similar 15:37:04 sure, we could do that for waiting on reporter... although we need to check that they really are 15:38:01 does that warrant an announcement to a list, just a note on the issue, or something else? 15:38:05 * wa1em stepping out for a bit will catch up on meeting minutes later 15:38:25 i think you can just close it and ask them to reopen if they respond 15:38:26 probibly note to issue... I can do so before the break and close after? 15:38:29 that's what i do 15:38:31 or that... 15:38:46 "closing due to inactivity, feel free to reopen if you can answer questions x,y,z" 15:38:57 bueno 15:39:11 ok anything else on this? 15:39:22 I can go through the waiting for upstream tickets and see if anything has happened that warrants a change 15:39:40 thanks aikidouke that would be great 15:40:48 .ticket 7229 15:40:50 aikidouke: Issue #7229: dist-git doesn't allow branches with unicode characters - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/7229 15:41:10 says waiting for implementation of repospanner, but is waiting for upstream? 15:41:12 thats going to be fixed when we switch away from gitolite 15:41:23 * aikidouke isnt going to do all these rn, just saw that one 15:41:49 is there a date for switching to repospanner? 15:41:56 or date-ish I guess 15:42:06 I would expect by February 15:42:11 early next year for src.fedoraproject.org 15:42:22 but I don't know if there is a specific date yet 15:42:27 ok - so it makes sense to leave that one alone then? 15:42:59 I suppose so yeah. I wish we had a magic notification service we could use so we could close things like this and it would notify them when the thing is actually done 15:43:12 but thats not easy 15:44:21 sounds like a job for smooge's bears 15:44:31 the magic bears! 15:44:38 right! 15:44:56 🐻 or perhaps 🧸 ? 15:45:03 I only see one of them 15:45:10 but not 🐨 15:45:20 ok so I think we move to 15:45:24 #topic open floor 15:45:25 the others are wearing their cloak of invisibility 15:45:52 OK so bear with me but I think this meeting and this year is coming to an end 15:45:54 I'd like to thank everyone for all their work this year... it's a great bunch of people you all are. ;) 15:46:03 nirik++ 15:46:25 I'd like to thank everyone for all of the advice I've been given over the years and esp recently 15:47:04 nirik++ Thanks all :) 15:47:27 Merry Christmas and Happy New Year :-) 15:47:32 OT - but I start my first real sys-admin job after Christmas and I owe alot, ok almost everything to the opportunity I have here 15:47:43 so thank you all 15:47:44 aikidouke: congrats on the job! 15:47:44 cool 15:47:47 congrats 15:47:52 grats 15:47:53 congratulations, aikidouke! 15:47:55 :) TY!! 15:48:16 congrats aikidouke 15:48:22 ty all 15:48:45 and especially stephen smoogen's magic ursine circus 15:48:59 srsly tho...you are all awesome 15:49:22 thank you all for coming and thank you for your work 15:49:41 next year my bears will be mauling sorry dancing with developers 15:49:46 thanks smooge 15:49:52 thanks smooge 15:49:59 and with that .. say good night Growlee 15:50:12 #endmeeting