14:59:49 <smooge> #startmeeting Infrastructure (2018-12-13)
14:59:49 <zodbot> Meeting started Thu Dec 13 14:59:49 2018 UTC.
14:59:49 <zodbot> This meeting is logged and archived in a public location.
14:59:49 <zodbot> The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:59:49 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
14:59:49 <zodbot> The meeting name has been set to 'infrastructure_(2018-12-13)'
14:59:49 <smooge> #meetingname infrastructure
14:59:49 <zodbot> The meeting name has been set to 'infrastructure'
14:59:49 <smooge> #topic aloha
14:59:49 <smooge> #chair nirik pingou puiterwijk relrod smooge tflink threebean cverna mkonecny
14:59:50 <zodbot> Current chairs: cverna mkonecny nirik pingou puiterwijk relrod smooge tflink threebean
15:00:05 <smooge> hello
15:00:30 <cverna> hello o/
15:00:37 <mkonecny> Hello
15:00:41 <nirik> morning
15:00:41 <aikidouke> hola
15:01:31 <bowlofeggs> .hello2
15:01:34 <zodbot> bowlofeggs: bowlofeggs 'Randy Barlow' <rbarlow@redhat.com>
15:02:11 <smooge> #topic New folks introductions
15:02:11 <smooge> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
15:02:11 <smooge> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
15:05:08 <smooge> #topic announcements and information
15:05:08 <smooge> #info tflink is on extended PTO
15:05:08 <smooge> #info nirik in PTO 2018-12-19 -> 2019-01-14
15:05:08 <smooge> #info smooge is on PTO 2018-12-27 -> 2019-01-07
15:05:08 <smooge> #info mkonecny is on PTO 2018-12-18 -> 2019-01-07
15:05:09 <smooge> #info puiterwijk is on PTO 2018-12-14 -> 2019-01-07
15:05:10 <smooge> #info there will be no infrastructure meeting 2018-12-20/2018-12-27
15:05:12 <smooge> #info devconf is happening from 2019-01-25 -> 2019-01-27
15:05:14 <smooge> #info minor outage: phx2 2018-12-13
15:05:16 <smooge> #info major outage: s390 2019-01-11 -> 2019-01-14
15:05:20 <smooge> #info Bodhi stakeholders' meeting next Tuesday https://apps.fedoraproject.org/calendar/meeting/9429/
15:05:23 <smooge> #info Proposed backwards incompatible changes in upcoming Bodhi 4.0.0: https://github.com/fedora-infra/bodhi/projects/6
15:05:26 <smooge> so a lot of upcoming outages
15:05:37 <nirik> yep
15:05:40 <nirik> and ptos
15:05:56 <smooge> I think I need to amend something
15:06:07 <smooge> #info there will be no infrastructure meeting 2018-12-20/2018-12-27/2019-01-03
15:06:35 <nirik> so this is our last meeting of 2018 then. :)
15:07:05 <smooge> I think that pretty much any problems between 2018-12-28->2019-01-07 are going to take a while to fix
15:07:38 <nirik> I should be around for any outages/issues... but hopefully it's nice and quiet.
15:07:53 <bowlofeggs> i'm going to deploy bodhi 4.0.0 on the 21st
15:07:57 <bowlofeggs> at 5 pm
15:08:06 <bowlofeggs> then sign off for the year
15:08:19 <smooge> bowlofeggs, I know where I can get a can of bear bait
15:08:23 <bowlofeggs> hahaha
15:08:28 * wa1em late
15:08:33 <wa1em> .hello2
15:08:34 <zodbot> wa1em: Sorry, but you don't exist
15:08:34 <smooge> and a bear
15:08:38 <wa1em> .hello2 linuxmodder
15:08:40 <zodbot> wa1em: Sorry, but you don't exist
15:08:58 <mkonecny> .hello linuxmodder
15:08:59 <zodbot> mkonecny: linuxmodder 'Corey W Sheldon' <sheldon.corey@gmail.com>
15:09:15 <smooge> well I think you have a bug to look for and fix wa1em :)
15:09:27 <nirik> bowlofeggs: meant to ask... rawhide gating would be 4.1? or 5.0? since I didn't see it in the 4.0.0 stuff?
15:09:43 <wa1em> no I'm using my ham call and it's not linking to my fasid
15:09:53 <wa1em> .hello linuxmodder
15:09:54 <zodbot> wa1em: linuxmodder 'Corey W Sheldon' <sheldon.corey@gmail.com>
15:10:20 <bowlofeggs> nirik: rawhide gating doesn't need a backwards incompatible change as far as i know, so it can happen anytime (even in the 3.x series)
15:10:36 <bowlofeggs> it'll probably be in the 4.x series
15:10:43 <bowlofeggs> but not necessarily 4.0
15:10:46 <smooge> sounds like something to turn on the 21st then
15:10:49 <nirik> ok, so that list was just the incompatible changes?
15:10:52 <bowlofeggs> 4.0 is about removing tons of code that is slwoing me down
15:11:10 <nirik> alright
15:11:13 <bowlofeggs> yeah it's just a bunch of removals
15:11:18 <cverna> removing code ++
15:11:26 <bowlofeggs> hopefully it'll speed me up
15:11:47 <smooge> ok so normally during a large 'outage' we are moving to a slush in infrastructure
15:12:29 <nirik> yeah, please see my email to the list about holidays. Try and not cause outages or alerts or problems... :)
15:12:38 <smooge> find a peer and get them to review/sign off for any major changes
15:12:53 <wa1em> ack
15:13:06 <smooge> I will put it quite clearly.. if you put in a change that breaks things, I WILL turn off that service until you are back
15:13:24 <smooge> or are dragged back from where-ever you are camping by my trained bears
15:13:42 * aikidouke wonders if he breaks the internet, would smooge just turn it off?
15:15:15 <smooge> #topic Oncall
15:15:15 <smooge> #info puiterwijk is on call from 2018-12-06->2018-12-13 (smooge has 2018-12-07)
15:15:15 <smooge> #info bowlofeggs is on call from 2018-12-13->2018-12-20
15:15:15 <smooge> #info smooge is on call from 2018-12-20->2018-12-27
15:15:16 <smooge> #info nirik is on call from 2018-12-27 -> 2019-01-03
15:15:17 <smooge> #info relrod is on call from 2019-01-03 -> 2019-01-10
15:15:19 <mkonecny> aikidouke: The smooge will sent his trained bears to do this for him :-)
15:15:20 <nirik> indeed
15:15:20 <smooge> #info smooge is on call from 2019-01-10 -> 2019-01-17
15:15:22 <smooge> #info ??? is on call from 2019-01-17 -> 2019-01-24
15:15:24 <smooge> #info ??? is on call from 2019-01-24 -> 2019-01-31
15:15:27 <smooge> #info Summary of last week: (from puiterwijk )
15:15:35 <aikidouke> hah :)
15:15:58 <nirik> looks like puiterwijk isn't around right now...
15:16:35 <smooge> ok this week has been pretty calm
15:16:50 <smooge> bowlofeggs, you are on call starting this meeting
15:17:10 <smooge> here is the bear bait
15:17:14 <bowlofeggs> here i go!
15:17:31 <bowlofeggs> we need a ceremony for changing of the oncall
15:17:52 <nirik> yeah, needs twirling of rifles and salutes
15:17:57 <wa1em> lol
15:18:17 * bcotton was on the JROTC color guard in high school...
15:18:22 <bowlofeggs> nirik: and a hand off of a coffee mug
15:18:40 * smooge takes the ceremonial red coffee mug from puiterwijk's clone and hands it to bowlofeggs
15:18:40 <wa1em> mug? more like a IV drip
15:18:46 <nirik> 💂 🥆 ☕
15:18:52 <bowlofeggs> haha
15:19:04 <smooge> the clone has green tea in it
15:19:10 <aikidouke> no swingline stapler?
15:19:13 <smooge> no
15:19:29 * aikidouke sighs
15:19:47 <smooge> that is the ceremony of the accountant
15:20:10 <smooge> ok back to the meeting
15:20:27 <smooge> #topic Monitoring discussion
15:20:27 <smooge> #info https://nagios.fedoraproject.org/nagios
15:20:27 <smooge> #info Go over existing out items and fix
15:20:38 <smooge> This is a new entry.
15:20:44 <puiterwijk> Sorry, back now..
15:20:47 <nirik> oh yeah, I was hoping for a clean monitoring slate before the holidays
15:21:17 <nirik> not going to be easy tho. ;)
15:21:33 <nirik> osuosl03 / ibiblio04 are down and need reinstalls
15:22:04 <nirik> busgateway is alerting on several fedmsgs it's not seen in a while... greenwave, bugzilla and askbot
15:22:33 <smooge> osuosl03/ibiblio04 will be rhel-8 test machines. I will remove them from ansible for the time being while working on that
15:22:34 <nirik> the zodbot fedmsg plugin is alerting... I have no idea why there.
15:22:49 <nirik> smooge: I don't think we should make them rhel8b
15:23:07 <nirik> there's no security updates for that and they are kinda exposed...
15:23:40 <bowlofeggs> nirik: i think greenwave has not been sending messages, isn't there an infra ticket about that?
15:23:59 <nirik> yep
15:24:03 <smooge> i was seeing that as a bonus for testing.. but I can reinstall them as normal boxes
15:25:09 <nirik> yeah, I think 7 is best... we can do testing on a cloud box or something.
15:25:33 <nirik> theres also various machines that have lost a drive from raid...
15:25:44 <nirik> some or all of those might just need reboots to add the drives back in?
15:26:09 <smooge> they will need reboots and someone on console to deal with the raid controller
15:26:11 <nirik> there is also a weird osbs alert:
15:26:21 <nirik> 
15:26:21 <nirik> Check OSBS build listing
15:26:22 <smooge> I can do that early next week
15:26:58 <nirik> cool. I can do some of them today/tomorrow too if you like.
15:27:22 <smooge> sure if you want. I was thinking you were focusing on shambolic cloud
15:27:42 <cverna> I can help with the OSBS alerts
15:27:59 <nirik> I've got many things going on... hard to focus on any one thing these days. ;)
15:28:21 <nirik> cverna: cool. Can provide details after meeting?
15:28:30 <cverna> nirik: sure
15:29:13 <smooge> understood. focusing lately has been the pits
15:29:55 <smooge> so my hope is that we will be down to few red lines by next friday
15:30:08 <smooge> #topic Tickets discussion
15:30:08 <smooge> #info https://pagure.io/fedora-infrastructure/report/Meetings%20ticket
15:31:07 <smooge> there are currently 111 open tickets
15:31:30 <nirik> yeah, they are piling up. I haven't been able to do much since I have been working on cloud and other things. ;(
15:31:49 <nirik> I would implore people who are assigned on tickets to see if they can resolve some. ;)
15:33:01 <smooge> i was wondering if in the new year we also start assigning tickets
15:33:08 <smooge> because most of the tickets don't seem assigned
15:33:19 <nirik> well, I am not sure that would help any
15:33:24 <nirik> but perhaps.
15:34:18 <nirik> with outages over those will go away. ;)
15:34:28 <aikidouke> would we consider putting some kind of time limit on responses from the original requestor?
15:34:47 <aikidouke> i know that doesn't help with incoming
15:34:55 <nirik> we could... how many do we have waiting on reporter?
15:35:03 <aikidouke> umm
15:35:14 <smooge> mostly we are waiting on assignee
15:35:16 <nirik> 6 it looks like
15:36:02 <bowlofeggs> on bodhi upstream, i give pull request people 1 month to respond then close
15:36:03 <aikidouke> yeah
15:36:25 <bowlofeggs> soemtimes people make a pull request and then don't respond to requests to change it
15:36:29 <bowlofeggs> kinda similar
15:37:04 <nirik> sure, we could do that for waiting on reporter... although we need to check that they really are
15:38:01 <aikidouke> does that warrant an announcement to a list, just a note on the issue, or something else?
15:38:05 * wa1em stepping out for a bit will catch up on meeting minutes later
15:38:25 <bowlofeggs> i think you can just close it and ask them to reopen if they respond
15:38:26 <nirik> probibly note to issue... I can do so before the break and close after?
15:38:29 <bowlofeggs> that's what i do
15:38:31 <nirik> or that...
15:38:46 <bowlofeggs> "closing due to inactivity, feel free to reopen if you can answer questions x,y,z"
15:38:57 <aikidouke> bueno
15:39:11 <smooge> ok anything else on this?
15:39:22 <aikidouke> I can go through the waiting for upstream tickets and see if anything has happened that warrants a change
15:39:40 <smooge> thanks aikidouke that would be great
15:40:48 <aikidouke> .ticket 7229
15:40:50 <zodbot> aikidouke: Issue #7229: dist-git doesn't allow branches with unicode characters - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/7229
15:41:10 <aikidouke> says waiting for implementation of repospanner, but is waiting for upstream?
15:41:12 <nirik> thats going to be fixed when we switch away from gitolite
15:41:23 * aikidouke isnt going to do all these rn, just saw that one
15:41:49 <aikidouke> is there a date for switching to repospanner?
15:41:56 <aikidouke> or date-ish I guess
15:42:06 <smooge> I would expect by February
15:42:11 <nirik> early next year for src.fedoraproject.org
15:42:22 <nirik> but I don't know if there is a specific date yet
15:42:27 <aikidouke> ok - so it makes sense to leave that one alone then?
15:42:59 <nirik> I suppose so yeah. I wish we had a magic notification service we could use so we could close things like this and it would notify them when the thing is actually done
15:43:12 <nirik> but thats not easy
15:44:21 <aikidouke> sounds like a job for smooge's bears
15:44:31 <nirik> the magic bears!
15:44:38 <aikidouke> right!
15:44:56 <nirik> 🐻 or perhaps 🧸 ?
15:45:03 <smooge> I only see one of them
15:45:10 <nirik> but not 🐨
15:45:20 <smooge> ok so I think we move to
15:45:24 <smooge> #topic open floor
15:45:25 <aikidouke> the others are wearing their cloak of invisibility
15:45:52 <smooge> OK so bear with me but I think this meeting and this year is coming to an end
15:45:54 <nirik> I'd like to thank everyone for all their work this year... it's a great bunch of people you all are. ;)
15:46:03 <aikidouke> nirik++
15:46:25 <aikidouke> I'd like to thank everyone for all of the advice I've been given over the years and esp recently
15:47:04 <cverna> nirik++ Thanks all :)
15:47:27 <mkonecny> Merry Christmas and Happy New Year :-)
15:47:32 <aikidouke> OT - but I start my first real sys-admin job after Christmas and I owe alot, ok almost everything to the opportunity I have here
15:47:43 <aikidouke> so thank you all
15:47:44 <nirik> aikidouke: congrats on the job!
15:47:44 <smooge> cool
15:47:47 <smooge> congrats
15:47:52 <mkonecny> grats
15:47:53 <bcotton> congratulations, aikidouke!
15:47:55 <aikidouke> :) TY!!
15:48:16 <bowlofeggs> congrats aikidouke
15:48:22 <aikidouke> ty all
15:48:45 <aikidouke> and especially stephen smoogen's magic ursine circus
15:48:59 <aikidouke> srsly tho...you are all awesome
15:49:22 <smooge> thank you all for coming and thank you for your work
15:49:41 <smooge> next year my bears will be mauling sorry dancing with developers
15:49:46 <mkonecny> thanks smooge
15:49:52 <aikidouke> thanks smooge
15:49:59 <smooge> and with that .. say good night Growlee
15:50:12 <smooge> #endmeeting