15:08:33 <nirik> #startmeeting Infrastructure (2020-02-13)
15:08:33 <zodbot> Meeting started Thu Feb 13 15:08:33 2020 UTC.
15:08:33 <zodbot> This meeting is logged and archived in a public location.
15:08:33 <zodbot> The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:08:33 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:08:33 <zodbot> The meeting name has been set to 'infrastructure_(2020-02-13)'
15:08:33 <nirik> #meetingname infrastructure
15:08:33 <zodbot> The meeting name has been set to 'infrastructure'
15:08:33 <nirik> #chair nirik pingou relrod smooge tflink cverna mizdebsk mkonecny abompard bowlofeggs
15:08:33 <zodbot> Current chairs: abompard bowlofeggs cverna mizdebsk mkonecny nirik pingou relrod smooge tflink
15:08:33 <nirik> #info Agenda is at: https://board.net/p/fedora-infra
15:08:33 <nirik> #topic aloha
15:08:44 <bowlofeggs> .hello2
15:08:46 <zodbot> bowlofeggs: bowlofeggs 'Randy Barlow' <randy@electronsweatshop.com>
15:08:52 <nirik> #topic Next chair
15:08:52 <nirik> #info magic eight ball says:
15:08:52 <nirik> #info 2020-02-13 - smooge
15:08:52 <nirik> #info 2020-02-20 - bowlofeggs
15:08:53 <nirik> #info 2020-02-27 - ???
15:09:02 <nirik> anyone want to take it on the 27th?
15:09:45 <pingou> .hello2
15:09:47 <zodbot> pingou: pingou 'Pierre-YvesChibon' <pingou@pingoured.fr>
15:10:00 <smooge> ah crap
15:10:11 <smooge> sorry been staring at a debugger
15:10:17 <smooge> thanks nirik
15:10:47 <nirik> If you stare into the debugger, the bugs will stare into you! </misquote>
15:11:17 <smooge> dhcpd was not working and so I was trying to figure out why
15:11:27 <smooge> #topic New folks introductions
15:11:27 <smooge> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
15:11:27 <smooge> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
15:11:33 <mymindstorm> Hello
15:11:40 <smooge> hello mymindstorm
15:12:24 <smooge> what are you interested in ?
15:12:41 <mymindstorm> I am interested in joining as an apprentice, I have packaging permissions if that means anything.
15:13:53 <smooge> welcome. send me your fas account and I will add you
15:14:19 <mkonecny> .hello zlopez
15:14:20 <zodbot> mkonecny: zlopez 'Michal Konečný' <michal.konecny@packetseekers.eu>
15:14:29 <mkonecny> Sorry, I'm a little late
15:15:10 <smooge> mkonecny, I didn't even start the meeting and I am supposed to be running it.. so no apologies needed
15:15:10 <nirik> welcome mymindstorm
15:16:31 <smooge> I will be doing a mass remove of people in apprentice later today I think
15:16:55 <austinpowered> Please don't delete me
15:17:07 <mymindstorm> ^
15:17:35 <smooge> the general rule will be I will check to see who has logged into bastion/batcave in the last month.
15:17:45 <smooge> those who have not will be removed
15:17:51 * austinpowered 'sitting quietly in the corner'
15:17:54 <smooge> #topic announcements and information
15:17:54 <smooge> #info ops folks are trying a 30min ticket triage every day at 19UTC in #fedora-admin
15:17:54 <smooge> #info Fedora Infrastructure will be moving in 2020-06 from its Phoenix Az datacenter to one near Herndon Va. A lot of planning will be involved on this. Please watch out for announcements on changes.
15:17:54 <smooge> #info Fedora 32 has been branched from rawhide.
15:18:32 <smooge> any other announcements
15:19:53 <nirik> oh, I have one
15:20:24 <nirik> #info Old openstack cloud is being decommissioned 2020-03-01. Working with instance owners to make sure everyone is migrated off.
15:20:46 <bowlofeggs> #info there are only 3,000 tigers left in the wild
15:21:17 <abompard> nirik: do you still plan on upgrading rabbitmq later today?
15:21:22 <mkonecny> bowlofeggs: This is sad :-(
15:21:37 <bowlofeggs> ☹
15:22:07 <nirik> abompard: yeah, I was gonna try... later tonight my time.
15:22:10 <nirik> in staging
15:22:14 <abompard> yes
15:22:15 <abompard> cool
15:22:30 <nirik> and it will be waiting for you with all it's issues (hopefully none)
15:22:47 <msuchy> bowlofeggs: there are even less fedora-infra team members left in the wild
15:22:55 <abompard> can't wait :-)
15:22:55 <smooge> I want to bring attention on the datacenter move. Mails to the community will be going out in the next week or so
15:23:06 <pingou> smooge++
15:23:26 <pingou> the short story: RH is leaving the PHX2 data-center where most of the Fedora Infra is hosted
15:23:34 <pingou> so we're working on how to deal with this
15:23:43 <pingou> the rest in announced emails :)
15:24:05 <bowlofeggs> msuchy: also true ☹
15:25:11 <smooge> #topic Oncall
15:25:11 <smooge> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
15:25:11 <smooge> #info smooge is oncall 2020-02-06 -> 2020-02-13
15:25:11 <smooge> #info nirik is oncall 2020-02-13 -> 2020-02-20
15:25:11 <smooge> #info cverna is oncall 2020-02-20 -> 2020-02-27
15:25:12 <smooge> #info ??? is oncall 2020-02-27 -> 2020-03-05
15:25:21 <nirik> .oncalltakeus
15:25:21 <zodbot> nirik: Kneel before zod!
15:25:34 <pingou> I could take that week, but I'm not 100% sure I'll be available on the 27th itself
15:26:01 <pingou> other than this, write we down for this
15:26:14 <smooge> #info Summary of last week: (from current oncall )
15:26:14 <smooge> #info DNS broke on 2020-02-11 due to signing problems.
15:26:40 <smooge> Otherwise it has been quiet
15:26:45 <nirik> BTW, openvpn broke sunday morning. ;(
15:27:00 <smooge> OH yeah.. my phone didn't take any pages
15:27:09 * nirik hopes for more quiet. we are getting process done on things for once. ;)
15:27:18 <smooge> #info OpenVPN died on 2020-02-09 due to keys expiration
15:27:33 <smooge> nirik++ for fixing that
15:27:52 <smooge> pingou++ for taking oncall
15:27:52 <zodbot> smooge: Karma for pingou changed to 16 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
15:28:08 <smooge> ok I think that is it for oncall
15:28:22 <smooge> #topic Monitoring discussion [nirik]
15:28:22 <smooge> #info https://nagios.fedoraproject.org/nagios
15:28:22 <smooge> #info Go over existing out items and fix
15:28:41 * nirik pulls up nagios
15:29:16 <nirik> the 2 down hosts I can now fix I think, we have a aarch64 emag up for those.
15:29:37 <smooge> cool. I should have the rest up today
15:29:44 <nirik> there's some proxy30 ones...
15:30:05 <nirik> might need to tell nagios it has no swap
15:30:17 <nirik> not sure about the registry one.
15:30:26 <smooge> the proxy30 needs the noswap and I need to work with relrod on standing up others
15:30:28 <nirik> is proxy30 in service?
15:30:38 <smooge> it is as far as I know
15:30:44 <smooge> that was my breakage last week
15:30:56 <nirik> it's also marked no alerts...
15:31:05 <nirik> we should re-enable and ack anything left.
15:31:34 * nirik goes and does that
15:32:17 <smooge> thanks.. kerberos and I are having an argument over my password
15:32:26 <nirik> and I see buildvmhost-aarch64-10.arm.fedoraproject.org has a low swap thing again
15:32:38 <smooge> ok I have a kernel to try on it
15:34:16 <nirik> ok, thats all nagios I see... the typical missing messages from fas and such
15:34:46 <smooge> #topic Tickets discussion [nirik]
15:34:46 <smooge> #info https://pagure.io/fedora-infrastructure/report/Meetings%20ticket
15:36:09 <smooge> We are doing daily standups to do meetings
15:36:18 <smooge> which have cut down the number of open tickets a bit.
15:36:26 <smooge> we were at 130 and are grinding down..
15:37:51 <nirik> yeah, I think we hit higher than that even...
15:37:56 <nirik> but making progress.
15:39:03 <smooge> #topic backlog discussion
15:39:03 <smooge> #info go over our backlog and discuss and determine priority
15:39:18 <smooge> cverna, did you have anything here
15:39:30 <nirik> he's on pto. ;)
15:39:43 <nirik> we do have some needs review ones... but we can do them at the normal meeting?
15:39:47 <nirik> or should we do them now?
15:39:58 <nirik> and cancel the standup. ;)
15:40:44 <nirik> Oh, I did have one... download-ib01 is full. I tried various things to free up space (hardlinking, etc)
15:41:03 <nirik> but it's just not going to be able to take the entire enchillada.
15:41:15 <smooge> nirik, I think we are just going to have archives on it anymore
15:41:18 <nirik> so, likely we need to cut something from it.
15:41:20 <smooge> what you said
15:41:35 <smooge> does it have alt?
15:41:43 <nirik> it has everything currently.
15:41:45 <smooge> because that could go if it does
15:41:49 <smooge> I would drop alt
15:42:02 <nirik> It might be able to limp along more when mboddu nukes the old rc's
15:42:05 <bowlofeggs> i can eat an entire enchillada
15:42:11 <bowlofeggs> and i will prove it if necessary
15:42:46 <nirik> ok, I'll look at nuking alt or something on it.
15:42:48 <smooge> I think alt or archives will have to go. I mean we knew this day would come someday.. there are only so many disks
15:42:55 <mboddu> I am nuking them
15:42:58 <smooge> and fedora just keeps growing
15:43:37 * bowlofeggs starts a cat /dev/urandom > /home/smooge/hello_smooge
15:44:06 <mkonecny> bowlofeggs: You are now a Perl developer?
15:44:07 <mboddu> Well, I removed most of it
15:44:12 <mboddu> There are only a couple remaining
15:44:14 <bowlofeggs> mkonecny: heh
15:44:15 * nirik starts a cat /dev/urandom > libreoffice-powerup-1.0-1.fc32.rpm
15:44:24 <bowlofeggs> haha
15:45:46 <nirik> anyhow, lets move on and deal with the needs review slater?
15:46:24 <smooge> #topic Open Floor
15:46:35 <smooge> sorry my browser crashed
15:46:40 <mkonecny> bowlofeggs: There is a rumor, if you do cat /dev/urandom | perl it will work till it hits 1; :-)
15:46:41 <smooge> and I am trying to restore 40 tabs
15:47:35 <smooge> any items for Fedora Infrastructure left for this week? I can close it out and people can get back to work
15:48:19 <mymindstorm> I had 2 quick questions if that's ok?
15:48:26 <smooge> sure
15:48:59 <mymindstorm> If static files are missing from the ansible repo, then should I add them as part of the patch?
15:49:14 <mymindstorm> for 8369
15:49:34 <nirik> .ticket 8369
15:49:35 <zodbot> nirik: Issue #8369: Fedorapeople CSS & HTML issues - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8369
15:49:46 <nirik> sure. I would think so
15:51:03 <mymindstorm> I was also wondering if y'all had any suggestions what tickets to work on?
15:51:27 <nirik> mymindstorm: well, depends on your tastes.
15:51:55 <nirik> If you are able to drop by our daily standup thing we usually discuss lots of tickets there. you can get more info, etc.
15:52:37 <mymindstorm> ok, thanks. everything here is pretty overwhelming
15:52:37 <nirik> or you can always ask in tickets about them...
15:52:44 <nirik> yeah. ;(
15:52:53 <nirik> there's a ton of stuff...
15:54:49 <smooge> ok I think we have covered everything we can this week
15:54:55 <smooge> #endmeeting