15:00:32 <smooge> #startmeeting Infrastructure (2019-04-04)
15:00:32 <zodbot> Meeting started Thu Apr  4 15:00:32 2019 UTC.
15:00:32 <zodbot> This meeting is logged and archived in a public location.
15:00:32 <zodbot> The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:32 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:00:32 <zodbot> The meeting name has been set to 'infrastructure_(2019-04-04)'
15:00:33 <smooge> #meetingname infrastructure
15:00:33 <smooge> #topic aloha
15:00:33 <smooge> #chair nirik pingou puiterwijk relrod smooge tflink cverna mizdebsk mkonecny abompard bowlofeggs
15:00:33 <zodbot> The meeting name has been set to 'infrastructure'
15:00:33 <zodbot> Current chairs: abompard bowlofeggs cverna mizdebsk mkonecny nirik pingou puiterwijk relrod smooge tflink
15:00:41 <bowlofeggs> .hello2
15:00:42 <zodbot> bowlofeggs: bowlofeggs 'Randy Barlow' <rbarlow@redhat.com>
15:00:48 <bowlofeggs> ooh, i made chair!
15:00:56 <pingou> .hello2
15:00:57 <zodbot> pingou: pingou 'Pierre-YvesChibon' <pingou@pingoured.fr>
15:01:00 * bowlofeggs sits down
15:01:17 * bowlofeggs puts up the leg rests (this is apparently a recliner)
15:01:20 <frailtyy> .hello2
15:01:22 * pingou only here for the first 1/2h
15:01:23 <zodbot> frailtyy: frailtyy 'Garrett Jansen' <garrett@jansen.sh>
15:01:42 * bowlofeggs reaches into the side of the chair and pulls out a beer (this is apparently one of those chairs that has a refrigerator in it)
15:01:51 <tflink> .hello2
15:01:52 <zodbot> tflink: tflink 'Tim Flink' <tflink@redhat.com>
15:02:06 <tflink> bowlofeggs: one of those kind of days already?
15:02:10 <bowlofeggs> hahaha
15:02:31 <bowlofeggs> actually i haven't even had my coffee yet :/
15:02:49 <smooge> #topic New folks introductions
15:02:49 <smooge> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
15:02:49 <smooge> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
15:03:20 <tflink> yikes, that sounds like a really bad day if you're going for beer before coffee on a work day
15:03:23 <mkonecny> .hello zlopez
15:03:24 <zodbot> mkonecny: zlopez 'Michal Konečný' <michal.konecny@packetseekers.eu>
15:03:29 <nirik> morning
15:04:15 <smooge> #topic announcements and information
15:04:16 <smooge> #info Beta Freeze Is Finished.
15:04:16 <smooge> #info Final Freeze Starts on 2019-04-16
15:04:16 <smooge> #info Taskotron major upgrade planned for 2019-04-09
15:04:23 <bowlofeggs> tflink: well, the chair was stocked with coffee porters, so maybe it can be both!
15:04:25 <smooge> Any other announcements
15:04:39 <bowlofeggs> #info bodhi-3.14.0 beta in staging, please test!
15:04:43 <tflink> bowlofeggs: 2 birds with one stone :-D
15:04:50 * nirik 💖s coffee porter
15:05:01 <bowlofeggs> #link https://bodhi.stg.fedoraproject.org/docs/user/release_notes.html#v3-14-0
15:05:21 <mkonecny> #info second blogpost about release-monitoring.org - https://communityblog.fedoraproject.org/stories-from-the-amazing-world-of-release-monitoring-org-2/
15:05:47 <nirik> #info 2 new aarch64 maintainer test machines ready and online
15:06:29 <smooge> nice blogpost mkonecny I read it this morning
15:06:38 <smooge> bowlofeggs, is this the last one before 4.0?
15:06:47 <bowlofeggs> smooge: that's the plan!
15:06:51 <mkonecny> thanks smooge, I already have next in my head
15:06:59 <bowlofeggs> smooge: i'm getting nervous about how much change is in 4.0 so i'd like to get it out to door
15:07:08 <smooge> understood
15:07:12 <smooge> ok next up
15:07:13 <bowlofeggs> going this long wihtout a release often means it'll be a rough one
15:07:22 <bowlofeggs> and 3.14.0 is mostly for otaylor
15:07:29 <bowlofeggs> (and a few other patches sprinkled in)
15:07:32 <smooge> so we need to do it this week before F30 final
15:07:48 <smooge> ok back to the agenda
15:07:55 <smooge> any other announcements?
15:08:22 * linuxmodder here
15:08:39 <smooge> #topic Oncall
15:08:39 <smooge> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
15:08:39 <smooge> #info smooge is on call from 2019-04-04 -> 2019-04-11
15:08:39 <smooge> #info ????? is on call from 2019-04-11 -> 2019-04-18
15:08:39 <smooge> #info ????? is on call from 2019-04-18 -> 2019-04-25
15:08:40 <smooge> #info Summary of last week: (from relrod )
15:08:52 <nirik> I can take the 4-11 to 4-18
15:09:01 <smooge> relrod, is studying for a test.. so I will cover what happened
15:09:09 <smooge> it was a quiet week I think
15:09:16 <nirik> I'd also like to take this time to mention more about oncall...
15:09:33 <nirik> we now have a wiki page: https://fedoraproject.org/wiki/Infrastructure/Oncall
15:10:03 <mkonecny> Nice
15:10:06 <nirik> and all of us on the ops side are going to try and be better about not answering things that should go to the oncall and just pointing them to the wiki page
15:11:11 <smooge> also when you are away from irc please set /away or have your znc do so
15:11:29 <nirik> yeah, I need to look into that.
15:12:42 <smooge> ok anything else here?
15:13:15 <smooge> #topic Monitoring discussion
15:13:15 <smooge> #info https://nagios.fedoraproject.org/nagios
15:13:15 <smooge> #info Go over existing out items and fix
15:13:37 <nirik> I ran the noc playbook last night...
15:13:45 <nirik> it of course needed some fixing up for things.
15:14:41 <nirik> I suppose we could now reboot pagure01 sometime off hours to fix the swap thing
15:14:55 <smooge> yep
15:15:01 <smooge> did you mean pkgs01?
15:15:11 <smooge> or pkgs02?
15:15:13 <nirik> and pagure01. they are both alerting for that
15:15:21 <nirik> pkgs02 needs disk space.
15:15:45 <nirik> but its needs will go down with repospanner, so perhaps we just watch it for now.
15:15:48 <smooge> ok I can put in outages later
15:16:24 <nirik> so... should we try and do a update/reboot cycle next week?
15:16:37 <nirik> or wait until after f30 is out...
15:16:39 <smooge> there are a lot of kernel nd other updates
15:16:57 <smooge> so I was going to say we look at staging boxes and see how ugly that is
15:17:28 <nirik> yeah, I haven't looked, but we can out of meeting...
15:17:53 <smooge> so I can put in a reboot notification for staging for tomorrow and we evaluate over weekend and see about doing rest next week?
15:19:10 <smooge> #topic Tickets discussion
15:19:10 <smooge> #info https://pagure.io/fedora-infrastructure/report/Meetings%20ticket
15:19:23 <smooge> I wanted to say that nirik did a LOT of tickets earlier today
15:19:55 <nirik> if tomorrow is enough notice for pkgs01.stg... since it's centos land important.
15:20:08 <mkonecny> I just want to say thanks for solving the ticket 7646 - src.stg.fp.o: unable to fork
15:20:08 <nirik> yeah, some good triage with pingou
15:20:41 <pingou> kevin++
15:21:51 <smooge> .ticket 7661
15:21:52 <zodbot> smooge: Issue #7661: GeoIP - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/7661
15:22:00 <bowlofeggs> are there any lottery tickets available in our ticket repo?
15:22:09 <nirik> sadly no
15:22:24 <nirik> smooge: relrod wanted to discuss that, but lets table until he's around?
15:22:25 <smooge> This ticket covers the fact that the old GeoIP interface was EOL in 2018
15:22:36 <smooge> we will discuss this hopefully next week
15:23:18 <smooge> #topic Priorities for next week?
15:23:19 <smooge> #info please put tickets needing to be focused on here
15:23:28 <smooge> OK no one put any tickets in here
15:23:40 <nirik> basically anything we want to get in before f30 final freeze we need to do next week
15:23:51 <smooge> 1. update/reboot needed services
15:24:06 <smooge> 2. update any systems in production which have been waiting in staging
15:24:44 <smooge> 3. ??
15:24:47 <nirik> I think we should try and get the flatpak namespace stuff done. which needs a new bodhi and a bunch of other stuff
15:25:09 <smooge> 3. flatpak namespace deployed
15:25:10 <nirik> .title https://pagure.io/fedora-infrastructure/issue/7496
15:25:11 <zodbot> nirik: Issue #7496: Configure a separate flatpaks/ namespace for Flatpaks - fedora-infrastructure - Pagure.io
15:27:51 <smooge> nirik, who does the PDC work?
15:28:28 <bowlofeggs> nirik: yeah the bodhi 3.14 in stg is for that
15:28:38 <bowlofeggs> nirik: we don't have to time the release though, because it's a config in 3.14
15:28:49 <bowlofeggs> so we need it deployed, and can just flip the config when we want to turn it on
15:28:55 <bowlofeggs> that way we can coordinate more easily
15:29:18 <bowlofeggs> otaylor: ^
15:30:18 <smooge> bowlofeggs, abompard how does monday sound for pie day?
15:30:44 <bowlofeggs> smooge: +1 from me, but abompard is planning to do the deployment so let's see what he says
15:30:55 <nirik> I could poke PDC, or mboddu perhaps?
15:31:27 <mboddu> I can work on pdc stuff
15:32:15 <smooge> ok will wait for abompard to agree on a deployment day later
15:32:22 <smooge> anything else here?
15:33:14 <tflink> as a heads up and to make sure this is clear, resultsdb will be down during the taskotron outage next week
15:33:23 <smooge> Ok what day was that?
15:33:34 <tflink> tuesday
15:33:39 <tflink> 2019-04-09
15:33:41 <nirik> this is getting them up to f29 right?
15:33:44 <tflink> yep
15:33:52 <smooge> #info Taskotron Outage is 2019-04-09 and it will affect resultsdb during that time
15:33:57 <tflink> and new buildbot, which was what took so long
15:33:58 * nirik notes there's 2 other sets of machines we need to upgrade badly.
15:34:01 <smooge> ok so we won't do qa outage during that
15:34:15 <smooge> well there's fas :)?
15:34:17 <nirik> sorry, more than 2.
15:34:33 <tflink> smooge: I don't follow - most of the qa bits will be down at that time
15:34:41 <nirik> autocloud*, mdapi*, modernpaste*, and notifs-backend
15:34:52 <smooge> tflink, well if I reboot your virthost-comm while you are doing a rebuild
15:35:01 <tflink> 2 of the qa machines need to be upgraded/destroyed badly
15:35:16 <tflink> smooge: yeah, that might be a problem
15:35:18 <abompard> smooge, bowlofeggs: works for me, Pie day is monday
15:35:31 <smooge> so we will need to coordinate on Tuesday to make sure we do not break you
15:35:33 <tflink> I was planning to update and reboot the majority of the qa machines at the same time while stuff is down, though
15:35:39 <bowlofeggs> too bad we didn't actually release on march 14
15:35:40 <smooge> ok that works for me
15:35:41 <bowlofeggs> oh well
15:36:03 <smooge> 4.4 is close enough to pi
15:36:12 <smooge> oh wait 4.8
15:36:19 <smooge> never mind
15:36:45 <abompard> is always "close". It's a rough estimate, for some definitions of rough.
15:36:54 <smooge> tflink, ok if you are going to do the qa systems, then that works
15:37:16 <smooge> alright next up
15:37:21 <smooge> #topic Open Floor
15:37:24 <smooge> since we are already there
15:37:25 <tflink> it makes sense in my mind since there's already an outage
15:37:38 <nirik> tflink: random query since you are around... are there any plans to work on beaker further? I heard a rumor that it's back to being developed...
15:38:02 <smooge> tflink, yep. it is fine I just didn't want to be doing something that breaks your ability to do an update/reboot
15:38:16 <tflink> nirik: not clear on that one - there's some movement going on in QE wrt automating tests so it's possible but I have my doubts about the public instance
15:38:20 <mkonecny> I wanted to ask, if there is any application with fedora-messaging already deployed on production?
15:38:30 * tflink needs to fix the ssl cert at som point, though
15:38:53 <nirik> tflink: I think the playbooks are broken currently, I don't recall why off hand.
15:38:55 <tflink> but yeah, there is a new team of folks inside RH working on beaker
15:39:17 <smooge> mkonecny, I do not know myself. I added it to retrace today but that really isn't us
15:39:55 <mkonecny> smooge: Ok, I will ask in #fedora-apps
15:40:10 <tflink> nirik: beaker services look like they're not starting properly - I tried running it yesterday
15:41:12 <nirik> ok
15:44:42 <smooge> ok I think we are don with this week.
15:44:52 <smooge> I hope you all have a good day and will see you next week.
15:44:59 <smooge> Same bat-channel. Same bat-time
15:45:07 <smooge> #endmeeting