15:01:39 <mkonecny> #startmeeting Infrastructure (2019-11-07)
15:01:39 <zodbot> Meeting started Thu Nov  7 15:01:39 2019 UTC.
15:01:39 <zodbot> This meeting is logged and archived in a public location.
15:01:39 <zodbot> The chair is mkonecny. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:39 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:01:39 <zodbot> The meeting name has been set to 'infrastructure_(2019-11-07)'
15:01:39 <mkonecny> #meetingname infrastructure
15:01:39 <zodbot> The meeting name has been set to 'infrastructure'
15:01:39 <mkonecny> #topic aloha
15:01:39 <mkonecny> #chair nirik pingou relrod smooge tflink cverna mizdebsk mkonecny abompard bowlofeggs
15:01:39 <zodbot> Current chairs: abompard bowlofeggs cverna mizdebsk mkonecny nirik pingou relrod smooge tflink
15:01:48 <mkonecny> .hello zlopez
15:01:50 <zodbot> mkonecny: zlopez 'Michal Konečný' <michal.konecny@packetseekers.eu>
15:01:55 <cverna> hello
15:02:11 <nirik> morning
15:02:14 <relrod> morning
15:02:22 <joalon> morning
15:02:44 <mkonecny> #topic Next chair
15:02:44 <mkonecny> #info magic eight ball says:
15:02:44 <mkonecny> 2019-11-07: mkonecny
15:02:44 <mkonecny> 2019-11-14: smooge
15:02:44 <mkonecny> 2019-11-21: pingou
15:02:55 <smooge> morning
15:03:29 <mkonecny> Do we want to nominate someone for 28th November?
15:04:23 <mkonecny> If not, we could decide this on next meeting
15:04:41 <cverna> Yes next meeting sounds good to me
15:04:52 <mkonecny> #topic New folks introductions
15:04:52 <mkonecny> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
15:04:52 <mkonecny> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
15:05:00 <mkonecny> Anybody new here?
15:06:07 <nirik> 28th nov is a holiday in the us... so we might skip that one.
15:06:24 <mkonecny> nirik: Oh, I forgot
15:07:16 <mkonecny> It looks like nobody new today, so moving to next topic
15:07:18 <mkonecny> #topic announcements and information
15:07:18 <mkonecny> #info f31 final freeze now over
15:07:18 <mkonecny> #info aarch64 hardware mostly sorted, will be installing soon
15:07:18 <mkonecny> #info koji backups are done on secondary host. Slowdown in koji still occurring.
15:07:18 <mkonecny> #info mass update/reboot cycle completed
15:07:47 <mkonecny> Any other announcement?
15:08:30 * nirik has nothing off hand.
15:09:37 <mkonecny> Moving to Oncall then
15:09:39 <mkonecny> #topic Oncall
15:09:39 <mkonecny> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
15:09:39 <mkonecny> #info nirik is on call 2019-10-31->2019-11-07
15:09:39 <mkonecny> #info cverna is on call 2019-11-07->2019-11-14
15:09:39 <mkonecny> #info pingou is on call 2019-11-14->2019-11-21
15:09:40 <mkonecny> #info Summary of last week: (from current oncall )
15:10:21 <nirik> wasn't too bad... really busy dealing with host updates/reboots, but not too many pings
15:10:35 <pingou> most were related to the reboot no?
15:11:02 <nirik> yeah, not too many otherwise.
15:11:13 <cverna> .takeoncalleu
15:11:33 <cverna> What is the magic command ?
15:11:40 <nirik> although I think we are backsliding sometimes (me anyhow) and answering people on irc too quickly/etc
15:11:58 <mkonecny> .oncalltakeus
15:11:58 <zodbot> mkonecny: Error: You don't have the alias.add capability. If you think that you should have this capability, be sure that you are identified before trying again. The 'whoami' command can tell you if you're identified.
15:12:07 <pingou> nirik: well, this week you were allowed ;-)
15:12:19 <mkonecny> I know the words, but I don't have right magic wand :-D
15:12:31 <pingou> try while wearing your hat! ;-)
15:12:38 <nirik> pingou: sure...
15:13:14 <cverna> I also don't want us time but Europe. Even if I don't sleep much 😛
15:13:40 <cverna> .oncalltakeeu
15:13:40 <zodbot> cverna: Kneel before zod!
15:13:41 <nirik> there is 'oncalltakeeu'
15:13:45 <nirik> :)
15:13:55 <mkonecny> I don't think we have separate oncall for us and eu :-D
15:13:55 <cverna> Cool 🙂
15:14:05 <cverna> .oncall
15:14:07 <zodbot> cverna is oncall. My normal hours are 0700 UTC to 1600 UTC Monday through Friday. If I do not answer or it is outside those hours, please file a ticket (https://pagure.io/fedora-infrastructure/issues)
15:14:28 <cverna> Ok I am all setup
15:14:30 <mkonecny> Oh, I was wrong then
15:14:43 <mkonecny> #topic Monitoring discussion
15:14:43 <mkonecny> #info https://nagios.fedoraproject.org/nagios
15:14:43 <mkonecny> #info Go over existing out items and fix
15:15:17 <nirik> so, lets see.
15:15:27 <nirik> we have the osbs-aarch64 boxes down (known)
15:15:38 <nirik> we have virthost-os02 down (it lost 2 drives)
15:16:10 <nirik> bugzilla messages are still not coming in. ;( I will poke that internal ticket again I guess...
15:16:54 * pingou forgot something for the announcements, will wait for the open floor
15:17:03 <nirik> the rabbitmq mts thing I am not sure on...
15:17:15 <nirik> perhaps the app needs a redeploy?
15:17:21 <mkonecny> mts?
15:17:23 <pingou> we could ping on the mts ticket
15:17:35 <pingou> module tagging service iirc
15:18:12 <mkonecny> thanks pingou
15:18:44 * nirik does a deploy
15:19:15 <nirik> I think thats most / all of them?
15:19:49 <mkonecny> What is the issue with bugzilla, those are fedmsg messages?
15:20:13 <nirik> yeah, we listen to a STOMP endpoint and inject fedmsgs...
15:20:18 <nirik> but it stopped sending to us.
15:21:15 <nirik> it might be a expired cert...
15:22:32 <mkonecny> The issue looks like it's on the bugzilla side, thanks for explanation
15:22:51 <mkonecny> #topic Tickets discussion
15:22:52 <mkonecny> #info https://pagure.io/fedora-infrastructure/report/Meetings%20ticket
15:22:52 <mkonecny> #info no tickets marked for the meeting
15:22:52 <mkonecny> #info Fedora CoreOS Team requests
15:22:52 <mkonecny> #link https://hackmd.io/5uB7hOJKSjGUt65iLgPnbA#Existing-requests-for-Fedora-Infra
15:23:26 <mkonecny> Do we have some tickets to be discussed?
15:23:56 <mkonecny> Is somebody from CoreOS Team here?
15:24:38 * nirik checks the link
15:24:40 <smooge> dustymabe, dustymabe, dustymabe
15:25:12 <nirik> yeah, all those are things we would like to get done. ;)
15:25:13 <smooge> hmmm doesn't look like summoning worked
15:25:17 * mkonecny beetlejuice, beetlejuice, beetlejuice
15:26:19 <mkonecny> Let's move to backlog
15:26:20 <mkonecny> #topic backlog discussion
15:26:20 <mkonecny> #info go over our backlog and discuss and determine priority
15:26:20 <mkonecny> #link https://pagure.io/fedora-infrastructure/issue/8178
15:26:20 <mkonecny> #info topic: provision new aarch64 builders
15:26:20 <mkonecny> #link https://pagure.io/fedora-infrastructure/issue/8157
15:26:21 <mkonecny> #info topic: ansible: enable ansible-report as a hook
15:27:25 <nirik> the aarch64 builders are finally all coming along...
15:27:47 <cverna> No progress on the ansible report
15:27:52 <nirik> on ansible-report... I think we were going to run it and send a initial to the list?
15:28:03 <nirik> so we could ideally clean up things?
15:28:24 <cverna> But since I am on call I'll take some time to do that
15:29:16 <mkonecny> What about doing this as part of the end year initiative?
15:29:25 <karsten> how long does one run take ?
15:29:54 <nirik> I think it's pretty quick?
15:30:06 <cverna> I did not manage to do a full run it add some errors
15:30:33 <cverna> It is quite picky on what you give as an input
15:30:34 <nirik> we had all errors fixed at the face2face meeting...
15:30:40 <nirik> but it had a ton of warnings
15:31:23 <cverna> Yes I think this what I saw lots of warnings
15:31:32 <nirik> you can supress those with -q
15:31:32 <mkonecny> Do we want to ignore them?
15:32:24 <nirik> I think we want to fix them, but perhaps only block on error.
15:32:41 * nirik can't remember how to run this thing... it expects that you cat everything to it?
15:33:28 <cverna> You need to use find and xargs
15:33:33 <nirik> perhaps there are better tools now. :)
15:33:48 <cverna> Find all the yml files and xargs them to the tool
15:34:13 <nirik> ah right
15:34:23 <mkonecny> Hm, not that straightforward like flake-8 in Python
15:34:35 <karsten> could someone put that info into the ticket so that anyone can reproduce the warnings/errors without having to ask here first ?
15:35:11 <mkonecny> karsten++
15:35:12 <zodbot> mkonecny: Karma for karsten changed to 1 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
15:35:12 <nirik> yeah, we should.
15:35:27 <cverna> Yes, although the info on how to run is on the GitHub repo
15:35:38 <cverna> But I am happy to update the ticket
15:36:29 * dustymabe waves
15:36:34 <mkonecny> cverna++
15:36:35 <zodbot> mkonecny: Karma for cverna changed to 5 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
15:36:38 <nirik> cverna++
15:36:39 <zodbot> nirik: Karma for cverna changed to 6 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
15:37:05 <mkonecny> Moving to last topic
15:37:06 <mkonecny> #topic Open Floor
15:37:08 <dustymabe> sorry I was late
15:37:13 <dustymabe> i've got nothing for  today
15:37:14 <cverna> Cookies :)
15:37:22 <pingou> #info I've updated ci-resultsdb-listener today
15:37:22 <nirik> dustymabe: welcome
15:37:30 <mkonecny> dustymabe: no worries than
15:37:39 <pingou> it now uses fedora-messaging and supports the new message format used by ci.centos
15:38:06 <pingou> this means, we will be able to decommission resultsdb.ci.centos.org soon
15:38:12 <pingou> as well as archive the rdbsync project
15:38:23 <mkonecny> Nice
15:38:32 <pingou> (rdbsync is already turned off on resultsdb01.qa)
15:38:52 <nirik> excellent.
15:38:58 * pingou is actually just removing it
15:39:10 <pingou> eol
15:39:56 <mkonecny> CentOS CI should start using fedora-messaging soon as well
15:40:06 <pingou> it was announced for next week
15:40:24 <pingou> they are planning on updating the messaging plugin today and change its configuration next week
15:40:36 <mkonecny> I was not sure about the dates
15:41:11 <nirik> so, I have a question on that timing, if we are on to that...
15:41:33 <pingou> I'm ok
15:42:34 <nirik> so next week, I heard... tuesday is prod rawhide gating work? that includes redeploying bodhi-backend01
15:42:49 <pingou> redeploying?
15:43:04 <nirik> reinstalling?
15:43:22 <nirik> or no?
15:43:28 <pingou> not that I am aware of
15:43:30 <pingou> abompard: ^
15:43:33 <mkonecny> I thought it is only an update
15:43:51 <pingou> for me as well
15:43:58 <cverna> Yes redeploying it
15:44:16 <nirik> well, either way is fine, although it's f29 and we need to move it off that before eol
15:44:23 <cverna> Because we need a more recent version of celery or something related
15:44:43 <mkonecny> cverna: I forgot about the celery in the new Bodhi
15:44:47 <nirik> ok, so it's going to f30? because f31 has not all the python2 needed for sigul
15:44:53 <pingou> bodhi-backend01.stg is F30
15:45:02 * nirik nods
15:45:07 <cverna> Yes f30
15:45:08 <pingou> prod is still f29 indeed
15:45:14 <nirik> 30 is fine, just want to know. ;)
15:45:26 <nirik> and finally to my question...
15:45:58 <pingou> yes on Tuesday for bodhi 5.0
15:46:02 <nirik> there's a new koji (1.19.0) I wanted to get updated/out, but... there's been issues with it, so they are doing a 1.19.1 next week hopefully. when would be a good time to do that that doesn't interfere
15:46:16 <nirik> and/or do you need 1.19.x koji as part of this?
15:46:32 <pingou> the koji that is stg seems to be working fine
15:46:41 <pingou> 1.18 I guess?
15:46:53 <nirik> I upgraded it to 1.19.0 monday. :)
15:46:55 <pingou> so we roll out the new bodhi on Tues
15:46:58 <pingou> ah :)
15:47:09 <pingou> well, still seems to perform fine for bodhi's needs
15:47:41 <nirik> well, prod right now is 1.18.1 (well, it's a mix, but thats what is on the hub)
15:47:43 <pingou> I'd recommend that we don't do koji 1.19.1 before Thu
15:48:05 <mkonecny> We could try it in staging first
15:48:17 <pingou> to give us a couple of days (Tue/Wed) to figure out if anything in prod is misbehaving
15:48:36 <pingou> but I'd also recomment we do koji 1.19.1 before announcing rawhide gating
15:48:41 <pingou> (which I have no ETA for atm)
15:48:54 <pingou> but I doubt it'll be before Nov 18th
15:49:02 <pingou> recommend*
15:49:16 <nirik> ok... I'm fine waiting and doing koji upgrade week after next
15:49:24 <mkonecny> Sounds like a reasonable date
15:49:34 <pingou> nirik: end of next week should be fine I think
15:49:39 <nirik> do we need a outage scheduled for tuesday? for bodhi at least/
15:49:50 <pingou> depending on how you like the idea to upgrade koji on a Thu or a Fri
15:50:10 <nirik> yeah, not great, but doable
15:50:14 <mkonecny> I think we should announce a outage, when we will do the redeploy of backend
15:50:32 <pingou> to be on the safe side we can announce it
15:50:37 <pingou> worst case, it's minimal
15:51:01 <nirik> cool... can someone take that as a action. ;) I'm happy to help
15:51:02 <pingou> or rather, worst case there is one, best case barely
15:51:08 <pingou> on it
15:51:22 <mkonecny> pingou++
15:51:22 <zodbot> mkonecny: Karma for pingou changed to 2 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
15:51:34 <pingou> hm I thought we had a template for outage on our issue tracker
15:51:55 <nirik> pingou: we do, but currently the repo attached to fedora-infrastructure is the repospanner one thats not working. ;)
15:52:04 <pingou> ahah, good point :)
15:52:07 <nirik> ie, we would need to move the old one back to get the template.
15:52:21 <pingou> yup
15:52:31 * pingou takes an old ticket as example
15:52:37 <nirik> anyhow, I had one more quick item:
15:53:13 <nirik> We are going to try and do some short projects before the end of the year... hopefully forming a few small teams and just getting them done.
15:53:31 <nirik> we should have a list of them next weekish and will share for everyone to see and provide input on
15:53:46 <nirik> EOF
15:53:58 <pingou> nirik++
15:54:40 <mkonecny> We should look at our backlog when planning these short projects
15:55:22 <mkonecny> There should be something reasonably small
15:55:55 <nirik> mkonecny: oh yeah, there's tons of things. ;)
15:56:01 <nirik> we have no lack of work
15:56:16 <mkonecny> I know, the time is our master
15:56:34 <mkonecny> Do we have something else for the last few minutes?
15:58:03 <mkonecny> In this case, I hereby declare this meeting as finished
15:58:04 <mkonecny> #endmeeting