15:04:57 <smooge> #startmeeting Infrastructure (2019-09-26)
15:04:57 <zodbot> Meeting started Thu Sep 26 15:04:57 2019 UTC.
15:04:57 <zodbot> This meeting is logged and archived in a public location.
15:04:57 <zodbot> The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:04:57 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:04:57 <zodbot> The meeting name has been set to 'infrastructure_(2019-09-26)'
15:04:57 <smooge> #meetingname infrastructure
15:04:57 <smooge> #topic aloha
15:04:57 <zodbot> The meeting name has been set to 'infrastructure'
15:04:57 <smooge> #chair nirik pingou relrod smooge tflink cverna mizdebsk mkonecny abompard bowlofeggs
15:04:57 <zodbot> Current chairs: abompard bowlofeggs cverna mizdebsk mkonecny nirik pingou relrod smooge tflink
15:05:10 <mizdebsk> hello
15:05:10 <relrod> morning
15:05:14 <smooge> My apologies for lateness. I got looking at disk space issues and missed the clock
15:05:18 <mkonecny> .hello zlopez
15:05:20 <zodbot> mkonecny: zlopez 'Michal Konečný' <michal.konecny@packetseekers.eu>
15:05:45 <mkonecny> smooge: Don't worry, this could happen to everybody
15:06:20 <smooge> nirik has power issues so may not be as available here as usual
15:06:23 * cverna waves
15:06:27 <smooge> #topic Next chair
15:06:27 <smooge> #info magic eight ball says:
15:06:49 <cverna> I ll take next week chair
15:07:05 <smooge> bowlofeggs did the meeting last week. I am doing it this week. Next week is cverna
15:07:14 <smooge> hey magic eight ball said it too
15:07:20 <cverna> +1
15:07:23 <cverna> :-)
15:07:30 <smooge> #info cverna will chair the next meeting
15:07:38 <smooge> #topic New folks introductions
15:07:38 <smooge> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
15:07:38 <smooge> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
15:07:51 <nirik> morning
15:08:17 <austinpowered> .hello
15:08:17 <zodbot> austinpowered: (hello <an alias, 1 argument>) -- Alias for "hellomynameis $1".
15:08:24 <nirik> yeah, I might drop off... but should be able to get back on hopefully.
15:10:18 <smooge> ok
15:10:20 <smooge> next up
15:10:42 <smooge> #topic announcements and information
15:10:42 <smooge> #info We are looking for people to maintain Fedocal and Nuancier - mkonecny
15:10:50 <mizdebsk> #info Koschei prod was moved to OpenSift - mizdebsk
15:10:52 <smooge> #info Final freeze starts soon
15:11:02 <smooge> congrats mizdebsk
15:11:43 <smooge> any other announcements this week?
15:11:45 <nirik> well, we have all next week... I don't know if we want to think about a update cycle, or not
15:11:58 <nirik> yeah, great work mizdebsk. Good to get more things in openshift.
15:13:06 <smooge> not sure.. that is 3 days concentrated on reboots
15:13:32 <smooge> is there anything else we need to land before the freeze?
15:13:52 <smooge> if not we could schedule it and do it for Tues->Thurs
15:14:16 <nirik> I'm not sure it's worth it... let me look at whats pending and we can discuss out of meeting?
15:14:22 <smooge> sure
15:14:26 <mkonecny> I will probably update Anitya and the-new-hotness in production
15:14:30 <mizdebsk> i am working on improving monitoring for koschei in openshift
15:14:39 <smooge> #topic Oncall
15:14:39 <smooge> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
15:14:39 <smooge> #info Summary of last week: (from bowlofeggs )
15:14:47 <nirik> I would like to give kojidb more memory... (which will need a really short outage). Will try and do it off hours sometime.
15:14:58 <mizdebsk> (tickets #7588 and #7115)
15:15:12 <mizdebsk> since we don't have staging nagios and nagios is frozen, this has to happen between freezes
15:15:38 * nirik nods
15:16:48 <smooge> i think bowlofeggs is unavailable so will go over what I saw. pretty quiet week
15:17:01 <smooge> who is going to cover starting today?
15:17:05 <nirik> yeah, not too crazy...
15:17:14 <nirik> our ticket queue has been growing again tho
15:17:28 <nirik> I could take it if no one else...
15:18:03 <smooge> I am unavailable this weekend but I can do next week
15:18:36 <smooge> #info nirik is on call 2019-09-26 -> 2019-10-03
15:18:36 <cverna> I can take it too
15:18:40 <pingou> I'm afk the coming 10 days, but I can take it from the 10th
15:18:43 <cverna> since I do the meeting
15:19:29 <smooge> cverna, can you do next week, I can take the next
15:19:48 <cverna> I do next week
15:19:50 <smooge> #info cverna is on call 2019-10-03 -> 2019-10-10
15:20:24 <smooge> #info pingou is on call 2019-10-10->2019-10-17
15:20:35 <smooge> #info smooge is on call 2019-10-17->2019-10-??
15:20:52 <pingou> 24
15:21:10 <nirik> final freeze starts on the 8th, FYI
15:21:27 <nirik> 22nd or 29th for release.
15:21:46 <smooge> nirik has a pilgrimage sometime around then
15:21:54 <bowlofeggs> sup bros - i'm here now (sorry)
15:21:58 <nirik> yeah, I might be away. need to plan that
15:22:04 <smooge> #topic Monitoring discussion
15:22:04 <smooge> #info https://nagios.fedoraproject.org/nagios
15:22:04 <smooge> #info Go over existing out items and fix
15:22:27 <smooge> hi bowlofeggs
15:23:00 <nirik> osbs is waiting reinstall or whatever
15:23:10 <mizdebsk> os-node04/05 alerts are caused by new monitoring for koschei at openshift
15:23:11 <cverna> yes I did not have time to do it :(
15:23:24 <nirik> 2 drives out are on their way to the DC
15:23:30 <mizdebsk> they will be fixed by me - more details in ticket 7115
15:23:39 <nirik> fas fedmsgs still needs us to add that to the openshift setup
15:23:42 <smooge> they will be fixed and we can hopefully clear those next week
15:23:52 <nirik> and thats all I think.
15:24:37 <smooge> we have 9 more emags which have arrived in the colocation which will need installation and adding. That should help on some of the osbs items
15:24:50 <smooge> #topic Monitoring discussion
15:24:50 <smooge> #info https://nagios.fedoraproject.org/nagios
15:24:50 <smooge> #info Go over existing out items and fix
15:25:04 <smooge> and that is what I pasted already geez
15:25:21 <smooge> #topic backlog discussion
15:25:21 <smooge> #info go over our backlog and discuss and determine priority
15:25:32 <smooge> nirik, your item
15:25:44 <nirik> yeah... hopefully I stay online. ;)
15:25:59 <nirik> so, I have marked some infra and some releng tickets a while back with 'backlog'
15:26:10 <nirik> these are all things that have been sitting there for a while.
15:26:28 <nirik> They may even not be something we want to do, or might be something we just keep never getting time for.
15:26:59 <nirik> I'd like to propose we start with a small list of them and try and take a few a week... in small teams
15:27:27 <nirik> that does mean we take more people's time, but I think it could be very good to help train people and also to have better ideas how to do things, etc.
15:27:45 <cverna> +1 also the oncall should act has a shield to protect others
15:27:52 <nirik> some of them may also be things that need privs, but even so, others can suggest actions, talk to the one doing the work, etc
15:28:02 <cverna> And allow them to focus on their tickets
15:28:08 * nirik nods.
15:28:25 <nirik> so, with that in mind. I have: https://board.net/p/fedora-infra-backlog that has all the backlog items I marked...
15:28:38 <nirik> (note: there's tons more, just these are oldish ones I thought we wanted to do)
15:29:00 <nirik> I'd like everyone to think that they have 5 x's...
15:29:08 * dustymabe late
15:29:19 <nirik> and place your x after the items you think are high pri or we should do first or whatever.
15:29:32 <nirik> ie, you could do xxxxx on something you really want, or spread them around.
15:29:48 <nirik> I'm happy to try and explain what the tickets are (If I know)
15:30:47 <nirik> although some actually are in progress already (since I marked them a while ago)
15:31:12 <nirik> 8158 pingou fixed. ;)
15:31:28 <cverna> \o/
15:31:54 <pingou> xx on that one :)
15:32:24 * nirik lost cout of his votes, but I think I am done. ;)
15:33:02 * cverna is done too
15:33:06 * pingou done
15:33:47 * relrod too
15:33:54 <nirik> The move builds thing is already in progress... but... It should go in ansible and I should show everyone how it works (documentation) and so forth, so I think there's still a good chunk there.
15:34:22 <nirik> ok, so, lets see.
15:34:33 <pingou> ansible-report
15:34:38 <pingou> registry.fp.o
15:34:40 <nirik> looks like a 3 way tie. ;)
15:34:53 <pingou> and new aarch64 builders
15:35:01 <pingou> the top 4
15:35:09 * mkonecny done with x's
15:35:11 <nirik> awesome.
15:35:59 <nirik> ok, so I will mark those and we can see who is interested in helping on each one (I can do that on the list out of meeting)
15:36:15 <mkonecny> We talked about 8167 today, when we helped with cverna moving the elections
15:36:34 <nirik> yeah, I am not sure how best to do that one...
15:37:02 <nirik> oh, might have an idea.
15:37:29 <nirik> smooge: can we sync rhos-15 ? (just came out). I bet it has the newer rabbitmq we need on el7
15:37:42 <smooge> i will see
15:37:53 <smooge> we need to clean up some of the old stuff we have
15:38:13 <pingou> how many do we want to pick and work on this week?
15:38:18 <pingou> 1 or 2, more?
15:38:29 <nirik> I think we should start small and work up.
15:38:40 <mkonecny> You mean for tomorrow?
15:39:10 <pingou> so the top 3 (w/ equal number of votes) are: aarch64, ansible-report and koji archiving
15:39:14 <nirik> also I think one thing we can do from the ops side is try and schedule a time to actually work on a specific one...
15:39:25 <nirik> so interested people can join on irc
15:39:39 * mkonecny likes the idea
15:40:18 * nirik already did try and install one aarch64 box, but it didn't work and I am not yet sure why.
15:40:37 <nirik> so, lets try and focus on those 3 this coming week (until next meeting)?
15:40:43 <pingou> +1
15:40:44 <smooge> that sounds good
15:40:53 <cverna> +1
15:41:00 <nirik> I am not sure the hook will be that hard... I haven;'t looked
15:41:26 <smooge> nirik, for the emag boxes we should talk with arrfab as they had to get them up and running for C8
15:41:54 <nirik> smooge: ok, good idea. They all have centos7alt on them, but I had no way to login. ;)
15:42:11 <nirik> I did get it to pxe boot... but then it went into la la land.
15:43:03 <nirik> so I have a meeting tomorrow from 16-16:30utc.
15:43:07 <cverna> Oh we should create a ticket for fasclient and python3 too so that it is visible
15:43:21 <nirik> but I could try and be available before or after that to work on any of these.
15:43:27 <nirik> cverna: oh! yes, great idea.
15:43:34 <nirik> I can file it?
15:43:37 <cverna> And then we can focus on that next week maybe
15:43:42 <nirik> oh wait, we do have one
15:43:57 <nirik> .ticket 8210
15:43:58 <zodbot> nirik: Issue #8210: FAS clients not installable on Fedora 31+ - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8210
15:44:03 <cverna> So let's tag it with backlog
15:44:09 <nirik> ok.
15:44:57 <nirik> for everyone following along: There's also a internal jira we are playing with that has these backlog items... but it's all manual syncing between them, and we are not putting anything other than a pointer to pagure in the jira tickets
15:46:09 <cverna> Yep pagure tickets are the source of truth
15:46:35 <nirik> so, perhaps tomorrow morning I will try and work on aarch64 installs... and will try and get input from anyone who wants to know the setup or help me out.
15:46:48 <smooge> ok anything else on this. there is a hard stop in this channel at 11:55 for other meetings
15:46:52 <nirik> I'd like to wait on the koji archiving until mboddu is around...
15:47:06 <nirik> I will also mail the list about this all... thats all I had
15:47:15 * nirik is amazed, the power is still up.
15:47:43 * mboddu reading back
15:48:44 * pingou would be happy to help w/ fasclient to py3
15:50:13 <mboddu> nirik: Re: koji archiving, what do you want from me and when do you want to work on it?
15:50:21 <nirik> just updated the ticket. I have been testing on rawhide-test box... so anyone with access there could try it.
15:50:53 <nirik> mboddu: just wanted you to be around when I explain how it works and get your help to figure out how to put it in ansible, etc. :)
15:51:17 <nirik> so, perhaps we find a time next week?
15:51:26 <mboddu> nirik: Ah okay, I would love to help and sure, that works
15:51:31 <smooge> #topic Fedora CoreOS related tickets
15:51:31 <smooge> #info for 2019-09-26 meeting
15:51:31 <smooge> #info https://github.com/coreos/fedora-coreos-tracker/blob/master/Fedora-Requests.md#existing-requests-for-fedora-infra
15:51:31 <smooge> \
15:52:00 <nirik> and there goes my power. ;)
15:52:05 <pingou> ó/
15:52:14 <smooge> I think this is other backlog we need to look at after we go to later
15:52:36 <pingou> smooge: that links gives me a 404
15:52:41 <pingou> .headers https://github.com/coreos/fedora-coreos-tracker/blob/master/Fedora-Requests.md#existing-requests-for-fedora-infra
15:52:42 <zodbot> pingou: HTTP Error 404: Not Found
15:52:50 <smooge> ok well that makes it easy to deal with tehn
15:53:01 <pingou> \ó/
15:53:06 <smooge> #topic Open Floor
15:53:29 <smooge> thank you all for coming. remember next week we will be working on the 3 high backlog items
15:53:29 <nirik> It was moved to hackmd
15:53:30 <pingou> it's moved to hackmd but the commit doesn't say where
15:53:45 <nirik> yeah. dunno.
15:53:58 <smooge> we have 2 minutes left
15:54:05 <smooge> anything else for this meeting?
15:54:25 * nirik has nothing off hand
15:54:39 <dustymabe> I do
15:54:45 <dustymabe> sorry
15:54:48 <dustymabe> the new link is
15:54:53 <dustymabe> https://hackmd.io/5uB7hOJKSjGUt65iLgPnbA
15:54:59 <nirik> thanks dustymabe
15:55:04 <dustymabe> sorry about that
15:55:22 <dustymabe> so #8218
15:55:33 <dustymabe> I think bowlofeggs said last time he was working on docs on how to add people to communishift
15:55:49 <nirik> yeah
15:55:51 <dustymabe> if I could get a peak at those docs maybe I could take a stab at https://pagure.io/fedora-infrastructure/issue/8218#comment-598925
15:56:02 <dustymabe> I think that basic thing should be easy to do ^^
15:56:16 * pingou has to step out
15:56:19 <pingou> cheers everyone!
15:56:34 <nirik> dustymabe: sure, can you update the ticket with that request?
15:56:43 <dustymabe> the request for the docs?
15:56:54 <nirik> yeah. I don't think they exist yet, he is collecting info for them
15:57:18 <dustymabe> done
15:57:21 * nirik listens to the chirp of the UPS. How long will it last...
15:57:29 <dustymabe> next item is #8208
15:58:02 <dustymabe> it seems that both tflink and the couros teams want this
15:58:04 <nirik> yeah, I have had no cycles to look at that.
15:58:13 <nirik> but I think it would be good to try and get working.
15:58:30 <dustymabe> yeah I need that working so I can run this one workload in communishift
15:58:41 <cverna> Can we tag this with the backlog tag
15:58:43 <nirik> yeah... ditto for qa
15:58:47 <nirik> cverna: sure!
15:58:54 <cverna> Then it will go in our queue
15:58:56 <nirik> cverna: can you sync the pagure backlog to the jira one?
15:59:01 <cverna> Yep
15:59:09 * nirik hears fast beeps... not long now.
15:59:42 <smooge> ok time to call this meeting
15:59:45 <smooge> #endmeeting