18:00:25 <nirik> #startmeeting Fedora Infrastructure Ops Daily Standup Meeting
18:00:25 <zodbot> Meeting started Mon Apr  4 18:00:25 2022 UTC.
18:00:25 <zodbot> This meeting is logged and archived in a public location.
18:00:25 <zodbot> The chair is nirik. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
18:00:25 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:25 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting'
18:00:25 <nirik> #chair mboddu nirik mobrien nb
18:00:25 <zodbot> Current chairs: mboddu mobrien nb nirik
18:00:25 <nirik> #meetingname fedora_infrastructure_ops_daily_standup_meeting
18:00:25 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting'
18:00:25 <nirik> #info meeting is 30 minutes MAX. At the end of 30, its stops
18:00:25 <nirik> #info agenda is at https://board.net/p/fedora-infra-daily
18:00:26 <nirik> #info reminder: speak up if you want to work on a ticket!
18:00:28 <nirik> #topic Tickets needing review
18:00:30 <nirik> #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1
18:00:45 <aheath1992> .hello aheath1992
18:00:46 <zodbot> aheath1992: aheath1992 'Andrew Heath' <aheath1992@gmail.com>
18:01:02 <SSmoogen> hello
18:01:13 <mboddu> Here here
18:01:44 <mobrien> Hello
18:01:55 <nirik> morning everyone.
18:02:01 <nirik> .ticket 10623
18:02:02 <zodbot> nirik: Issue #10623: ask.fp.o rejects auth from Red Hat internal network due to too many requests from the same IP - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10623
18:02:22 <nirik> med med ops I guess? need mattdm I think to do it, I don't think I have any visibility into those settings. ;(
18:03:56 <nirik> .ticket 10625
18:03:58 <mobrien> +1
18:04:06 <nirik> I just closed this one. sharkcz fixed them! :)
18:04:08 <zodbot> nirik: Issue #10625: z/vm s390x builders down - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10625
18:04:20 <nirik> .ticket 10626
18:04:21 <zodbot> nirik: Issue #10626: Outage: Upgrade of Copr servers - 2022-04-05 07:00 UTC - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10626
18:04:29 <nirik> just needs waiting for asignee?
18:04:34 <SSmoogen> +1
18:04:40 <nirik> well, and low low ops I suppose.
18:05:39 <mboddu> Done
18:05:47 <nirik> thats it for new infra... any new releng?
18:06:17 <mboddu> Yup
18:06:33 <mboddu> .releng 10730
18:06:34 <zodbot> mboddu: Issue #10730: Mass "add to admin" needed for some rubygems - releng - Pagure.io - https://pagure.io/releng/issue/10730
18:06:55 <mboddu> The task is low, low, ops, but there is no script to mass update them
18:07:24 <mboddu> And jednorozec is working on writing the script
18:07:31 <nirik> ah, cool.
18:07:41 <nirik> perhaps med trouble then?
18:08:08 <mboddu> Maybe...
18:08:34 <mboddu> .releng 10731
18:08:35 <zodbot> mboddu: Issue #10731: F37: System wide change: Signed RPM Contents - releng - Pagure.io - https://pagure.io/releng/issue/10731
18:08:53 <mboddu> This requires some changes on releng side as well but mostly robosig config
18:09:01 <mboddu> high gain, low trouble, ops?
18:09:17 <SSmoogen> need a high-devel-list
18:09:31 <mboddu> Yeah, it depends on the approval
18:10:03 <nirik> I'm not sure it's high gain. ;)
18:10:14 <nirik> but doesn't matter I guess...
18:10:24 <SSmoogen> med-gain,med-trouble,ops,fesco-approval?
18:10:26 <nirik> At least we know it all worked before.
18:11:11 <mboddu> From security point of view, its high gain :)
18:11:39 <mboddu> But I do understand, so, med gain, low trouble, ops
18:11:51 <SSmoogen> I had one item for meeting
18:11:52 <mboddu> There is no fesco approval tag :(
18:12:28 <nirik> well, itself it's not really... it depends on something using it and setting up policies.
18:12:37 <nirik> +1 to med/low/ops
18:12:59 <SSmoogen> +1 to that also
18:14:08 <mboddu> .releng 10733
18:14:09 <zodbot> mboddu: Issue #10733: Stalled EPEL package: tinyproxy - releng - Pagure.io - https://pagure.io/releng/issue/10733
18:14:14 <mboddu> .releng 10734
18:14:15 <zodbot> mboddu: Issue #10734: Stalled EPEL Package: xsel - releng - Pagure.io - https://pagure.io/releng/issue/10734
18:14:20 <mboddu> low, low, ops for both
18:15:12 <nirik> +1
18:15:53 <SSmoogen> same +1
18:15:58 <mboddu> .releng 10735
18:15:59 <zodbot> mboddu: Issue #10735: Link to current $releasever in https://dl.fedoraproject.org/pub/archive/epel/? - releng - Pagure.io - https://pagure.io/releng/issue/10735
18:16:34 <nirik> I think this person is confused about what archive means there.
18:16:40 <nirik> or what we intend archive to mean
18:16:57 <mboddu> Maybe we should move 8/ to 8.0/ ? But both 8/ and 8.1 are symlinked to 8.1.2020-04-22/
18:17:25 <mboddu> Yeah, but to avoid the confusion, should we move 8/ to 8.0 ?
18:17:42 <nirik> archive is the OLD releases. It would make no sense to put current releases where.
18:17:43 <nirik> there
18:18:16 <nirik> sure, I guess we could do that...
18:18:26 <nirik> I don't know if it will break anything.
18:18:41 <SSmoogen> actually removing 8 is ok
18:18:55 <SSmoogen> 8 should not be there as there was no saving of 8.0
18:19:11 <nirik> huh, so whats in that dir?
18:19:16 <SSmoogen> it also links to 8.1
18:19:32 <SSmoogen> because by the time we started archiving it.. that was what was there
18:19:57 <mboddu> Yup, which confused as well
18:20:05 <SSmoogen> lrwxrwxrwx.  1 root root   15 Apr 22  2020 8 -> 8.1.2020-04-22/
18:20:05 <SSmoogen> lrwxrwxrwx.  1 root root   15 Apr 22  2020 8.1 -> 8.1.2020-04-22/
18:20:05 <SSmoogen> drwxr-xr-x.  4 root root 4096 Jun 16  2021 8.1.2020-04-22
18:20:24 <nirik> ok, I'm fine deleting the 8 link then.
18:20:34 <nirik> since we don't have a 8.0 archive.
18:21:23 <nirik> if others are ok with that... who will bell the cat?
18:21:48 <SSmoogen> wait
18:21:55 <SSmoogen> oh sorry different bell
18:22:00 <SSmoogen> I can bell the cat
18:22:05 <mboddu> +1 to remove 8/ dir
18:22:19 <SSmoogen> I am quite happy to tell someone they are right and we fixed it by removing it
18:22:49 <SSmoogen> took the ticket. going to see what box I can log into to clean it up
18:22:50 <nirik> ha. best kind of right. ;)
18:23:31 <SSmoogen> done
18:23:41 <nirik> mboddu: next?
18:24:09 <mboddu> Thanks SSmoogen
18:24:14 <mboddu> And yes, one last one
18:24:35 <mboddu> .releng 10736
18:24:36 <zodbot> mboddu: Issue #10736: Salvage KOJI build marked for deletion by garbage collector - releng - Pagure.io - https://pagure.io/releng/issue/10736
18:25:01 <mboddu> low, low, ops
18:25:01 <SSmoogen> the title of that sounds like fun
18:25:08 <mboddu> Yeah :)
18:25:11 <SSmoogen> is that a nope
18:25:42 <mboddu> We can tag them in candidate tag and write the signed rpms back
18:25:44 <nirik> they need to be tagged into the candidate tag
18:25:50 <SSmoogen> ah not one which has been past that date
18:25:56 <nirik> bodhi should take care of signing when an update is made
18:26:35 <nirik> if they were 'deleted' nothing we could do, but they are still there.
18:26:43 <nirik> mboddu: you got the tagging/ticket? or want me to?
18:26:49 <mboddu> nirik: I can take it
18:26:59 <nirik> cool. Also untag them from tashcan
18:27:12 <mboddu> Yup
18:27:12 <nirik> is that it on tickets?
18:27:15 <SSmoogen> two items from me 1) https://pagure.io/fedora-infra/ansible/pull-request/1026 2) RDU download server.. still need help?
18:27:17 <mboddu> Yup
18:27:22 <nirik> #topic Upcoming / Planning
18:27:44 <SSmoogen> oh sorry can wait til open floor
18:27:46 <nirik> SSmoogen: I was gonna make another pass of PR's today
18:28:03 <nirik> and yes, download-rdu01 is still down. I meant to file a ticket on it, but never got to it. ;(
18:28:42 <nirik> #info F36 final freeze starts tomorrow
18:29:19 <nirik> #topic Open Flood
18:29:21 <SSmoogen> ok I am without a car until Thursday/Friday. I can work on it after the car gets back if still needed
18:29:58 <nirik> SSmoogen: thanks. I can try and get a ticket in today and cc you... hopefully they can get it, but if not...
18:30:22 <aheath1992> With ticket 10623 if we need reach out to Red Hat IT let me know.
18:30:29 <SSmoogen> ok it may be a 'just put a vm on that other dell'
18:30:39 <mobrien> Just an FYI from me. I had to disable mail sending for rbac today it broke again
18:31:59 <nirik> SSmoogen: yeah, but we will see.
18:32:14 <nirik> aheath1992: ok.
18:32:19 <SSmoogen> okie dokie
18:32:33 <nirik> mobrien: yeah, we need to poke it... I need to update the ansible upgrade ticket with my findings...
18:32:51 <aheath1992> nirik, as Red Hat IT i may be able to help move it along if needed
18:32:59 <nirik> I'm wondering if we might not try AWX again now. Should be easier to install these days as a operator
18:33:28 <mboddu> ^ we might or we might not ?
18:33:30 <nirik> aheath1992: well, it's a weird one. it's in a IT cage in RDU2... I don't know who all has access to it.
18:34:25 <SSmoogen> I had access to it but might not anymore. PNT does not have access to it. IT dcops does have access to it
18:34:34 <nirik> mboddu: yeah, that sounds odd, but I meant try. "might not try" I think is a weird englishism meaning "do it"...
18:35:01 <aheath1992> I know a few people who have access to it.
18:35:11 <mboddu> nirik: Ah okay, I am all up for trying to get awx in fedora infra
18:35:14 <nirik> aheath1992: can cc you...
18:35:35 <aheath1992> Sure
18:35:45 <aheath1992> and +1 to AWX
18:35:52 <nirik> mboddu: There's a lot of questions about how it will work with our setup, but if we can get it working, that would give us access control for people.... we could ditch rbac-playbook.
18:36:33 <nirik> anyhow, we are over time...
18:36:34 <phsmoura> little update on infra 10532, I'm seeing uploads without errors in fedimg. posted a comment on to ticket to the requester. Dont know how it got fixed tho
18:37:01 <nirik> phsmoura: yeah, I guess it could have been my playbook run... but I am pretty sure we ran the playbook in the past before and it didn't help
18:37:02 <mobrien> Awx sounds like an interesting option. Never really looked at it before
18:37:46 <nirik> mobrien: we tried it years ago when it first came out... it was... rough. We never even got it pully deployed.
18:38:12 <phsmoura> nirik: I did some changes there on Friday too, but I'm not sure if it was it that fixed
18:38:45 <nirik> mobrien: oh, whats the status on that power9 box in rdu? still on your list?
18:40:05 <nirik> and... I broke mdapi. :) whee....
18:40:43 <aheath1992> mobrien, AWX is neat ive never used it on OpenShift but I have used it before
18:40:55 <SSmoogen> what is mdapi?  markdown api server?
18:41:05 * mboddu has something as well
18:41:26 <nirik> metadata api... it takes repos of packages and exposes them as an api you can query.
18:41:40 <nirik> mboddu: go... we are already way over. ;)
18:41:40 <mboddu> #info Signing server might be a bit busy to sign ~300 modules in eln
18:42:13 <nirik> eln has 300 modules? woah.
18:42:21 <mboddu> Yup
18:42:33 <nirik> thats more than rhel has isn't it?
18:43:22 <mboddu> Well, 300 module builds in total and not all of them are latest and since there isn't an easy way to find all the latest builds for each stream, the easiest way is "sign them all" :(
18:43:47 <nirik> ugh, ok
18:44:06 <nirik> ok, anything else?
18:44:25 <mboddu> `koji list-tagged --latest` considers ruby-2.5 and ruby-3.1 as one package and showing the latest of the two instead latest of each
18:45:00 <nirik> yeah, modules were never very fullly baked in koji
18:45:00 * SSmoogen rolls some dice to see how large a critical failure this is
18:45:54 * mboddu sees a 1 ;)
18:46:04 <nirik> #endmeeting