18:00:25 <nirik> #startmeeting Fedora Infrastructure Ops Daily Standup Meeting 18:00:25 <zodbot> Meeting started Mon Apr 4 18:00:25 2022 UTC. 18:00:25 <zodbot> This meeting is logged and archived in a public location. 18:00:25 <zodbot> The chair is nirik. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions. 18:00:25 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:00:25 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting' 18:00:25 <nirik> #chair mboddu nirik mobrien nb 18:00:25 <zodbot> Current chairs: mboddu mobrien nb nirik 18:00:25 <nirik> #meetingname fedora_infrastructure_ops_daily_standup_meeting 18:00:25 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting' 18:00:25 <nirik> #info meeting is 30 minutes MAX. At the end of 30, its stops 18:00:25 <nirik> #info agenda is at https://board.net/p/fedora-infra-daily 18:00:26 <nirik> #info reminder: speak up if you want to work on a ticket! 18:00:28 <nirik> #topic Tickets needing review 18:00:30 <nirik> #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1 18:00:45 <aheath1992> .hello aheath1992 18:00:46 <zodbot> aheath1992: aheath1992 'Andrew Heath' <aheath1992@gmail.com> 18:01:02 <SSmoogen> hello 18:01:13 <mboddu> Here here 18:01:44 <mobrien> Hello 18:01:55 <nirik> morning everyone. 18:02:01 <nirik> .ticket 10623 18:02:02 <zodbot> nirik: Issue #10623: ask.fp.o rejects auth from Red Hat internal network due to too many requests from the same IP - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10623 18:02:22 <nirik> med med ops I guess? need mattdm I think to do it, I don't think I have any visibility into those settings. ;( 18:03:56 <nirik> .ticket 10625 18:03:58 <mobrien> +1 18:04:06 <nirik> I just closed this one. sharkcz fixed them! :) 18:04:08 <zodbot> nirik: Issue #10625: z/vm s390x builders down - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10625 18:04:20 <nirik> .ticket 10626 18:04:21 <zodbot> nirik: Issue #10626: Outage: Upgrade of Copr servers - 2022-04-05 07:00 UTC - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10626 18:04:29 <nirik> just needs waiting for asignee? 18:04:34 <SSmoogen> +1 18:04:40 <nirik> well, and low low ops I suppose. 18:05:39 <mboddu> Done 18:05:47 <nirik> thats it for new infra... any new releng? 18:06:17 <mboddu> Yup 18:06:33 <mboddu> .releng 10730 18:06:34 <zodbot> mboddu: Issue #10730: Mass "add to admin" needed for some rubygems - releng - Pagure.io - https://pagure.io/releng/issue/10730 18:06:55 <mboddu> The task is low, low, ops, but there is no script to mass update them 18:07:24 <mboddu> And jednorozec is working on writing the script 18:07:31 <nirik> ah, cool. 18:07:41 <nirik> perhaps med trouble then? 18:08:08 <mboddu> Maybe... 18:08:34 <mboddu> .releng 10731 18:08:35 <zodbot> mboddu: Issue #10731: F37: System wide change: Signed RPM Contents - releng - Pagure.io - https://pagure.io/releng/issue/10731 18:08:53 <mboddu> This requires some changes on releng side as well but mostly robosig config 18:09:01 <mboddu> high gain, low trouble, ops? 18:09:17 <SSmoogen> need a high-devel-list 18:09:31 <mboddu> Yeah, it depends on the approval 18:10:03 <nirik> I'm not sure it's high gain. ;) 18:10:14 <nirik> but doesn't matter I guess... 18:10:24 <SSmoogen> med-gain,med-trouble,ops,fesco-approval? 18:10:26 <nirik> At least we know it all worked before. 18:11:11 <mboddu> From security point of view, its high gain :) 18:11:39 <mboddu> But I do understand, so, med gain, low trouble, ops 18:11:51 <SSmoogen> I had one item for meeting 18:11:52 <mboddu> There is no fesco approval tag :( 18:12:28 <nirik> well, itself it's not really... it depends on something using it and setting up policies. 18:12:37 <nirik> +1 to med/low/ops 18:12:59 <SSmoogen> +1 to that also 18:14:08 <mboddu> .releng 10733 18:14:09 <zodbot> mboddu: Issue #10733: Stalled EPEL package: tinyproxy - releng - Pagure.io - https://pagure.io/releng/issue/10733 18:14:14 <mboddu> .releng 10734 18:14:15 <zodbot> mboddu: Issue #10734: Stalled EPEL Package: xsel - releng - Pagure.io - https://pagure.io/releng/issue/10734 18:14:20 <mboddu> low, low, ops for both 18:15:12 <nirik> +1 18:15:53 <SSmoogen> same +1 18:15:58 <mboddu> .releng 10735 18:15:59 <zodbot> mboddu: Issue #10735: Link to current $releasever in https://dl.fedoraproject.org/pub/archive/epel/? - releng - Pagure.io - https://pagure.io/releng/issue/10735 18:16:34 <nirik> I think this person is confused about what archive means there. 18:16:40 <nirik> or what we intend archive to mean 18:16:57 <mboddu> Maybe we should move 8/ to 8.0/ ? But both 8/ and 8.1 are symlinked to 8.1.2020-04-22/ 18:17:25 <mboddu> Yeah, but to avoid the confusion, should we move 8/ to 8.0 ? 18:17:42 <nirik> archive is the OLD releases. It would make no sense to put current releases where. 18:17:43 <nirik> there 18:18:16 <nirik> sure, I guess we could do that... 18:18:26 <nirik> I don't know if it will break anything. 18:18:41 <SSmoogen> actually removing 8 is ok 18:18:55 <SSmoogen> 8 should not be there as there was no saving of 8.0 18:19:11 <nirik> huh, so whats in that dir? 18:19:16 <SSmoogen> it also links to 8.1 18:19:32 <SSmoogen> because by the time we started archiving it.. that was what was there 18:19:57 <mboddu> Yup, which confused as well 18:20:05 <SSmoogen> lrwxrwxrwx. 1 root root 15 Apr 22 2020 8 -> 8.1.2020-04-22/ 18:20:05 <SSmoogen> lrwxrwxrwx. 1 root root 15 Apr 22 2020 8.1 -> 8.1.2020-04-22/ 18:20:05 <SSmoogen> drwxr-xr-x. 4 root root 4096 Jun 16 2021 8.1.2020-04-22 18:20:24 <nirik> ok, I'm fine deleting the 8 link then. 18:20:34 <nirik> since we don't have a 8.0 archive. 18:21:23 <nirik> if others are ok with that... who will bell the cat? 18:21:48 <SSmoogen> wait 18:21:55 <SSmoogen> oh sorry different bell 18:22:00 <SSmoogen> I can bell the cat 18:22:05 <mboddu> +1 to remove 8/ dir 18:22:19 <SSmoogen> I am quite happy to tell someone they are right and we fixed it by removing it 18:22:49 <SSmoogen> took the ticket. going to see what box I can log into to clean it up 18:22:50 <nirik> ha. best kind of right. ;) 18:23:31 <SSmoogen> done 18:23:41 <nirik> mboddu: next? 18:24:09 <mboddu> Thanks SSmoogen 18:24:14 <mboddu> And yes, one last one 18:24:35 <mboddu> .releng 10736 18:24:36 <zodbot> mboddu: Issue #10736: Salvage KOJI build marked for deletion by garbage collector - releng - Pagure.io - https://pagure.io/releng/issue/10736 18:25:01 <mboddu> low, low, ops 18:25:01 <SSmoogen> the title of that sounds like fun 18:25:08 <mboddu> Yeah :) 18:25:11 <SSmoogen> is that a nope 18:25:42 <mboddu> We can tag them in candidate tag and write the signed rpms back 18:25:44 <nirik> they need to be tagged into the candidate tag 18:25:50 <SSmoogen> ah not one which has been past that date 18:25:56 <nirik> bodhi should take care of signing when an update is made 18:26:35 <nirik> if they were 'deleted' nothing we could do, but they are still there. 18:26:43 <nirik> mboddu: you got the tagging/ticket? or want me to? 18:26:49 <mboddu> nirik: I can take it 18:26:59 <nirik> cool. Also untag them from tashcan 18:27:12 <mboddu> Yup 18:27:12 <nirik> is that it on tickets? 18:27:15 <SSmoogen> two items from me 1) https://pagure.io/fedora-infra/ansible/pull-request/1026 2) RDU download server.. still need help? 18:27:17 <mboddu> Yup 18:27:22 <nirik> #topic Upcoming / Planning 18:27:44 <SSmoogen> oh sorry can wait til open floor 18:27:46 <nirik> SSmoogen: I was gonna make another pass of PR's today 18:28:03 <nirik> and yes, download-rdu01 is still down. I meant to file a ticket on it, but never got to it. ;( 18:28:42 <nirik> #info F36 final freeze starts tomorrow 18:29:19 <nirik> #topic Open Flood 18:29:21 <SSmoogen> ok I am without a car until Thursday/Friday. I can work on it after the car gets back if still needed 18:29:58 <nirik> SSmoogen: thanks. I can try and get a ticket in today and cc you... hopefully they can get it, but if not... 18:30:22 <aheath1992> With ticket 10623 if we need reach out to Red Hat IT let me know. 18:30:29 <SSmoogen> ok it may be a 'just put a vm on that other dell' 18:30:39 <mobrien> Just an FYI from me. I had to disable mail sending for rbac today it broke again 18:31:59 <nirik> SSmoogen: yeah, but we will see. 18:32:14 <nirik> aheath1992: ok. 18:32:19 <SSmoogen> okie dokie 18:32:33 <nirik> mobrien: yeah, we need to poke it... I need to update the ansible upgrade ticket with my findings... 18:32:51 <aheath1992> nirik, as Red Hat IT i may be able to help move it along if needed 18:32:59 <nirik> I'm wondering if we might not try AWX again now. Should be easier to install these days as a operator 18:33:28 <mboddu> ^ we might or we might not ? 18:33:30 <nirik> aheath1992: well, it's a weird one. it's in a IT cage in RDU2... I don't know who all has access to it. 18:34:25 <SSmoogen> I had access to it but might not anymore. PNT does not have access to it. IT dcops does have access to it 18:34:34 <nirik> mboddu: yeah, that sounds odd, but I meant try. "might not try" I think is a weird englishism meaning "do it"... 18:35:01 <aheath1992> I know a few people who have access to it. 18:35:11 <mboddu> nirik: Ah okay, I am all up for trying to get awx in fedora infra 18:35:14 <nirik> aheath1992: can cc you... 18:35:35 <aheath1992> Sure 18:35:45 <aheath1992> and +1 to AWX 18:35:52 <nirik> mboddu: There's a lot of questions about how it will work with our setup, but if we can get it working, that would give us access control for people.... we could ditch rbac-playbook. 18:36:33 <nirik> anyhow, we are over time... 18:36:34 <phsmoura> little update on infra 10532, I'm seeing uploads without errors in fedimg. posted a comment on to ticket to the requester. Dont know how it got fixed tho 18:37:01 <nirik> phsmoura: yeah, I guess it could have been my playbook run... but I am pretty sure we ran the playbook in the past before and it didn't help 18:37:02 <mobrien> Awx sounds like an interesting option. Never really looked at it before 18:37:46 <nirik> mobrien: we tried it years ago when it first came out... it was... rough. We never even got it pully deployed. 18:38:12 <phsmoura> nirik: I did some changes there on Friday too, but I'm not sure if it was it that fixed 18:38:45 <nirik> mobrien: oh, whats the status on that power9 box in rdu? still on your list? 18:40:05 <nirik> and... I broke mdapi. :) whee.... 18:40:43 <aheath1992> mobrien, AWX is neat ive never used it on OpenShift but I have used it before 18:40:55 <SSmoogen> what is mdapi? markdown api server? 18:41:05 * mboddu has something as well 18:41:26 <nirik> metadata api... it takes repos of packages and exposes them as an api you can query. 18:41:40 <nirik> mboddu: go... we are already way over. ;) 18:41:40 <mboddu> #info Signing server might be a bit busy to sign ~300 modules in eln 18:42:13 <nirik> eln has 300 modules? woah. 18:42:21 <mboddu> Yup 18:42:33 <nirik> thats more than rhel has isn't it? 18:43:22 <mboddu> Well, 300 module builds in total and not all of them are latest and since there isn't an easy way to find all the latest builds for each stream, the easiest way is "sign them all" :( 18:43:47 <nirik> ugh, ok 18:44:06 <nirik> ok, anything else? 18:44:25 <mboddu> `koji list-tagged --latest` considers ruby-2.5 and ruby-3.1 as one package and showing the latest of the two instead latest of each 18:45:00 <nirik> yeah, modules were never very fullly baked in koji 18:45:00 * SSmoogen rolls some dice to see how large a critical failure this is 18:45:54 * mboddu sees a 1 ;) 18:46:04 <nirik> #endmeeting