2024-07-23 15:59:59 <@patrikp:matrix.org> !startmeeting RELENG (2024-07-23)
2024-07-23 16:00:01 <@meetbot:fedora.im> Meeting started at 2024-07-23 15:59:59 UTC
2024-07-23 16:00:02 <@meetbot:fedora.im> The Meeting name is 'RELENG (2024-07-23)'
2024-07-23 16:00:06 <@patrikp:matrix.org> !meetingname releng
2024-07-23 16:00:07 <@smilner:fedora.im> !hi
2024-07-23 16:00:07 <@zodbot:fedora.im> None (smilner)
2024-07-23 16:00:11 <@meetbot:fedora.im> The Meeting Name is now releng
2024-07-23 16:00:13 <@patrikp:matrix.org> !chair nirik phsmoura dustymabe jednorozec jnsamyak patrikp
2024-07-23 16:00:23 <@patrikp:matrix.org> !info Meeting is 60 minutes MAX. At the end of 60, it stops
2024-07-23 16:00:31 <@patrikp:matrix.org> !info agenda is at https://hackmd.io/vm6biLBcTYKtkQUH5kQkmw
2024-07-23 16:00:32 <@nirik:matrix.scrye.com> morning
2024-07-23 16:00:39 <@patrikp:matrix.org> !topic init process
2024-07-23 16:00:43 <@patrikp:matrix.org> Hello to all!
2024-07-23 16:00:59 <@patrikp:matrix.org> Do we have anything for the init? Any blockers, tasks, issues, requests, features that needs releng intervention?
2024-07-23 16:01:09 <@jnsamyak:matrix.org> 0/
2024-07-23 16:01:21 <@jnsamyak:matrix.org> I’m on PTO so joining from my phone 
2024-07-23 16:01:29 <@jnsamyak:matrix.org> Just have one thing mass rebuild is done 
2024-07-23 16:01:37 <@jnsamyak:matrix.org> Samyak has to send email about it 
2024-07-23 16:01:49 <@jnsamyak:matrix.org> And I need to merge the mass rebuild tag
2024-07-23 16:01:56 <@humaton:fedora.im> I think Kevin already did that
2024-07-23 16:02:04 <@jnsamyak:matrix.org> Oh :3
2024-07-23 16:02:10 <@humaton:fedora.im> the email I think I saw it on the devel list
2024-07-23 16:02:13 <@jnsamyak:matrix.org> I guess that’s why the untag request that explains 
2024-07-23 16:02:30 <@jnsamyak:matrix.org> Thanks need to check 
2024-07-23 16:02:47 <@nirik:matrix.scrye.com> yeah, I merged it and sent email. ;)
2024-07-23 16:03:12 <@nirik:matrix.scrye.com> but... the rawhide compose today finished and then got OOM killed before it could sync. ;( I am manually syncing it now.
2024-07-23 16:03:58 <@humaton:fedora.im> since we are talking mass rebuild 
2024-07-23 16:04:03 <@humaton:fedora.im> !releng 12214
2024-07-23 16:04:04 <@zodbot:fedora.im> 
2024-07-23 16:04:04 <@zodbot:fedora.im> **releng #12214** (https://pagure.io/releng/issue/12214):**Investigate and untag packages that failed gating but were merged in via mass rebuild**
2024-07-23 16:04:04 <@zodbot:fedora.im> ● **Last Updated:** 5 hours ago
2024-07-23 16:04:04 <@zodbot:fedora.im> ● **Assignee:** Not Assigned
2024-07-23 16:04:04 <@zodbot:fedora.im> ● **Opened:** 11 hours ago by adamwill
2024-07-23 16:04:35 <@humaton:fedora.im> so my script returns empty list for builds in bodhi updates that are tagged in f41
2024-07-23 16:04:57 <@humaton:fedora.im> can you of you check if my logic in the loop and bodhi query is correct?
2024-07-23 16:05:35 <@nirik:matrix.scrye.com> yeah, they wouldn't be in bodhi...
2024-07-23 16:05:41 <@nirik:matrix.scrye.com> well, I mean...
2024-07-23 16:06:20 <@nirik:matrix.scrye.com> foo-1.0-1.fc41 is made and a bodhi update made for it. It fails gating. mass rebuild bumps and rebuilds it, so foo-1.0-2.fc41 is tagged in directly to f41...
2024-07-23 16:06:23 <@nirik:matrix.scrye.com> thats the case right?
2024-07-23 16:06:37 <@humaton:fedora.im> ahhh
2024-07-23 16:06:52 <@humaton:fedora.im> ok I will update the script
2024-07-23 16:06:56 <@nirik:matrix.scrye.com> but i am not sure how we can get this... I guess it has to be failed gating + anything after?
2024-07-23 16:07:29 <@humaton:fedora.im> or I can get th NVR string rpmbump it and query koji for the build
2024-07-23 16:08:18 <@nirik:matrix.scrye.com> yeah, so I guess ideally, the mass tag script would query bodhi for failed gating updates and make a list, then when tagging check against that list and if package matches not tag it?
2024-07-23 16:08:26 <@nirik:matrix.scrye.com> complicated, but possible I guess.
2024-07-23 16:09:04 <@humaton:fedora.im> it is but we should check for the gating fails
2024-07-23 16:09:39 <@humaton:fedora.im> Ok, first I will update the script and untag all the "bad" NVRs and look into updating the tagging script later
2024-07-23 16:09:47 <@nirik:matrix.scrye.com> it might be nice if bodhi had a query for this.
2024-07-23 16:10:20 <@humaton:fedora.im> they have gating failed and you can filter it
2024-07-23 16:10:26 <@nirik:matrix.scrye.com> I mean a direct query... things in testing that failed tests. but I guess for rawhide anything in testing failed tests.
2024-07-23 16:10:34 <@nirik:matrix.scrye.com> ah, cool.
2024-07-23 16:10:37 <@humaton:fedora.im> so there is some data there I dont know how much 
2024-07-23 16:10:52 <@nirik:matrix.scrye.com> I think we need to be careful about untagging anything at this point... since rawhide is already going out with these...
2024-07-23 16:11:07 <@nirik:matrix.scrye.com> but we may need to. I guess we should look at the list.
2024-07-23 16:11:15 <@humaton:fedora.im> yeah
2024-07-23 16:11:28 <@humaton:fedora.im> I will produce the list first and comment on the ticket
2024-07-23 16:12:31 <@humaton:fedora.im> shoul I just continue with another ticket?
2024-07-23 16:12:50 <@nirik:matrix.scrye.com> one more thing
2024-07-23 16:13:12 <@nirik:matrix.scrye.com> a lot of the tests are currently not working right, we need to regenerate the openqa images after rawhide syncs...
2024-07-23 16:13:25 <@nirik:matrix.scrye.com> so after that the tests can be re-run, some of those might pass
2024-07-23 16:13:52 <@humaton:fedora.im> ok so does it mean its ok to produce the list tomorrow?\
2024-07-23 16:14:34 <@humaton:fedora.im> I will keep in mind that the list might change during the night
2024-07-23 16:14:53 <@humaton:fedora.im> !releng 12208
2024-07-23 16:14:54 <@zodbot:fedora.im> **releng #12208** (https://pagure.io/releng/issue/12208):**Retire golang-github-src-d-gcfg**
2024-07-23 16:14:54 <@zodbot:fedora.im> ● **Last Updated:** 5 hours ago
2024-07-23 16:14:54 <@zodbot:fedora.im> ● **Opened:** 6 days ago by mikelo2
2024-07-23 16:14:54 <@zodbot:fedora.im> 
2024-07-23 16:14:54 <@zodbot:fedora.im> ● **Assignee:** patrikp
2024-07-23 16:15:09 <@humaton:fedora.im> patrikp: can you please do the retire manualy for them? 
2024-07-23 16:15:18 <@patrikp:matrix.org> I wanted to bring this up as well.
2024-07-23 16:15:31 <@humaton:fedora.im> I have opened ticket with fedpkg its error on their side and we should not block them on this...
2024-07-23 16:15:50 <@humaton:fedora.im> its retired and now its ending in composes whitch is not good.
2024-07-23 16:15:59 <@humaton:fedora.im> its retired and now its ending in composes which is not good.
2024-07-23 16:17:04 <@nirik:matrix.scrye.com> There has been a problem with retirements, I think lenkaseg was looking into it... but it might be fixed, I am not sure.
2024-07-23 16:17:19 <@humaton:fedora.im> so there was 2 problems
2024-07-23 16:17:31 <@humaton:fedora.im> one was toddlers not working properly I think that is what lenka fixed
2024-07-23 16:17:39 <@patrikp:matrix.org> Interestingly `fedpkg retire` didn't even orphan it.
2024-07-23 16:17:39 <@patrikp:matrix.org> ?
2024-07-23 16:17:39 <@patrikp:matrix.org> 4) Update PDC
2024-07-23 16:17:39 <@patrikp:matrix.org> 3) Block rawhide tag in Koji
2024-07-23 16:17:39 <@patrikp:matrix.org> 2) Revert the removal of the dead.package file
2024-07-23 16:17:39 <@patrikp:matrix.org> 1) Orphan it
2024-07-23 16:17:39 <@patrikp:matrix.org> The steps here in this specific case would be:
2024-07-23 16:18:24 <@humaton:fedora.im> except you have to create the dead.package and you should not need to update PDC
2024-07-23 16:18:36 <@humaton:fedora.im> the retired status will be refreshed nightly
2024-07-23 16:19:09 <@nirik:matrix.scrye.com> jednorozec: so retirements should be working then with the toddler? But we need to catch up on any we missed right? or does something sync it?
2024-07-23 16:19:38 <@patrikp:matrix.org> dead.package was there (this step of fedpkg retire worked) and the requester manually removed it. So revert that commit?
2024-07-23 16:20:13 <@nirik:matrix.scrye.com> yeah, it should have a dead.package.
2024-07-23 16:20:24 <@humaton:fedora.im> when it failed it stopped processing and restarted on last failed msg
2024-07-23 16:21:30 <@nirik:matrix.scrye.com> jednorozec: hum... ok, I can check the releng tickets about missing retirements and see if it caught up...
2024-07-23 16:21:44 <@nirik:matrix.scrye.com> oh, and that fedpkg error from this package is likely because there's no dead.package file.
2024-07-23 16:22:08 <@nirik:matrix.scrye.com> well, or any files.
2024-07-23 16:22:35 <@humaton:fedora.im> Let me rephrase the retirement, .dead package is created. Before nightly compose all repost with dead.package are added to the list for each branch, toddlers is checking for the dead.package as well and it blocks the package in koji
2024-07-23 16:22:50 <@patrikp:matrix.org> And I wanted to ask how one would get such a list, i.e. all packages that failed to be retired and need to be processed manually.
2024-07-23 16:22:50 <@patrikp:matrix.org> 
2024-07-23 16:22:50 <@patrikp:matrix.org> https://pagure.io/releng/issue/12192#comment-919797
2024-07-23 16:22:50 <@patrikp:matrix.org> Here they ask us to process the other ones.
2024-07-23 16:24:00 <@nirik:matrix.scrye.com> The ones in the initial comment are still not blocked.
2024-07-23 16:24:14 <@humaton:fedora.im> Ok I will run koji block for rawhide, this is the list of retired packages from last night https://src.fedoraproject.org/lookaside/retired_in_rawhide.json
2024-07-23 16:24:26 <@humaton:fedora.im> those packages have dead.package on rawhide branch
2024-07-23 16:25:20 <@humaton:fedora.im> this is the list for f40 and so on https://src.fedoraproject.org/lookaside/retired_in_f40.json
2024-07-23 16:25:20 <@nirik:matrix.scrye.com> Also ones in https://pagure.io/releng/issue/12212
2024-07-23 16:25:56 <@nirik:matrix.scrye.com> if it's syncing based on that shouldn't it be blocking things? something still seems wrong. We should fix the toddler and let it clean up things rather than doing anything manually. ;)
2024-07-23 16:25:58 <@nirik:matrix.scrye.com> IMHO
2024-07-23 16:26:32 <@humaton:fedora.im> the koji blocking in the toddler is fixed
2024-07-23 16:27:21 <@humaton:fedora.im> but it wont catch up until its redeployed in prod, michal have pr for it
2024-07-23 16:27:47 <@humaton:fedora.im> do we want to wait? or just make sure all packages from the listws are blocked befroe the nightly?
2024-07-23 16:28:30 <@nirik:matrix.scrye.com> pr on toddlers? or ? I can't see one...
2024-07-23 16:28:51 <@humaton:fedora.im> https://pagure.io/fedora-infra/toddlers/pull-request/214
2024-07-23 16:29:01 <@humaton:fedora.im> that will deploy the fixed things on prod
2024-07-23 16:30:59 <@humaton:fedora.im> this is the actual fix for the broken koji blocking https://pagure.io/fedora-infra/toddlers/pull-request/210#request_diff
2024-07-23 16:31:11 <@nirik:matrix.scrye.com> ok, if thats ready, we should just push it. ;)
2024-07-23 16:31:35 <@humaton:fedora.im> I am not sure its ready mmichal pushed one more commit hour ago
2024-07-23 16:31:45 <@humaton:fedora.im> I am not sure its ready michal pushed one more commit hour ago
2024-07-23 16:32:01 <@nirik:matrix.scrye.com> ah, ok, we can ask him I guess... it would be very nice to get this fixed. ;)
2024-07-23 16:32:17 <@humaton:fedora.im> I will bring it up on morning standup
2024-07-23 16:32:42 <@humaton:fedora.im> but should I go and block the stuff that was missed so far?
2024-07-23 16:32:49 <@nirik:matrix.scrye.com> I'd say we can just wait then for that.
2024-07-23 16:32:57 <@humaton:fedora.im> since this was my initiative I dont mind cleaning up things...
2024-07-23 16:33:25 <@humaton:fedora.im> ok, I will help to make it deployed 
2024-07-23 16:33:28 <@nirik:matrix.scrye.com> well, it's been broken since feb 3rd... so another day really doesn't seem that big a deal to me... but if you all want to, sure..
2024-07-23 16:33:35 <@nirik:matrix.scrye.com> jul 3rd that is.
2024-07-23 16:33:50 <@humaton:fedora.im> ok
2024-07-23 16:34:02 <@humaton:fedora.im> I was little worried that thing are broken for that long
2024-07-23 16:34:44 <@nirik:matrix.scrye.com> ¯\\\_(ツ)\_/¯ most of them shouldn't have any deps or anything I wouldn't think. Since they are being retired.
2024-07-23 16:35:39 <@humaton:fedora.im> That is all I have, will take the cleanup actions and assign the ticket to myself and see everything is fixed.
2024-07-23 16:36:00 <@nirik:matrix.scrye.com> so close to death of pdc. (I might get a cake)
2024-07-23 16:36:28 <@nirik:matrix.scrye.com> Oh, jednorozec: did you get a chance to update pagure-dist-git in prod? or want me to?
2024-07-23 16:36:42 <@humaton:fedora.im> oh let me do that right now
2024-07-23 16:41:34 <@patrikp:matrix.org> I'll process the ticket that's assigned to me then and write a small SOP about manual retirements as a fallback in case we need it in the future...
2024-07-23 16:42:58 <@nirik:matrix.scrye.com> Sounds good. I think we might have a sop already, but it likely needs adjustment for no pdc
2024-07-23 16:43:19 <@patrikp:matrix.org> Do you happen to know where? I looked through the releng misc guide and couldn't find it.
2024-07-23 16:43:39 <@nirik:matrix.scrye.com> hum... perhaps I was imaging it... let me see
2024-07-23 16:45:15 <@nirik:matrix.scrye.com> yeah, not finding it. Oh well.
2024-07-23 16:45:31 <@nirik:matrix.scrye.com> I guess because it was usually just automated so nothing to do.
2024-07-23 16:45:57 <@patrikp:matrix.org> Yeah, I'll put in a disclaimer that it's just for "last resort" purposes.
2024-07-23 16:46:48 <@humaton:fedora.im> its running
2024-07-23 16:47:03 <@patrikp:matrix.org> What access level/group membership is required for the steps?
2024-07-23 16:47:17 <@humaton:fedora.im> so from now on fedpkg push is not touching pdc anymore
2024-07-23 16:48:24 <@patrikp:matrix.org> I mean these steps minus PDC.
2024-07-23 16:48:52 <@jnsamyak:matrix.org> jednorozec: finally noice
2024-07-23 16:49:08 <@nirik:matrix.scrye.com> 1. admin on pkgs/src pagure, 2. scmadmin/provenpackager, 3. koji admin, 4, pdc admin token.
2024-07-23 16:49:15 <@nirik:matrix.scrye.com> so simple our infra is. ;)
2024-07-23 16:49:26 <@patrikp:matrix.org> Thanks!
2024-07-23 16:49:49 <@jnsamyak:matrix.org> hehe
2024-07-23 16:49:49 <@jnsamyak:matrix.org> >so simple our infra is. ;)
2024-07-23 16:49:55 <@jnsamyak:matrix.org> hehe
2024-07-23 16:49:55 <@jnsamyak:matrix.org> > so simple our infra is. ;)
2024-07-23 16:51:49 <@patrikp:matrix.org> Alright, less than 10 minutes to go. Any other tickets folks would like to bring up?
2024-07-23 16:52:29 <@jnsamyak:matrix.org> i have nothing for today
2024-07-23 16:52:43 <@jnsamyak:matrix.org> thanks for running it today :D
2024-07-23 16:53:09 <@patrikp:matrix.org> Let's go to open floor then.
2024-07-23 16:53:22 <@patrikp:matrix.org> Ah, chair!
2024-07-23 16:53:25 <@patrikp:matrix.org> !topic choose next chair
2024-07-23 16:53:30 <@patrikp:matrix.org> !info next chair Jul 30 - ?
2024-07-23 16:53:35 <@jnsamyak:matrix.org> i can do that :D
2024-07-23 16:53:39 <@patrikp:matrix.org> Sold!
2024-07-23 16:53:44 <@patrikp:matrix.org> !info next chair Jul 30 - Samyak
2024-07-23 16:53:54 <@patrikp:matrix.org> !topic Discussion / Any other Business / Blockers
2024-07-23 16:54:28 <@nirik:matrix.scrye.com> reminder that flock is coming up fast... week after next
2024-07-23 16:56:41 <@patrikp:matrix.org> If there is nothing else you may have a couple minutes of your time back.
2024-07-23 16:56:46 <@patrikp:matrix.org> !info Thank you all for coming.
2024-07-23 16:56:51 <@patrikp:matrix.org> !endmeeting