<@patrikp:matrix.org>
15:59:59
!startmeeting RELENG (2024-07-23)
<@meetbot:fedora.im>
16:00:01
Meeting started at 2024-07-23 15:59:59 UTC
<@meetbot:fedora.im>
16:00:02
The Meeting name is 'RELENG (2024-07-23)'
<@patrikp:matrix.org>
16:00:06
!meetingname releng
<@smilner:fedora.im>
16:00:07
!hi
<@zodbot:fedora.im>
16:00:07
None (smilner)
<@meetbot:fedora.im>
16:00:11
The Meeting Name is now releng
<@patrikp:matrix.org>
16:00:13
!chair nirik phsmoura dustymabe jednorozec jnsamyak patrikp
<@patrikp:matrix.org>
16:00:23
!info Meeting is 60 minutes MAX. At the end of 60, it stops
<@patrikp:matrix.org>
16:00:31
!info agenda is at https://hackmd.io/vm6biLBcTYKtkQUH5kQkmw
<@nirik:matrix.scrye.com>
16:00:32
morning
<@patrikp:matrix.org>
16:00:39
!topic init process
<@patrikp:matrix.org>
16:00:43
Hello to all!
<@patrikp:matrix.org>
16:00:59
Do we have anything for the init? Any blockers, tasks, issues, requests, features that needs releng intervention?
<@jnsamyak:matrix.org>
16:01:09
0/
<@jnsamyak:matrix.org>
16:01:21
I’m on PTO so joining from my phone
<@jnsamyak:matrix.org>
16:01:29
Just have one thing mass rebuild is done
<@jnsamyak:matrix.org>
16:01:37
Samyak has to send email about it
<@jnsamyak:matrix.org>
16:01:49
And I need to merge the mass rebuild tag
<@humaton:fedora.im>
16:01:56
I think Kevin already did that
<@jnsamyak:matrix.org>
16:02:04
Oh :3
<@humaton:fedora.im>
16:02:10
the email I think I saw it on the devel list
<@jnsamyak:matrix.org>
16:02:13
I guess that’s why the untag request that explains
<@jnsamyak:matrix.org>
16:02:30
Thanks need to check
<@nirik:matrix.scrye.com>
16:02:47
yeah, I merged it and sent email. ;)
<@nirik:matrix.scrye.com>
16:03:12
but... the rawhide compose today finished and then got OOM killed before it could sync. ;( I am manually syncing it now.
<@humaton:fedora.im>
16:03:58
since we are talking mass rebuild
<@humaton:fedora.im>
16:04:03
!releng 12214
<@zodbot:fedora.im>
16:04:04
<@zodbot:fedora.im>
16:04:04
**releng #12214** (https://pagure.io/releng/issue/12214):**Investigate and untag packages that failed gating but were merged in via mass rebuild**
<@zodbot:fedora.im>
16:04:04
● **Last Updated:** 5 hours ago
<@zodbot:fedora.im>
16:04:04
● **Assignee:** Not Assigned
<@zodbot:fedora.im>
16:04:04
● **Opened:** 11 hours ago by adamwill
<@humaton:fedora.im>
16:04:35
so my script returns empty list for builds in bodhi updates that are tagged in f41
<@humaton:fedora.im>
16:04:57
can you of you check if my logic in the loop and bodhi query is correct?
<@nirik:matrix.scrye.com>
16:05:35
yeah, they wouldn't be in bodhi...
<@nirik:matrix.scrye.com>
16:05:41
well, I mean...
<@nirik:matrix.scrye.com>
16:06:20
foo-1.0-1.fc41 is made and a bodhi update made for it. It fails gating. mass rebuild bumps and rebuilds it, so foo-1.0-2.fc41 is tagged in directly to f41...
<@nirik:matrix.scrye.com>
16:06:23
thats the case right?
<@humaton:fedora.im>
16:06:37
ahhh
<@humaton:fedora.im>
16:06:52
ok I will update the script
<@nirik:matrix.scrye.com>
16:06:56
but i am not sure how we can get this... I guess it has to be failed gating + anything after?
<@humaton:fedora.im>
16:07:29
or I can get th NVR string rpmbump it and query koji for the build
<@nirik:matrix.scrye.com>
16:08:18
yeah, so I guess ideally, the mass tag script would query bodhi for failed gating updates and make a list, then when tagging check against that list and if package matches not tag it?
<@nirik:matrix.scrye.com>
16:08:26
complicated, but possible I guess.
<@humaton:fedora.im>
16:09:04
it is but we should check for the gating fails
<@humaton:fedora.im>
16:09:39
Ok, first I will update the script and untag all the "bad" NVRs and look into updating the tagging script later
<@nirik:matrix.scrye.com>
16:09:47
it might be nice if bodhi had a query for this.
<@humaton:fedora.im>
16:10:20
they have gating failed and you can filter it
<@nirik:matrix.scrye.com>
16:10:26
I mean a direct query... things in testing that failed tests. but I guess for rawhide anything in testing failed tests.
<@nirik:matrix.scrye.com>
16:10:34
ah, cool.
<@humaton:fedora.im>
16:10:37
so there is some data there I dont know how much
<@nirik:matrix.scrye.com>
16:10:52
I think we need to be careful about untagging anything at this point... since rawhide is already going out with these...
<@nirik:matrix.scrye.com>
16:11:07
but we may need to. I guess we should look at the list.
<@humaton:fedora.im>
16:11:15
yeah
<@humaton:fedora.im>
16:11:28
I will produce the list first and comment on the ticket
<@humaton:fedora.im>
16:12:31
shoul I just continue with another ticket?
<@nirik:matrix.scrye.com>
16:12:50
one more thing
<@nirik:matrix.scrye.com>
16:13:12
a lot of the tests are currently not working right, we need to regenerate the openqa images after rawhide syncs...
<@nirik:matrix.scrye.com>
16:13:25
so after that the tests can be re-run, some of those might pass
<@humaton:fedora.im>
16:13:52
ok so does it mean its ok to produce the list tomorrow?\
<@humaton:fedora.im>
16:14:34
I will keep in mind that the list might change during the night
<@humaton:fedora.im>
16:14:53
!releng 12208
<@zodbot:fedora.im>
16:14:54
**releng #12208** (https://pagure.io/releng/issue/12208):**Retire golang-github-src-d-gcfg**
<@zodbot:fedora.im>
16:14:54
● **Last Updated:** 5 hours ago
<@zodbot:fedora.im>
16:14:54
● **Opened:** 6 days ago by mikelo2
<@zodbot:fedora.im>
16:14:54
<@zodbot:fedora.im>
16:14:54
● **Assignee:** patrikp
<@humaton:fedora.im>
16:15:09
patrikp: can you please do the retire manualy for them?
<@patrikp:matrix.org>
16:15:18
I wanted to bring this up as well.
<@humaton:fedora.im>
16:15:31
I have opened ticket with fedpkg its error on their side and we should not block them on this...
<@humaton:fedora.im>
16:15:50
its retired and now its ending in composes whitch is not good.
<@humaton:fedora.im>
16:15:59
its retired and now its ending in composes which is not good.
<@nirik:matrix.scrye.com>
16:17:04
There has been a problem with retirements, I think lenkaseg was looking into it... but it might be fixed, I am not sure.
<@humaton:fedora.im>
16:17:19
so there was 2 problems
<@humaton:fedora.im>
16:17:31
one was toddlers not working properly I think that is what lenka fixed
<@patrikp:matrix.org>
16:17:39
Interestingly `fedpkg retire` didn't even orphan it.
<@patrikp:matrix.org>
16:17:39
?
<@patrikp:matrix.org>
16:17:39
4) Update PDC
<@patrikp:matrix.org>
16:17:39
3) Block rawhide tag in Koji
<@patrikp:matrix.org>
16:17:39
2) Revert the removal of the dead.package file
<@patrikp:matrix.org>
16:17:39
1) Orphan it
<@patrikp:matrix.org>
16:17:39
The steps here in this specific case would be:
<@humaton:fedora.im>
16:18:24
except you have to create the dead.package and you should not need to update PDC
<@humaton:fedora.im>
16:18:36
the retired status will be refreshed nightly
<@nirik:matrix.scrye.com>
16:19:09
jednorozec: so retirements should be working then with the toddler? But we need to catch up on any we missed right? or does something sync it?
<@patrikp:matrix.org>
16:19:38
dead.package was there (this step of fedpkg retire worked) and the requester manually removed it. So revert that commit?
<@nirik:matrix.scrye.com>
16:20:13
yeah, it should have a dead.package.
<@humaton:fedora.im>
16:20:24
when it failed it stopped processing and restarted on last failed msg
<@nirik:matrix.scrye.com>
16:21:30
jednorozec: hum... ok, I can check the releng tickets about missing retirements and see if it caught up...
<@nirik:matrix.scrye.com>
16:21:44
oh, and that fedpkg error from this package is likely because there's no dead.package file.
<@nirik:matrix.scrye.com>
16:22:08
well, or any files.
<@humaton:fedora.im>
16:22:35
Let me rephrase the retirement, .dead package is created. Before nightly compose all repost with dead.package are added to the list for each branch, toddlers is checking for the dead.package as well and it blocks the package in koji
<@patrikp:matrix.org>
16:22:50
And I wanted to ask how one would get such a list, i.e. all packages that failed to be retired and need to be processed manually.
<@patrikp:matrix.org>
16:22:50
<@patrikp:matrix.org>
16:22:50
https://pagure.io/releng/issue/12192#comment-919797
<@patrikp:matrix.org>
16:22:50
Here they ask us to process the other ones.
<@nirik:matrix.scrye.com>
16:24:00
The ones in the initial comment are still not blocked.
<@humaton:fedora.im>
16:24:14
Ok I will run koji block for rawhide, this is the list of retired packages from last night https://src.fedoraproject.org/lookaside/retired_in_rawhide.json
<@humaton:fedora.im>
16:24:26
those packages have dead.package on rawhide branch
<@humaton:fedora.im>
16:25:20
this is the list for f40 and so on https://src.fedoraproject.org/lookaside/retired_in_f40.json
<@nirik:matrix.scrye.com>
16:25:20
Also ones in https://pagure.io/releng/issue/12212
<@nirik:matrix.scrye.com>
16:25:56
if it's syncing based on that shouldn't it be blocking things? something still seems wrong. We should fix the toddler and let it clean up things rather than doing anything manually. ;)
<@nirik:matrix.scrye.com>
16:25:58
IMHO
<@humaton:fedora.im>
16:26:32
the koji blocking in the toddler is fixed
<@humaton:fedora.im>
16:27:21
but it wont catch up until its redeployed in prod, michal have pr for it
<@humaton:fedora.im>
16:27:47
do we want to wait? or just make sure all packages from the listws are blocked befroe the nightly?
<@nirik:matrix.scrye.com>
16:28:30
pr on toddlers? or ? I can't see one...
<@humaton:fedora.im>
16:28:51
https://pagure.io/fedora-infra/toddlers/pull-request/214
<@humaton:fedora.im>
16:29:01
that will deploy the fixed things on prod
<@humaton:fedora.im>
16:30:59
this is the actual fix for the broken koji blocking https://pagure.io/fedora-infra/toddlers/pull-request/210#request_diff
<@nirik:matrix.scrye.com>
16:31:11
ok, if thats ready, we should just push it. ;)
<@humaton:fedora.im>
16:31:35
I am not sure its ready mmichal pushed one more commit hour ago
<@humaton:fedora.im>
16:31:45
I am not sure its ready michal pushed one more commit hour ago
<@nirik:matrix.scrye.com>
16:32:01
ah, ok, we can ask him I guess... it would be very nice to get this fixed. ;)
<@humaton:fedora.im>
16:32:17
I will bring it up on morning standup
<@humaton:fedora.im>
16:32:42
but should I go and block the stuff that was missed so far?
<@nirik:matrix.scrye.com>
16:32:49
I'd say we can just wait then for that.
<@humaton:fedora.im>
16:32:57
since this was my initiative I dont mind cleaning up things...
<@humaton:fedora.im>
16:33:25
ok, I will help to make it deployed
<@nirik:matrix.scrye.com>
16:33:28
well, it's been broken since feb 3rd... so another day really doesn't seem that big a deal to me... but if you all want to, sure..
<@nirik:matrix.scrye.com>
16:33:35
jul 3rd that is.
<@humaton:fedora.im>
16:33:50
ok
<@humaton:fedora.im>
16:34:02
I was little worried that thing are broken for that long
<@nirik:matrix.scrye.com>
16:34:44
¯\\\_(ツ)\_/¯ most of them shouldn't have any deps or anything I wouldn't think. Since they are being retired.
<@humaton:fedora.im>
16:35:39
That is all I have, will take the cleanup actions and assign the ticket to myself and see everything is fixed.
<@nirik:matrix.scrye.com>
16:36:00
so close to death of pdc. (I might get a cake)
<@nirik:matrix.scrye.com>
16:36:28
Oh, jednorozec: did you get a chance to update pagure-dist-git in prod? or want me to?
<@humaton:fedora.im>
16:36:42
oh let me do that right now
<@patrikp:matrix.org>
16:41:34
I'll process the ticket that's assigned to me then and write a small SOP about manual retirements as a fallback in case we need it in the future...
<@nirik:matrix.scrye.com>
16:42:58
Sounds good. I think we might have a sop already, but it likely needs adjustment for no pdc
<@patrikp:matrix.org>
16:43:19
Do you happen to know where? I looked through the releng misc guide and couldn't find it.
<@nirik:matrix.scrye.com>
16:43:39
hum... perhaps I was imaging it... let me see
<@nirik:matrix.scrye.com>
16:45:15
yeah, not finding it. Oh well.
<@nirik:matrix.scrye.com>
16:45:31
I guess because it was usually just automated so nothing to do.
<@patrikp:matrix.org>
16:45:57
Yeah, I'll put in a disclaimer that it's just for "last resort" purposes.
<@humaton:fedora.im>
16:46:48
its running
<@patrikp:matrix.org>
16:47:03
What access level/group membership is required for the steps?
<@humaton:fedora.im>
16:47:17
so from now on fedpkg push is not touching pdc anymore
<@patrikp:matrix.org>
16:48:24
I mean these steps minus PDC.
<@jnsamyak:matrix.org>
16:48:52
jednorozec: finally noice
<@nirik:matrix.scrye.com>
16:49:08
1. admin on pkgs/src pagure, 2. scmadmin/provenpackager, 3. koji admin, 4, pdc admin token.
<@nirik:matrix.scrye.com>
16:49:15
so simple our infra is. ;)
<@patrikp:matrix.org>
16:49:26
Thanks!
<@jnsamyak:matrix.org>
16:49:49
hehe
<@jnsamyak:matrix.org>
16:49:49
>so simple our infra is. ;)
<@jnsamyak:matrix.org>
16:49:55
hehe
<@jnsamyak:matrix.org>
16:49:55
> so simple our infra is. ;)
<@patrikp:matrix.org>
16:51:49
Alright, less than 10 minutes to go. Any other tickets folks would like to bring up?
<@jnsamyak:matrix.org>
16:52:29
i have nothing for today
<@jnsamyak:matrix.org>
16:52:43
thanks for running it today :D
<@patrikp:matrix.org>
16:53:09
Let's go to open floor then.
<@patrikp:matrix.org>
16:53:22
Ah, chair!
<@patrikp:matrix.org>
16:53:25
!topic choose next chair
<@patrikp:matrix.org>
16:53:30
!info next chair Jul 30 - ?
<@jnsamyak:matrix.org>
16:53:35
i can do that :D
<@patrikp:matrix.org>
16:53:39
Sold!
<@patrikp:matrix.org>
16:53:44
!info next chair Jul 30 - Samyak
<@patrikp:matrix.org>
16:53:54
!topic Discussion / Any other Business / Blockers
<@nirik:matrix.scrye.com>
16:54:28
reminder that flock is coming up fast... week after next
<@patrikp:matrix.org>
16:56:41
If there is nothing else you may have a couple minutes of your time back.
<@patrikp:matrix.org>
16:56:46
!info Thank you all for coming.
<@patrikp:matrix.org>
16:56:51
!endmeeting