<@jnsamyak:matrix.org>
16:00:50
!startmeeting RELENG (2024-07-30)
<@meetbot:fedora.im>
16:00:52
Meeting started at 2024-07-30 16:00:50 UTC
<@meetbot:fedora.im>
16:00:52
The Meeting name is 'RELENG (2024-07-30)'
<@jnsamyak:matrix.org>
16:01:04
!meetingname releng
<@meetbot:fedora.im>
16:01:05
The Meeting Name is now releng
<@jnsamyak:matrix.org>
16:01:14
!chair nirik jednorozec jnsamyak patrikp
<@jnsamyak:matrix.org>
16:01:21
!info Meeting is 60 minutes MAX. At the end of 60, it stops
<@jnsamyak:matrix.org>
16:01:21
!info agenda is at https://hackmd.io/vm6biLBcTYKtkQUH5kQkmw
<@jnsamyak:matrix.org>
16:01:21
!topic init process
<@jnsamyak:matrix.org>
16:01:31
Hi folks how is everyone doing today?
<@patrikp:matrix.org>
16:01:39
Hello. π
<@jnsamyak:matrix.org>
16:01:39
Welcome to releng weekly partayyy
<@Zlopez:matrix.org>
16:02:17
!hi
<@jnsamyak:matrix.org>
16:02:19
Do we have anything for the init? Any blockers, tasks, issues, requests, features that needs releng intervention?
<@zodbot:fedora.im>
16:02:20
Michal Konecny (zlopez)
<@nirik:matrix.scrye.com>
16:02:20
morning
<@jnsamyak:matrix.org>
16:03:09
I have some things for the init - some relates to mass branching
<@jnsamyak:matrix.org>
16:03:44
!releng 12210
<@zodbot:fedora.im>
16:03:47
β **Assignee:** jnsamyak
<@zodbot:fedora.im>
16:03:47
β **Last Updated:** 23 hours ago
<@zodbot:fedora.im>
16:03:47
β **Opened:** a week ago by petersen
<@zodbot:fedora.im>
16:03:47
<@zodbot:fedora.im>
16:03:47
**releng #12210** (https://pagure.io/releng/issue/12210):**a few mass rebuild bumps failed to git push - script should retry or error**
<@jnsamyak:matrix.org>
16:03:59
I want this ticket to become a feature ticket instead of ops
<@jnsamyak:matrix.org>
16:04:30
I was discussing this with tomas today, and we should add a check for retrying the failed push things
<@nirik:matrix.scrye.com>
16:04:41
well, yeah, it's sort of development, we need to improve the script
<@jnsamyak:matrix.org>
16:06:22
so i was planning to do introduce a function in another script which collects the list of packages and then retry those packages to push and build
<@jnsamyak:matrix.org>
16:06:36
or should i do it in the same script?
<@smilner:fedora.im>
16:06:52
!hi
<@carlwgeorge:matrix.org>
16:06:53
!hi
<@zodbot:fedora.im>
16:06:53
None (smilner)
<@zodbot:fedora.im>
16:06:54
Carl George (carlwgeorge) - he / him / his
<@jnsamyak:matrix.org>
16:07:28
<@jnsamyak:matrix.org>
16:07:28
The logic is to check the massbuild output and compare it with the failure
<@jnsamyak:matrix.org>
16:07:28
```
<@jnsamyak:matrix.org>
16:07:28
def parse_massbuild_log(logfile):
<@jnsamyak:matrix.org>
16:07:28
"""Parse the massbuild.out log file to find packages that failed to push."""
<@jnsamyak:matrix.org>
16:07:28
failed_packages = []
<@jnsamyak:matrix.org>
16:07:28
with open(logfile, 'r') as file:
<@jnsamyak:matrix.org>
16:07:28
lines = file.readlines()
<@jnsamyak:matrix.org>
16:07:28
for line in lines:
<@jnsamyak:matrix.org>
16:07:28
if 'failed push' in line:
<@jnsamyak:matrix.org>
16:07:28
pkg = line.split()[0]
<@jnsamyak:matrix.org>
16:07:28
failed_packages.append(pkg)
<@jnsamyak:matrix.org>
16:07:28
return failed_packages
<@jnsamyak:matrix.org>
16:07:28
<@jnsamyak:matrix.org>
16:07:28
```
<@nirik:matrix.scrye.com>
16:07:31
My laptop seems unhappy with the xz I started. On via phone
<@jnsamyak:matrix.org>
16:07:34
something like this ^
<@nirik:matrix.scrye.com>
16:08:22
Yeah, that could work... Or just a retry loop in the script? But might get stuck I guess
<@jnsamyak:matrix.org>
16:08:43
but yeah something like this, we can discuss more when i have the pr ready for this or we can discuss this in the ticket, the only thing is i wanted it to convert from ops to dev
<@jnsamyak:matrix.org>
16:08:59
Hey Steve and Carl George π
<@blackwell:fedora.im>
16:09:03
!hi
<@zodbot:fedora.im>
16:09:03
Jason Blackwell (blackwell)
<@jnsamyak:matrix.org>
16:09:21
Anything that you folks want to bring up before we dive int releng tickets later on :D
<@jnsamyak:matrix.org>
16:09:42
Anything that you folks want to bring up before we dive in releng tickets later on :D
<@carlwgeorge:matrix.org>
16:09:59
i mainly wanted to be available for interactive questions on my epel10 activity
<@carlwgeorge:matrix.org>
16:10:08
thanks for the quick merges and actions on those recent issues/prs
<@jnsamyak:matrix.org>
16:10:16
\0/
<@carlwgeorge:matrix.org>
16:10:20
thanks for the quick merges and actions on those recent prs/issues
<@jnsamyak:matrix.org>
16:10:51
Carl George: okay that brings me to: https://pagure.io/releng/fedora-scm-requests/issue/63884
<@carlwgeorge:matrix.org>
16:10:57
getting epel10.0 in bodhi let me pass the fedpkg check to request the first epel10 branch, but it showed me i missed setting the branch slas in toddlers
<@nirik:matrix.scrye.com>
16:11:06
I think perhaps a check for error in the script and a sleep 5; retry might solve most of it, but sure... either way
<@jnsamyak:matrix.org>
16:11:09
yes
<@carlwgeorge:matrix.org>
16:11:28
almost got the pr to set the slas and hopefully get the branch creation fixed
<@Zlopez:matrix.org>
16:12:11
Carl George: Just deployed the changes for bodhi for EPEL 10
<@Zlopez:matrix.org>
16:12:25
nirik: Running the whole playbook helped
<@carlwgeorge:matrix.org>
16:12:31
awesome
<@jnsamyak:matrix.org>
16:13:07
nice nice
<@jnsamyak:matrix.org>
16:13:44
let's move on then to our next topic?
<@jnsamyak:matrix.org>
16:15:03
!topic scheduled actions coming up in the next week
<@jnsamyak:matrix.org>
16:15:09
!info Here we list/discuss anything about items that are due to be done in the next week
<@jnsamyak:matrix.org>
16:15:52
I did the RAWHIDEFTBFS filing last day!
<@nirik:matrix.scrye.com>
16:16:14
Thanks!
<@nirik:matrix.scrye.com>
16:16:37
!info next week is flock
<@jnsamyak:matrix.org>
16:16:46
woohoo :F
<@jnsamyak:matrix.org>
16:16:50
woohoo :D
<@patrikp:matrix.org>
16:17:05
Who is going to Flock? (from people present here)
<@carlwgeorge:matrix.org>
16:17:15
i was super bummed to see that the infra/releng hackfest is scheduled at the same time as the epel10 hackfest
<@smilner:fedora.im>
16:17:32
π I'll be there.
<@carlwgeorge:matrix.org>
16:17:39
i'll be there
<@humaton:fedora.im>
16:17:42
!hi
<@zodbot:fedora.im>
16:17:43
TomΓ‘Ε‘ HrΔka (humaton) - he / him / his
<@nirik:matrix.scrye.com>
16:18:02
Carl George: yeah, it got moved. ;(
<@Zlopez:matrix.org>
16:18:26
I'll be on Flock as well
<@smilner:fedora.im>
16:18:29
Carl George: nirik is it possible for us to ask for it to move or are things in cement at this point?
<@nirik:matrix.scrye.com>
16:18:44
I am not sure. ;) would need to ask Justin W. Flory (he/him)
<@jnsamyak:matrix.org>
16:19:04
click good pictures, for the people not attending :3
<@carlwgeorge:matrix.org>
16:19:09
was just asking him
<@jnsamyak:matrix.org>
16:20:03
let's take this to open floor, later on?
<@jnsamyak:matrix.org>
16:20:17
!link https://fedorapeople.org/groups/schedule/f-41/f-41-key-tasks.html
<@nirik:matrix.scrye.com>
16:20:19
sure. Didn't mean to sidetrack
<@jnsamyak:matrix.org>
16:21:23
so branching is scheduled for a week later, but i plan to open the issue tracker and the prs one week prior for us to review
<@nirik:matrix.scrye.com>
16:21:34
https://fedorapeople.org/groups/schedule/f-41/f-41-releng-tasks.html might be better / additional
<@jnsamyak:matrix.org>
16:21:34
and learn from the things that went bad last time
<@jnsamyak:matrix.org>
16:21:46
yes
<@jnsamyak:matrix.org>
16:22:01
One thing that folks questioned was bodhi outage
<@jnsamyak:matrix.org>
16:22:46
but we should ask them (in form a mail) prior that we are going to do our branching, and schedule that in status.fpo?
<@jnsamyak:matrix.org>
16:23:21
so people know the outage is gonna happen, and there builds etc will be affected meanwhile branching happens
<@jnsamyak:matrix.org>
16:25:06
nirik: I guess this is the ticket for that?
<@jnsamyak:matrix.org>
16:25:08
!releng 11943
<@zodbot:fedora.im>
16:25:09
β **Assignee:** Not Assigned
<@zodbot:fedora.im>
16:25:09
β **Last Updated:** 5 months ago
<@zodbot:fedora.im>
16:25:09
β **Opened:** 5 months ago by decathorpe
<@zodbot:fedora.im>
16:25:09
β **Closed: It's all good** 5 months ago by kevin
<@zodbot:fedora.im>
16:25:09
<@zodbot:fedora.im>
16:25:09
**releng #11943** (https://pagure.io/releng/issue/11943):**builds / updates in inconsistent state due to mass branching**
<@nirik:matrix.scrye.com>
16:25:33
yeah, I am not sure how to make it better off hand. I guess we can revisit and ponder on it and see if we can come up with any better ways
<@nirik:matrix.scrye.com>
16:26:12
I had on my list to file a ticket on bodhi (which I haven't yet done) to see if it could handle these state changes better.
<@nirik:matrix.scrye.com>
16:26:33
but implementing that would be it's own set of work
<@jnsamyak:matrix.org>
16:27:17
makes sense, so a mailing and branching outage (status.fpo) is we can do for users I guess
<@jnsamyak:matrix.org>
16:27:22
what say?
<@nirik:matrix.scrye.com>
16:28:27
well the problem isn't that people don't know about it...
<@nirik:matrix.scrye.com>
16:28:34
consider this case:
<@nirik:matrix.scrye.com>
16:28:51
package foo is built and submitted, but fails gating, so it's sitting there.
<@nirik:matrix.scrye.com>
16:29:25
then branching happens and it's still there, but bodhi still thinks it's going into rawhide
<@nirik:matrix.scrye.com>
16:29:53
or people build things while the branching is happening and it's a f40 branch with a rawhide build.
<@jnsamyak:matrix.org>
16:29:57
ahhh but it should ideally go to rawhide-1
<@jnsamyak:matrix.org>
16:30:13
makes sense
<@nirik:matrix.scrye.com>
16:30:24
there's all kinds of weird cases...
<@nirik:matrix.scrye.com>
16:30:51
and it's hard to just block everything branching/rawhide without blocking other releases or ourselves
<@nirik:matrix.scrye.com>
16:31:17
but I suppose we could do that... just block external access to koji while branching is happening.
<@jnsamyak:matrix.org>
16:32:02
yeah I'll put this in the issue tracker for branching
<@jnsamyak:matrix.org>
16:32:15
so I can keep this in mind
<@nirik:matrix.scrye.com>
16:32:40
I'll try and ponder on it some and see if I can think of a better way to do it.
<@jnsamyak:matrix.org>
16:33:13
when you say - external access what do you mean? Only koji-admin can submit builds? Or sysadmin-main folks?
<@jnsamyak:matrix.org>
16:33:25
if we restricts for others
<@nirik:matrix.scrye.com>
16:34:32
I mean block at the proxies access to koji.fedoraproject.org (it would show as being down to users)
<@nirik:matrix.scrye.com>
16:34:42
but internally we could still access it
<@nirik:matrix.scrye.com>
16:35:02
people might get mad about that tho. ;)
<@jnsamyak:matrix.org>
16:35:12
ha!
<@jnsamyak:matrix.org>
16:35:46
let's discuss this more on the ticket and mailing list dont want to eat time with finding a way now
<@jnsamyak:matrix.org>
16:35:56
that brings me to one more important step missing from our schedule, I need to remention to Aoife to add these to schedule
<@jnsamyak:matrix.org>
16:36:00
!link https://pagure.io/releng/issue/12001
<@jnsamyak:matrix.org>
16:36:10
We need to create f42 keys
<@nirik:matrix.scrye.com>
16:36:34
ah yep.
<@nirik:matrix.scrye.com>
16:39:30
I can do that and/or if you like walk you through doing it.
<@jnsamyak:matrix.org>
16:39:57
is there an sop?
<@nirik:matrix.scrye.com>
16:40:07
There could well be
<@jflory7:fedora.im>
16:40:13
Let's discuss in #flock:fedoraproject.org? I think all the hackfests will be in the afternoon and workshops in the morning. But maybe we can find a way to make it work.
<@jnsamyak:matrix.org>
16:40:24
i can try to do that if it's not complicated with my new powers ;)
<@nirik:matrix.scrye.com>
16:42:22
hum, the sop if any will need adjustment too actually.
<@nirik:matrix.scrye.com>
16:42:37
we need to make new ima certs for the new key...
<@humaton:fedora.im>
16:42:51
I think the ima keys are in the SOP
<@nirik:matrix.scrye.com>
16:43:14
keys yes, certs no
<@nirik:matrix.scrye.com>
16:43:17
it's new.
<@patrikp:matrix.org>
16:43:25
Do you happen to have a link to the SOP?
<@nirik:matrix.scrye.com>
16:43:26
and we have to send them to kernel maintainers.
<@humaton:fedora.im>
16:43:29
oh
<@nirik:matrix.scrye.com>
16:43:31
I'm looking
<@nirik:matrix.scrye.com>
16:43:39
net is being slow for me today. ;(
<@jnsamyak:matrix.org>
16:43:57
okay can you do it this time and tell the steps so i can create an sop? we can do it as a form of ticket also
<@nirik:matrix.scrye.com>
16:44:20
I really thought there was a sop... but I will keep looking.
<@nirik:matrix.scrye.com>
16:44:30
might be it didn't move over from the old docs?
<@humaton:fedora.im>
16:44:41
hmm
<@humaton:fedora.im>
16:44:47
It was in the new docs for sure
<@nirik:matrix.scrye.com>
16:44:48
but yes, I can update the sop and you can then use it and let me know if it's failing anywhere. ;)
<@humaton:fedora.im>
16:44:54
part of the branching process I think
<@jnsamyak:matrix.org>
16:45:20
jednorozec: branching doesn't seem to have parts for creating new keys?
<@humaton:fedora.im>
16:45:29
but it did
<@humaton:fedora.im>
16:45:36
it might got lost?
<@humaton:fedora.im>
16:45:54
I have noticed some pages were not rebased but rewriten some time ago
<@humaton:fedora.im>
16:45:58
let me find the commit
<@nirik:matrix.scrye.com>
16:46:04
we can track it down after meeting?
<@jnsamyak:matrix.org>
16:46:21
yes
<@jnsamyak:matrix.org>
16:46:57
<@jnsamyak:matrix.org>
16:46:57
brings me to one last thing: jednorozec we can skip this part right now since PDC is ta ta byeb bye
<@jnsamyak:matrix.org>
16:46:57
https://docs.fedoraproject.org/en-US/infra/release_guide/sop_mass_branching/#_pdc
<@nirik:matrix.scrye.com>
16:47:21
yes, we will need to remove pdc from our docs.
<@humaton:fedora.im>
16:47:28
yup
<@humaton:fedora.im>
16:47:34
skip&remove
<@jnsamyak:matrix.org>
16:47:38
I'll remove it from the release process
<@jnsamyak:matrix.org>
16:48:13
moving on to our next item, thanks for the good discussion with the release items
<@jnsamyak:matrix.org>
16:48:14
!topic Tickets needing attention
<@jnsamyak:matrix.org>
16:48:20
!info this is the time where you can bring releng tickets to discuss if there are blockers etc
<@jnsamyak:matrix.org>
16:48:26
Any tickets?
<@humaton:fedora.im>
16:48:45
!link https://pagure.io/releng/issue/12228
<@humaton:fedora.im>
16:49:05
maybe some toddler or something not working properly because of PDC?
<@humaton:fedora.im>
16:50:06
maybe just some glitch? checking other requests today and this was the only complaint so far
<@humaton:fedora.im>
16:50:47
hm but the other requests didnt comit anything yet
<@jnsamyak:matrix.org>
16:51:09
yeah
<@humaton:fedora.im>
16:51:41
hm actually there is one
<@humaton:fedora.im>
16:51:52
requested 10hrs ago and build landed in f41
<@humaton:fedora.im>
16:52:03
https://koji.fedoraproject.org/koji/packageinfo?packageID=40592
<@jnsamyak:matrix.org>
16:52:53
https://pagure.io/releng/fedora-scm-requests/issue/63880
<@jnsamyak:matrix.org>
16:52:54
yes
<@jnsamyak:matrix.org>
16:53:39
maybe some timeout thing? We can check toddlers logs but it seemed OK to me on quick look
<@jnsamyak:matrix.org>
16:54:36
Are there any other tickets, I just want to reserve last 3-4 minutes for open floor!
<@Zlopez:matrix.org>
16:54:37
I'm watching the logs today for toddlers and didn't noticed any error
<@nirik:matrix.scrye.com>
16:54:50
we need a map of where things are done. :0
<@nirik:matrix.scrye.com>
16:54:58
this used to be on bodhi-backend in a listener.
<@humaton:fedora.im>
16:55:35
I can map things I know about
<@humaton:fedora.im>
16:55:45
but that will not be 100% of the processes
<@jnsamyak:matrix.org>
16:56:06
that would be really really good for onboarding as well to infra&releng
<@Zlopez:matrix.org>
16:56:10
Some of the automated things could be confusing sometimes
<@jnsamyak:matrix.org>
16:56:42
!topic Discussion / Any other Business / Blockers
<@nirik:matrix.scrye.com>
16:57:22
ok, it is still on bodhi-backend01.
<@nirik:matrix.scrye.com>
16:57:26
it's got some errros.
<@jnsamyak:matrix.org>
16:57:28
so should we skip next releng call because of flock?
<@nirik:matrix.scrye.com>
16:57:44
Jul 30 16:56:56 bodhi-backend01.iad2.fedoraproject.org fedora-messaging[2178760]: RuntimeError: ['/usr/local/bin/owner-sync-pagure', '--package', 'golang-github-apache-arrow12', '--verbose', 'f41', 'f41-container', 'f41-flatpak', 'f40', 'f40-container', 'f40-flatpak', 'f39', 'f39-container', 'f39-flatpak', 'epel10.0', 'epel9', 'epel9-next', 'epel8'] gave return code 1
<@nirik:matrix.scrye.com>
16:57:52
yes, I think so
<@jnsamyak:matrix.org>
16:58:10
+1/-1 (s)
<@carlwgeorge:matrix.org>
16:58:12
abompard was asking if he or someone in relent should run a certain command related to epel7 retirement
<@carlwgeorge:matrix.org>
16:58:16
!link https://pagure.io/releng/issue/12185
<@carlwgeorge:matrix.org>
16:58:28
abompard was asking if he or someone in releng should run a certain command related to epel7 retirement
<@jnsamyak:matrix.org>
17:00:44
i think sysadmin-main or sysdmin-releng can? OR just who can sudo on mirrormanger machine right nirik ?
<@jnsamyak:matrix.org>
17:01:12
Okay we are on time, I plan to cancel next releng call because of flock
<@jnsamyak:matrix.org>
17:01:19
!topic choose next chair
<@jnsamyak:matrix.org>
17:01:23
!info next chair Aug 13 - Samyak
<@nirik:matrix.scrye.com>
17:01:30
anyone who can run the playbook, yes.
<@jnsamyak:matrix.org>
17:01:30
!info Thank you all for coming.
<@Zlopez:matrix.org>
17:01:39
nirik: That is the new version I just deployed
<@jnsamyak:matrix.org>
17:01:44
meeting timeout error :X
<@jnsamyak:matrix.org>
17:01:46
lol
<@jnsamyak:matrix.org>
17:01:52
!endmeeting