2024-07-30 16:00:50 <@jnsamyak:matrix.org> !startmeeting RELENG (2024-07-30)

2024-07-30 16:00:52 <@meetbot:fedora.im> Meeting started at 2024-07-30 16:00:50 UTC
2024-07-30 16:00:52 <@meetbot:fedora.im> The Meeting name is 'RELENG (2024-07-30)'
2024-07-30 16:01:04 <@jnsamyak:matrix.org> !meetingname releng
2024-07-30 16:01:05 <@meetbot:fedora.im> The Meeting Name is now releng
2024-07-30 16:01:14 <@jnsamyak:matrix.org> !chair nirik jednorozec jnsamyak patrikp
2024-07-30 16:01:21 <@jnsamyak:matrix.org> !info Meeting is 60 minutes MAX. At the end of 60, it stops
2024-07-30 16:01:21 <@jnsamyak:matrix.org> !info agenda is at https://hackmd.io/vm6biLBcTYKtkQUH5kQkmw
2024-07-30 16:01:21 <@jnsamyak:matrix.org> !topic init process
2024-07-30 16:01:31 <@jnsamyak:matrix.org> Hi folks how is everyone doing today?
2024-07-30 16:01:39 <@patrikp:matrix.org> Hello. 👋
2024-07-30 16:01:39 <@jnsamyak:matrix.org> Welcome to releng weekly partayyy
2024-07-30 16:02:17 <@Zlopez:matrix.org> !hi
2024-07-30 16:02:19 <@jnsamyak:matrix.org> Do we have anything for the init? Any blockers, tasks, issues, requests, features that needs releng intervention?
2024-07-30 16:02:20 <@zodbot:fedora.im> Michal Konecny (zlopez)
2024-07-30 16:02:20 <@nirik:matrix.scrye.com> morning
2024-07-30 16:03:09 <@jnsamyak:matrix.org> I have some things for the init - some relates to mass branching
2024-07-30 16:03:44 <@jnsamyak:matrix.org> !releng 12210
2024-07-30 16:03:47 <@zodbot:fedora.im> ● **Assignee:** jnsamyak
2024-07-30 16:03:47 <@zodbot:fedora.im> ● **Last Updated:** 23 hours ago
2024-07-30 16:03:47 <@zodbot:fedora.im> ● **Opened:** a week ago by petersen
2024-07-30 16:03:47 <@zodbot:fedora.im> 
2024-07-30 16:03:47 <@zodbot:fedora.im> **releng #12210** (https://pagure.io/releng/issue/12210):**a few mass rebuild bumps failed to git push - script should retry or error**
2024-07-30 16:03:59 <@jnsamyak:matrix.org> I want this ticket to become a feature ticket instead of ops
2024-07-30 16:04:30 <@jnsamyak:matrix.org> I was discussing this with tomas today, and we should add a check for retrying the failed push things
2024-07-30 16:04:41 <@nirik:matrix.scrye.com> well, yeah, it's sort of development, we need to improve the script
2024-07-30 16:06:22 <@jnsamyak:matrix.org> so i was planning to do introduce a function in another script which collects the list of packages and then retry those packages to push and build
2024-07-30 16:06:36 <@jnsamyak:matrix.org> or should i do it in the same script?
2024-07-30 16:06:52 <@smilner:fedora.im> !hi
2024-07-30 16:06:53 <@carlwgeorge:matrix.org> !hi
2024-07-30 16:06:53 <@zodbot:fedora.im> None (smilner)
2024-07-30 16:06:54 <@zodbot:fedora.im> Carl George (carlwgeorge) - he / him / his
2024-07-30 16:07:28 <@jnsamyak:matrix.org> 
2024-07-30 16:07:28 <@jnsamyak:matrix.org> The logic is to check the massbuild output and compare it with the failure
2024-07-30 16:07:28 <@jnsamyak:matrix.org> ```
2024-07-30 16:07:28 <@jnsamyak:matrix.org> def parse_massbuild_log(logfile):
2024-07-30 16:07:28 <@jnsamyak:matrix.org>     """Parse the massbuild.out log file to find packages that failed to push."""
2024-07-30 16:07:28 <@jnsamyak:matrix.org>     failed_packages = []
2024-07-30 16:07:28 <@jnsamyak:matrix.org>     with open(logfile, 'r') as file:
2024-07-30 16:07:28 <@jnsamyak:matrix.org>         lines = file.readlines()
2024-07-30 16:07:28 <@jnsamyak:matrix.org>         for line in lines:
2024-07-30 16:07:28 <@jnsamyak:matrix.org>             if 'failed push' in line:
2024-07-30 16:07:28 <@jnsamyak:matrix.org>                 pkg = line.split()[0]
2024-07-30 16:07:28 <@jnsamyak:matrix.org>                 failed_packages.append(pkg)
2024-07-30 16:07:28 <@jnsamyak:matrix.org>     return failed_packages
2024-07-30 16:07:28 <@jnsamyak:matrix.org> 
2024-07-30 16:07:28 <@jnsamyak:matrix.org> ```
2024-07-30 16:07:31 <@nirik:matrix.scrye.com> My laptop seems unhappy with the xz I started. On via phone
2024-07-30 16:07:34 <@jnsamyak:matrix.org> something like this ^
2024-07-30 16:08:22 <@nirik:matrix.scrye.com> Yeah, that could work... Or just a retry loop in the script? But might get stuck I guess
2024-07-30 16:08:43 <@jnsamyak:matrix.org> but yeah something like this, we can discuss more when i have the pr ready for this or we can discuss this in the ticket, the only thing is i wanted it to convert from ops to dev
2024-07-30 16:08:59 <@jnsamyak:matrix.org> Hey Steve and Carl George 👋
2024-07-30 16:09:03 <@blackwell:fedora.im> !hi
2024-07-30 16:09:03 <@zodbot:fedora.im> Jason Blackwell (blackwell)
2024-07-30 16:09:21 <@jnsamyak:matrix.org> Anything that you folks want to bring up before we dive int releng tickets later on :D
2024-07-30 16:09:42 <@jnsamyak:matrix.org> Anything that you folks want to bring up before we dive in releng tickets later on :D
2024-07-30 16:09:59 <@carlwgeorge:matrix.org> i mainly wanted to be available for interactive questions on my epel10 activity
2024-07-30 16:10:08 <@carlwgeorge:matrix.org> thanks for the quick merges and actions on those recent issues/prs
2024-07-30 16:10:16 <@jnsamyak:matrix.org> \0/
2024-07-30 16:10:20 <@carlwgeorge:matrix.org> thanks for the quick merges and actions on those recent prs/issues
2024-07-30 16:10:51 <@jnsamyak:matrix.org> Carl George: okay that brings me to: https://pagure.io/releng/fedora-scm-requests/issue/63884
2024-07-30 16:10:57 <@carlwgeorge:matrix.org> getting epel10.0 in bodhi let me pass the fedpkg check to request the first epel10 branch, but it showed me i missed setting the branch slas in toddlers
2024-07-30 16:11:06 <@nirik:matrix.scrye.com> I think perhaps a check for error in the script and a sleep 5; retry might solve most of it, but sure... either way
2024-07-30 16:11:09 <@jnsamyak:matrix.org> yes
2024-07-30 16:11:28 <@carlwgeorge:matrix.org> almost got the pr to set the slas and hopefully get the branch creation fixed
2024-07-30 16:12:11 <@Zlopez:matrix.org> Carl George: Just deployed the changes for bodhi for EPEL 10
2024-07-30 16:12:25 <@Zlopez:matrix.org> nirik: Running the whole playbook helped
2024-07-30 16:12:31 <@carlwgeorge:matrix.org> awesome
2024-07-30 16:13:07 <@jnsamyak:matrix.org> nice nice
2024-07-30 16:13:44 <@jnsamyak:matrix.org> let's move on then to our next topic? 
2024-07-30 16:15:03 <@jnsamyak:matrix.org> !topic scheduled actions coming up in the next week
2024-07-30 16:15:09 <@jnsamyak:matrix.org> !info Here we list/discuss anything about items that are due to be done in the next week
2024-07-30 16:15:52 <@jnsamyak:matrix.org> I did the RAWHIDEFTBFS filing last day!
2024-07-30 16:16:14 <@nirik:matrix.scrye.com> Thanks!
2024-07-30 16:16:37 <@nirik:matrix.scrye.com> !info next week is flock
2024-07-30 16:16:46 <@jnsamyak:matrix.org> woohoo :F
2024-07-30 16:16:50 <@jnsamyak:matrix.org> woohoo :D
2024-07-30 16:17:05 <@patrikp:matrix.org> Who is going to Flock? (from people present here)
2024-07-30 16:17:15 <@carlwgeorge:matrix.org> i was super bummed to see that the infra/releng hackfest is scheduled at the same time as the epel10 hackfest
2024-07-30 16:17:32 <@smilner:fedora.im> 👋 I'll be there.
2024-07-30 16:17:39 <@carlwgeorge:matrix.org> i'll be there
2024-07-30 16:17:42 <@humaton:fedora.im> !hi
2024-07-30 16:17:43 <@zodbot:fedora.im> Tomáš Hrčka (humaton) - he / him / his
2024-07-30 16:18:02 <@nirik:matrix.scrye.com> Carl George: yeah, it got moved. ;(
2024-07-30 16:18:26 <@Zlopez:matrix.org> I'll be on Flock as well
2024-07-30 16:18:29 <@smilner:fedora.im> Carl George: nirik is it possible for us to ask for it to move or are things in cement at this point?
2024-07-30 16:18:44 <@nirik:matrix.scrye.com> I am not sure. ;) would need to ask Justin W. Flory (he/him)
2024-07-30 16:19:04 <@jnsamyak:matrix.org> click good pictures, for the people not attending :3 
2024-07-30 16:19:09 <@carlwgeorge:matrix.org> was just asking him
2024-07-30 16:20:03 <@jnsamyak:matrix.org> let's take this to open floor, later on?
2024-07-30 16:20:17 <@jnsamyak:matrix.org> !link https://fedorapeople.org/groups/schedule/f-41/f-41-key-tasks.html
2024-07-30 16:20:19 <@nirik:matrix.scrye.com> sure. Didn't mean to sidetrack
2024-07-30 16:21:23 <@jnsamyak:matrix.org> so branching is scheduled for a week later, but i plan to open the issue tracker and the prs one week prior for us to review 
2024-07-30 16:21:34 <@nirik:matrix.scrye.com> https://fedorapeople.org/groups/schedule/f-41/f-41-releng-tasks.html might be better / additional
2024-07-30 16:21:34 <@jnsamyak:matrix.org> and learn from the things that went bad last time
2024-07-30 16:21:46 <@jnsamyak:matrix.org> yes
2024-07-30 16:22:01 <@jnsamyak:matrix.org> One thing that folks questioned was bodhi outage
2024-07-30 16:22:46 <@jnsamyak:matrix.org> but we should ask them (in form a mail) prior that we are going to do our branching, and schedule that in status.fpo?
2024-07-30 16:23:21 <@jnsamyak:matrix.org> so people know the outage is gonna happen, and there builds etc will be affected meanwhile branching happens
2024-07-30 16:25:06 <@jnsamyak:matrix.org> nirik: I guess this is the ticket for that?
2024-07-30 16:25:08 <@jnsamyak:matrix.org> !releng 11943
2024-07-30 16:25:09 <@zodbot:fedora.im> ● **Assignee:** Not Assigned
2024-07-30 16:25:09 <@zodbot:fedora.im> ● **Last Updated:** 5 months ago
2024-07-30 16:25:09 <@zodbot:fedora.im> ● **Opened:** 5 months ago by decathorpe
2024-07-30 16:25:09 <@zodbot:fedora.im> ● **Closed: It's all good** 5 months ago by kevin
2024-07-30 16:25:09 <@zodbot:fedora.im> 
2024-07-30 16:25:09 <@zodbot:fedora.im> **releng #11943** (https://pagure.io/releng/issue/11943):**builds / updates in inconsistent state due to mass branching**
2024-07-30 16:25:33 <@nirik:matrix.scrye.com> yeah, I am not sure how to make it better off hand. I guess we can revisit and ponder on it and see if we can come up with any better ways
2024-07-30 16:26:12 <@nirik:matrix.scrye.com> I had on my list to file a ticket on bodhi (which I haven't yet done) to see if it could handle these state changes better.
2024-07-30 16:26:33 <@nirik:matrix.scrye.com> but implementing that would be it's own set of work
2024-07-30 16:27:17 <@jnsamyak:matrix.org> makes sense, so a mailing and branching outage (status.fpo) is we can do for users I guess
2024-07-30 16:27:22 <@jnsamyak:matrix.org> what say?
2024-07-30 16:28:27 <@nirik:matrix.scrye.com> well the problem isn't that people don't know about it...
2024-07-30 16:28:34 <@nirik:matrix.scrye.com> consider this case:
2024-07-30 16:28:51 <@nirik:matrix.scrye.com> package foo is built and submitted, but fails gating, so it's sitting there.
2024-07-30 16:29:25 <@nirik:matrix.scrye.com> then branching happens and it's still there, but bodhi still thinks it's going into rawhide
2024-07-30 16:29:53 <@nirik:matrix.scrye.com> or people build things while the branching is happening and it's a f40 branch with a rawhide build.
2024-07-30 16:29:57 <@jnsamyak:matrix.org> ahhh but it should ideally go to rawhide-1
2024-07-30 16:30:13 <@jnsamyak:matrix.org> makes sense
2024-07-30 16:30:24 <@nirik:matrix.scrye.com> there's all kinds of weird cases...
2024-07-30 16:30:51 <@nirik:matrix.scrye.com> and it's hard to just block everything branching/rawhide without blocking other releases or ourselves
2024-07-30 16:31:17 <@nirik:matrix.scrye.com> but I suppose we could do that... just block external access to koji while branching is happening.
2024-07-30 16:32:02 <@jnsamyak:matrix.org> yeah I'll put this in the issue tracker for branching
2024-07-30 16:32:15 <@jnsamyak:matrix.org> so I can keep this in mind
2024-07-30 16:32:40 <@nirik:matrix.scrye.com> I'll try and ponder on it some and see if I can think of a better way to do it.
2024-07-30 16:33:13 <@jnsamyak:matrix.org> when you say - external access what do you mean? Only koji-admin can submit builds? Or sysadmin-main folks?
2024-07-30 16:33:25 <@jnsamyak:matrix.org> if we restricts for others
2024-07-30 16:34:32 <@nirik:matrix.scrye.com> I mean block at the proxies access to koji.fedoraproject.org (it would show as being down to users)
2024-07-30 16:34:42 <@nirik:matrix.scrye.com> but internally we could still access it
2024-07-30 16:35:02 <@nirik:matrix.scrye.com> people might get mad about that tho. ;)
2024-07-30 16:35:12 <@jnsamyak:matrix.org> ha!
2024-07-30 16:35:46 <@jnsamyak:matrix.org> let's discuss this more on the ticket and mailing list dont want to eat time with finding a way now
2024-07-30 16:35:56 <@jnsamyak:matrix.org> that brings me to one more important step missing from our schedule, I need to remention to Aoife to add these to schedule
2024-07-30 16:36:00 <@jnsamyak:matrix.org> !link https://pagure.io/releng/issue/12001
2024-07-30 16:36:10 <@jnsamyak:matrix.org> We need to create f42 keys
2024-07-30 16:36:34 <@nirik:matrix.scrye.com> ah yep.
2024-07-30 16:39:30 <@nirik:matrix.scrye.com> I can do that and/or if you like walk you through doing it.
2024-07-30 16:39:57 <@jnsamyak:matrix.org> is there an sop? 
2024-07-30 16:40:07 <@nirik:matrix.scrye.com> There could well be
2024-07-30 16:40:13 <@jflory7:fedora.im> Let's discuss in #flock:fedoraproject.org? I think all the hackfests will be in the afternoon and workshops in the morning. But maybe we can find a way to make it work.
2024-07-30 16:40:24 <@jnsamyak:matrix.org> i can try to do that if it's not complicated with my new powers ;)
2024-07-30 16:42:22 <@nirik:matrix.scrye.com> hum, the sop if any will need adjustment too actually.
2024-07-30 16:42:37 <@nirik:matrix.scrye.com> we need to make new ima certs for the new key...
2024-07-30 16:42:51 <@humaton:fedora.im> I think the ima keys are in the SOP
2024-07-30 16:43:14 <@nirik:matrix.scrye.com> keys yes, certs no
2024-07-30 16:43:17 <@nirik:matrix.scrye.com> it's new.
2024-07-30 16:43:25 <@patrikp:matrix.org> Do you happen to have a link to the SOP?
2024-07-30 16:43:26 <@nirik:matrix.scrye.com> and we have to send them to kernel maintainers.
2024-07-30 16:43:29 <@humaton:fedora.im> oh
2024-07-30 16:43:31 <@nirik:matrix.scrye.com> I'm looking
2024-07-30 16:43:39 <@nirik:matrix.scrye.com> net is being slow for me today. ;(
2024-07-30 16:43:57 <@jnsamyak:matrix.org> okay can you do it this time and tell the steps so i can create an sop? we can do it as a form of ticket also
2024-07-30 16:44:20 <@nirik:matrix.scrye.com> I really thought there was a sop... but I will keep looking.
2024-07-30 16:44:30 <@nirik:matrix.scrye.com> might be it didn't move over from the old docs?
2024-07-30 16:44:41 <@humaton:fedora.im> hmm
2024-07-30 16:44:47 <@humaton:fedora.im> It was in the new docs for sure
2024-07-30 16:44:48 <@nirik:matrix.scrye.com> but yes, I can update the sop and you can then use it and let me know if it's failing anywhere. ;)
2024-07-30 16:44:54 <@humaton:fedora.im> part of the branching process I think
2024-07-30 16:45:20 <@jnsamyak:matrix.org> jednorozec: branching doesn't seem to have parts for creating new keys?
2024-07-30 16:45:29 <@humaton:fedora.im> but it did
2024-07-30 16:45:36 <@humaton:fedora.im> it might got lost?
2024-07-30 16:45:54 <@humaton:fedora.im> I have noticed some pages were not rebased but rewriten some time ago
2024-07-30 16:45:58 <@humaton:fedora.im> let me find the commit
2024-07-30 16:46:04 <@nirik:matrix.scrye.com> we can track it down after meeting?
2024-07-30 16:46:21 <@jnsamyak:matrix.org> yes
2024-07-30 16:46:57 <@jnsamyak:matrix.org> 
2024-07-30 16:46:57 <@jnsamyak:matrix.org> brings me to one last thing: jednorozec we can skip this part right now since PDC is ta ta byeb bye
2024-07-30 16:46:57 <@jnsamyak:matrix.org> https://docs.fedoraproject.org/en-US/infra/release_guide/sop_mass_branching/#_pdc
2024-07-30 16:47:21 <@nirik:matrix.scrye.com> yes, we will need to remove pdc from our docs.
2024-07-30 16:47:28 <@humaton:fedora.im> yup
2024-07-30 16:47:34 <@humaton:fedora.im> skip&remove
2024-07-30 16:47:38 <@jnsamyak:matrix.org> I'll remove it from the release process
2024-07-30 16:48:13 <@jnsamyak:matrix.org> moving on to our next item, thanks for the good discussion with the release items
2024-07-30 16:48:14 <@jnsamyak:matrix.org> !topic Tickets needing attention
2024-07-30 16:48:20 <@jnsamyak:matrix.org> !info this is the time where you can bring releng tickets to discuss if there are blockers etc
2024-07-30 16:48:26 <@jnsamyak:matrix.org> Any tickets?
2024-07-30 16:48:45 <@humaton:fedora.im> !link https://pagure.io/releng/issue/12228
2024-07-30 16:49:05 <@humaton:fedora.im> maybe some toddler or something not working properly because of PDC?
2024-07-30 16:50:06 <@humaton:fedora.im> maybe just some glitch? checking other requests today and this was the only complaint so far
2024-07-30 16:50:47 <@humaton:fedora.im> hm but the other requests didnt comit anything yet
2024-07-30 16:51:09 <@jnsamyak:matrix.org> yeah
2024-07-30 16:51:41 <@humaton:fedora.im> hm actually there is one
2024-07-30 16:51:52 <@humaton:fedora.im> requested 10hrs ago and build landed in f41
2024-07-30 16:52:03 <@humaton:fedora.im> https://koji.fedoraproject.org/koji/packageinfo?packageID=40592
2024-07-30 16:52:53 <@jnsamyak:matrix.org> https://pagure.io/releng/fedora-scm-requests/issue/63880
2024-07-30 16:52:54 <@jnsamyak:matrix.org> yes
2024-07-30 16:53:39 <@jnsamyak:matrix.org> maybe some timeout thing? We can check toddlers logs but it seemed OK to me on quick look
2024-07-30 16:54:36 <@jnsamyak:matrix.org> Are there any other tickets, I just want to reserve last 3-4 minutes for open floor!
2024-07-30 16:54:37 <@Zlopez:matrix.org> I'm watching the logs today for toddlers and didn't noticed any error
2024-07-30 16:54:50 <@nirik:matrix.scrye.com> we need a map of where things are done. :0
2024-07-30 16:54:58 <@nirik:matrix.scrye.com> this used to be on bodhi-backend in a listener.
2024-07-30 16:55:35 <@humaton:fedora.im> I can map things I know about
2024-07-30 16:55:45 <@humaton:fedora.im> but that will not be 100% of the processes
2024-07-30 16:56:06 <@jnsamyak:matrix.org> that would be really really good for onboarding as well to infra&releng
2024-07-30 16:56:10 <@Zlopez:matrix.org> Some of the automated things could be confusing sometimes
2024-07-30 16:56:42 <@jnsamyak:matrix.org> !topic Discussion / Any other Business / Blockers
2024-07-30 16:57:22 <@nirik:matrix.scrye.com> ok, it is still on bodhi-backend01.
2024-07-30 16:57:26 <@nirik:matrix.scrye.com> it's got some errros.
2024-07-30 16:57:28 <@jnsamyak:matrix.org> so should we skip next releng call because of flock?
2024-07-30 16:57:44 <@nirik:matrix.scrye.com> Jul 30 16:56:56 bodhi-backend01.iad2.fedoraproject.org fedora-messaging[2178760]: RuntimeError: ['/usr/local/bin/owner-sync-pagure', '--package', 'golang-github-apache-arrow12', '--verbose', 'f41', 'f41-container', 'f41-flatpak', 'f40', 'f40-container', 'f40-flatpak', 'f39', 'f39-container', 'f39-flatpak', 'epel10.0', 'epel9', 'epel9-next', 'epel8'] gave return code 1
2024-07-30 16:57:52 <@nirik:matrix.scrye.com> yes, I think so
2024-07-30 16:58:10 <@jnsamyak:matrix.org> +1/-1 (s)
2024-07-30 16:58:12 <@carlwgeorge:matrix.org> abompard was asking if he or someone in relent should run a certain command related to epel7 retirement
2024-07-30 16:58:16 <@carlwgeorge:matrix.org> !link https://pagure.io/releng/issue/12185
2024-07-30 16:58:28 <@carlwgeorge:matrix.org> abompard was asking if he or someone in releng should run a certain command related to epel7 retirement
2024-07-30 17:00:44 <@jnsamyak:matrix.org> i think sysadmin-main or sysdmin-releng can? OR just who can sudo on mirrormanger machine right nirik ?
2024-07-30 17:01:12 <@jnsamyak:matrix.org> Okay we are on time, I plan to cancel next releng call because of flock
2024-07-30 17:01:19 <@jnsamyak:matrix.org> !topic choose next chair
2024-07-30 17:01:23 <@jnsamyak:matrix.org> !info next chair Aug 13 - Samyak
2024-07-30 17:01:30 <@nirik:matrix.scrye.com> anyone who can run the playbook, yes.
2024-07-30 17:01:30 <@jnsamyak:matrix.org> !info Thank you all for coming.
2024-07-30 17:01:39 <@Zlopez:matrix.org> nirik: That is the new version I just deployed
2024-07-30 17:01:44 <@jnsamyak:matrix.org> meeting timeout error :X
2024-07-30 17:01:46 <@jnsamyak:matrix.org> lol
2024-07-30 17:01:52 <@jnsamyak:matrix.org> !endmeeting