2025-09-15 15:06:03 <@jnsamyak:matrix.org> !startmeeting RELENG (2025-09-15)

2025-09-15 15:06:04 <@meetbot:fedora.im> Meeting started at 2025-09-15 15:06:03 UTC
2025-09-15 15:06:05 <@meetbot:fedora.im> The Meeting name is 'RELENG (2025-09-15)'
2025-09-15 15:06:09 <@jnsamyak:matrix.org> !meetingname releng
2025-09-15 15:06:10 <@meetbot:fedora.im> The Meeting Name is now releng
2025-09-15 15:06:12 <@jnsamyak:matrix.org> !chair nirik jnsamyak patrikp amedvede
2025-09-15 15:06:18 <@jnsamyak:matrix.org> !info Meeting is 60 minutes MAX. At the end of 60, it STOPS.
2025-09-15 15:06:26 <@jnsamyak:matrix.org> !info Agenda is at https://hackmd.io/vm6biLBcTYKtkQUH5kQkmw.
2025-09-15 15:06:26 <@jnsamyak:matrix.org> 
2025-09-15 15:06:31 <@jnsamyak:matrix.org> !topic Init process.
2025-09-15 15:06:35 <@jnsamyak:matrix.org> 
2025-09-15 15:06:35 <@jnsamyak:matrix.org> Do we have anything for the init? Any blockers/tasks/issues/requests/features that need releng intervention?
2025-09-15 15:06:49 <@jnsamyak:matrix.org> Okay we are all set, I forgot I had to chair today
2025-09-15 15:06:58 <@jnsamyak:matrix.org> Hello again!
2025-09-15 15:07:13 <@nirik:matrix.scrye.com> morning again
2025-09-15 15:07:18 <@patrikp:matrix.org> 👋
2025-09-15 15:07:29 <@jnsamyak:matrix.org> What do we have for init?
2025-09-15 15:07:44 <@amedvede:fedora.im> guten morgen
2025-09-15 15:08:35 <@amedvede:fedora.im> nothing from me
2025-09-15 15:09:20 <@jnsamyak:matrix.org> releng #12934 and #12935
2025-09-15 15:09:20 <@jnsamyak:matrix.org> 
2025-09-15 15:09:20 <@jnsamyak:matrix.org> I have one thing to bring up - reviews are open for the tickets:
2025-09-15 15:09:46 <@naraiank:fedora.im> !hi
2025-09-15 15:09:48 <@zodbot:fedora.im> Naraian K (naraiank) - he / him / his
2025-09-15 15:10:09 <@jnsamyak:matrix.org> both Anton Medvedev and patrikp are done with there investigations about where to move the scripts, and remove if needed. I'm doing my reviews gradualy but dealing with beta release first.
2025-09-15 15:10:25 <@jnsamyak:matrix.org> both Anton Medvedev and patrikp are done with their investigations about where to move the scripts, and remove if needed. I'm doing my reviews gradually, but dealing with the beta release first.
2025-09-15 15:10:43 <@jnsamyak:matrix.org> cf: https://pagure.io/releng/issue/12935 and https://pagure.io/releng/issue/12934
2025-09-15 15:11:32 <@jnsamyak:matrix.org> Once, these are closed, I'll start with moving and closing https://pagure.io/releng/issue/12925 in order to start working on https://pagure.io/releng/issue/12931
2025-09-15 15:11:46 <@nirik:matrix.scrye.com> I had no time to look friday. I can try again today...
2025-09-15 15:11:56 <@patrikp:matrix.org> I left some comments about the structure on your ticket. I was also wondering if it would be worthwhile having a `tests` subdirectory. At this point we only have the one unit test file and it doesn't work, perhaps we can improve that over time.
2025-09-15 15:11:58 <@jnsamyak:matrix.org> Yeah - same!
2025-09-15 15:12:04 <@nirik:matrix.scrye.com> but it's also fine if we just do it, we can always adjust more later if something ends up in a non ideal place.
2025-09-15 15:12:22 <@jnsamyak:matrix.org> let me see, unit tests which script?
2025-09-15 15:12:57 <@patrikp:matrix.org> Right now we only have `mass-rebuild-test.py`.
2025-09-15 15:13:14 <@jnsamyak:matrix.org> I really want to atleast do one review so it always two pair of eyes rather than one, I will finish it by wednesday if all goes clear tomorrow :D
2025-09-15 15:13:30 <@amedvede:fedora.im> I took a look on sheet that you sent jnsamyak , looks correct
2025-09-15 15:13:38 <@jnsamyak:matrix.org> Anton Medvedev: thanks man!
2025-09-15 15:14:36 <@amedvede:fedora.im> I think after you finish your review, it would be nice if all of us will jump to meet to just get through list and confirm opinion on each script
2025-09-15 15:14:54 <@patrikp:matrix.org> And in an ideal scenario we would have tests for all scripts that are critical for the release process.
2025-09-15 15:15:11 <@jnsamyak:matrix.org> 
2025-09-15 15:15:11 <@jnsamyak:matrix.org> │   ├│   ├── tests/
2025-09-15 15:15:11 <@jnsamyak:matrix.org> │   ├── mass-rebuilds/
2025-09-15 15:15:11 <@jnsamyak:matrix.org> release_process/
2025-09-15 15:15:11 <@jnsamyak:matrix.org> Right if required - 
2025-09-15 15:15:11 <@jnsamyak:matrix.org> we can move it like that
2025-09-15 15:15:11 <@jnsamyak:matrix.org> 
2025-09-15 15:15:21 <@nirik:matrix.scrye.com> yeah, although some things are hard to test.
2025-09-15 15:15:30 <@jnsamyak:matrix.org> correct! 
2025-09-15 15:15:46 <@jnsamyak:matrix.org> that is the goal :)
2025-09-15 15:16:11 <@jnsamyak:matrix.org> I mentioned it before once the review happens, we can always do a hour call to settle on anf finalise the moving details
2025-09-15 15:16:17 <@patrikp:matrix.org> Some are but some are not. We may at least go for the low hanging fruit. But I'm just brainstorming here.
2025-09-15 15:16:28 <@jnsamyak:matrix.org> I mentioned it before, once the review happens, we can always do an hour call to settle on and finalise the moving details
2025-09-15 15:16:42 <@nirik:matrix.scrye.com> sure
2025-09-15 15:17:04 <@jnsamyak:matrix.org> sure but that can be another independent post migration work to bring up
2025-09-15 15:17:14 <@patrikp:matrix.org> Yes, just wanted to mention it.
2025-09-15 15:17:18 <@jnsamyak:matrix.org> Alright anything else?
2025-09-15 15:18:02 <@patrikp:matrix.org> Is there any particular reason why `release_process` uses an underscore but nothing else does?
2025-09-15 15:18:33 <@amedvede:fedora.im> if we want to cover scripts by tests, the best point to start is from scripts connected to SOP's imho
2025-09-15 15:18:42 <@jnsamyak:matrix.org> it was a typo 
2025-09-15 15:18:45 <@jnsamyak:matrix.org> I'll fix it
2025-09-15 15:18:52 <@jnsamyak:matrix.org> adding a comment wait
2025-09-15 15:21:17 <@jnsamyak:matrix.org> Sure, added a comment.
2025-09-15 15:21:23 <@jnsamyak:matrix.org> Anything ekse?
2025-09-15 15:21:27 <@jnsamyak:matrix.org> Anything else?
2025-09-15 15:21:31 <@patrikp:matrix.org> Nothing from me.
2025-09-15 15:22:04 <@patrikp:matrix.org> Will we do the rest of the retrospective today? Seems like all issues are assigned.
2025-09-15 15:22:13 <@jnsamyak:matrix.org> yes
2025-09-15 15:22:31 <@jnsamyak:matrix.org> 
2025-09-15 15:22:31 <@jnsamyak:matrix.org> > Seems like all issues are assigned.
2025-09-15 15:22:31 <@jnsamyak:matrix.org> in what terms?
2025-09-15 15:25:06 <@patrikp:matrix.org> In that I don't see any new issue on the issue tracker that is not assigned.
2025-09-15 15:25:13 <@jnsamyak:matrix.org> if you mean from new issues, we have a backlog 102 issues pick which are open 
2025-09-15 15:25:23 <@jnsamyak:matrix.org> pick it up groom it
2025-09-15 15:25:32 <@jnsamyak:matrix.org> start working
2025-09-15 15:25:49 <@jnsamyak:matrix.org> oh you meant in term of issues being discussed?
2025-09-15 15:25:57 <@jnsamyak:matrix.org> today in meeting?
2025-09-15 15:26:09 <@patrikp:matrix.org> Yeah, I know that we have a backlog but I don't see any new ones, that's all I meant.
2025-09-15 15:26:17 <@jnsamyak:matrix.org> sure
2025-09-15 15:26:21 <@patrikp:matrix.org> Didn't have some deep insight.
2025-09-15 15:26:21 <@jnsamyak:matrix.org> !topic Scheduled actions coming up in the next week.
2025-09-15 15:26:27 <@jnsamyak:matrix.org> !info Here we list/discuss anything about items that are due to be done in the next week.
2025-09-15 15:26:33 <@jnsamyak:matrix.org> 
2025-09-15 15:26:33 <@jnsamyak:matrix.org> !link https://fedorapeople.org/groups/schedule/f-43/f-43-releng-tasks.html
2025-09-15 15:26:43 <@jnsamyak:matrix.org> F43 Beta Release is scheduled for tomorrow
2025-09-15 15:27:40 <@jnsamyak:matrix.org> Most of the things are done, stage release and checksum synced; Torrents are also 
2025-09-15 15:27:45 <@jnsamyak:matrix.org> done
2025-09-15 15:28:04 <@nirik:matrix.scrye.com> excellent
2025-09-15 15:28:43 <@amedvede:fedora.im> So there are left only run a playbooks and post beta cleanup?
2025-09-15 15:29:07 <@jnsamyak:matrix.org> F40 torrents were still there - so i cleaned up those
2025-09-15 15:29:45 <@nirik:matrix.scrye.com> so... there's something I see thats not there, but not sure we want to add it.
2025-09-15 15:29:56 <@jnsamyak:matrix.org> playbooks were done last friday and releng lift was lifted, I did this discussion on releng there 
2025-09-15 15:30:06 <@jnsamyak:matrix.org> playbooks were done last friday and releng lift was lifted, I did this discussion on releng channel there 
2025-09-15 15:31:01 <@jnsamyak:matrix.org> shoot, I see there was -t roll out changes from bodhi, torrents creation mention was missing.
2025-09-15 15:31:01 <@jnsamyak:matrix.org> > so... there's something I see thats not there, but not sure we want to add it.
2025-09-15 15:31:01 <@jnsamyak:matrix.org> 
2025-09-15 15:31:04 <@jnsamyak:matrix.org> anything else?
2025-09-15 15:31:47 <@jnsamyak:matrix.org> Can we look into adam's ticket and review
2025-09-15 15:31:47 <@jnsamyak:matrix.org>  https://pagure.io/fedora-infra/ansible/pull-request/2841 and https://pagure.io/infra-docs-fpo/pull-request/410
2025-09-15 15:32:20 <@nirik:matrix.scrye.com> The night before a release we do a 'bit flip' and make permissions readable on the various release dirs on the mirrors. We do this in advance of the release because if we just switched it right at release time many non tier1 mirrors may not have synced the content and all of them will hit at once with all the users. So we do it the night before so mirrors have a chance to sync up... but we don't want to really advertise it or tell users it's there until the actual release...
2025-09-15 15:32:53 <@jnsamyak:matrix.org> so 12 hours before the release?
2025-09-15 15:33:28 <@patrikp:matrix.org> What is it? Or what is not where? 
2025-09-15 15:33:34 <@jnsamyak:matrix.org> yeah but if it is not readable people can't access it right?
2025-09-15 15:33:34 <@nirik:matrix.scrye.com> yeah, something like that. In the past I have done it before going to bed... but someone else could do it their morning.
2025-09-15 15:34:09 <@nirik:matrix.scrye.com> it will be readable? not sure I follow
2025-09-15 15:34:19 <@jnsamyak:matrix.org> done as part of stage-release script
2025-09-15 15:34:19 <@jnsamyak:matrix.org> The dl.alt.fpo has staged rc  that are released 
2025-09-15 15:35:07 <@nirik:matrix.scrye.com> yes, with mode 750 on the directories so only tier1 mirrors can read them
2025-09-15 15:35:14 <@nirik:matrix.scrye.com> or see them
2025-09-15 15:35:40 <@jnsamyak:matrix.org> I mean, in terms of access, even if we don't make it 750, and it is there in SOP public viewer should not be able to see it right but internally it can pick up? 
2025-09-15 15:35:42 <@nirik:matrix.scrye.com> drwxr-x---. 15 263 263 4096 Sep 12 17:59 43_Beta-1.3/
2025-09-15 15:35:53 <@jnsamyak:matrix.org> and we can open it at the time of release
2025-09-15 15:36:03 <@nirik:matrix.scrye.com> but then we have the problem I described.
2025-09-15 15:36:04 <@jnsamyak:matrix.org> and we can open it at the time of release (I mean 12 hour before)
2025-09-15 15:36:27 <@jnsamyak:matrix.org> yeah :/
2025-09-15 15:36:38 <@nirik:matrix.scrye.com> a mass of NON tier1 mirrors will try and sync all the content that suddenly appeared ALONG with all the users who read the announcement.
2025-09-15 15:37:13 <@jnsamyak:matrix.org> I think we can just add a generic note to make it accessible at 12 hour before? Not exactly mentioning paths? 
2025-09-15 15:37:23 <@nirik:matrix.scrye.com> so to mitigate that we have opened it before to allow for mirrors (who have crons and automation) to get it before folks see an announcement
2025-09-15 15:37:52 <@nirik:matrix.scrye.com> sure...
2025-09-15 15:38:35 <@jnsamyak:matrix.org> yeah or just remember, I know I had to do it tomorrow morning, I remember me and you asks this every cycle at the time of your night or my morning whoever want's do it just do it and get it done with it
2025-09-15 15:38:36 <@nirik:matrix.scrye.com> I guess if people start pointing at it we can try and do something else.
2025-09-15 15:39:10 <@nirik:matrix.scrye.com> sure.
2025-09-15 15:39:13 <@patrikp:matrix.org> I understand what you guys are saying but I'm still not sure I get what you meant in the beginning Kevin. Something that is not there and not sure if we want to add it. Do you mean a step in the SOP? And since people can read the docs they would know?
2025-09-15 15:39:32 <@nirik:matrix.scrye.com> yes, exactly.
2025-09-15 15:39:56 <@nirik:matrix.scrye.com> we could add it and adjust if it becomes a problem I guess
2025-09-15 15:40:15 <@patrikp:matrix.org> They can read the release engineering Matrix channel too though. I think all should be in SOPs.
2025-09-15 15:41:17 <@jnsamyak:matrix.org> Let's get in so we can see what happens.
2025-09-15 15:41:22 <@jnsamyak:matrix.org> Let's get in sop we can see what happens.
2025-09-15 15:41:37 <@jnsamyak:matrix.org> anything else for release process?
2025-09-15 15:41:40 <@jnsamyak:matrix.org> Or we can move on?
2025-09-15 15:42:23 <@patrikp:matrix.org> Retro time!
2025-09-15 15:43:07 <@jnsamyak:matrix.org> before that let's just quickly choose chair so we dont loose it in the end
2025-09-15 15:43:22 <@jnsamyak:matrix.org> !topic Choose next chair.
2025-09-15 15:43:37 <@jnsamyak:matrix.org> !info Next chair September 22nd: ??
2025-09-15 15:43:40 <@patrikp:matrix.org> 🖐️
2025-09-15 15:43:45 <@jnsamyak:matrix.org> Any volunteers?
2025-09-15 15:43:47 <@jnsamyak:matrix.org> Okay
2025-09-15 15:43:53 <@jnsamyak:matrix.org> you are first :D
2025-09-15 15:44:06 <@jnsamyak:matrix.org> !info Next chair September 22nd: patrikp 
2025-09-15 15:44:20 <@jnsamyak:matrix.org> !topic Fedora 43 Retrospective part 2
2025-09-15 15:44:50 <@jnsamyak:matrix.org> I think we covered on Communication with Website Team and OpenH264 Repository Missing
2025-09-15 15:46:04 <@jnsamyak:matrix.org> For the openh264 as an action item I did update the documentation: https://docs.fedoraproject.org/en-US/infra/release_guide/sop_mass_branching/#_openh264_repository_update_after_branching But not sure if we can close it now, it has been unresponsive for couple of days
2025-09-15 15:46:19 <@jnsamyak:matrix.org> The ticket: https://pagure.io/releng/issue/12880
2025-09-15 15:48:51 <@jnsamyak:matrix.org> Any thoughts? Or should we jump into the second item?
2025-09-15 15:49:25 <@patrikp:matrix.org> Looks good to me and I think the ticket can be closed.
2025-09-15 15:49:48 <@jnsamyak:matrix.org> +1 okay
2025-09-15 15:49:57 <@jnsamyak:matrix.org> !info MirrorManager Updates
2025-09-15 15:50:07 <@jnsamyak:matrix.org> Needed to manually set ctime=0, update the fullfiletimelist, and trigger the usual mirror updates for the changes to reflect.
2025-09-15 15:51:09 <@jnsamyak:matrix.org> Because this time the mirrormanager did not pick up with the repository updates when we did those
2025-09-15 15:51:28 <@jnsamyak:matrix.org> 
2025-09-15 15:51:28 <@jnsamyak:matrix.org> My Recommendation: Document this as a standard post-branching step.
2025-09-15 15:51:56 <@jnsamyak:matrix.org> What are your opinions?
2025-09-15 15:52:13 <@patrikp:matrix.org> +1, for sure it should be in the docs.
2025-09-15 15:52:50 <@amedvede:fedora.im> +1, yep lest update teh doc
2025-09-15 15:53:17 <@amedvede:fedora.im> should we create ticket on it, or quick PR will be enough?
2025-09-15 15:53:31 <@nirik:matrix.scrye.com> yeah. +1
2025-09-15 15:53:34 <@jnsamyak:matrix.org> quick pr for this is okay there is mass branching ticket is on
2025-09-15 15:53:40 <@jnsamyak:matrix.org> Post-Branch Monitoring
2025-09-15 15:54:02 <@jnsamyak:matrix.org> - Monitor the branched compose status.
2025-09-15 15:54:02 <@jnsamyak:matrix.org> - Fire off the first compose as soon as the branch opens.
2025-09-15 15:54:02 <@jnsamyak:matrix.org> We need better-structured post-branch steps:
2025-09-15 15:54:14 <@jnsamyak:matrix.org> This will help identify any repository gaps or configuration mismatches earlier.
2025-09-15 15:54:21 <@jnsamyak:matrix.org> We need to add it to post branching work
2025-09-15 15:55:14 <@jnsamyak:matrix.org> So, one more thing to add is documentation. 
2025-09-15 15:56:18 <@jnsamyak:matrix.org> So, one more thing to add to documentation. 
2025-09-15 15:56:36 <@jnsamyak:matrix.org> this sums up the retro stuff, but if folks want to bring something else in terms of retro
2025-09-15 15:56:53 <@jnsamyak:matrix.org> please bring up
2025-09-15 15:59:17 <@patrikp:matrix.org> I was on PTO so nothing to add but I would like to say that I found the retro useful and am glad the mass branching itself went relatively smoothly. Good job.
2025-09-15 15:59:57 <@patrikp:matrix.org> And looks like we're at time, thanks for running the meeting.
2025-09-15 16:00:49 <@jnsamyak:matrix.org> My network just went-out agian 
2025-09-15 16:01:12 <@jnsamyak:matrix.org> Sigh it has been an irritating day with Internet 
2025-09-15 16:01:16 <@nirik:matrix.scrye.com> yeah, retros are good...
2025-09-15 16:01:32 <@jnsamyak:matrix.org> !info thank you everyone for coming 
2025-09-15 16:01:44 <@jnsamyak:matrix.org> Thanks patrikp nirik 
2025-09-15 16:01:53 <@jnsamyak:matrix.org> !endmeeting