2025-07-14 15:04:17 <@patrikp:matrix.org> !startmeeting RELENG (2025-07-14)
2025-07-14 15:04:18 <@meetbot:fedora.im> Meeting started at 2025-07-14 15:04:17 UTC
2025-07-14 15:04:19 <@meetbot:fedora.im> The Meeting name is 'RELENG (2025-07-14)'
2025-07-14 15:04:26 <@patrikp:matrix.org> !info Meeting is 60 minutes MAX. At the end of 60, it STOPS.
2025-07-14 15:04:26 <@patrikp:matrix.org> !info Agenda is at https://hackmd.io/vm6biLBcTYKtkQUH5kQkmw.
2025-07-14 15:04:26 <@patrikp:matrix.org> !meetingname releng
2025-07-14 15:04:26 <@patrikp:matrix.org> !chair nirik jnsamyak patrikp amedvede
2025-07-14 15:04:27 <@meetbot:fedora.im> The Meeting Name is now releng
2025-07-14 15:04:37 <@nirik:matrix.scrye.com> morning. 👋
2025-07-14 15:04:45 <@jnsamyak:matrix.org> oh he just texted in my slack like 15 mins back 
2025-07-14 15:04:50 <@jnsamyak:matrix.org> he couldntmak eit
2025-07-14 15:05:04 <@patrikp:matrix.org> I'll put him in for next week. 👍️ Init time?
2025-07-14 15:05:10 <@patrikp:matrix.org> !topic Init process.
2025-07-14 15:05:10 <@patrikp:matrix.org> Do we have anything for the init? Any blockers/tasks/issues/requests/features that need releng intervention?
2025-07-14 15:06:01 <@nirik:matrix.scrye.com> I have one thing to bring up (possibly here)?
2025-07-14 15:06:06 <@jnsamyak:matrix.org> Okay i have one thing regarding sigul
2025-07-14 15:06:08 <@patrikp:matrix.org> Go for it.
2025-07-14 15:06:16 <@jnsamyak:matrix.org> but nirik you beat me to it
2025-07-14 15:06:18 <@jnsamyak:matrix.org> so you go!
2025-07-14 15:06:31 <@nirik:matrix.scrye.com> There's a problem with updates composes ( https://pagure.io/releng/issue/12813 )
2025-07-14 15:06:51 <@nirik:matrix.scrye.com> I think it might be a tcp timeout thing... but haven't had a chance to try fixes on that
2025-07-14 15:07:01 <@nirik:matrix.scrye.com> there's currently 2 failed composes if someone wants to resume them
2025-07-14 15:08:11 <@jnsamyak:matrix.org> I looked it this morning, I was away since wednesday last week, but I guessed it was a timeout issue too but is it failing with every compose?
2025-07-14 15:08:26 <@nirik:matrix.scrye.com> it's most of them, not everyone...
2025-07-14 15:08:31 <@nirik:matrix.scrye.com> well, some, not most
2025-07-14 15:08:31 <@jnsamyak:matrix.org> What did you do to resume it?
2025-07-14 15:08:46 <@jnsamyak:matrix.org> I can take a look at it
2025-07-14 15:08:52 <@jnsamyak:matrix.org> and fix it after this call
2025-07-14 15:09:18 <@nirik:matrix.scrye.com> the usual thing... on bodhi-backend01: sudo -u apache bodhi-push --username (yourusername) --resume
2025-07-14 15:09:22 <@patrikp:matrix.org> Just what I was about to ask. Do we have any words written down about how the composes are done and where to look if they're failing (at least in which direction)? If I recall correctly it runs as a script as a cron job or something like that?
2025-07-14 15:09:34 <@patrikp:matrix.org> Or where is it explained how this is set up?
2025-07-14 15:10:04 <@nirik:matrix.scrye.com> https://docs.fedoraproject.org/en-US/infra/releng_misc_guide/sop_pushing_updates/
2025-07-14 15:10:08 <@jnsamyak:matrix.org> The compose setup for updates has been explained several times, is that what you are referrinfg to?
2025-07-14 15:10:12 <@nirik:matrix.scrye.com> but as usual, it needs some updates. ;)
2025-07-14 15:10:12 <@jnsamyak:matrix.org> it's also documented
2025-07-14 15:10:32 <@jnsamyak:matrix.org> ahhhh theseeee!
2025-07-14 15:10:47 <@jnsamyak:matrix.org> I thought about something else, leave this to me never mind
2025-07-14 15:11:32 <@nirik:matrix.scrye.com> anyhow, thats it on this. Just wanted to raise it up and be aware... we will need to resume updates every day until we fix it.
2025-07-14 15:11:42 <@jnsamyak:matrix.org> nirik: from the fixing it for future do you think increasing the timeout will help? Or is it flaky because of recent changes!
2025-07-14 15:11:43 <@patrikp:matrix.org> But out of curiosity where is it explained?
2025-07-14 15:13:47 <@nirik:matrix.scrye.com> jnsamyak: my current theory is that it's a tcp timeout like we hit with pkgs in the new datacenter... basically bodhi opens up some database connections, and then goes to run pungi and other things, and it takes > 30m... it comes back and tries to use one of it's db connections but the firewall between vlans there has killed the connection because it was idle. ;( At least thats my current theory. I think we can tell sqlalchemy to only have connections for a shorter time and restablish them... like what was done for pagure / pkgs.
2025-07-14 15:14:43 <@jnsamyak:matrix.org> I explained it to you myself, I think I also remember writing a document a year ago about this that i wrote for my understanding, and if you want i can host it too! But Anyways, I can explain it later once again in my 1:1
2025-07-14 15:14:47 <@jnsamyak:matrix.org> For the sigul 
2025-07-14 15:15:23 <@patrikp:matrix.org> Unfortunately I don't have perfect memory so I have to consult documentation from time to time. Would it be possible to put in a doc that could be published perhaps?
2025-07-14 15:15:25 <@jnsamyak:matrix.org> list keys before signing to check if it was working but it just accepts the password and do nothibg about it 
2025-07-14 15:15:25 <@jnsamyak:matrix.org> I have some questions - I tried running basic commands for it 
2025-07-14 15:15:42 <@patrikp:matrix.org> But that's last comment on the topic, go ahead with the questions, sorry.
2025-07-14 15:16:42 <@jnsamyak:matrix.org> so nirik do I need to redo the setup again or the regular stuff should have just worked?
2025-07-14 15:19:48 <@nirik:matrix.scrye.com> grrrrrrrrrrrrrrrrrrrrrrrrrrrrrr. my main ups overloaded and took out all my servers and internet.
2025-07-14 15:20:24 <@nirik:matrix.scrye.com> I would have expected it to just work. It just hangs?
2025-07-14 15:20:33 <@jnsamyak:matrix.org> yeah
2025-07-14 15:20:36 <@jnsamyak:matrix.org> :(
2025-07-14 15:21:11 <@nirik:matrix.scrye.com> we can debug in #releng:fedoraproject.org when you have some time?
2025-07-14 15:21:20 <@jnsamyak:matrix.org> yes
2025-07-14 15:21:30 <@jnsamyak:matrix.org> please continue 
2025-07-14 15:21:38 <@patrikp:matrix.org> Anything else for the init?
2025-07-14 15:21:53 <@patrikp:matrix.org> !link https://fedorapeople.org/groups/schedule/f-42/f-42-releng-tasks.html
2025-07-14 15:21:53 <@patrikp:matrix.org> !topic Scheduled actions coming up in the next week.
2025-07-14 15:21:53 <@patrikp:matrix.org> !info Here we list/discuss anything about items that are due to be done in the next week.
2025-07-14 15:21:53 <@patrikp:matrix.org> !link https://fedorapeople.org/groups/schedule/f-43/f-43-releng-tasks.html
2025-07-14 15:22:19 <@patrikp:matrix.org> Mass rebuild soon.
2025-07-14 15:22:33 <@patrikp:matrix.org> !link https://docs.fedoraproject.org/en-US/infra/release_guide/mass_rebuild/
2025-07-14 15:22:35 <@jnsamyak:matrix.org> nothing for me
2025-07-14 15:22:45 <@jnsamyak:matrix.org> from*
2025-07-14 15:22:56 <@nirik:matrix.scrye.com> yeah, next week?
2025-07-14 15:23:33 <@patrikp:matrix.org> So, the plan is, for each step in the schedule we will have a corresponding SOP with the same title and some context (why it's important etc.), yes? By the end of the F43 release cycle.
2025-07-14 15:23:54 <@jnsamyak:matrix.org> I guess you would be working on it?
2025-07-14 15:24:01 <@patrikp:matrix.org> Aoife is the only person with access to the schedule but we can open tickets.
2025-07-14 15:24:11 <@patrikp:matrix.org> Yes but I will need a second and a third set of eyes and to ask questions.
2025-07-14 15:24:12 <@jnsamyak:matrix.org> Open tickets where?
2025-07-14 15:24:40 <@patrikp:matrix.org> https://pagure.io/fedora-pgm/schedule/issues
2025-07-14 15:24:40 <@patrikp:matrix.org> If we want to update anything about the schedule we can do so via opening a ticket here:
2025-07-14 15:24:48 <@jnsamyak:matrix.org> sure I keep explaining those process and what i do everytime on releng channel if you want you can take insipirations from there 
2025-07-14 15:24:49 <@patrikp:matrix.org> Just for info.
2025-07-14 15:25:31 <@patrikp:matrix.org> Nothing apart from the mass rebuild coming up. Tickets?
2025-07-14 15:25:41 <@jnsamyak:matrix.org> not from me
2025-07-14 15:26:07 <@nirik:matrix.scrye.com> oh, I think I found the sigul problem. It was stuck... ;( try your commands anytime now...
2025-07-14 15:26:22 <@jnsamyak:matrix.org> oh thank you!
2025-07-14 15:26:36 <@nirik:matrix.scrye.com> weird error on the vault. ;( Hope it was a one off
2025-07-14 15:27:33 <@patrikp:matrix.org> Do you have any words of wisdom for the upcoming actions Kevin? How to prepare mentally and/or physically? Or we can go to tickets.
2025-07-14 15:28:19 <@nirik:matrix.scrye.com> nothing off hand. we should perhaps look over the SOP's for things upcoming before them and see if we see anything that needs adjustment...
2025-07-14 15:28:32 <@jnsamyak:matrix.org> :)
2025-07-14 15:28:43 <@nirik:matrix.scrye.com> but otherwise, just go carefully along and get things done on time. ;)
2025-07-14 15:29:04 <@patrikp:matrix.org> !topic Tickets that need attention.
2025-07-14 15:29:04 <@patrikp:matrix.org> !link https://pagure.io/releng/issues
2025-07-14 15:29:04 <@patrikp:matrix.org> !info This is the time where you can bring up releng tickets to discuss if there are blockers, etc.
2025-07-14 15:29:28 <@patrikp:matrix.org> !releng 12825
2025-07-14 15:29:29 <@zodbot:fedora.im> ● **Last Updated:** Never
2025-07-14 15:29:29 <@zodbot:fedora.im> ● **Assignee:** Not Assigned
2025-07-14 15:29:29 <@zodbot:fedora.im> **releng #12825** (https://pagure.io/releng/issue/12825):**When processing stalled EPEL package requests, make the requester EPEL bugzilla override**
2025-07-14 15:29:29 <@zodbot:fedora.im> 
2025-07-14 15:29:29 <@zodbot:fedora.im> ● **Opened:** 4 hours ago by churchyard
2025-07-14 15:29:58 <@patrikp:matrix.org> So I processed those, but I just set the collaborators in dist-git (we don't have a doc on processing those tickets). Sounds like something has to be manually set in Bugzilla?
2025-07-14 15:30:17 <@nirik:matrix.scrye.com> it's in src.fedoraproject.org...
2025-07-14 15:30:45 <@jnsamyak:matrix.org> What are the steps do you follow while doing stalled epels?
2025-07-14 15:30:48 <@nirik:matrix.scrye.com> https://src.fedoraproject.org/rpms/koji on the side there's a 'bugzilla asignee' section...
2025-07-14 15:31:07 <@nirik:matrix.scrye.com> but I am not sure how to set it via api/command line...
2025-07-14 15:31:27 <@jnsamyak:matrix.org> EXACTLY because you are missing these and you have been doing these stalled for longer time, are you missing those on every ticket>
2025-07-14 15:31:34 <@jnsamyak:matrix.org> EXACTLY because you are missing these and you have been doing these stalled for longer time, are you missing those on every ticket?
2025-07-14 15:31:42 <@patrikp:matrix.org> So all I do is go to src.fpo settings and add a user as collaborator for specific branches. But sounds like this is simple to do now that we know about it, it can be set in the web UI.
2025-07-14 15:31:48 <@patrikp:matrix.org> I guess so.
2025-07-14 15:32:13 <@jnsamyak:matrix.org> :) Can you go back to all your stalled package tickets and fix it please!
2025-07-14 15:32:20 <@patrikp:matrix.org> Let me assign to myself and take a look...
2025-07-14 15:32:26 <@patrikp:matrix.org> Next ticket?
2025-07-14 15:33:45 <@jnsamyak:matrix.org> Also, it would be good if you can try to write an automation around it, start simple, and then implement it as a toddler! Zlopez can we do that?
2025-07-14 15:34:00 <@patrikp:matrix.org> Automation for what?
2025-07-14 15:34:02 <@jnsamyak:matrix.org> I can help this groom with Zlopez and then we can see if I can put it to work
2025-07-14 15:34:18 <@patrikp:matrix.org> Nope, it's a matter of a couple of clicks. We don't need more tech debt.
2025-07-14 15:34:35 <@jnsamyak:matrix.org> For stalled packages so people can just go and request it from fedora-scm-requests
2025-07-14 15:34:58 <@patrikp:matrix.org> I don't know... This is something that should be done manually by a human.
2025-07-14 15:35:02 <@jnsamyak:matrix.org> it is not tbh, this saves a lot of time and human errors - it will not be more than a week work
2025-07-14 15:35:11 <@jnsamyak:matrix.org> may i know why?
2025-07-14 15:35:42 <@jnsamyak:matrix.org> nirik: it still stucks :(
2025-07-14 15:35:52 <@jnsamyak:matrix.org> but maybe my net is flaky
2025-07-14 15:36:05 <@jnsamyak:matrix.org> I have a hard stop in 9 mins
2025-07-14 15:36:44 <@jnsamyak:matrix.org> !releng 12782
2025-07-14 15:36:47 <@zodbot:fedora.im> 
2025-07-14 15:36:47 <@zodbot:fedora.im> **releng #12782** (https://pagure.io/releng/issue/12782):**Change proposal: Drop i686 support**
2025-07-14 15:36:47 <@zodbot:fedora.im> ● **Opened:** 3 weeks ago by decathorpe
2025-07-14 15:36:47 <@zodbot:fedora.im> ● **Assignee:** Not Assigned
2025-07-14 15:36:47 <@zodbot:fedora.im> ● **Last Updated:** 2 weeks ago
2025-07-14 15:36:51 <@jnsamyak:matrix.org> This is something I want to discuss
2025-07-14 15:36:54 <@patrikp:matrix.org> You need to verify the Bugzilla ticket. It's not about clicking in the UI, it's about the stalled process overall. Clicking is just the final step.
2025-07-14 15:37:26 <@jnsamyak:matrix.org> We can always implement a logic there :)
2025-07-14 15:37:41 <@patrikp:matrix.org> Go ahead if you feel like it'd be worthwhile.
2025-07-14 15:38:06 <@jnsamyak:matrix.org> For that, I'll discuss it with jednorozec!
2025-07-14 15:38:14 <@jnsamyak:matrix.org> And see if our PO accepts :D
2025-07-14 15:38:42 <@jnsamyak:matrix.org> This is something I want to discuss becaure some of the composes should be affected by it, iirc
2025-07-14 15:38:49 <@jnsamyak:matrix.org> so it will involve releng 
2025-07-14 15:39:30 <@jnsamyak:matrix.org> But it is for F44 so I want everyone to keep an eye and get involved in those discussion from releng too
2025-07-14 15:40:17 <@jnsamyak:matrix.org> !releng 12736
2025-07-14 15:40:18 <@zodbot:fedora.im> **releng #12736** (https://pagure.io/releng/issue/12736):**Fedora-KDE-42-1.1-x86_64-CHECKSUM has wrong ISO filename **
2025-07-14 15:40:18 <@zodbot:fedora.im> ● **Assignee:** amedvede
2025-07-14 15:40:18 <@zodbot:fedora.im> ● **Last Updated:** a day ago
2025-07-14 15:40:18 <@zodbot:fedora.im> ● **Opened:** 2 months ago by ananas-comosus
2025-07-14 15:40:18 <@zodbot:fedora.im> 
2025-07-14 15:40:27 <@jnsamyak:matrix.org> Anton Medvedev: What's the status for this? 
2025-07-14 15:40:39 <@jnsamyak:matrix.org> But i guess he is not here, so it is of no news
2025-07-14 15:41:17 <@jnsamyak:matrix.org> But that's all from me
2025-07-14 15:41:51 <@patrikp:matrix.org> All from me too.
2025-07-14 15:41:57 <@patrikp:matrix.org> !topic Choose next chair.
2025-07-14 15:41:57 <@patrikp:matrix.org> !info Next chair July 21st: Anton
2025-07-14 15:42:04 <@patrikp:matrix.org> !topic Discussion / Any other business / Open floor.
2025-07-14 15:42:41 <@patrikp:matrix.org> So, the OpenH264 SOP is (finally) published here.
2025-07-14 15:42:41 <@patrikp:matrix.org> https://docs.fedoraproject.org/en-US/infra/releng_misc_guide/sop_openh264/
2025-07-14 15:45:18 <@nirik:matrix.scrye.com> the i686 proposal was pulled... we can close that ticket. ;)
2025-07-14 15:45:25 <@jnsamyak:matrix.org> Thanks for this 
2025-07-14 15:46:57 <@nirik:matrix.scrye.com> yeah, on that checksum thing. I tried to update the whitelist, but it didn't work...
2025-07-14 15:47:05 <@nirik:matrix.scrye.com> but I could be missing something there...
2025-07-14 15:48:42 <@jnsamyak:matrix.org> it was anton to figure it out did he ask for help?
2025-07-14 15:48:57 <@patrikp:matrix.org> This is the step where something went wrong?
2025-07-14 15:48:57 <@patrikp:matrix.org> https://docs.fedoraproject.org/en-US/infra/release_guide/torrentrelease/#_torrent_release
2025-07-14 15:49:01 <@nirik:matrix.scrye.com> jnsamyak: sigul is single threaded. it's trying to sign a large package right now, so yeah, things will hang until your request is at the top of the queue
2025-07-14 15:49:33 <@nirik:matrix.scrye.com> He was going to look, but... right now no one can download kde torrents, which is kinda bad. So, we should try and fix it as soon as we can.
2025-07-14 15:50:07 <@jnsamyak:matrix.org> Okay so we need to escalte it why it's not being handled by him if it's urgent if he can't accomodate it he should ask for help
2025-07-14 15:50:09 <@nirik:matrix.scrye.com> that sop strangely doesn't mention the whitelist. ;(
2025-07-14 15:50:12 <@jnsamyak:matrix.org> This is not good
2025-07-14 15:50:43 <@jnsamyak:matrix.org> but for the issue I'll take a look at it tomorrow morning it was not on my notice, sorry about it! 
2025-07-14 15:51:41 <@patrikp:matrix.org> Anything else anybody would like to bring up? A good joke perhaps to end it on a light note?
2025-07-14 15:51:51 <@patrikp:matrix.org> I am a grim person so I don't know any.
2025-07-14 15:52:29 <@nirik:matrix.scrye.com> I never seem to remember jokes...
2025-07-14 15:52:52 <@patrikp:matrix.org> And on that terrible disappointment...
2025-07-14 15:53:02 <@patrikp:matrix.org> !endmeeting
2025-07-14 15:53:02 <@patrikp:matrix.org> !info Thank you all for coming!