2025-10-20 15:00:26 <@amedvede:fedora.im> !startmeeting RELENG (2025-10-20)
2025-10-20 15:00:31 <@amedvede:fedora.im> !meetingname releng
2025-10-20 15:00:31 <@meetbot:fedora.im> Meeting started at 2025-10-20 15:00:26 UTC
2025-10-20 15:00:32 <@meetbot:fedora.im> The Meeting name is 'RELENG (2025-10-20)'
2025-10-20 15:00:32 <@meetbot:fedora.im> The Meeting Name is now releng
2025-10-20 15:00:35 <@amedvede:fedora.im> !chair nirik jnsamyak patrikp amedvede
2025-10-20 15:00:40 <@amedvede:fedora.im> !info Meeting is 60 minutes MAX. At the end of 60, it STOPS.
2025-10-20 15:00:45 <@amedvede:fedora.im> !info Agenda is at https://hackmd.io/vm6biLBcTYKtkQUH5kQkmw.
2025-10-20 15:01:01 <@amedvede:fedora.im> Hello everyone!
2025-10-20 15:01:57 <@patrikp:matrix.org> Hi.
2025-10-20 15:02:08 <@amedvede:fedora.im> )
2025-10-20 15:02:17 <@amedvede:fedora.im> let's move to init
2025-10-20 15:02:20 <@nirik:matrix.scrye.com> morning
2025-10-20 15:02:38 <@amedvede:fedora.im> !topic Init process.
2025-10-20 15:02:43 <@amedvede:fedora.im> Do we have anything for the init? Any blockers/tasks/issues/requests/features that need releng intervention?
2025-10-20 15:02:59 <@amedvede:fedora.im> I have few things for init
2025-10-20 15:03:51 <@nirik:matrix.scrye.com> I have one item for sometime... doesn't have to be init. ;)
2025-10-20 15:04:21 <@amedvede:fedora.im> 1) can someone take a look on my ansible PR: https://pagure.io/fedora-infra/ansible/pull-request/2909 I'm making ansible tasks for first time so it might not work)
2025-10-20 15:05:38 <@amedvede:fedora.im> 2. I'm planning to take PTO next week so I won't be able to chair
2025-10-20 15:05:44 <@patrikp:matrix.org> This raises an interesting question about where we stand with Nagios and Zabbix.
2025-10-20 15:05:58 <@patrikp:matrix.org> The ultimate plan is still to phase out Nagios eventually?
2025-10-20 15:06:12 <@gwmngilfen:fedora.im> yep 🙂
2025-10-20 15:07:25 <@amedvede:fedora.im> true, goodl question, if we will move from nagios to zabbix, its even better because i won't be afraid to break something in nagios xdd
2025-10-20 15:09:07 <@nirik:matrix.scrye.com> cloning the entire releng repo for that one script seems overkill, but having two copies is also bad...
2025-10-20 15:09:10 <@nirik:matrix.scrye.com> so not sure
2025-10-20 15:09:26 <@gwmngilfen:fedora.im> i'd be happy to work with someone on figuring out how to do that in Zabbix
2025-10-20 15:10:53 <@gwmngilfen:fedora.im> to be clear, we're still defining the zabbix config in ansible, so your problem doesn't go away 😛
2025-10-20 15:11:06 <@amedvede:fedora.im> true, cloning whole repo might be too much, there is other way, i can copy script to the machine where we run the playbooks and just copy this script in particular to the nagios during deplot
2025-10-20 15:12:53 <@amedvede:fedora.im> I'd be happy to try implement it in zabbix but not sure if its needed there
2025-10-20 15:13:16 <@amedvede:fedora.im> let me do some investigation and I will let you know
2025-10-20 15:13:51 <@nirik:matrix.scrye.com> So, this is emiting messages on nagios alerts...
2025-10-20 15:13:59 <@nirik:matrix.scrye.com> in zabbix we may have another way to do that?
2025-10-20 15:14:06 <@gwmngilfen:fedora.im> well, if monitoring is needed at all, it'll be needed in zabbix eventually
2025-10-20 15:14:45 <@gwmngilfen:fedora.im> most likely - I'm not familiar with what's being done here, so it's hard to comment, but zabbix is flexible
2025-10-20 15:16:49 <@amedvede:fedora.im> here script was used to send nagios alerts and notifications to the fedora messages bus 
2025-10-20 15:17:07 <@amedvede:fedora.im> if it is also usefull for zabbix we can do it
2025-10-20 15:17:38 <@amedvede:fedora.im> anyway, thats it from me for init, so if you have something else -> right time and place
2025-10-20 15:19:29 <@nirik:matrix.scrye.com> The one thing I had was around failing updates pushes... every few days we hit https://github.com/fedora-infra/bodhi/issues/5935 and all the updates pushes fail.
2025-10-20 15:19:55 <@nirik:matrix.scrye.com> To fix it you need to restart bodhi-celery on bodhi-backend01 and then resume the composes.
2025-10-20 15:20:13 <@nirik:matrix.scrye.com> I've been doing this, but if other people in eariler time zones could do it it would be nice. ;)
2025-10-20 15:20:23 <@nirik:matrix.scrye.com> and of course fixing the bug would be nice too.
2025-10-20 15:21:48 <@gwmngilfen:fedora.im> Anton Medvedev: patrikp I'll try to catch you in some non-meeting time and you can talk me through the requirements. I'm sure we can sort it, and it'll be a good test case
2025-10-20 15:21:52 <@nirik:matrix.scrye.com> it's currently broken right now if someone wants to walk thru the resume.
2025-10-20 15:22:17 <@patrikp:matrix.org> Forgive my ignorance but how do you find out when this happens and becomes an issue?
2025-10-20 15:23:28 <@patrikp:matrix.org> Found at least some docs that we have...
2025-10-20 15:23:28 <@patrikp:matrix.org> https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/bodhi/
2025-10-20 15:23:49 <@amedvede:fedora.im> !releng 13023 does this issue also connected to the problem you nirik  mentioned?
2025-10-20 15:23:50 <@zodbot:fedora.im> Issue querying Pagure: Issue not found
2025-10-20 15:24:28 <@nirik:matrix.scrye.com> email to bodhiadmin-members@fedoraproject.org, output from bodhi composes list...
2025-10-20 15:25:30 <@nirik:matrix.scrye.com> no, 13023 is likely related to https://pagure.io/fedora-infrastructure/issue/12814 which has taken up all my time for weeks and I can't figure it out.
2025-10-20 15:25:39 <@patrikp:matrix.org> If a push fails for some reason, you can easily resume it on bodhi-backend01 by running:
2025-10-20 15:25:39 <@patrikp:matrix.org> `sudo -u apache bodhi-push --resume`
2025-10-20 15:25:39 <@patrikp:matrix.org> There is a section that says titled "Resuming a failed push" that says:
2025-10-20 15:26:04 <@nirik:matrix.scrye.com> yep.
2025-10-20 15:26:39 <@nirik:matrix.scrye.com> but you _MUST_ 'sudo systemctl restart bodhi-celery' first... or it will all just fail for the same reason it failed before.
2025-10-20 15:27:40 <@amedvede:fedora.im> okay, I'll keep an eye on it during my timezone if it appears 
2025-10-20 15:28:18 <@nirik:matrix.scrye.com> would one of you all like to restart it now? or shall I just do so?
2025-10-20 15:28:54 <@amedvede:fedora.im> I'll do it if Patrik is not processing it right now
2025-10-20 15:30:02 <@nirik:matrix.scrye.com> ok, please let me know if you run into any issues or have any questions on it.
2025-10-20 15:31:56 <@amedvede:fedora.im> one more questions regarding it, so the logs about failures comes to this mailing list?
2025-10-20 15:34:25 <@patrikp:matrix.org> Oof, there's A LOT of Bodhi error e-mails, looking at my inbox here.
2025-10-20 15:34:29 <@patrikp:matrix.org> Do you check all of them Kevin?
2025-10-20 15:34:35 <@nirik:matrix.scrye.com> it's an alias, but yeah.
2025-10-20 15:34:49 <@nirik:matrix.scrye.com> no... if they happen a lot I report them upstream...
2025-10-20 15:36:36 <@patrikp:matrix.org> Well, looking at the commit history it seems to me Bodhi is pretty much unmaintained. Occasionally someone touches something.
2025-10-20 15:36:38 <@nirik:matrix.scrye.com> there might be a more targeted failure email for this case...
2025-10-20 15:37:00 <@nirik:matrix.scrye.com> there's a community person who works on it, or we do when we need something from it.
2025-10-20 15:39:23 <@amedvede:fedora.im> do we like to resume all sugested tags? `Resume <Compose: F43F testing>? [y/N]:`
2025-10-20 15:39:38 <@nirik:matrix.scrye.com> yep. say y to all of them
2025-10-20 15:39:58 <@amedvede:fedora.im> done
2025-10-20 15:40:27 <@amedvede:fedora.im> none issue, bodhi-celery status is running
2025-10-20 15:41:02 <@nirik:matrix.scrye.com> thanks
2025-10-20 15:42:04 <@amedvede:fedora.im> anyway, i didn't get how to see updates failes?
2025-10-20 15:42:18 <@patrikp:matrix.org> I still don't understand that either.
2025-10-20 15:42:30 <@patrikp:matrix.org> I understand that it's an e-mail.
2025-10-20 15:42:53 <@patrikp:matrix.org> But many e-mails are sent to bodhiadmin-members each day and I reckon you have a filter for that so they may skip the inbox altogether. What is the workflow here for checking?
2025-10-20 15:42:56 <@nirik:matrix.scrye.com> 'bodhi composes list' from your local machine will show the failed composes. You can see more detailed output with journalctl -u bodhi-celery on bodhi-backend01
2025-10-20 15:43:33 <@patrikp:matrix.org> So you discover it when you manually check?
2025-10-20 15:43:58 <@amedvede:fedora.im> and you following it? that tough
2025-10-20 15:44:35 <@amedvede:fedora.im> its cool that we can check it from local machine at least
2025-10-20 15:45:27 <@amedvede:fedora.im> and if we see something strange there we should mail the logs to bodhiadmin-members@fedoraproject.org right?
2025-10-20 15:46:14 <@patrikp:matrix.org> No, that is automated. Are you subscribed to the mailing list?
2025-10-20 15:46:33 <@patrikp:matrix.org> Question I have is how to sift through those e-mails to discover the important bits.
2025-10-20 15:47:00 <@amedvede:fedora.im> nope, i can't find it withing my mails, and I don't see such topic on hyperkitty
2025-10-20 15:47:01 <@amedvede:fedora.im> https://lists.fedoraproject.org/archives/
2025-10-20 15:47:21 <@amedvede:fedora.im> nope, i can't find it within my mails, and I don't see such topic on hyperkitty
2025-10-20 15:47:58 <@amedvede:fedora.im> anyway, we short in time so probably should rapidly go through sections that left
2025-10-20 15:48:54 <@patrikp:matrix.org> I think you need to be in `bodhiadmin` FAS group for that.
2025-10-20 15:49:07 <@patrikp:matrix.org> https://accounts.fedoraproject.org/group/bodhiadmin/
2025-10-20 15:51:04 <@amedvede:fedora.im> might be)
2025-10-20 15:51:23 <@amedvede:fedora.im> I'll create a ticket with request if so
2025-10-20 15:53:17 <@amedvede:fedora.im> lets go through other sections
2025-10-20 15:53:18 <@amedvede:fedora.im> !topic Scheduled actions coming up in the next week.
2025-10-20 15:54:06 <@amedvede:fedora.im> tomorrow go/nogo meet, after that realease after that postrelease cleanup, now wainting for gonogo 
2025-10-20 15:54:24 <@nirik:matrix.scrye.com> thats not a mailing list. ;) it's all the members of the bodhiadmins fas group.
2025-10-20 15:54:41 <@nirik:matrix.scrye.com> yeah, next go/nogo is thursday.
2025-10-20 15:54:53 <@amedvede:fedora.im> oh
2025-10-20 15:55:30 <@amedvede:fedora.im> okay, it's all that was planned for this week i believe
2025-10-20 15:56:14 <@amedvede:fedora.im> !topic Fedora releng backlog refinement
2025-10-20 15:56:45 <@amedvede:fedora.im> we don't have time for many tickets, we can skip or take a look on one
2025-10-20 15:57:08 <@patrikp:matrix.org> The meeting is already pretty much over I guess?
2025-10-20 15:57:57 <@amedvede:fedora.im> true, anyway we have work to do for this week so might be better to leave it until next releng weekly
2025-10-20 15:58:28 <@amedvede:fedora.im> !topic Tickets that need attention.
2025-10-20 15:58:47 <@amedvede:fedora.im> we went through it already
2025-10-20 15:58:57 <@amedvede:fedora.im> !topic Choose next chair.
2025-10-20 15:59:03 <@amedvede:fedora.im> thats the interesting one
2025-10-20 15:59:10 <@amedvede:fedora.im> i'll be off next week
2025-10-20 15:59:20 <@amedvede:fedora.im> same as Patrik based on calendar
2025-10-20 15:59:24 <@amedvede:fedora.im> so
2025-10-20 15:59:24 <@patrikp:matrix.org> I can take it.
2025-10-20 15:59:30 <@amedvede:fedora.im> okay
2025-10-20 15:59:39 <@amedvede:fedora.im> ty
2025-10-20 15:59:58 <@amedvede:fedora.im> !info Next chair October 27th: Patrik
2025-10-20 16:00:13 <@patrikp:matrix.org> Time.
2025-10-20 16:00:26 <@amedvede:fedora.im> !info Thank you all for coming!
2025-10-20 16:00:31 <@amedvede:fedora.im> !endmeeting