<@amedvede:fedora.im>
15:00:26
!startmeeting RELENG (2025-10-20)
<@amedvede:fedora.im>
15:00:31
!meetingname releng
<@meetbot:fedora.im>
15:00:31
Meeting started at 2025-10-20 15:00:26 UTC
<@meetbot:fedora.im>
15:00:32
The Meeting name is 'RELENG (2025-10-20)'
<@meetbot:fedora.im>
15:00:32
The Meeting Name is now releng
<@amedvede:fedora.im>
15:00:35
!chair nirik jnsamyak patrikp amedvede
<@amedvede:fedora.im>
15:00:40
!info Meeting is 60 minutes MAX. At the end of 60, it STOPS.
<@amedvede:fedora.im>
15:00:45
!info Agenda is at https://hackmd.io/vm6biLBcTYKtkQUH5kQkmw.
<@amedvede:fedora.im>
15:01:01
Hello everyone!
<@patrikp:matrix.org>
15:01:57
Hi.
<@amedvede:fedora.im>
15:02:08
)
<@amedvede:fedora.im>
15:02:17
let's move to init
<@nirik:matrix.scrye.com>
15:02:20
morning
<@amedvede:fedora.im>
15:02:38
!topic Init process.
<@amedvede:fedora.im>
15:02:43
Do we have anything for the init? Any blockers/tasks/issues/requests/features that need releng intervention?
<@amedvede:fedora.im>
15:02:59
I have few things for init
<@nirik:matrix.scrye.com>
15:03:51
I have one item for sometime... doesn't have to be init. ;)
<@amedvede:fedora.im>
15:04:21
1) can someone take a look on my ansible PR: https://pagure.io/fedora-infra/ansible/pull-request/2909 I'm making ansible tasks for first time so it might not work)
<@amedvede:fedora.im>
15:05:38
2. I'm planning to take PTO next week so I won't be able to chair
<@patrikp:matrix.org>
15:05:44
This raises an interesting question about where we stand with Nagios and Zabbix.
<@patrikp:matrix.org>
15:05:58
The ultimate plan is still to phase out Nagios eventually?
<@gwmngilfen:fedora.im>
15:06:12
yep 🙂
<@amedvede:fedora.im>
15:07:25
true, goodl question, if we will move from nagios to zabbix, its even better because i won't be afraid to break something in nagios xdd
<@nirik:matrix.scrye.com>
15:09:07
cloning the entire releng repo for that one script seems overkill, but having two copies is also bad...
<@nirik:matrix.scrye.com>
15:09:10
so not sure
<@gwmngilfen:fedora.im>
15:09:26
i'd be happy to work with someone on figuring out how to do that in Zabbix
<@gwmngilfen:fedora.im>
15:10:53
to be clear, we're still defining the zabbix config in ansible, so your problem doesn't go away 😛
<@amedvede:fedora.im>
15:11:06
true, cloning whole repo might be too much, there is other way, i can copy script to the machine where we run the playbooks and just copy this script in particular to the nagios during deplot
<@amedvede:fedora.im>
15:12:53
I'd be happy to try implement it in zabbix but not sure if its needed there
<@amedvede:fedora.im>
15:13:16
let me do some investigation and I will let you know
<@nirik:matrix.scrye.com>
15:13:51
So, this is emiting messages on nagios alerts...
<@nirik:matrix.scrye.com>
15:13:59
in zabbix we may have another way to do that?
<@gwmngilfen:fedora.im>
15:14:06
well, if monitoring is needed at all, it'll be needed in zabbix eventually
<@gwmngilfen:fedora.im>
15:14:45
most likely - I'm not familiar with what's being done here, so it's hard to comment, but zabbix is flexible
<@amedvede:fedora.im>
15:16:49
here script was used to send nagios alerts and notifications to the fedora messages bus
<@amedvede:fedora.im>
15:17:07
if it is also usefull for zabbix we can do it
<@amedvede:fedora.im>
15:17:38
anyway, thats it from me for init, so if you have something else -> right time and place
<@nirik:matrix.scrye.com>
15:19:29
The one thing I had was around failing updates pushes... every few days we hit https://github.com/fedora-infra/bodhi/issues/5935 and all the updates pushes fail.
<@nirik:matrix.scrye.com>
15:19:55
To fix it you need to restart bodhi-celery on bodhi-backend01 and then resume the composes.
<@nirik:matrix.scrye.com>
15:20:13
I've been doing this, but if other people in eariler time zones could do it it would be nice. ;)
<@nirik:matrix.scrye.com>
15:20:23
and of course fixing the bug would be nice too.
<@gwmngilfen:fedora.im>
15:21:48
Anton Medvedev: patrikp I'll try to catch you in some non-meeting time and you can talk me through the requirements. I'm sure we can sort it, and it'll be a good test case
<@nirik:matrix.scrye.com>
15:21:52
it's currently broken right now if someone wants to walk thru the resume.
<@patrikp:matrix.org>
15:22:17
Forgive my ignorance but how do you find out when this happens and becomes an issue?
<@patrikp:matrix.org>
15:23:28
Found at least some docs that we have...
<@patrikp:matrix.org>
15:23:28
https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/bodhi/
<@amedvede:fedora.im>
15:23:49
!releng 13023 does this issue also connected to the problem you nirik mentioned?
<@zodbot:fedora.im>
15:23:50
Issue querying Pagure: Issue not found
<@nirik:matrix.scrye.com>
15:24:28
email to bodhiadmin-members@fedoraproject.org, output from bodhi composes list...
<@nirik:matrix.scrye.com>
15:25:30
no, 13023 is likely related to https://pagure.io/fedora-infrastructure/issue/12814 which has taken up all my time for weeks and I can't figure it out.
<@patrikp:matrix.org>
15:25:39
If a push fails for some reason, you can easily resume it on bodhi-backend01 by running:
<@patrikp:matrix.org>
15:25:39
`sudo -u apache bodhi-push --resume`
<@patrikp:matrix.org>
15:25:39
There is a section that says titled "Resuming a failed push" that says:
<@nirik:matrix.scrye.com>
15:26:04
yep.
<@nirik:matrix.scrye.com>
15:26:39
but you _MUST_ 'sudo systemctl restart bodhi-celery' first... or it will all just fail for the same reason it failed before.
<@amedvede:fedora.im>
15:27:40
okay, I'll keep an eye on it during my timezone if it appears
<@nirik:matrix.scrye.com>
15:28:18
would one of you all like to restart it now? or shall I just do so?
<@amedvede:fedora.im>
15:28:54
I'll do it if Patrik is not processing it right now
<@nirik:matrix.scrye.com>
15:30:02
ok, please let me know if you run into any issues or have any questions on it.
<@amedvede:fedora.im>
15:31:56
one more questions regarding it, so the logs about failures comes to this mailing list?
<@patrikp:matrix.org>
15:34:25
Oof, there's A LOT of Bodhi error e-mails, looking at my inbox here.
<@patrikp:matrix.org>
15:34:29
Do you check all of them Kevin?
<@nirik:matrix.scrye.com>
15:34:35
it's an alias, but yeah.
<@nirik:matrix.scrye.com>
15:34:49
no... if they happen a lot I report them upstream...
<@patrikp:matrix.org>
15:36:36
Well, looking at the commit history it seems to me Bodhi is pretty much unmaintained. Occasionally someone touches something.
<@nirik:matrix.scrye.com>
15:36:38
there might be a more targeted failure email for this case...
<@nirik:matrix.scrye.com>
15:37:00
there's a community person who works on it, or we do when we need something from it.
<@amedvede:fedora.im>
15:39:23
do we like to resume all sugested tags? `Resume <Compose: F43F testing>? [y/N]:`
<@nirik:matrix.scrye.com>
15:39:38
yep. say y to all of them
<@amedvede:fedora.im>
15:39:58
done
<@amedvede:fedora.im>
15:40:27
none issue, bodhi-celery status is running
<@nirik:matrix.scrye.com>
15:41:02
thanks
<@amedvede:fedora.im>
15:42:04
anyway, i didn't get how to see updates failes?
<@patrikp:matrix.org>
15:42:18
I still don't understand that either.
<@patrikp:matrix.org>
15:42:30
I understand that it's an e-mail.
<@patrikp:matrix.org>
15:42:53
But many e-mails are sent to bodhiadmin-members each day and I reckon you have a filter for that so they may skip the inbox altogether. What is the workflow here for checking?
<@nirik:matrix.scrye.com>
15:42:56
'bodhi composes list' from your local machine will show the failed composes. You can see more detailed output with journalctl -u bodhi-celery on bodhi-backend01
<@patrikp:matrix.org>
15:43:33
So you discover it when you manually check?
<@amedvede:fedora.im>
15:43:58
and you following it? that tough
<@amedvede:fedora.im>
15:44:35
its cool that we can check it from local machine at least
<@amedvede:fedora.im>
15:45:27
and if we see something strange there we should mail the logs to bodhiadmin-members@fedoraproject.org right?
<@patrikp:matrix.org>
15:46:14
No, that is automated. Are you subscribed to the mailing list?
<@patrikp:matrix.org>
15:46:33
Question I have is how to sift through those e-mails to discover the important bits.
<@amedvede:fedora.im>
15:47:00
nope, i can't find it withing my mails, and I don't see such topic on hyperkitty
<@amedvede:fedora.im>
15:47:01
https://lists.fedoraproject.org/archives/
<@amedvede:fedora.im>
15:47:21
nope, i can't find it within my mails, and I don't see such topic on hyperkitty
<@amedvede:fedora.im>
15:47:58
anyway, we short in time so probably should rapidly go through sections that left
<@patrikp:matrix.org>
15:48:54
I think you need to be in `bodhiadmin` FAS group for that.
<@patrikp:matrix.org>
15:49:07
https://accounts.fedoraproject.org/group/bodhiadmin/
<@amedvede:fedora.im>
15:51:04
might be)
<@amedvede:fedora.im>
15:51:23
I'll create a ticket with request if so
<@amedvede:fedora.im>
15:53:17
lets go through other sections
<@amedvede:fedora.im>
15:53:18
!topic Scheduled actions coming up in the next week.
<@amedvede:fedora.im>
15:54:06
tomorrow go/nogo meet, after that realease after that postrelease cleanup, now wainting for gonogo
<@nirik:matrix.scrye.com>
15:54:24
thats not a mailing list. ;) it's all the members of the bodhiadmins fas group.
<@nirik:matrix.scrye.com>
15:54:41
yeah, next go/nogo is thursday.
<@amedvede:fedora.im>
15:54:53
oh
<@amedvede:fedora.im>
15:55:30
okay, it's all that was planned for this week i believe
<@amedvede:fedora.im>
15:56:14
!topic Fedora releng backlog refinement
<@amedvede:fedora.im>
15:56:45
we don't have time for many tickets, we can skip or take a look on one
<@patrikp:matrix.org>
15:57:08
The meeting is already pretty much over I guess?
<@amedvede:fedora.im>
15:57:57
true, anyway we have work to do for this week so might be better to leave it until next releng weekly
<@amedvede:fedora.im>
15:58:28
!topic Tickets that need attention.
<@amedvede:fedora.im>
15:58:47
we went through it already
<@amedvede:fedora.im>
15:58:57
!topic Choose next chair.
<@amedvede:fedora.im>
15:59:03
thats the interesting one
<@amedvede:fedora.im>
15:59:10
i'll be off next week
<@amedvede:fedora.im>
15:59:20
same as Patrik based on calendar
<@amedvede:fedora.im>
15:59:24
so
<@patrikp:matrix.org>
15:59:24
I can take it.
<@amedvede:fedora.im>
15:59:30
okay
<@amedvede:fedora.im>
15:59:39
ty
<@amedvede:fedora.im>
15:59:58
!info Next chair October 27th: Patrik
<@patrikp:matrix.org>
16:00:13
Time.
<@amedvede:fedora.im>
16:00:26
!info Thank you all for coming!
<@amedvede:fedora.im>
16:00:31
!endmeeting