<@meetbot:fedora.im>
08:24:27
HTML Minutes: https://meetbot.fedoraproject.org/meeting-3_matrix_fedoraproject-org/2024-10-10/cpe-infra-releng-daily-standup.2024-10-10-08.02.html
<@Zlopez:matrix.org>
16:00:14
!startmeeting Infrastructure (2024-10-10)
<@meetbot:fedora.im>
16:00:16
Meeting started at 2024-10-10 16:00:14 UTC
<@meetbot:fedora.im>
16:00:17
The Meeting name is 'Infrastructure (2024-10-10)'
<@Zlopez:matrix.org>
16:00:27
!topic ahoy
<@Zlopez:matrix.org>
16:00:27
!info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
<@Zlopez:matrix.org>
16:00:27
!info About our team: https://docs.fedoraproject.org/en-US/cpe/
<@Zlopez:matrix.org>
16:00:27
!info Agenda is at: https://board.net/p/fedora-infra
<@Zlopez:matrix.org>
16:00:27
!chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp
<@Zlopez:matrix.org>
16:00:27
!meetingname infrastructure
<@meetbot:fedora.im>
16:00:50
The Meeting Name is now infrastructure
<@nirik:matrix.scrye.com>
16:01:04
morning
<@james:fedora.im>
16:01:06
!hi
<@Zlopez:matrix.org>
16:01:06
Hello everyone
<@zodbot:fedora.im>
16:01:07
James Antill (james)
<@Zlopez:matrix.org>
16:01:17
And welcome to today infra meeting
<@Zlopez:matrix.org>
16:01:21
!hi
<@zodbot:fedora.im>
16:01:23
Michal Konecny (zlopez)
<@zardian:matrix.org>
16:01:37
!hi
<@zodbot:fedora.im>
16:01:38
Aman Singh (zardian)
<@Zlopez:matrix.org>
16:01:54
Are you ready for a bit of infrastructure in your lives?
<@zardian:matrix.org>
16:02:44
Hi everyone...!
<@nirik:matrix.scrye.com>
16:03:31
who wouldn't be?
<@Zlopez:matrix.org>
16:04:45
So let's start with newcomers :-)
<@Zlopez:matrix.org>
16:04:53
!info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
<@Zlopez:matrix.org>
16:04:53
!topic New folks introductions
<@Zlopez:matrix.org>
16:04:53
!info Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/
<@Zlopez:matrix.org>
16:05:05
Do we have anybody new here?
<@carlwgeorge:matrix.org>
16:05:20
!hi
<@zodbot:fedora.im>
16:05:21
Carl George (carlwgeorge) - he / him / his
<@carlwgeorge:matrix.org>
16:05:30
not exactly new, but not a regular attendee either
<@Zlopez:matrix.org>
16:05:49
Hi Carl :-)
<@Zlopez:matrix.org>
16:05:52
Nice to see you here
<@zardian:matrix.org>
16:06:14
I am new but did attended the last meeting also.
<@Zlopez:matrix.org>
16:06:35
I remember welcoming you
<@zardian:matrix.org>
16:06:44
yes
<@nirik:matrix.scrye.com>
16:07:14
I treat each day as a new adventure, does that count?
<@Zlopez:matrix.org>
16:07:45
So you are going on Adventure time?
<@Zlopez:matrix.org>
16:08:36
What about chairing this meeting?
<@Zlopez:matrix.org>
16:08:44
!info chair 2024-10-17 - lenkaseg
<@Zlopez:matrix.org>
16:08:44
!info magic eight ball says:
<@Zlopez:matrix.org>
16:08:44
!info chair 2024-10-24 - ???
<@Zlopez:matrix.org>
16:08:44
!info chair 2024-10-10 - zlopez
<@Zlopez:matrix.org>
16:08:44
!topic Next chair
<@Zlopez:matrix.org>
16:09:09
Any volunteer for 24th October?
<@nirik:matrix.scrye.com>
16:10:36
I can if no one else, or we could just leave it to figure out until next time?
<@Zlopez:matrix.org>
16:11:04
Let's leave it for next week
<@Zlopez:matrix.org>
16:11:13
!info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
<@Zlopez:matrix.org>
16:11:13
!topic announcements and information
<@Zlopez:matrix.org>
16:11:13
!info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
<@Zlopez:matrix.org>
16:11:22
Anything else to announce?
<@nirik:matrix.scrye.com>
16:12:07
!info f41 final freeze starts next tuesday. Get changes in before then!
<@Zlopez:matrix.org>
16:12:40
That's important one
<@Zlopez:matrix.org>
16:15:30
It doesn't seem that there is anything else to announce, let's continue with oncall
<@Zlopez:matrix.org>
16:15:36
!info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
<@Zlopez:matrix.org>
16:15:36
!info https://fedoraproject.org/wiki/Infrastructure/Oncall
<@Zlopez:matrix.org>
16:15:36
!topic Oncall
<@Zlopez:matrix.org>
16:15:42
!info lenkaseg is on call from 2024-10-03 to 2024-10-10
<@Zlopez:matrix.org>
16:15:42
!info zardian is on call from 2024-10-10 to 2024-10-17
<@zardian:matrix.org>
16:16:14
oh, its me this week....
<@Zlopez:matrix.org>
16:16:17
!info ??? is on call from 2024-10-18 to 2024-10-24
<@Zlopez:matrix.org>
16:16:31
@zardian:matrix.org Let me update the oncall bot
<@zardian:matrix.org>
16:17:21
did we had any tickets created by lenkaseg based on the oncall function this week?
<@Zlopez:matrix.org>
16:17:37
That is the next topic
<@zardian:matrix.org>
16:17:46
oh, ok.
<@Zlopez:matrix.org>
16:18:05
Right now we are looking for volunteer from 2024-10-18 to 2024-10-24
<@nirik:matrix.scrye.com>
16:18:32
again, I can take it if no one else. ;)
<@Zlopez:matrix.org>
16:19:19
It's yours
<@Zlopez:matrix.org>
16:19:29
!info nirik is on call from 2024-10-18 to 2024-10-24
<@Zlopez:matrix.org>
16:19:49
!info Summary of last week: (from current oncall)
<@Zlopez:matrix.org>
16:20:01
@lenkaseg:fedora.im Anything to report?
<@Zlopez:matrix.org>
16:20:42
She is probably not here
<@nirik:matrix.scrye.com>
16:21:18
I don't think there were any...
<@Zlopez:matrix.org>
16:23:05
I see a ping when there were the switch issues, but it was resolved by itself
<@Zlopez:matrix.org>
16:23:36
s/issues/reboots
<@nirik:matrix.scrye.com>
16:23:51
yeah, I don't see any off hand
<@zardian:matrix.org>
16:24:30
need explanation, where such pings and previous created issued can be checked. pagure?
<@Zlopez:matrix.org>
16:25:22
The pings are !oncall, it will also ping you directly once you are added to oncall duty
<@Zlopez:matrix.org>
16:25:27
It's a zodbot command
<@nirik:matrix.scrye.com>
16:25:36
yeah, it's matrix rooms, #admin:fedoraproject.org mostly
<@Zlopez:matrix.org>
16:25:53
The tickets should be created on https://pagure.io/fedora-infrastructure/issues
<@Zlopez:matrix.org>
16:26:42
The rooms we are usually watching are #admin:fedoraproject.org #noc:fedoraproject.org #releng:fedoraproject.org #apps:fedoraproject.org
<@zardian:matrix.org>
16:27:05
Thanks, understood.
<@Zlopez:matrix.org>
16:27:41
Let's continue with monitoring discussion
<@Zlopez:matrix.org>
16:27:43
!info Go over existing items and fix them
<@Zlopez:matrix.org>
16:27:43
!topic Monitoring discussion [nirik]
<@Zlopez:matrix.org>
16:27:43
!info https://nagios.fedoraproject.org/nagios
<@nirik:matrix.scrye.com>
16:28:03
probibly can't reach it due to switch reboots...
<@nirik:matrix.scrye.com>
16:28:19
yeah. ;(
<@nirik:matrix.scrye.com>
16:28:25
skip today?
<@Zlopez:matrix.org>
16:28:31
Hopefully that will be resolved soon
<@Zlopez:matrix.org>
16:28:36
But let's skip it for today
<@Zlopez:matrix.org>
16:29:01
Let's do a backlog refinement today
<@Zlopez:matrix.org>
16:29:10
!link https://pagure.io/fedora-infrastructure/issues?status=Open&order_key=last_updated&order=asc
<@Zlopez:matrix.org>
16:29:10
!topic Fedora Infra backlog refinement
<@Zlopez:matrix.org>
16:29:10
!info Refine oldest tickets on Fedora Infra tracker
<@nirik:matrix.scrye.com>
16:29:36
sure.
<@Zlopez:matrix.org>
16:29:39
!ticket 11683
<@Zlopez:matrix.org>
16:29:54
It seems that we lost the zodbot
<@nirik:matrix.scrye.com>
16:29:55
bot can't reach pagure... ;)
<@Zlopez:matrix.org>
16:30:08
!link https://pagure.io/fedora-infrastructure/issue/11683
<@Zlopez:matrix.org>
16:30:12
Let it do like that
<@nirik:matrix.scrye.com>
16:30:39
it will catch up when it reconnects
<@Zlopez:matrix.org>
16:30:53
It seems that this is waiting for mattdm
<@nirik:matrix.scrye.com>
16:31:11
Yeah, I am completely not sure what the status of this is.
<@nirik:matrix.scrye.com>
16:31:19
I can bring it up with him my next meeting...
<@Zlopez:matrix.org>
16:32:01
I will ping him in the ticket as well, I assume he just doesn't have time for this
<@Zlopez:matrix.org>
16:32:18
!link https://pagure.io/fedora-infrastructure/issue/11393
<@nirik:matrix.scrye.com>
16:32:47
yeah, unclear...
<@Zlopez:matrix.org>
16:32:49
This one is being worked on, but the last update on the ticket is 7 months old
<@zardian:matrix.org>
16:33:13
need explanation, where such pings and previous created issues can be checked. pagure?
<@zodbot:fedora.im>
16:33:47
● **Opened:** 10 months ago by mattdm
<@zodbot:fedora.im>
16:33:47
● **Last Updated:** 2 minutes ago
<@zodbot:fedora.im>
16:33:47
● **Assignee:** Not Assigned
<@zodbot:fedora.im>
16:33:47
<@zodbot:fedora.im>
16:33:47
**fedora-infrastructure #11683** (https://pagure.io/fedora-infrastructure/issue/11683):**discourse s3 backup buckets no longer active**
<@Zlopez:matrix.org>
16:34:58
I know we have a bot sending zabbix alerts to a channel on matrix, but not sure about much else
<@nirik:matrix.scrye.com>
16:35:21
which one are we on now?
<@Zlopez:matrix.org>
16:35:53
!ticket 11393
<@zodbot:fedora.im>
16:35:55
● **Opened:** a year ago by zlopez
<@zodbot:fedora.im>
16:35:55
<@zodbot:fedora.im>
16:35:55
**fedora-infrastructure #11393** (https://pagure.io/fedora-infrastructure/issue/11393):**Replace Nagios with Zabbix in Fedora Infrastructure**
<@zodbot:fedora.im>
16:35:55
● **Last Updated:** 7 months ago
<@zodbot:fedora.im>
16:35:55
● **Assignee:** dkirwan
<@nirik:matrix.scrye.com>
16:37:40
ah right.
<@nirik:matrix.scrye.com>
16:37:56
yeah, so we have messages going, but we need to adjust them to be less noisy.
<@james:fedora.im>
16:38:14
I did that for the production one
<@james:fedora.im>
16:38:38
Or at least ... all the disk read timeouts I could find. But I can't do it on stg atm.
<@james:fedora.im>
16:39:10
And things aren't updating properly on the production zabbix side since the big update last week.
<@james:fedora.im>
16:39:53
tl;dr ... it's moving fwd slowly, but it's not at the top of anyone's todo list.
<@nirik:matrix.scrye.com>
16:40:19
yeah.
<@Zlopez:matrix.org>
16:40:19
@james:fedora.im Could you add updates to the ticket?
<@james:fedora.im>
16:40:21
Also hopefully soon we'll get the big update with the digital clock ;)
<@nirik:matrix.scrye.com>
16:40:28
ha. yeah
<@james:fedora.im>
16:40:32
Sure
<@Zlopez:matrix.org>
16:40:49
That would be real lifechanger :-)
<@Zlopez:matrix.org>
16:41:16
Let's go to next one than
<@Zlopez:matrix.org>
16:41:23
!ticket 11958
<@zodbot:fedora.im>
16:41:24
**fedora-infrastructure #11958** (https://pagure.io/fedora-infrastructure/issue/11958):**Add fedora-l10n pagure group as an admin to the fedora-l10n-docs namespace projects**
<@zodbot:fedora.im>
16:41:24
● **Assignee:** Not Assigned
<@zodbot:fedora.im>
16:41:24
● **Last Updated:** 4 months ago
<@zodbot:fedora.im>
16:41:24
● **Opened:** 5 months ago by peartown
<@zodbot:fedora.im>
16:41:24
<@Zlopez:matrix.org>
16:42:12
I don't think this should be hanging there for 5 months :-/
<@nirik:matrix.scrye.com>
16:42:24
so...
<@nirik:matrix.scrye.com>
16:42:37
this one I was hoping ryanlerch had a script to do it.
<@nirik:matrix.scrye.com>
16:42:46
but failing that, someone else could write something to do it?
<@Zlopez:matrix.org>
16:43:03
I can ping him and ask him, so we can move this forward
<@nirik:matrix.scrye.com>
16:43:50
ok
<@Zlopez:matrix.org>
16:44:19
In any case at least we can continue on this
<@Zlopez:matrix.org>
16:44:41
!ticket 11144
<@zodbot:fedora.im>
16:44:43
● **Opened:** a year ago by zlopez
<@zodbot:fedora.im>
16:44:43
<@zodbot:fedora.im>
16:44:43
**fedora-infrastructure #11144** (https://pagure.io/fedora-infrastructure/issue/11144):**Create monitoring tool for rabbitmq certificates**
<@zodbot:fedora.im>
16:44:43
● **Last Updated:** 4 months ago
<@zodbot:fedora.im>
16:44:43
● **Assignee:** t0xic0der
<@Zlopez:matrix.org>
16:44:59
So this is blocked on https://pagure.io/fedora-infrastructure/issue/12066
<@Zlopez:matrix.org>
16:45:33
I will update the ticket with that, it should file issues to repository, but there is issue with creating it
<@nirik:matrix.scrye.com>
16:46:05
I wonder...
<@nirik:matrix.scrye.com>
16:46:38
I was wondering if it was related to https://pagure.io/fedora-infrastructure/issue/11869 but I guess thats a different namespace
<@nirik:matrix.scrye.com>
16:47:05
we could just make it a top level to avoid this issue?
<@Zlopez:matrix.org>
16:47:08
Yeah, that is different issue
<@Zlopez:matrix.org>
16:47:32
I would like to look in those two and try few things to get them deleted
<@Zlopez:matrix.org>
16:47:48
Where is the pagure.io db actually hosted?
<@Zlopez:matrix.org>
16:48:08
I noticed that the pagure PostgreSQL db on db01 is in fack dist-git
<@Zlopez:matrix.org>
16:48:21
I noticed that the pagure PostgreSQL db on db01 is in fact dist-git
<@nirik:matrix.scrye.com>
16:48:53
it's on pagure02
<@Zlopez:matrix.org>
16:49:17
Oh, it's directly on the machine
<@nirik:matrix.scrye.com>
16:49:25
the 11869 thing is a bug in agile board stuff. I tried to poke the db to fix it, but wasn't successfull
<@nirik:matrix.scrye.com>
16:49:27
yes
<@Zlopez:matrix.org>
16:49:52
OK, I will try to look into that tomorrow on next week
<@Zlopez:matrix.org>
16:50:00
OK, I will try to look into that tomorrow or next week
<@nirik:matrix.scrye.com>
16:50:35
that would be great!
<@Zlopez:matrix.org>
16:51:00
I finally have some spare cycles, so I'm trying to go through tickets on infra tracker
<@Zlopez:matrix.org>
16:51:14
!ticket 11815
<@zodbot:fedora.im>
16:51:15
<@zodbot:fedora.im>
16:51:15
● **Assignee:** zlopez
<@zodbot:fedora.im>
16:51:15
● **Last Updated:** 3 months ago
<@zodbot:fedora.im>
16:51:15
● **Opened:** 7 months ago by zlopez
<@zodbot:fedora.im>
16:51:15
**fedora-infrastructure #11815** (https://pagure.io/fedora-infrastructure/issue/11815):**rhel7 eol**
<@nirik:matrix.scrye.com>
16:51:30
Yeah, to fedmsg and github2fedmsg are the last left
<@Zlopez:matrix.org>
16:51:38
This will be solved when https://pagure.io/fedora-infrastructure/issue/11804
<@nirik:matrix.scrye.com>
16:51:42
but I wonder... do we have a timeline on retiring github2fedmsg?
<@Zlopez:matrix.org>
16:51:57
The webhook2fedmsg is ready, but it wasn't announced yet
<@Zlopez:matrix.org>
16:52:29
That should happen soon and then we should give some time for people to migrate from fedmsg and github2fedmsg
<@nirik:matrix.scrye.com>
16:52:29
on fedmsg we have to wait a month...
<@nirik:matrix.scrye.com>
16:52:37
but it would be nice to set a clear timeline
<@Zlopez:matrix.org>
16:52:51
I would do the same for both, once the replacement is in place
<@nirik:matrix.scrye.com>
16:52:58
yeah.
<@Zlopez:matrix.org>
16:53:21
Give people a month to migrate from github2fedmsg and fedmsg
<@nirik:matrix.scrye.com>
16:54:00
right, and we retire that and fedmsg at the same time whenever the deadline is
<@Zlopez:matrix.org>
16:54:04
It would be a reason to celebrate wehen we finally retire fedmsg :-)
<@Zlopez:matrix.org>
16:54:19
It would be a reason to celebrate when we finally retire fedmsg :-)
<@Zlopez:matrix.org>
16:56:08
Updated the ticket
<@Zlopez:matrix.org>
16:56:22
We have only few minutes left, so let's change to open floor
<@Zlopez:matrix.org>
16:56:40
!topic Open Floor
<@Zlopez:matrix.org>
16:56:53
Anything to discuss for the last few minutes?
<@Zlopez:matrix.org>
16:57:34
I just wanted to thank everybody for their work on Fedora Infra, we are getting close to 60 tickets on the tracker
<@Zlopez:matrix.org>
16:58:06
I think this is the lowest number I ever saw on the tracker 👍️
<@nirik:matrix.scrye.com>
16:58:21
we have been lower, but not in a while
<@nirik:matrix.scrye.com>
16:58:31
I guess I need to file some more tickets. ;)
<@Zlopez:matrix.org>
16:59:23
One question, does https://pagure.io/fedora-infrastructure/issue/12158 still needs reinstalling of ipa02 and ipa03?
<@nirik:matrix.scrye.com>
16:59:47
well, they are failing backups every day and the check for replication doesn't work on them.
<@nirik:matrix.scrye.com>
16:59:59
but not on ipa01.stg either...
<@nirik:matrix.scrye.com>
17:00:18
so the suggestion was to reinstall them, but I am not sure that will fix the problem
<@Zlopez:matrix.org>
17:00:29
I will try to reinstall them then and see
<@nirik:matrix.scrye.com>
17:01:16
I think the 'can't check replication status' thing has always been happening, but the failed backups are due to them not having something after the re-replica issue.
<@nirik:matrix.scrye.com>
17:01:42
make sure to make a good backup on 01 just in case. ;)
<@Zlopez:matrix.org>
17:01:54
!endmeeting