2024-10-10 08:24:27 <@meetbot:fedora.im> HTML Minutes: https://meetbot.fedoraproject.org/meeting-3_matrix_fedoraproject-org/2024-10-10/cpe-infra-releng-daily-standup.2024-10-10-08.02.html
2024-10-10 16:00:14 <@Zlopez:matrix.org> !startmeeting Infrastructure (2024-10-10)
2024-10-10 16:00:16 <@meetbot:fedora.im> Meeting started at 2024-10-10 16:00:14 UTC
2024-10-10 16:00:17 <@meetbot:fedora.im> The Meeting name is 'Infrastructure (2024-10-10)'
2024-10-10 16:00:27 <@Zlopez:matrix.org> !topic ahoy
2024-10-10 16:00:27 <@Zlopez:matrix.org> !info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
2024-10-10 16:00:27 <@Zlopez:matrix.org> !info About our team: https://docs.fedoraproject.org/en-US/cpe/
2024-10-10 16:00:27 <@Zlopez:matrix.org> !info Agenda is at: https://board.net/p/fedora-infra
2024-10-10 16:00:27 <@Zlopez:matrix.org> !chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp
2024-10-10 16:00:27 <@Zlopez:matrix.org> !meetingname infrastructure
2024-10-10 16:00:50 <@meetbot:fedora.im> The Meeting Name is now infrastructure
2024-10-10 16:01:04 <@nirik:matrix.scrye.com> morning
2024-10-10 16:01:06 <@james:fedora.im> !hi
2024-10-10 16:01:06 <@Zlopez:matrix.org> Hello everyone
2024-10-10 16:01:07 <@zodbot:fedora.im> James Antill (james)
2024-10-10 16:01:17 <@Zlopez:matrix.org> And welcome to today infra meeting
2024-10-10 16:01:21 <@Zlopez:matrix.org> !hi
2024-10-10 16:01:23 <@zodbot:fedora.im> Michal Konecny (zlopez)
2024-10-10 16:01:37 <@zardian:matrix.org> !hi
2024-10-10 16:01:38 <@zodbot:fedora.im> Aman Singh (zardian)
2024-10-10 16:01:54 <@Zlopez:matrix.org> Are you ready for a bit of infrastructure in your lives?
2024-10-10 16:02:44 <@zardian:matrix.org> Hi everyone...!
2024-10-10 16:03:31 <@nirik:matrix.scrye.com> who wouldn't be?
2024-10-10 16:04:45 <@Zlopez:matrix.org> So let's start with newcomers :-)
2024-10-10 16:04:53 <@Zlopez:matrix.org> !info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
2024-10-10 16:04:53 <@Zlopez:matrix.org> !topic New folks introductions
2024-10-10 16:04:53 <@Zlopez:matrix.org> !info Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/
2024-10-10 16:05:05 <@Zlopez:matrix.org> Do we have anybody new here?
2024-10-10 16:05:20 <@carlwgeorge:matrix.org> !hi
2024-10-10 16:05:21 <@zodbot:fedora.im> Carl George (carlwgeorge) - he / him / his
2024-10-10 16:05:30 <@carlwgeorge:matrix.org> not exactly new, but not a regular attendee either
2024-10-10 16:05:49 <@Zlopez:matrix.org> Hi Carl :-)
2024-10-10 16:05:52 <@Zlopez:matrix.org> Nice to see you here
2024-10-10 16:06:14 <@zardian:matrix.org> I am new but did attended the last meeting also.
2024-10-10 16:06:35 <@Zlopez:matrix.org> I remember welcoming you
2024-10-10 16:06:44 <@zardian:matrix.org> yes
2024-10-10 16:07:14 <@nirik:matrix.scrye.com> I treat each day as a new adventure, does that count?
2024-10-10 16:07:45 <@Zlopez:matrix.org> So you are going on Adventure time?
2024-10-10 16:08:36 <@Zlopez:matrix.org> What about chairing this meeting?
2024-10-10 16:08:44 <@Zlopez:matrix.org> !info chair 2024-10-17 - lenkaseg
2024-10-10 16:08:44 <@Zlopez:matrix.org> !info magic eight ball says:
2024-10-10 16:08:44 <@Zlopez:matrix.org> !info chair 2024-10-24 - ???
2024-10-10 16:08:44 <@Zlopez:matrix.org> !info chair 2024-10-10 - zlopez
2024-10-10 16:08:44 <@Zlopez:matrix.org> !topic Next chair
2024-10-10 16:09:09 <@Zlopez:matrix.org> Any volunteer for 24th October?
2024-10-10 16:10:36 <@nirik:matrix.scrye.com> I can if no one else, or we could just leave it to figure out until next time?
2024-10-10 16:11:04 <@Zlopez:matrix.org> Let's leave it for next week
2024-10-10 16:11:13 <@Zlopez:matrix.org> !info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
2024-10-10 16:11:13 <@Zlopez:matrix.org> !topic announcements and information
2024-10-10 16:11:13 <@Zlopez:matrix.org> !info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
2024-10-10 16:11:22 <@Zlopez:matrix.org> Anything else to announce?
2024-10-10 16:12:07 <@nirik:matrix.scrye.com> !info f41 final freeze starts next tuesday. Get changes in before then!
2024-10-10 16:12:40 <@Zlopez:matrix.org> That's important one
2024-10-10 16:15:30 <@Zlopez:matrix.org> It doesn't seem that there is anything else to announce, let's continue with oncall
2024-10-10 16:15:36 <@Zlopez:matrix.org> !info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
2024-10-10 16:15:36 <@Zlopez:matrix.org> !info https://fedoraproject.org/wiki/Infrastructure/Oncall
2024-10-10 16:15:36 <@Zlopez:matrix.org> !topic Oncall
2024-10-10 16:15:42 <@Zlopez:matrix.org> !info lenkaseg is on call from 2024-10-03 to 2024-10-10
2024-10-10 16:15:42 <@Zlopez:matrix.org> !info zardian is on call from 2024-10-10 to 2024-10-17
2024-10-10 16:16:14 <@zardian:matrix.org> oh, its me this week....
2024-10-10 16:16:17 <@Zlopez:matrix.org> !info ??? is on call from 2024-10-18 to 2024-10-24
2024-10-10 16:16:31 <@Zlopez:matrix.org> @zardian:matrix.org Let me update the oncall bot
2024-10-10 16:17:21 <@zardian:matrix.org> did we had any tickets created by lenkaseg based on the oncall function this week?
2024-10-10 16:17:37 <@Zlopez:matrix.org> That is the next topic
2024-10-10 16:17:46 <@zardian:matrix.org> oh, ok.
2024-10-10 16:18:05 <@Zlopez:matrix.org> Right now we are looking for volunteer from 2024-10-18 to 2024-10-24
2024-10-10 16:18:32 <@nirik:matrix.scrye.com> again, I can take it if no one else. ;)
2024-10-10 16:19:19 <@Zlopez:matrix.org> It's yours
2024-10-10 16:19:29 <@Zlopez:matrix.org> !info nirik is on call from 2024-10-18 to 2024-10-24
2024-10-10 16:19:49 <@Zlopez:matrix.org> !info Summary of last week: (from current oncall)
2024-10-10 16:20:01 <@Zlopez:matrix.org> @lenkaseg:fedora.im Anything to report?
2024-10-10 16:20:42 <@Zlopez:matrix.org> She is probably not here
2024-10-10 16:21:18 <@nirik:matrix.scrye.com> I don't think there were any...
2024-10-10 16:23:05 <@Zlopez:matrix.org> I see a ping when there were the switch issues, but it was resolved by itself
2024-10-10 16:23:36 <@Zlopez:matrix.org> s/issues/reboots
2024-10-10 16:23:51 <@nirik:matrix.scrye.com> yeah, I don't see any off hand
2024-10-10 16:24:30 <@zardian:matrix.org> need explanation, where such pings and previous created issued can be checked. pagure?
2024-10-10 16:25:22 <@Zlopez:matrix.org> The pings are !oncall, it will also ping you directly once you are added to oncall duty
2024-10-10 16:25:27 <@Zlopez:matrix.org> It's a zodbot command
2024-10-10 16:25:36 <@nirik:matrix.scrye.com> yeah, it's matrix rooms, #admin:fedoraproject.org mostly
2024-10-10 16:25:53 <@Zlopez:matrix.org> The tickets should be created on https://pagure.io/fedora-infrastructure/issues
2024-10-10 16:26:42 <@Zlopez:matrix.org> The rooms we are usually watching are #admin:fedoraproject.org #noc:fedoraproject.org #releng:fedoraproject.org #apps:fedoraproject.org
2024-10-10 16:27:05 <@zardian:matrix.org> Thanks, understood.
2024-10-10 16:27:41 <@Zlopez:matrix.org> Let's continue with monitoring discussion
2024-10-10 16:27:43 <@Zlopez:matrix.org> !info Go over existing items and fix them
2024-10-10 16:27:43 <@Zlopez:matrix.org> !topic Monitoring discussion [nirik]
2024-10-10 16:27:43 <@Zlopez:matrix.org> !info https://nagios.fedoraproject.org/nagios 
2024-10-10 16:28:03 <@nirik:matrix.scrye.com> probibly can't reach it due to switch reboots...
2024-10-10 16:28:19 <@nirik:matrix.scrye.com> yeah. ;(
2024-10-10 16:28:25 <@nirik:matrix.scrye.com> skip today?
2024-10-10 16:28:31 <@Zlopez:matrix.org> Hopefully that will be resolved soon
2024-10-10 16:28:36 <@Zlopez:matrix.org> But let's skip it for today
2024-10-10 16:29:01 <@Zlopez:matrix.org> Let's do a backlog refinement today
2024-10-10 16:29:10 <@Zlopez:matrix.org> !link https://pagure.io/fedora-infrastructure/issues?status=Open&order_key=last_updated&order=asc
2024-10-10 16:29:10 <@Zlopez:matrix.org> !topic Fedora Infra backlog refinement
2024-10-10 16:29:10 <@Zlopez:matrix.org> !info Refine oldest tickets on Fedora Infra tracker
2024-10-10 16:29:36 <@nirik:matrix.scrye.com> sure.
2024-10-10 16:29:39 <@Zlopez:matrix.org> !ticket 11683
2024-10-10 16:29:54 <@Zlopez:matrix.org> It seems that we lost the zodbot
2024-10-10 16:29:55 <@nirik:matrix.scrye.com> bot can't reach pagure... ;)
2024-10-10 16:30:08 <@Zlopez:matrix.org> !link https://pagure.io/fedora-infrastructure/issue/11683
2024-10-10 16:30:12 <@Zlopez:matrix.org> Let it do like that
2024-10-10 16:30:39 <@nirik:matrix.scrye.com> it will catch up when it reconnects
2024-10-10 16:30:53 <@Zlopez:matrix.org> It seems that this is waiting for mattdm
2024-10-10 16:31:11 <@nirik:matrix.scrye.com> Yeah, I am completely not sure what the status of this is.
2024-10-10 16:31:19 <@nirik:matrix.scrye.com> I can bring it up with him my next meeting...
2024-10-10 16:32:01 <@Zlopez:matrix.org> I will ping him in the ticket as well, I assume he just doesn't have time for this
2024-10-10 16:32:18 <@Zlopez:matrix.org> !link https://pagure.io/fedora-infrastructure/issue/11393
2024-10-10 16:32:47 <@nirik:matrix.scrye.com> yeah, unclear...
2024-10-10 16:32:49 <@Zlopez:matrix.org> This one is being worked on, but the last update on the ticket is 7 months old
2024-10-10 16:33:13 <@zardian:matrix.org> need explanation, where such pings and previous created issues can be checked. pagure?
2024-10-10 16:33:47 <@zodbot:fedora.im> ● **Opened:** 10 months ago by mattdm
2024-10-10 16:33:47 <@zodbot:fedora.im> ● **Last Updated:** 2 minutes ago
2024-10-10 16:33:47 <@zodbot:fedora.im> ● **Assignee:** Not Assigned
2024-10-10 16:33:47 <@zodbot:fedora.im> 
2024-10-10 16:33:47 <@zodbot:fedora.im> **fedora-infrastructure #11683** (https://pagure.io/fedora-infrastructure/issue/11683):**discourse s3 backup buckets no longer active**
2024-10-10 16:34:58 <@Zlopez:matrix.org> I know we have a bot sending zabbix alerts to a channel on matrix, but not sure about much else
2024-10-10 16:35:21 <@nirik:matrix.scrye.com> which one are we on now?
2024-10-10 16:35:53 <@Zlopez:matrix.org> !ticket 11393
2024-10-10 16:35:55 <@zodbot:fedora.im> ● **Opened:** a year ago by zlopez
2024-10-10 16:35:55 <@zodbot:fedora.im> 
2024-10-10 16:35:55 <@zodbot:fedora.im> **fedora-infrastructure #11393** (https://pagure.io/fedora-infrastructure/issue/11393):**Replace Nagios with Zabbix in Fedora Infrastructure**
2024-10-10 16:35:55 <@zodbot:fedora.im> ● **Last Updated:** 7 months ago
2024-10-10 16:35:55 <@zodbot:fedora.im> ● **Assignee:** dkirwan
2024-10-10 16:37:40 <@nirik:matrix.scrye.com> ah right.
2024-10-10 16:37:56 <@nirik:matrix.scrye.com> yeah, so we have messages going, but we need to adjust them to be less noisy.
2024-10-10 16:38:14 <@james:fedora.im> I did that for the production one
2024-10-10 16:38:38 <@james:fedora.im> Or at least ... all the disk read timeouts I could find. But I can't do it on stg atm.
2024-10-10 16:39:10 <@james:fedora.im> And things aren't updating properly on the production zabbix side since the big update last week.
2024-10-10 16:39:53 <@james:fedora.im> tl;dr ... it's moving fwd slowly, but it's not at the top of anyone's todo list.
2024-10-10 16:40:19 <@nirik:matrix.scrye.com> yeah.
2024-10-10 16:40:19 <@Zlopez:matrix.org> @james:fedora.im Could you add updates to the ticket?
2024-10-10 16:40:21 <@james:fedora.im> Also hopefully soon we'll get the big update with the digital clock ;)
2024-10-10 16:40:28 <@nirik:matrix.scrye.com> ha. yeah
2024-10-10 16:40:32 <@james:fedora.im> Sure
2024-10-10 16:40:49 <@Zlopez:matrix.org> That would be real lifechanger :-)
2024-10-10 16:41:16 <@Zlopez:matrix.org> Let's go to next one than
2024-10-10 16:41:23 <@Zlopez:matrix.org> !ticket 11958
2024-10-10 16:41:24 <@zodbot:fedora.im> **fedora-infrastructure #11958** (https://pagure.io/fedora-infrastructure/issue/11958):**Add fedora-l10n pagure group as an admin to the fedora-l10n-docs namespace projects**
2024-10-10 16:41:24 <@zodbot:fedora.im> ● **Assignee:** Not Assigned
2024-10-10 16:41:24 <@zodbot:fedora.im> ● **Last Updated:** 4 months ago
2024-10-10 16:41:24 <@zodbot:fedora.im> ● **Opened:** 5 months ago by peartown
2024-10-10 16:41:24 <@zodbot:fedora.im> 
2024-10-10 16:42:12 <@Zlopez:matrix.org> I don't think this should be hanging there for 5 months :-/
2024-10-10 16:42:24 <@nirik:matrix.scrye.com> so...
2024-10-10 16:42:37 <@nirik:matrix.scrye.com> this one I was hoping ryanlerch had a script to do it.
2024-10-10 16:42:46 <@nirik:matrix.scrye.com> but failing that, someone else could write something to do it?
2024-10-10 16:43:03 <@Zlopez:matrix.org> I can ping him and ask him, so we can move this forward
2024-10-10 16:43:50 <@nirik:matrix.scrye.com> ok
2024-10-10 16:44:19 <@Zlopez:matrix.org> In any case at least we can continue on this
2024-10-10 16:44:41 <@Zlopez:matrix.org> !ticket 11144
2024-10-10 16:44:43 <@zodbot:fedora.im> ● **Opened:** a year ago by zlopez
2024-10-10 16:44:43 <@zodbot:fedora.im> 
2024-10-10 16:44:43 <@zodbot:fedora.im> **fedora-infrastructure #11144** (https://pagure.io/fedora-infrastructure/issue/11144):**Create monitoring tool for rabbitmq certificates**
2024-10-10 16:44:43 <@zodbot:fedora.im> ● **Last Updated:** 4 months ago
2024-10-10 16:44:43 <@zodbot:fedora.im> ● **Assignee:** t0xic0der
2024-10-10 16:44:59 <@Zlopez:matrix.org> So this is blocked on https://pagure.io/fedora-infrastructure/issue/12066
2024-10-10 16:45:33 <@Zlopez:matrix.org> I will update the ticket with that, it should file issues to repository, but there is issue with creating it
2024-10-10 16:46:05 <@nirik:matrix.scrye.com> I wonder...
2024-10-10 16:46:38 <@nirik:matrix.scrye.com> I was wondering if it was related to https://pagure.io/fedora-infrastructure/issue/11869 but I guess thats a different namespace
2024-10-10 16:47:05 <@nirik:matrix.scrye.com> we could just make it a top level to avoid this issue?
2024-10-10 16:47:08 <@Zlopez:matrix.org> Yeah, that is different issue
2024-10-10 16:47:32 <@Zlopez:matrix.org> I would like to look in those two and try few things to get them deleted
2024-10-10 16:47:48 <@Zlopez:matrix.org> Where is the pagure.io db actually hosted?
2024-10-10 16:48:08 <@Zlopez:matrix.org> I noticed that the pagure PostgreSQL db on db01 is in fack dist-git
2024-10-10 16:48:21 <@Zlopez:matrix.org> I noticed that the pagure PostgreSQL db on db01 is in fact dist-git
2024-10-10 16:48:53 <@nirik:matrix.scrye.com> it's on pagure02
2024-10-10 16:49:17 <@Zlopez:matrix.org> Oh, it's directly on the machine
2024-10-10 16:49:25 <@nirik:matrix.scrye.com> the 11869 thing is a bug in agile board stuff. I tried to poke the db to fix it, but wasn't successfull
2024-10-10 16:49:27 <@nirik:matrix.scrye.com> yes
2024-10-10 16:49:52 <@Zlopez:matrix.org> OK, I will try to look into that tomorrow on next week
2024-10-10 16:50:00 <@Zlopez:matrix.org> OK, I will try to look into that tomorrow or next week
2024-10-10 16:50:35 <@nirik:matrix.scrye.com> that would be great!
2024-10-10 16:51:00 <@Zlopez:matrix.org> I finally have some spare cycles, so I'm trying to go through tickets on infra tracker
2024-10-10 16:51:14 <@Zlopez:matrix.org> !ticket 11815
2024-10-10 16:51:15 <@zodbot:fedora.im> 
2024-10-10 16:51:15 <@zodbot:fedora.im> ● **Assignee:** zlopez
2024-10-10 16:51:15 <@zodbot:fedora.im> ● **Last Updated:** 3 months ago
2024-10-10 16:51:15 <@zodbot:fedora.im> ● **Opened:** 7 months ago by zlopez
2024-10-10 16:51:15 <@zodbot:fedora.im> **fedora-infrastructure #11815** (https://pagure.io/fedora-infrastructure/issue/11815):**rhel7 eol**
2024-10-10 16:51:30 <@nirik:matrix.scrye.com> Yeah, to fedmsg and github2fedmsg are the last left
2024-10-10 16:51:38 <@Zlopez:matrix.org> This will be solved when https://pagure.io/fedora-infrastructure/issue/11804
2024-10-10 16:51:42 <@nirik:matrix.scrye.com> but I wonder... do we have a timeline on retiring github2fedmsg?
2024-10-10 16:51:57 <@Zlopez:matrix.org> The webhook2fedmsg is ready, but it wasn't announced yet
2024-10-10 16:52:29 <@Zlopez:matrix.org> That should happen soon and then we should give some time for people to migrate from fedmsg and github2fedmsg
2024-10-10 16:52:29 <@nirik:matrix.scrye.com> on fedmsg we have to wait a month...
2024-10-10 16:52:37 <@nirik:matrix.scrye.com> but it would be nice to set a clear timeline
2024-10-10 16:52:51 <@Zlopez:matrix.org> I would do the same for both, once the replacement is in place
2024-10-10 16:52:58 <@nirik:matrix.scrye.com> yeah.
2024-10-10 16:53:21 <@Zlopez:matrix.org> Give people a month to migrate from github2fedmsg and fedmsg
2024-10-10 16:54:00 <@nirik:matrix.scrye.com> right, and we retire that and fedmsg at the same time whenever the deadline is
2024-10-10 16:54:04 <@Zlopez:matrix.org> It would be a reason to celebrate wehen we finally retire fedmsg :-)
2024-10-10 16:54:19 <@Zlopez:matrix.org> It would be a reason to celebrate when we finally retire fedmsg :-)
2024-10-10 16:56:08 <@Zlopez:matrix.org> Updated the ticket
2024-10-10 16:56:22 <@Zlopez:matrix.org> We have only few minutes left, so let's change to open floor
2024-10-10 16:56:40 <@Zlopez:matrix.org> !topic Open Floor
2024-10-10 16:56:53 <@Zlopez:matrix.org> Anything to discuss for the last few minutes?
2024-10-10 16:57:34 <@Zlopez:matrix.org> I just wanted to thank everybody for their work on Fedora Infra, we are getting close to 60 tickets on the tracker
2024-10-10 16:58:06 <@Zlopez:matrix.org> I think this is the lowest number I ever saw on the tracker 👍️
2024-10-10 16:58:21 <@nirik:matrix.scrye.com> we have been lower, but not in a while
2024-10-10 16:58:31 <@nirik:matrix.scrye.com> I guess I need to file some more tickets. ;)
2024-10-10 16:59:23 <@Zlopez:matrix.org> One question, does https://pagure.io/fedora-infrastructure/issue/12158 still needs reinstalling of ipa02 and ipa03?
2024-10-10 16:59:47 <@nirik:matrix.scrye.com> well, they are failing backups every day and the check for replication doesn't work on them.
2024-10-10 16:59:59 <@nirik:matrix.scrye.com> but not on ipa01.stg either...
2024-10-10 17:00:18 <@nirik:matrix.scrye.com> so the suggestion was to reinstall them, but I am not sure that will fix the problem
2024-10-10 17:00:29 <@Zlopez:matrix.org> I will try to reinstall them then and see
2024-10-10 17:01:16 <@nirik:matrix.scrye.com> I think the 'can't check replication status' thing has always been happening, but the failed backups are due to them not having something after the re-replica issue.
2024-10-10 17:01:42 <@nirik:matrix.scrye.com> make sure to make a good backup on 01 just in case. ;)
2024-10-10 17:01:54 <@Zlopez:matrix.org> !endmeeting