2024-10-10 08:24:27 <@meetbot:fedora.im> HTML Minutes: https://meetbot.fedoraproject.org/meeting-3_matrix_fedoraproject-org/2024-10-10/cpe-infra-releng-daily-standup.2024-10-10-08.02.html 2024-10-10 16:00:14 <@Zlopez:matrix.org> !startmeeting Infrastructure (2024-10-10) 2024-10-10 16:00:16 <@meetbot:fedora.im> Meeting started at 2024-10-10 16:00:14 UTC 2024-10-10 16:00:17 <@meetbot:fedora.im> The Meeting name is 'Infrastructure (2024-10-10)' 2024-10-10 16:00:27 <@Zlopez:matrix.org> !topic ahoy 2024-10-10 16:00:27 <@Zlopez:matrix.org> !info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra 2024-10-10 16:00:27 <@Zlopez:matrix.org> !info About our team: https://docs.fedoraproject.org/en-US/cpe/ 2024-10-10 16:00:27 <@Zlopez:matrix.org> !info Agenda is at: https://board.net/p/fedora-infra 2024-10-10 16:00:27 <@Zlopez:matrix.org> !chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp 2024-10-10 16:00:27 <@Zlopez:matrix.org> !meetingname infrastructure 2024-10-10 16:00:50 <@meetbot:fedora.im> The Meeting Name is now infrastructure 2024-10-10 16:01:04 <@nirik:matrix.scrye.com> morning 2024-10-10 16:01:06 <@james:fedora.im> !hi 2024-10-10 16:01:06 <@Zlopez:matrix.org> Hello everyone 2024-10-10 16:01:07 <@zodbot:fedora.im> James Antill (james) 2024-10-10 16:01:17 <@Zlopez:matrix.org> And welcome to today infra meeting 2024-10-10 16:01:21 <@Zlopez:matrix.org> !hi 2024-10-10 16:01:23 <@zodbot:fedora.im> Michal Konecny (zlopez) 2024-10-10 16:01:37 <@zardian:matrix.org> !hi 2024-10-10 16:01:38 <@zodbot:fedora.im> Aman Singh (zardian) 2024-10-10 16:01:54 <@Zlopez:matrix.org> Are you ready for a bit of infrastructure in your lives? 2024-10-10 16:02:44 <@zardian:matrix.org> Hi everyone...! 2024-10-10 16:03:31 <@nirik:matrix.scrye.com> who wouldn't be? 2024-10-10 16:04:45 <@Zlopez:matrix.org> So let's start with newcomers :-) 2024-10-10 16:04:53 <@Zlopez:matrix.org> !info This is a place where people who are interested in Fedora Infrastructure can introduce themselves 2024-10-10 16:04:53 <@Zlopez:matrix.org> !topic New folks introductions 2024-10-10 16:04:53 <@Zlopez:matrix.org> !info Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/ 2024-10-10 16:05:05 <@Zlopez:matrix.org> Do we have anybody new here? 2024-10-10 16:05:20 <@carlwgeorge:matrix.org> !hi 2024-10-10 16:05:21 <@zodbot:fedora.im> Carl George (carlwgeorge) - he / him / his 2024-10-10 16:05:30 <@carlwgeorge:matrix.org> not exactly new, but not a regular attendee either 2024-10-10 16:05:49 <@Zlopez:matrix.org> Hi Carl :-) 2024-10-10 16:05:52 <@Zlopez:matrix.org> Nice to see you here 2024-10-10 16:06:14 <@zardian:matrix.org> I am new but did attended the last meeting also. 2024-10-10 16:06:35 <@Zlopez:matrix.org> I remember welcoming you 2024-10-10 16:06:44 <@zardian:matrix.org> yes 2024-10-10 16:07:14 <@nirik:matrix.scrye.com> I treat each day as a new adventure, does that count? 2024-10-10 16:07:45 <@Zlopez:matrix.org> So you are going on Adventure time? 2024-10-10 16:08:36 <@Zlopez:matrix.org> What about chairing this meeting? 2024-10-10 16:08:44 <@Zlopez:matrix.org> !info chair 2024-10-17 - lenkaseg 2024-10-10 16:08:44 <@Zlopez:matrix.org> !info magic eight ball says: 2024-10-10 16:08:44 <@Zlopez:matrix.org> !info chair 2024-10-24 - ??? 2024-10-10 16:08:44 <@Zlopez:matrix.org> !info chair 2024-10-10 - zlopez 2024-10-10 16:08:44 <@Zlopez:matrix.org> !topic Next chair 2024-10-10 16:09:09 <@Zlopez:matrix.org> Any volunteer for 24th October? 2024-10-10 16:10:36 <@nirik:matrix.scrye.com> I can if no one else, or we could just leave it to figure out until next time? 2024-10-10 16:11:04 <@Zlopez:matrix.org> Let's leave it for next week 2024-10-10 16:11:13 <@Zlopez:matrix.org> !info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org 2024-10-10 16:11:13 <@Zlopez:matrix.org> !topic announcements and information 2024-10-10 16:11:13 <@Zlopez:matrix.org> !info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org 2024-10-10 16:11:22 <@Zlopez:matrix.org> Anything else to announce? 2024-10-10 16:12:07 <@nirik:matrix.scrye.com> !info f41 final freeze starts next tuesday. Get changes in before then! 2024-10-10 16:12:40 <@Zlopez:matrix.org> That's important one 2024-10-10 16:15:30 <@Zlopez:matrix.org> It doesn't seem that there is anything else to announce, let's continue with oncall 2024-10-10 16:15:36 <@Zlopez:matrix.org> !info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/ 2024-10-10 16:15:36 <@Zlopez:matrix.org> !info https://fedoraproject.org/wiki/Infrastructure/Oncall 2024-10-10 16:15:36 <@Zlopez:matrix.org> !topic Oncall 2024-10-10 16:15:42 <@Zlopez:matrix.org> !info lenkaseg is on call from 2024-10-03 to 2024-10-10 2024-10-10 16:15:42 <@Zlopez:matrix.org> !info zardian is on call from 2024-10-10 to 2024-10-17 2024-10-10 16:16:14 <@zardian:matrix.org> oh, its me this week.... 2024-10-10 16:16:17 <@Zlopez:matrix.org> !info ??? is on call from 2024-10-18 to 2024-10-24 2024-10-10 16:16:31 <@Zlopez:matrix.org> @zardian:matrix.org Let me update the oncall bot 2024-10-10 16:17:21 <@zardian:matrix.org> did we had any tickets created by lenkaseg based on the oncall function this week? 2024-10-10 16:17:37 <@Zlopez:matrix.org> That is the next topic 2024-10-10 16:17:46 <@zardian:matrix.org> oh, ok. 2024-10-10 16:18:05 <@Zlopez:matrix.org> Right now we are looking for volunteer from 2024-10-18 to 2024-10-24 2024-10-10 16:18:32 <@nirik:matrix.scrye.com> again, I can take it if no one else. ;) 2024-10-10 16:19:19 <@Zlopez:matrix.org> It's yours 2024-10-10 16:19:29 <@Zlopez:matrix.org> !info nirik is on call from 2024-10-18 to 2024-10-24 2024-10-10 16:19:49 <@Zlopez:matrix.org> !info Summary of last week: (from current oncall) 2024-10-10 16:20:01 <@Zlopez:matrix.org> @lenkaseg:fedora.im Anything to report? 2024-10-10 16:20:42 <@Zlopez:matrix.org> She is probably not here 2024-10-10 16:21:18 <@nirik:matrix.scrye.com> I don't think there were any... 2024-10-10 16:23:05 <@Zlopez:matrix.org> I see a ping when there were the switch issues, but it was resolved by itself 2024-10-10 16:23:36 <@Zlopez:matrix.org> s/issues/reboots 2024-10-10 16:23:51 <@nirik:matrix.scrye.com> yeah, I don't see any off hand 2024-10-10 16:24:30 <@zardian:matrix.org> need explanation, where such pings and previous created issued can be checked. pagure? 2024-10-10 16:25:22 <@Zlopez:matrix.org> The pings are !oncall, it will also ping you directly once you are added to oncall duty 2024-10-10 16:25:27 <@Zlopez:matrix.org> It's a zodbot command 2024-10-10 16:25:36 <@nirik:matrix.scrye.com> yeah, it's matrix rooms, #admin:fedoraproject.org mostly 2024-10-10 16:25:53 <@Zlopez:matrix.org> The tickets should be created on https://pagure.io/fedora-infrastructure/issues 2024-10-10 16:26:42 <@Zlopez:matrix.org> The rooms we are usually watching are #admin:fedoraproject.org #noc:fedoraproject.org #releng:fedoraproject.org #apps:fedoraproject.org 2024-10-10 16:27:05 <@zardian:matrix.org> Thanks, understood. 2024-10-10 16:27:41 <@Zlopez:matrix.org> Let's continue with monitoring discussion 2024-10-10 16:27:43 <@Zlopez:matrix.org> !info Go over existing items and fix them 2024-10-10 16:27:43 <@Zlopez:matrix.org> !topic Monitoring discussion [nirik] 2024-10-10 16:27:43 <@Zlopez:matrix.org> !info https://nagios.fedoraproject.org/nagios 2024-10-10 16:28:03 <@nirik:matrix.scrye.com> probibly can't reach it due to switch reboots... 2024-10-10 16:28:19 <@nirik:matrix.scrye.com> yeah. ;( 2024-10-10 16:28:25 <@nirik:matrix.scrye.com> skip today? 2024-10-10 16:28:31 <@Zlopez:matrix.org> Hopefully that will be resolved soon 2024-10-10 16:28:36 <@Zlopez:matrix.org> But let's skip it for today 2024-10-10 16:29:01 <@Zlopez:matrix.org> Let's do a backlog refinement today 2024-10-10 16:29:10 <@Zlopez:matrix.org> !link https://pagure.io/fedora-infrastructure/issues?status=Open&order_key=last_updated&order=asc 2024-10-10 16:29:10 <@Zlopez:matrix.org> !topic Fedora Infra backlog refinement 2024-10-10 16:29:10 <@Zlopez:matrix.org> !info Refine oldest tickets on Fedora Infra tracker 2024-10-10 16:29:36 <@nirik:matrix.scrye.com> sure. 2024-10-10 16:29:39 <@Zlopez:matrix.org> !ticket 11683 2024-10-10 16:29:54 <@Zlopez:matrix.org> It seems that we lost the zodbot 2024-10-10 16:29:55 <@nirik:matrix.scrye.com> bot can't reach pagure... ;) 2024-10-10 16:30:08 <@Zlopez:matrix.org> !link https://pagure.io/fedora-infrastructure/issue/11683 2024-10-10 16:30:12 <@Zlopez:matrix.org> Let it do like that 2024-10-10 16:30:39 <@nirik:matrix.scrye.com> it will catch up when it reconnects 2024-10-10 16:30:53 <@Zlopez:matrix.org> It seems that this is waiting for mattdm 2024-10-10 16:31:11 <@nirik:matrix.scrye.com> Yeah, I am completely not sure what the status of this is. 2024-10-10 16:31:19 <@nirik:matrix.scrye.com> I can bring it up with him my next meeting... 2024-10-10 16:32:01 <@Zlopez:matrix.org> I will ping him in the ticket as well, I assume he just doesn't have time for this 2024-10-10 16:32:18 <@Zlopez:matrix.org> !link https://pagure.io/fedora-infrastructure/issue/11393 2024-10-10 16:32:47 <@nirik:matrix.scrye.com> yeah, unclear... 2024-10-10 16:32:49 <@Zlopez:matrix.org> This one is being worked on, but the last update on the ticket is 7 months old 2024-10-10 16:33:13 <@zardian:matrix.org> need explanation, where such pings and previous created issues can be checked. pagure? 2024-10-10 16:33:47 <@zodbot:fedora.im> ● **Opened:** 10 months ago by mattdm 2024-10-10 16:33:47 <@zodbot:fedora.im> ● **Last Updated:** 2 minutes ago 2024-10-10 16:33:47 <@zodbot:fedora.im> ● **Assignee:** Not Assigned 2024-10-10 16:33:47 <@zodbot:fedora.im> 2024-10-10 16:33:47 <@zodbot:fedora.im> **fedora-infrastructure #11683** (https://pagure.io/fedora-infrastructure/issue/11683):**discourse s3 backup buckets no longer active** 2024-10-10 16:34:58 <@Zlopez:matrix.org> I know we have a bot sending zabbix alerts to a channel on matrix, but not sure about much else 2024-10-10 16:35:21 <@nirik:matrix.scrye.com> which one are we on now? 2024-10-10 16:35:53 <@Zlopez:matrix.org> !ticket 11393 2024-10-10 16:35:55 <@zodbot:fedora.im> ● **Opened:** a year ago by zlopez 2024-10-10 16:35:55 <@zodbot:fedora.im> 2024-10-10 16:35:55 <@zodbot:fedora.im> **fedora-infrastructure #11393** (https://pagure.io/fedora-infrastructure/issue/11393):**Replace Nagios with Zabbix in Fedora Infrastructure** 2024-10-10 16:35:55 <@zodbot:fedora.im> ● **Last Updated:** 7 months ago 2024-10-10 16:35:55 <@zodbot:fedora.im> ● **Assignee:** dkirwan 2024-10-10 16:37:40 <@nirik:matrix.scrye.com> ah right. 2024-10-10 16:37:56 <@nirik:matrix.scrye.com> yeah, so we have messages going, but we need to adjust them to be less noisy. 2024-10-10 16:38:14 <@james:fedora.im> I did that for the production one 2024-10-10 16:38:38 <@james:fedora.im> Or at least ... all the disk read timeouts I could find. But I can't do it on stg atm. 2024-10-10 16:39:10 <@james:fedora.im> And things aren't updating properly on the production zabbix side since the big update last week. 2024-10-10 16:39:53 <@james:fedora.im> tl;dr ... it's moving fwd slowly, but it's not at the top of anyone's todo list. 2024-10-10 16:40:19 <@nirik:matrix.scrye.com> yeah. 2024-10-10 16:40:19 <@Zlopez:matrix.org> @james:fedora.im Could you add updates to the ticket? 2024-10-10 16:40:21 <@james:fedora.im> Also hopefully soon we'll get the big update with the digital clock ;) 2024-10-10 16:40:28 <@nirik:matrix.scrye.com> ha. yeah 2024-10-10 16:40:32 <@james:fedora.im> Sure 2024-10-10 16:40:49 <@Zlopez:matrix.org> That would be real lifechanger :-) 2024-10-10 16:41:16 <@Zlopez:matrix.org> Let's go to next one than 2024-10-10 16:41:23 <@Zlopez:matrix.org> !ticket 11958 2024-10-10 16:41:24 <@zodbot:fedora.im> **fedora-infrastructure #11958** (https://pagure.io/fedora-infrastructure/issue/11958):**Add fedora-l10n pagure group as an admin to the fedora-l10n-docs namespace projects** 2024-10-10 16:41:24 <@zodbot:fedora.im> ● **Assignee:** Not Assigned 2024-10-10 16:41:24 <@zodbot:fedora.im> ● **Last Updated:** 4 months ago 2024-10-10 16:41:24 <@zodbot:fedora.im> ● **Opened:** 5 months ago by peartown 2024-10-10 16:41:24 <@zodbot:fedora.im> 2024-10-10 16:42:12 <@Zlopez:matrix.org> I don't think this should be hanging there for 5 months :-/ 2024-10-10 16:42:24 <@nirik:matrix.scrye.com> so... 2024-10-10 16:42:37 <@nirik:matrix.scrye.com> this one I was hoping ryanlerch had a script to do it. 2024-10-10 16:42:46 <@nirik:matrix.scrye.com> but failing that, someone else could write something to do it? 2024-10-10 16:43:03 <@Zlopez:matrix.org> I can ping him and ask him, so we can move this forward 2024-10-10 16:43:50 <@nirik:matrix.scrye.com> ok 2024-10-10 16:44:19 <@Zlopez:matrix.org> In any case at least we can continue on this 2024-10-10 16:44:41 <@Zlopez:matrix.org> !ticket 11144 2024-10-10 16:44:43 <@zodbot:fedora.im> ● **Opened:** a year ago by zlopez 2024-10-10 16:44:43 <@zodbot:fedora.im> 2024-10-10 16:44:43 <@zodbot:fedora.im> **fedora-infrastructure #11144** (https://pagure.io/fedora-infrastructure/issue/11144):**Create monitoring tool for rabbitmq certificates** 2024-10-10 16:44:43 <@zodbot:fedora.im> ● **Last Updated:** 4 months ago 2024-10-10 16:44:43 <@zodbot:fedora.im> ● **Assignee:** t0xic0der 2024-10-10 16:44:59 <@Zlopez:matrix.org> So this is blocked on https://pagure.io/fedora-infrastructure/issue/12066 2024-10-10 16:45:33 <@Zlopez:matrix.org> I will update the ticket with that, it should file issues to repository, but there is issue with creating it 2024-10-10 16:46:05 <@nirik:matrix.scrye.com> I wonder... 2024-10-10 16:46:38 <@nirik:matrix.scrye.com> I was wondering if it was related to https://pagure.io/fedora-infrastructure/issue/11869 but I guess thats a different namespace 2024-10-10 16:47:05 <@nirik:matrix.scrye.com> we could just make it a top level to avoid this issue? 2024-10-10 16:47:08 <@Zlopez:matrix.org> Yeah, that is different issue 2024-10-10 16:47:32 <@Zlopez:matrix.org> I would like to look in those two and try few things to get them deleted 2024-10-10 16:47:48 <@Zlopez:matrix.org> Where is the pagure.io db actually hosted? 2024-10-10 16:48:08 <@Zlopez:matrix.org> I noticed that the pagure PostgreSQL db on db01 is in fack dist-git 2024-10-10 16:48:21 <@Zlopez:matrix.org> I noticed that the pagure PostgreSQL db on db01 is in fact dist-git 2024-10-10 16:48:53 <@nirik:matrix.scrye.com> it's on pagure02 2024-10-10 16:49:17 <@Zlopez:matrix.org> Oh, it's directly on the machine 2024-10-10 16:49:25 <@nirik:matrix.scrye.com> the 11869 thing is a bug in agile board stuff. I tried to poke the db to fix it, but wasn't successfull 2024-10-10 16:49:27 <@nirik:matrix.scrye.com> yes 2024-10-10 16:49:52 <@Zlopez:matrix.org> OK, I will try to look into that tomorrow on next week 2024-10-10 16:50:00 <@Zlopez:matrix.org> OK, I will try to look into that tomorrow or next week 2024-10-10 16:50:35 <@nirik:matrix.scrye.com> that would be great! 2024-10-10 16:51:00 <@Zlopez:matrix.org> I finally have some spare cycles, so I'm trying to go through tickets on infra tracker 2024-10-10 16:51:14 <@Zlopez:matrix.org> !ticket 11815 2024-10-10 16:51:15 <@zodbot:fedora.im> 2024-10-10 16:51:15 <@zodbot:fedora.im> ● **Assignee:** zlopez 2024-10-10 16:51:15 <@zodbot:fedora.im> ● **Last Updated:** 3 months ago 2024-10-10 16:51:15 <@zodbot:fedora.im> ● **Opened:** 7 months ago by zlopez 2024-10-10 16:51:15 <@zodbot:fedora.im> **fedora-infrastructure #11815** (https://pagure.io/fedora-infrastructure/issue/11815):**rhel7 eol** 2024-10-10 16:51:30 <@nirik:matrix.scrye.com> Yeah, to fedmsg and github2fedmsg are the last left 2024-10-10 16:51:38 <@Zlopez:matrix.org> This will be solved when https://pagure.io/fedora-infrastructure/issue/11804 2024-10-10 16:51:42 <@nirik:matrix.scrye.com> but I wonder... do we have a timeline on retiring github2fedmsg? 2024-10-10 16:51:57 <@Zlopez:matrix.org> The webhook2fedmsg is ready, but it wasn't announced yet 2024-10-10 16:52:29 <@Zlopez:matrix.org> That should happen soon and then we should give some time for people to migrate from fedmsg and github2fedmsg 2024-10-10 16:52:29 <@nirik:matrix.scrye.com> on fedmsg we have to wait a month... 2024-10-10 16:52:37 <@nirik:matrix.scrye.com> but it would be nice to set a clear timeline 2024-10-10 16:52:51 <@Zlopez:matrix.org> I would do the same for both, once the replacement is in place 2024-10-10 16:52:58 <@nirik:matrix.scrye.com> yeah. 2024-10-10 16:53:21 <@Zlopez:matrix.org> Give people a month to migrate from github2fedmsg and fedmsg 2024-10-10 16:54:00 <@nirik:matrix.scrye.com> right, and we retire that and fedmsg at the same time whenever the deadline is 2024-10-10 16:54:04 <@Zlopez:matrix.org> It would be a reason to celebrate wehen we finally retire fedmsg :-) 2024-10-10 16:54:19 <@Zlopez:matrix.org> It would be a reason to celebrate when we finally retire fedmsg :-) 2024-10-10 16:56:08 <@Zlopez:matrix.org> Updated the ticket 2024-10-10 16:56:22 <@Zlopez:matrix.org> We have only few minutes left, so let's change to open floor 2024-10-10 16:56:40 <@Zlopez:matrix.org> !topic Open Floor 2024-10-10 16:56:53 <@Zlopez:matrix.org> Anything to discuss for the last few minutes? 2024-10-10 16:57:34 <@Zlopez:matrix.org> I just wanted to thank everybody for their work on Fedora Infra, we are getting close to 60 tickets on the tracker 2024-10-10 16:58:06 <@Zlopez:matrix.org> I think this is the lowest number I ever saw on the tracker 👍️ 2024-10-10 16:58:21 <@nirik:matrix.scrye.com> we have been lower, but not in a while 2024-10-10 16:58:31 <@nirik:matrix.scrye.com> I guess I need to file some more tickets. ;) 2024-10-10 16:59:23 <@Zlopez:matrix.org> One question, does https://pagure.io/fedora-infrastructure/issue/12158 still needs reinstalling of ipa02 and ipa03? 2024-10-10 16:59:47 <@nirik:matrix.scrye.com> well, they are failing backups every day and the check for replication doesn't work on them. 2024-10-10 16:59:59 <@nirik:matrix.scrye.com> but not on ipa01.stg either... 2024-10-10 17:00:18 <@nirik:matrix.scrye.com> so the suggestion was to reinstall them, but I am not sure that will fix the problem 2024-10-10 17:00:29 <@Zlopez:matrix.org> I will try to reinstall them then and see 2024-10-10 17:01:16 <@nirik:matrix.scrye.com> I think the 'can't check replication status' thing has always been happening, but the failed backups are due to them not having something after the re-replica issue. 2024-10-10 17:01:42 <@nirik:matrix.scrye.com> make sure to make a good backup on 01 just in case. ;) 2024-10-10 17:01:54 <@Zlopez:matrix.org> !endmeeting