2024-10-31 16:00:43 <@Zlopez:matrix.org> !startmeeting Infrastructure (2024-10-31)
2024-10-31 16:00:46 <@meetbot:fedora.im> Meeting started at 2024-10-31 16:00:43 UTC
2024-10-31 16:00:46 <@meetbot:fedora.im> The Meeting name is 'Infrastructure (2024-10-31)'
2024-10-31 16:00:53 <@Zlopez:matrix.org> !info About our team: https://docs.fedoraproject.org/en-US/cpe/
2024-10-31 16:00:53 <@Zlopez:matrix.org> !info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
2024-10-31 16:00:53 <@Zlopez:matrix.org> !meetingname infrastructure
2024-10-31 16:00:53 <@Zlopez:matrix.org> !topic ahoy
2024-10-31 16:00:53 <@Zlopez:matrix.org> !chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp
2024-10-31 16:00:53 <@Zlopez:matrix.org> !info Agenda is at: https://board.net/p/fedora-infra
2024-10-31 16:00:54 <@meetbot:fedora.im> The Meeting Name is now infrastructure
2024-10-31 16:01:13 <@nirik:matrix.scrye.com> 👻spooky morning everyone 👻
2024-10-31 16:01:49 <@Zlopez:matrix.org> It's dark outside here :-D
2024-10-31 16:02:11 <@Zlopez:matrix.org> But spooky halloween everyone
2024-10-31 16:02:21 <@nirik:matrix.scrye.com> 🌁 and raining here, but it is morning.
2024-10-31 16:02:33 <@Zlopez:matrix.org> I will be your seriall killer for this meeting 🗡️
2024-10-31 16:02:48 <@Zlopez:matrix.org> I will be your serial killer for this meeting 🗡️
2024-10-31 16:03:24 <@seddik:fedora.im> .hi 
2024-10-31 16:03:40 <@seddik:fedora.im> !hi
2024-10-31 16:03:41 <@zodbot:fedora.im> seddik alaouiismaili (seddik)
2024-10-31 16:03:44 <@Zlopez:matrix.org> Which service will be my first victim today :-D
2024-10-31 16:05:16 <@nirik:matrix.scrye.com> could be any one? do we have any walking alone in the dark?
2024-10-31 16:05:28 <@james:fedora.im> All of them
2024-10-31 16:05:32 <@phsmoura:fedora.im> hello
2024-10-31 16:06:03 <@thisisyaash:fedora.im> !hi
2024-10-31 16:06:04 <@zodbot:fedora.im> Yashwanth Rathakrishnan (thisisyaash)
2024-10-31 16:06:49 <@Zlopez:matrix.org> Hi @thisisyaash:fedora.im, thanks for the PR for statusfpo :-)
2024-10-31 16:07:19 <@thisisyaash:fedora.im> thanks for merging it :)
2024-10-31 16:07:34 <@Zlopez:matrix.org> So let's go to the first victim of this meeting
2024-10-31 16:07:35 <@Zlopez:matrix.org> !info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
2024-10-31 16:07:35 <@Zlopez:matrix.org> !topic New folks introductions
2024-10-31 16:07:35 <@Zlopez:matrix.org> !info Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/
2024-10-31 16:08:06 <@Zlopez:matrix.org> So who is here for first time?
2024-10-31 16:09:49 <@nirik:matrix.scrye.com> I was here last week, but it's the first time I am here this week. ;)
2024-10-31 16:10:44 <@Zlopez:matrix.org> Let's move on, my first victim is eluding me
2024-10-31 16:10:52 <@Zlopez:matrix.org> !topic Next chair
2024-10-31 16:10:52 <@Zlopez:matrix.org> !info magic eight ball says:
2024-10-31 16:10:52 <@Zlopez:matrix.org> !info chair 2024-10-31 - zlopez
2024-10-31 16:10:52 <@Zlopez:matrix.org> !info chair 2024-11-07 - ?
2024-10-31 16:10:52 <@Zlopez:matrix.org> !info chair 2024-11-14 - ?
2024-10-31 16:11:41 <@Zlopez:matrix.org> So who wants to be my next victim, I wanted to say chair the next meeting
2024-10-31 16:13:48 <@Zlopez:matrix.org> Any volunteer?
2024-10-31 16:14:48 <@nirik:matrix.scrye.com> I suppose I can if no one else is wanting it.
2024-10-31 16:16:06 <@Zlopez:matrix.org> We should probably try to clean the chairs and just rotate through them
2024-10-31 16:16:23 <@Zlopez:matrix.org> What do you think?
2024-10-31 16:17:15 <@Zlopez:matrix.org> But for now I will assign it to you
2024-10-31 16:17:17 <@Zlopez:matrix.org> !info chair 2024-11-07 - nirik
2024-10-31 16:17:44 <@nirik:matrix.scrye.com> It's a great way for new folks to get more involved too... but yeah, if it's always the same people we should figure out something else. ;)
2024-10-31 16:18:15 <@seddik:fedora.im> i can take 14-11 
2024-10-31 16:18:21 <@seddik:fedora.im> I can take 14-11 
2024-10-31 16:18:39 <@Zlopez:matrix.org> We can always assign the chair to volunteer but if nobody is volunteering just pick the next one on chair list
2024-10-31 16:18:51 <@Zlopez:matrix.org> @seddik:fedora.im It's yours :-)
2024-10-31 16:19:00 <@Zlopez:matrix.org> !info chair 2024-11-14 - saibug
2024-10-31 16:19:42 <@Zlopez:matrix.org> I think we are covered for now
2024-10-31 16:19:55 <@Zlopez:matrix.org> Let's go haunting somewhere else :-D
2024-10-31 16:20:13 <@Zlopez:matrix.org> !info CLE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0900 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
2024-10-31 16:20:13 <@Zlopez:matrix.org> !info CLE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
2024-10-31 16:20:13 <@Zlopez:matrix.org> !topic announcements and information
2024-10-31 16:20:27 <@Zlopez:matrix.org> !info F41 freeze is over
2024-10-31 16:20:30 <@nirik:matrix.scrye.com> !info we are out of f41 final freeze now
2024-10-31 16:21:11 <@james:fedora.im> Change all the things!
2024-10-31 16:21:19 <@Zlopez:matrix.org> !info dist-git has issues returning 502/503 - we are aware of that and working on it
2024-10-31 16:22:00 <@nirik:matrix.scrye.com> Yeah, thats a puzzling one. ;(
2024-10-31 16:22:36 <@Zlopez:matrix.org> I will continue investigating that tomorrow, spend some time on that today, but got distracted by other things
2024-10-31 16:23:15 <@nirik:matrix.scrye.com> !info pagure.io outage in a while (hopefully short)
2024-10-31 16:25:17 <@Zlopez:matrix.org> Let's continue to next one
2024-10-31 16:25:27 <@Zlopez:matrix.org> !info ? is on call from 2024-11-08 to 2024-11-14
2024-10-31 16:25:27 <@Zlopez:matrix.org> !info ? is on call from 2024-11-01 to 2024-11-07
2024-10-31 16:25:27 <@Zlopez:matrix.org> !topic Oncall
2024-10-31 16:25:27 <@Zlopez:matrix.org> !info https://fedoraproject.org/wiki/Infrastructure/Oncall
2024-10-31 16:25:27 <@Zlopez:matrix.org> !info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
2024-10-31 16:25:27 <@Zlopez:matrix.org> !info lenkaseg is on call from 2024-10-25 to 2024-10-31
2024-10-31 16:25:38 <@Zlopez:matrix.org> I will take the next one
2024-10-31 16:25:49 <@Zlopez:matrix.org> !info zlopez is on call from 2024-11-01 to 2024-11-07
2024-10-31 16:27:16 <@Zlopez:matrix.org> Hm, there was still zardian in oncall
2024-10-31 16:27:22 <@Zlopez:matrix.org> !infra oncall list
2024-10-31 16:27:23 <@zodbot:fedora.im> ● @Zlopez:matrix.org (zlopez) Current Time for them: 17:27 (Europe/Prague)
2024-10-31 16:27:23 <@zodbot:fedora.im> The following people are oncall:
2024-10-31 16:27:23 <@zodbot:fedora.im> 
2024-10-31 16:27:23 <@zodbot:fedora.im> If they do not respond, please file a ticket (https://pagure.io/fedora-infrastructure/issues)
2024-10-31 16:27:31 <@Zlopez:matrix.org> I added myself now
2024-10-31 16:28:05 <@Zlopez:matrix.org> Anybody wants to take oncall for 2024-11-08 to 2024-11-14
2024-10-31 16:28:28 <@Zlopez:matrix.org> ?
2024-10-31 16:28:37 <@nirik:matrix.scrye.com> I could
2024-10-31 16:29:41 <@Zlopez:matrix.org> !info nirik is on call from 2024-11-08 to 2024-11-14
2024-10-31 16:30:01 <@Zlopez:matrix.org> I think we can use the same process as for chairs in future
2024-10-31 16:30:11 <@Zlopez:matrix.org> I will look into that for next week
2024-10-31 16:30:37 <@nirik:matrix.scrye.com> feel free to propose a new process... ;)
2024-10-31 16:31:04 <@nirik:matrix.scrye.com> side note: I see status is failing to build now, not due to any of the recent changes, seems to be a github actions change?
2024-10-31 16:31:07 <@Zlopez:matrix.org> Will do and we can try it next week
2024-10-31 16:31:31 <@Zlopez:matrix.org> I assume they changed the container they are using for running them
2024-10-31 16:31:39 <@nirik:matrix.scrye.com> not sure.
2024-10-31 16:31:39 <@Zlopez:matrix.org> I was looking at that as well
2024-10-31 16:32:00 <@nirik:matrix.scrye.com> well, we should discuss the change before just doing it, having a concrete proposal would be good for that
2024-10-31 16:32:11 <@Zlopez:matrix.org> It should be just change of the pip command we are using as one of the arguments is not recognised anymore
2024-10-31 16:32:29 <@Zlopez:matrix.org> Where should I submit the proposal?
2024-10-31 16:32:39 <@Zlopez:matrix.org> Infra mailing list?
2024-10-31 16:32:47 <@nirik:matrix.scrye.com> right, but ryan had to add that option I think to make it work before... :)
2024-10-31 16:32:53 <@nirik:matrix.scrye.com> sure! or discussion?
2024-10-31 16:33:47 <@Zlopez:matrix.org> I don't want to automate that just make the decision process easier :-)
2024-10-31 16:33:57 <@Zlopez:matrix.org> Let me submit it to both
2024-10-31 16:35:39 <@Zlopez:matrix.org> Let us continue
2024-10-31 16:35:44 <@Zlopez:matrix.org> !info Summary of last week: (from current oncall)
2024-10-31 16:36:11 <@Zlopez:matrix.org> As @lenkaseg:matrix.org wasn't even assigned to oncall I don't know if she did notice any
2024-10-31 16:36:24 <@Zlopez:matrix.org> I didn't saw any oncall ping in last week
2024-10-31 16:37:00 <@nirik:matrix.scrye.com> yeah. I don't recall any
2024-10-31 16:37:45 <@Zlopez:matrix.org> Let's move on then
2024-10-31 16:37:53 <@Zlopez:matrix.org> !topic Monitoring discussion [nirik]
2024-10-31 16:37:53 <@Zlopez:matrix.org> !info Go over existing items and fix them
2024-10-31 16:37:53 <@Zlopez:matrix.org> !info https://nagios.fedoraproject.org/nagios 
2024-10-31 16:38:24 <@nirik:matrix.scrye.com> lets see.
2024-10-31 16:39:00 <@nirik:matrix.scrye.com> theres a fmn alert, but thats Aurélien B working on the caching
2024-10-31 16:39:16 <@nirik:matrix.scrye.com> There's 2 down aarch64 machines, but I am going to shuffle things around and they should be gone soon
2024-10-31 16:39:35 <@nirik:matrix.scrye.com> pagure02 disk space (but that should be fixed in an hour or so)
2024-10-31 16:39:40 <@Zlopez:matrix.org> The FMN is now processing again, as the amount of messages in queue is getting lower
2024-10-31 16:39:54 <@nirik:matrix.scrye.com> disk space on ipa01.stg I haven't looked at... logs of logs?
2024-10-31 16:40:26 <@nirik:matrix.scrye.com> pagure is being hit hard again, I'm going to block some more ips
2024-10-31 16:40:27 <@Zlopez:matrix.org> I remember that there were some logs that weren't compressed, so I just did the compression
2024-10-31 16:40:54 <@Zlopez:matrix.org> It seems that logrotate doesn't work as it should
2024-10-31 16:41:05 <@nirik:matrix.scrye.com> :(
2024-10-31 16:41:22 <@nirik:matrix.scrye.com> anyhow, thats all the current alerts... hopefully we can have some quiet time soon...
2024-10-31 16:41:34 <@Zlopez:matrix.org> Or maybe there is just more traffic on staging as the dist-git test instances are using staging accounts
2024-10-31 16:42:25 <@Zlopez:matrix.org> One thing to add I restarted openqa01 today as it was stuck on some CPU stall and didn't wanted to even shutdown
2024-10-31 16:42:52 <@Zlopez:matrix.org> But it's a VM, so I just shut it down forcefully and started it again
2024-10-31 16:43:00 <@nirik:matrix.scrye.com> ok. Odd.
2024-10-31 16:43:14 <@Zlopez:matrix.org> It's running OK now
2024-10-31 16:43:39 <@Zlopez:matrix.org> What do we want to do now?
2024-10-31 16:43:49 <@nirik:matrix.scrye.com> I also had to muck with bvmhost-x86-03 yesterday... it went unresponsive and then had a disk drop out and had some raid issues... took a while to get it back. It's up and down a disk, but I need to look and see if that disk is really bad or not, because some things say it's fine.
2024-10-31 16:44:01 <@nirik:matrix.scrye.com> sorry, thats bvmhost-x86-03.stg
2024-10-31 16:44:52 <@Zlopez:matrix.org> I saw the messages in noc channel, it looked really strange
2024-10-31 16:45:17 <@Zlopez:matrix.org> Let us vote by giving thumbs up
2024-10-31 16:45:32 <@Zlopez:matrix.org> Revisit blocked tickets
2024-10-31 16:45:37 <@Zlopez:matrix.org> Fedora Infra backlog refinement
2024-10-31 16:45:47 <@Zlopez:matrix.org> Think of learning topic
2024-10-31 16:45:49 <@Zlopez:matrix.org> Open floor
2024-10-31 16:45:53 <@Zlopez:matrix.org> End early
2024-10-31 16:46:12 <@nirik:matrix.scrye.com> we don't have too much time left...
2024-10-31 16:46:31 <@nirik:matrix.scrye.com> we could hit on blocked really quick then open floor?
2024-10-31 16:46:38 <@Zlopez:matrix.org> So let's skip to open floor
2024-10-31 16:46:44 <@Zlopez:matrix.org> !topic Open Floor
2024-10-31 16:47:41 <@thisisyaash:fedora.im> actually i noticed that the recently merged PR in github is failing the checks
2024-10-31 16:48:12 <@nirik:matrix.scrye.com> for status? yeah, it seems to have changed and no longer accepts a flag we were passing to pip
2024-10-31 16:48:14 <@thisisyaash:fedora.im> actually i noticed that the recently merged PR in github is failing the checks (the check is failed)
2024-10-31 16:48:20 <@Zlopez:matrix.org> Today on Fedora prioritisation meeting we talked about zabbix and what is the situation around it
2024-10-31 16:48:37 <@Zlopez:matrix.org> It's not related to your PR
2024-10-31 16:48:42 <@nirik:matrix.scrye.com> I was gonna try and get to that, but it was too early for me. ;)
2024-10-31 16:49:32 <@thisisyaash:fedora.im> i see! 
2024-10-31 16:49:32 <@thisisyaash:fedora.im> i thought i messed up bcz after my PR getting merged
2024-10-31 16:49:32 <@thisisyaash:fedora.im> 
2024-10-31 16:49:32 <@thisisyaash:fedora.im> and the check is failed
2024-10-31 16:49:32 <@thisisyaash:fedora.im> so that's why!
2024-10-31 16:50:21 <@Zlopez:matrix.org> From what I understand the zabbix work is now more or less waiting for us to start using it and start converting the checks we have for nagios
2024-10-31 16:51:03 <@nirik:matrix.scrye.com> well, it's still really noisy, we need to make it not alert every 2 minutes. ;)
2024-10-31 16:51:28 <@nirik:matrix.scrye.com> althought I guess james did some of that. I should look again.
2024-10-31 16:51:50 <@nirik:matrix.scrye.com> There's also a 7.0 version out we should look at to see if we can move to
2024-10-31 16:51:55 <@Zlopez:matrix.org> It is really noisy that's true
2024-10-31 16:52:52 <@nirik:matrix.scrye.com> but then yeah, we should look at whats in nagios and make sure it's in zabbix.
2024-10-31 16:53:18 <@nirik:matrix.scrye.com> I'm not sure if zabbix is monitoring all the hosts nagios is? but it might be.
2024-10-31 16:54:03 <@Zlopez:matrix.org> It would be nice to check
2024-10-31 16:54:07 <@nirik:matrix.scrye.com> perhaps we could break up things into milestones or something and track them?
2024-10-31 16:54:27 <@nirik:matrix.scrye.com> I don't know if thats more tickets, or just some planning in the existing one(s)...
2024-10-31 16:54:49 <@Zlopez:matrix.org> I would like as to move this forward, but there are so much things happening it's hard to move forward everything
2024-10-31 16:55:15 <@nirik:matrix.scrye.com> very much so yeah.
2024-10-31 16:55:36 <@nirik:matrix.scrye.com> lots of big big things on the horizon (mostly next year, but need some planning this year)
2024-10-31 16:55:55 <@Zlopez:matrix.org> There are so many interesting things happening in Fedora and not only in infrastructure, it's hard to keep track of everything :-)
2024-10-31 16:56:06 <@nirik:matrix.scrye.com> it's a firehose for sure.
2024-10-31 16:56:25 <@Zlopez:matrix.org> We should at least finish the RHEL 7 EOL till end of year
2024-10-31 16:56:46 <@nirik:matrix.scrye.com> yes, I would very much like to finish that off.
2024-10-31 16:56:58 <@Zlopez:matrix.org> I need to poke the work on webhook2fedmsg and announce sunset of github2fedmsg
2024-10-31 16:57:14 <@nirik:matrix.scrye.com> I think there was a article / announcement?
2024-10-31 16:57:38 <@Zlopez:matrix.org> Yes, I scheduled that one
2024-10-31 16:57:54 <@nirik:matrix.scrye.com> https://communityblog.fedoraproject.org/announcing-webhook-to-fedora-messaging/
2024-10-31 16:58:31 <@Zlopez:matrix.org> I see only one request for migration
2024-10-31 16:58:52 <@Zlopez:matrix.org> https://pagure.io/fedora-infra/w2fm-migration/issues
2024-10-31 16:59:01 <@nirik:matrix.scrye.com> yeah, thats not great. ;( Probibly should reach out to each person that has github2fedmsg enabled.
2024-10-31 16:59:25 <@nirik:matrix.scrye.com> also, coming up: we need to move some f39 things to f40 before f39 goes eol
2024-10-31 16:59:39 <@Zlopez:matrix.org> +1 on that
2024-10-31 16:59:45 <@nirik:matrix.scrye.com> and also we need to setup process for communishift cleanup/announce/etc
2024-10-31 17:00:05 <@Zlopez:matrix.org> @dkirwan:fedora.im is working on that
2024-10-31 17:00:24 <@nirik:matrix.scrye.com> yep.
2024-10-31 17:00:49 <@nirik:matrix.scrye.com> we were gonna do that at f39 eol tho, so 1 month left and need to give people time to react
2024-10-31 17:00:50 <@Zlopez:matrix.org> I will check with him
2024-10-31 17:02:14 <@Zlopez:matrix.org> Could we get the list of users of github2fedmsg from db?
2024-10-31 17:02:23 <@Zlopez:matrix.org> !endmeeting