2024-09-26 16:00:10 <@patrikp:matrix.org> !startmeeting Infrastructure (2024-09-26)
2024-09-26 16:00:11 <@meetbot:fedora.im> Meeting started at 2024-09-26 16:00:10 UTC
2024-09-26 16:00:12 <@meetbot:fedora.im> The Meeting name is 'Infrastructure (2024-09-26)'
2024-09-26 16:00:17 <@patrikp:matrix.org> !meetingname infrastructure
2024-09-26 16:00:18 <@meetbot:fedora.im> The Meeting Name is now infrastructure
2024-09-26 16:00:23 <@patrikp:matrix.org> !chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp
2024-09-26 16:00:29 <@patrikp:matrix.org> !info Agenda is at: https://board.net/p/fedora-infra
2024-09-26 16:00:36 <@patrikp:matrix.org> !info About our team: https://docs.fedoraproject.org/en-US/cpe/
2024-09-26 16:00:42 <@patrikp:matrix.org> !info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
2024-09-26 16:00:47 <@nirik:matrix.scrye.com> morning
2024-09-26 16:00:49 <@patrikp:matrix.org> !topic ahoy
2024-09-26 16:00:50 <@james:fedora.im> !hi
2024-09-26 16:00:51 <@zodbot:fedora.im> James Antill (james)
2024-09-26 16:00:52 <@patrikp:matrix.org> Hello!
2024-09-26 16:00:55 <@Zlopez:matrix.org> !hi
2024-09-26 16:00:57 <@zodbot:fedora.im> Michal Konecny (zlopez)
2024-09-26 16:01:25 <@patrikp:matrix.org> !topic New folks introductions
2024-09-26 16:01:30 <@patrikp:matrix.org> !info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
2024-09-26 16:01:35 <@patrikp:matrix.org> !info Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/
2024-09-26 16:01:45 <@patrikp:matrix.org> Do we have anyone new joining us today? Come and say hi. 👋
2024-09-26 16:03:04 <@Zlopez:matrix.org> @patrikp:matrix.org Feel free to copy over multiple lines, only line that needs to be pasted separately is !startmeeting
2024-09-26 16:03:36 <@patrikp:matrix.org> Does it? Alright, let me try. Looks like nobody new is here, we can move on...
2024-09-26 16:03:49 <@patrikp:matrix.org> !topic Next chair
2024-09-26 16:03:49 <@patrikp:matrix.org> !info chair 2024-10-03 - lenkaseg
2024-09-26 16:03:49 <@patrikp:matrix.org> !info chair 2024-09-26 - patrikp
2024-09-26 16:03:49 <@patrikp:matrix.org> !info magic eight ball says:
2024-09-26 16:03:49 <@patrikp:matrix.org> !info chair 2024-10-10 - ???
2024-09-26 16:04:04 <@patrikp:matrix.org> Any takers for October 10?
2024-09-26 16:04:20 <@Zlopez:matrix.org> I can take it
2024-09-26 16:04:37 <@patrikp:matrix.org> !info chair 2024-10-10 - zlopez
2024-09-26 16:05:00 <@patrikp:matrix.org> !info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
2024-09-26 16:05:00 <@patrikp:matrix.org> !info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
2024-09-26 16:05:00 <@patrikp:matrix.org> !topic announcements and information
2024-09-26 16:05:46 <@patrikp:matrix.org> There are announcements about F41 Beta but I suppose they were already covered last week.
2024-09-26 16:05:58 <@Zlopez:matrix.org> !Info CPE Update for Q2 published https://communityblog.fedoraproject.org/cpe-update-q2-2024/
2024-09-26 16:05:58 <@patrikp:matrix.org> Any other announcements?
2024-09-26 16:06:25 <@nirik:matrix.scrye.com> !info nirik out on pto tomorrow
2024-09-26 16:06:50 <@Zlopez:matrix.org> It seems meetbot skipped my message
2024-09-26 16:07:05 <@patrikp:matrix.org> Is it because of the capital I?
2024-09-26 16:07:35 <@nirik:matrix.scrye.com> could be
2024-09-26 16:07:38 <@patrikp:matrix.org> !info CPE Update for Q2 published https://communityblog.fedoraproject.org/cpe-update-q2-2024/
2024-09-26 16:07:49 <@Zlopez:matrix.org> Didn't realized that
2024-09-26 16:08:22 <@patrikp:matrix.org> Any other announcements?
2024-09-26 16:08:37 <@patrikp:matrix.org> Moving on...
2024-09-26 16:08:44 <@patrikp:matrix.org> !info https://fedoraproject.org/wiki/Infrastructure/Oncall
2024-09-26 16:08:44 <@patrikp:matrix.org> !info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
2024-09-26 16:08:44 <@patrikp:matrix.org> !topic Oncall
2024-09-26 16:08:53 <@patrikp:matrix.org> !info lenkaseg is on call from 2024-09-20 to 2024-09-27
2024-09-26 16:08:53 <@patrikp:matrix.org> !info ? is on call from 2024-09-27 to 2024-10-03
2024-09-26 16:09:13 <@patrikp:matrix.org> Any takers for the next week in regards to oncall?
2024-09-26 16:10:02 <@nirik:matrix.scrye.com> I think I forgot to move it to lenkaseg last week, so I have had it two weeks. ;)
2024-09-26 16:10:30 <@patrikp:matrix.org> Ah, alright, so Lenka has Oct 3 - 10?
2024-09-26 16:11:27 <@nirik:matrix.scrye.com> not sure, wouldn't want to sign her up for something without checking.
2024-09-26 16:11:43 <@Zlopez:matrix.org> I don't see her on the meeting, so I'm not sure if it's fine to assign the oncall to her when she doesn't know about it
2024-09-26 16:11:53 <@patrikp:matrix.org> Good point.
2024-09-26 16:12:14 <@patrikp:matrix.org> Any takers for Sep 27 to Oct 3?
2024-09-26 16:12:57 <@Zlopez:matrix.org> I can take it
2024-09-26 16:13:17 <@Zlopez:matrix.org> Hopefully it will work better for me this time
2024-09-26 16:13:40 <@patrikp:matrix.org> !info zlopez is on call from 2024-09-27 to 2024-10-03
2024-09-26 16:14:01 <@patrikp:matrix.org> !info Summary of last week: (from current oncall)
2024-09-26 16:14:37 <@nirik:matrix.scrye.com> lets see... several oncall calls.
2024-09-26 16:14:44 <@nirik:matrix.scrye.com> some for read only koji builders.
2024-09-26 16:15:11 <@nirik:matrix.scrye.com> one from adamw about proxy01.stg not responding...
2024-09-26 16:15:16 <@leo:fedora.im> hey!
2024-09-26 16:16:23 <@lenkaseg:matrix.org> Hi all!
2024-09-26 16:16:24 <@nirik:matrix.scrye.com> one from when bvmhost-x86-03 was unresponsive... taking down koji and registry (those vm's are on it)
2024-09-26 16:16:47 <@Zlopez:matrix.org> Hi @leo:fedora.im @lenkaseg:matrix.org
2024-09-26 16:17:15 <@lenkaseg:matrix.org> Assign the oncall to me please!
2024-09-26 16:17:34 <@nirik:matrix.scrye.com> actually it was two from that bvmhost thing. ;)
2024-09-26 16:17:38 <@nirik:matrix.scrye.com> thats all of them that I saw.
2024-09-26 16:18:03 <@Zlopez:matrix.org> @lenkaseg:fedora.im I already assigned it to me, but let's add you for next week
2024-09-26 16:18:14 <@lenkaseg:matrix.org> (Sorry, the new school year started and I'm a bit busy around this time moving kids to trainings)
2024-09-26 16:18:19 <@patrikp:matrix.org> Would you like Oct 3 - 10 Lenka?
2024-09-26 16:18:26 <@lenkaseg:matrix.org> Yup!
2024-09-26 16:18:40 <@patrikp:matrix.org> !info lenkaseg is on call from 2024-10-03 to 2024-10-10
2024-09-26 16:19:11 <@Zlopez:matrix.org> I forgot that in Spain it's starting at end of September, here the kids are already going to school for three weeks
2024-09-26 16:19:48 <@patrikp:matrix.org> Moving on then.
2024-09-26 16:19:56 <@patrikp:matrix.org> !info Go over existing items and fix them
2024-09-26 16:19:56 <@patrikp:matrix.org> !info https://nagios.fedoraproject.org/nagios 
2024-09-26 16:19:56 <@patrikp:matrix.org> !topic Monitoring discussion [nirik]
2024-09-26 16:21:40 <@nirik:matrix.scrye.com> lets see.
2024-09-26 16:21:59 <@nirik:matrix.scrye.com> 2 aarch64 machines down with bad memory.
2024-09-26 16:22:26 <@Zlopez:matrix.org> I see only one in nagios
2024-09-26 16:22:35 <@nirik:matrix.scrye.com> pagure02 close to disk alerting... usually this can be fixed up by nuking the autogenerated releases made for forks...
2024-09-26 16:22:53 <@nirik:matrix.scrye.com> buildhw-a64-23.iad2.fedoraproject.org  and openqa-a64-worker01.iad2.fedoraproject.org
2024-09-26 16:23:28 <@nirik:matrix.scrye.com> openqa-p09-worker02.iad2.fedoraproject.org is alerting on /boot space... this machine just needs reinstalled to make /boot bigger sometime.
2024-09-26 16:23:30 <@Zlopez:matrix.org> Oh, I was looking at the services
2024-09-26 16:25:07 <@nirik:matrix.scrye.com> thats all I see.. several things got cleared this last week.
2024-09-26 16:25:17 <@nirik:matrix.scrye.com> the planet messages are working now. kudos phsmoura
2024-09-26 16:25:31 <@nirik:matrix.scrye.com> and a drive was replaced that was bad. kudos David Kirwan
2024-09-26 16:26:22 <@nirik:matrix.scrye.com> Oh, one more monitoring note:
2024-09-26 16:26:48 <@nirik:matrix.scrye.com> We now have staging zabiix sending to a staging room and production to another room...
2024-09-26 16:27:02 <@nirik:matrix.scrye.com> it's very noisy and we need to fix that, but if you want to see what zabbix is alerting on you can join those.
2024-09-26 16:27:12 <@nirik:matrix.scrye.com> #noc-zabbix-testing:fedora.im
2024-09-26 16:27:29 <@Zlopez:matrix.org> Already joined
2024-09-26 16:27:51 <@nirik:matrix.scrye.com> The staging one I think we should move. it's in the staging bots room right now.
2024-09-26 16:28:49 <@nirik:matrix.scrye.com> thats all on monitoring.
2024-09-26 16:29:42 <@patrikp:matrix.org> Alright, next topic. Now I'm a bit confused as to what should be next. Looks like we don't have a learning topic for this week.
2024-09-26 16:30:02 <@patrikp:matrix.org> RHEL7 EOL? Blocked tickets?
2024-09-26 16:30:02 <@Zlopez:matrix.org> I don't think we have
2024-09-26 16:30:03 <@nirik:matrix.scrye.com> yeah, we usually decide. Or a while back we used to alternate.
2024-09-26 16:30:32 <@Zlopez:matrix.org> We didn't do a Backlog refinement for some time
2024-09-26 16:30:51 <@nirik:matrix.scrye.com> sure, I'm game to do whichever...
2024-09-26 16:31:02 <@patrikp:matrix.org> !info Refine oldest tickets on Fedora Infra tracker
2024-09-26 16:31:02 <@patrikp:matrix.org> !topic Fedora Infra backlog refinement
2024-09-26 16:31:02 <@patrikp:matrix.org> !link https://pagure.io/fedora-infrastructure/issues?status=Open&order_key=last_updated&order=asc
2024-09-26 16:32:08 <@nirik:matrix.scrye.com> Zlopez: you want to run things? or patrikp? we usually go one by one and discuss.
2024-09-26 16:32:23 <@Zlopez:matrix.org> I can run it
2024-09-26 16:32:29 <@Zlopez:matrix.org> !ticket 10458
2024-09-26 16:32:31 <@zodbot:fedora.im> ● **Last Updated:** 9 months ago
2024-09-26 16:32:31 <@zodbot:fedora.im> **fedora-infrastructure #10458** (https://pagure.io/fedora-infrastructure/issue/10458):**Help me move my discourse bots to production?**
2024-09-26 16:32:31 <@zodbot:fedora.im> 
2024-09-26 16:32:31 <@zodbot:fedora.im> ● **Opened:** 2 years ago by mattdm
2024-09-26 16:32:31 <@zodbot:fedora.im> ● **Assignee:** Not Assigned
2024-09-26 16:32:49 <@Zlopez:matrix.org> This one is waiting for mattdm
2024-09-26 16:32:51 <@nirik:matrix.scrye.com> So, this is on my list of 'projects looking for people' that I intend to send today
2024-09-26 16:33:07 <@nirik:matrix.scrye.com> I'd like to get someone on our side that can 'own' it and nag him about it and get the info, etc.
2024-09-26 16:33:36 <@Zlopez:matrix.org> That sounds good, I think we have a few tickets like that
2024-09-26 16:34:20 <@Zlopez:matrix.org> !ticket 12204
2024-09-26 16:34:21 <@nirik:matrix.scrye.com> yep.
2024-09-26 16:34:22 <@zodbot:fedora.im> **fedora-infrastructure #12204** (https://pagure.io/fedora-infrastructure/issue/12204):**buildvm-x86-05.iad2.fedoraproject.org: Read-only file system: '/var/tmp/koji/tasks/2642/123992642'**
2024-09-26 16:34:22 <@zodbot:fedora.im> 
2024-09-26 16:34:22 <@zodbot:fedora.im> ● **Opened:** 6 hours ago by ppisar
2024-09-26 16:34:22 <@zodbot:fedora.im> ● **Last Updated:** 6 hours ago
2024-09-26 16:34:22 <@zodbot:fedora.im> ● **Assignee:** Not Assigned
2024-09-26 16:34:29 <@Zlopez:matrix.org> Wrong one
2024-09-26 16:34:46 <@Zlopez:matrix.org> !ticket 11013
2024-09-26 16:34:47 <@zodbot:fedora.im> 
2024-09-26 16:34:47 <@zodbot:fedora.im> ● **Opened:** a year ago by adamwill
2024-09-26 16:34:47 <@zodbot:fedora.im> ● **Last Updated:** 7 months ago
2024-09-26 16:34:47 <@zodbot:fedora.im> **fedora-infrastructure #11013** (https://pagure.io/fedora-infrastructure/issue/11013):**Redeploy openQA workers with consistent storage configuration**
2024-09-26 16:34:47 <@zodbot:fedora.im> ● **Assignee:** Not Assigned
2024-09-26 16:34:47 <@nirik:matrix.scrye.com> so many tickets. ;)
2024-09-26 16:35:19 <@nirik:matrix.scrye.com> oh yeah, we should ask adamwill about this and when we can do it...
2024-09-26 16:35:28 <@nirik:matrix.scrye.com> I also want to upgrade the qa vmhosts.
2024-09-26 16:35:45 <@nirik:matrix.scrye.com> I am not sure we have the desired configuration decided yet tho.
2024-09-26 16:35:51 <@Zlopez:matrix.org> It's marked as low-trouble and sitting there for more than 2 years :/
2024-09-26 16:36:24 <@nirik:matrix.scrye.com> yeah. people or machines are always busy. ;(
2024-09-26 16:37:22 <@Zlopez:matrix.org> It seems that adamwill did some investigation around that
2024-09-26 16:37:34 <@nirik:matrix.scrye.com> how about we try and schedule it for the 8th/9th... then we have a goal. ;)
2024-09-26 16:37:41 <@nirik:matrix.scrye.com> (the week before final freeze starts)
2024-09-26 16:38:26 <@nirik:matrix.scrye.com> or at least propose that in the ticket. ;)
2024-09-26 16:38:47 <@nirik:matrix.scrye.com> it will probibly need to be announced since updates won't flow without gating tests.
2024-09-26 16:39:12 <@Zlopez:matrix.org> That sounds like good idea
2024-09-26 16:39:44 <@nirik:matrix.scrye.com> shall I update the ticket? or you want to?
2024-09-26 16:40:54 <@Zlopez:matrix.org> Could you update it, it seems you know what you want to do
2024-09-26 16:41:02 <@Zlopez:matrix.org> Could you update it? It seems you know what you want to do
2024-09-26 16:41:21 <@nirik:matrix.scrye.com> yep. I can.
2024-09-26 16:41:54 <@Zlopez:matrix.org> Next ticket
2024-09-26 16:42:01 <@Zlopez:matrix.org> !ticket 11512
2024-09-26 16:42:03 <@zodbot:fedora.im> ● **Assignee:** Not Assigned
2024-09-26 16:42:03 <@zodbot:fedora.im> ● **Last Updated:** 7 months ago
2024-09-26 16:42:03 <@zodbot:fedora.im> ● **Opened:** a year ago by kevin
2024-09-26 16:42:03 <@zodbot:fedora.im> 
2024-09-26 16:42:03 <@zodbot:fedora.im> **fedora-infrastructure #11512** (https://pagure.io/fedora-infrastructure/issue/11512):**container registry pruning**
2024-09-26 16:42:50 <@Zlopez:matrix.org> Should we still consider this being an issue when most of our container workflow is ending up in quay.io?
2024-09-26 16:43:21 <@nirik:matrix.scrye.com> well... it's a short term problem still for us...
2024-09-26 16:43:30 <@nirik:matrix.scrye.com> let me get the graph
2024-09-26 16:44:18 <@nirik:matrix.scrye.com> I mean, it could be just from more use and perhaps we should just not worry about it too much
2024-09-26 16:44:47 <@nirik:matrix.scrye.com> There was someone who said they wanted to look at it, but I didn't see any updates from them
2024-09-26 16:45:30 <@nirik:matrix.scrye.com> so, I guess I'm ok just closing it... but we should try and push forward on the quay.io move.
2024-09-26 16:45:46 <@Zlopez:matrix.org> Probably didn't have spare time to look into it
2024-09-26 16:45:56 <@james:fedora.im> What is the rough netapp limit to expand to?
2024-09-26 16:47:41 <@nirik:matrix.scrye.com> looks like there's 20T more on that aggregate, but of course we couldn't allocate it all.
2024-09-26 16:47:51 <@nirik:matrix.scrye.com> it gets mad if you go over 90% usage
2024-09-26 16:48:05 <@nirik:matrix.scrye.com> in any case, there's still room.
2024-09-26 16:48:35 <@Zlopez:matrix.org> Let's wait with closing that till flatpaks are moved to quay.io
2024-09-26 16:48:43 <@james:fedora.im> yeh, I'd just worry about it more if it was 10T and a lot less at ~2 years of expansion.
2024-09-26 16:49:35 <@nirik:matrix.scrye.com> I think now we should just close it...
2024-09-26 16:50:00 <@nirik:matrix.scrye.com> FYI, there's 459 fedora flatpaks now
2024-09-26 16:50:27 <@Zlopez:matrix.org> Much more than I thought 👍️
2024-09-26 16:50:34 <@Zlopez:matrix.org> Let me close it than
2024-09-26 16:50:52 <@nirik:matrix.scrye.com> yselkowitz is a flatpak building superstar. ;)
2024-09-26 16:51:42 <@Zlopez:matrix.org> I tried to build one in past, but I hit some roadblock
2024-09-26 16:51:55 <@Zlopez:matrix.org> I still have 2 flatpaks on flathub
2024-09-26 16:52:37 <@patrikp:matrix.org> Alright, less than 8 minutes to go. You want to do one more or shall we go to open floor?
2024-09-26 16:52:48 <@Zlopez:matrix.org> Issue closed as Will not/Can not fix
2024-09-26 16:53:11 <@Zlopez:matrix.org> @nirik:matrix.scrye.com One more?
2024-09-26 16:53:28 <@nirik:matrix.scrye.com> sure
2024-09-26 16:53:52 <@Zlopez:matrix.org> !ticket 11807
2024-09-26 16:53:53 <@zodbot:fedora.im> ● **Opened:** 7 months ago by kevin
2024-09-26 16:53:53 <@zodbot:fedora.im> ● **Assignee:** Not Assigned
2024-09-26 16:53:53 <@zodbot:fedora.im> ● **Last Updated:** 7 months ago
2024-09-26 16:53:53 <@zodbot:fedora.im> **fedora-infrastructure #11807** (https://pagure.io/fedora-infrastructure/issue/11807):**move resultsdb-ci-listener deployment pipeline**
2024-09-26 16:53:53 <@zodbot:fedora.im> 
2024-09-26 16:54:18 <@Zlopez:matrix.org> I think this one just fall through the cracks
2024-09-26 16:54:30 <@Zlopez:matrix.org> As it seems easy to do
2024-09-26 16:55:22 <@nirik:matrix.scrye.com> yeah, someone just needs to do it.
2024-09-26 16:56:04 <@Zlopez:matrix.org> I will share it on monday standup as it's a good way to learn how the s2i in openshift works and it's easy to do
2024-09-26 16:56:32 <@nirik:matrix.scrye.com> 👍
2024-09-26 16:57:06 <@Zlopez:matrix.org> Or if anybody here want's to take it feel free I can guide you through it
2024-09-26 16:57:16 <@Zlopez:matrix.org> Or if anybody here wants to take it feel free I can guide you through it
2024-09-26 16:57:32 <@patrikp:matrix.org> I will assign it to myself.
2024-09-26 16:58:09 <@patrikp:matrix.org> And with that, we are pretty much at time. Any closing thoughts?
2024-09-26 16:58:29 <@Zlopez:matrix.org> Nice work with the tickets today :-)
2024-09-26 16:59:16 <@patrikp:matrix.org> I suppose that's it then. Have a good rest of the day everyone!
2024-09-26 16:59:21 <@patrikp:matrix.org> !endmeeting