2025-03-24 19:01:30 <@phsmoura:fedora.im> !startmeeting Fedora Infrastructure Ops Daily Standup Meeting
2025-03-24 19:01:31 <@meetbot:fedora.im> Meeting started at 2025-03-24 19:01:30 UTC
2025-03-24 19:01:31 <@meetbot:fedora.im> The Meeting name is 'Fedora Infrastructure Ops Daily Standup Meeting'
2025-03-24 19:01:33 <@smilner:fedora.im> !hi
2025-03-24 19:01:35 <@phsmoura:fedora.im> !info agenda is at https://board.net/p/fedora-infra-daily
2025-03-24 19:01:35 <@phsmoura:fedora.im> !info reminder: speak up if you want to work on a ticket!
2025-03-24 19:01:35 <@phsmoura:fedora.im> !chair nirik phsmoura james
2025-03-24 19:01:35 <@phsmoura:fedora.im> !meetingname fedora_infrastructure_ops_daily_standup_meeting
2025-03-24 19:01:35 <@phsmoura:fedora.im> !info meeting is 30 minutes MAX. At the end of 30, its stops
2025-03-24 19:01:35 <@meetbot:fedora.im> The Meeting Name is now fedora_infrastructure_ops_daily_standup_meeting
2025-03-24 19:01:36 <@zodbot:fedora.im> None (smilner)
2025-03-24 19:01:37 <@phsmoura:fedora.im> hello
2025-03-24 19:01:52 <@nirik:matrix.scrye.com> morning
2025-03-24 19:01:55 <@james:fedora.im> !hi
2025-03-24 19:01:56 <@zodbot:fedora.im> James Antill (james)
2025-03-24 19:02:38 <@phsmoura:fedora.im> !topic Tickets needing review
2025-03-24 19:02:38 <@phsmoura:fedora.im> !info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1
2025-03-24 19:02:51 <@james:fedora.im> Steve: If you edit: https://accounts.fedoraproject.org/user/smilner/ ... you'll get a name instead of None ;)
2025-03-24 19:03:08 <@phsmoura:fedora.im> !ticket 12457
2025-03-24 19:03:08 <@zodbot:fedora.im> **fedora-infrastructure #12457** (https://pagure.io/fedora-infrastructure/issue/12457):**User samcday cannot be added as co-maintainer**
2025-03-24 19:03:08 <@zodbot:fedora.im> 
2025-03-24 19:03:08 <@zodbot:fedora.im> ● **Opened:** 6 hours ago by samcday
2025-03-24 19:03:08 <@zodbot:fedora.im> ● **Last Updated:** 15 minutes ago
2025-03-24 19:03:08 <@zodbot:fedora.im> ● **Assignee:** Not Assigned
2025-03-24 19:03:17 <@smilner:fedora.im> Fair :-) I'll do that!
2025-03-24 19:03:52 <@nirik:matrix.scrye.com> I think it's the 'private' setting.
2025-03-24 19:04:06 <@nirik:matrix.scrye.com> so, on this one, it seems like it might be solved, but we aren't sure... low low ops?
2025-03-24 19:04:12 <@james:fedora.im> low / low
2025-03-24 19:04:46 <@james:fedora.im> yeh, we both spent some time on it and then it maybe fixed itself ... or due to logouts. But seems good
2025-03-24 19:05:50 <@nirik:matrix.scrye.com> yeah, was puzzling, but perhaps the login fixed it...
2025-03-24 19:06:52 <@phsmoura:fedora.im> !ticket 12458
2025-03-24 19:06:54 <@zodbot:fedora.im> ● **Assignee:** Not Assigned
2025-03-24 19:06:54 <@zodbot:fedora.im> **fedora-infrastructure #12458** (https://pagure.io/fedora-infrastructure/issue/12458):**Fedora Copr Outage - Updating to a March 2025 Version**
2025-03-24 19:06:54 <@zodbot:fedora.im> 
2025-03-24 19:06:54 <@zodbot:fedora.im> ● **Opened:** 49 minutes ago by nikromen
2025-03-24 19:06:54 <@zodbot:fedora.im> ● **Last Updated:** 47 minutes ago
2025-03-24 19:07:14 <@nirik:matrix.scrye.com> med med outage I guess... this is just tracking for the copr team
2025-03-24 19:07:19 <@smilner:fedora.im> Looks like an FYI?
2025-03-24 19:07:27 <@smilner:fedora.im> ah
2025-03-24 19:08:25 <@nirik:matrix.scrye.com> yeah, mostly so
2025-03-24 19:08:49 <@phsmoura:fedora.im> !topic Planning, Upcoming work and Open floor
2025-03-24 19:09:22 <@nirik:matrix.scrye.com> !info mass update/reboot cycle this week
2025-03-24 19:09:46 <@nirik:matrix.scrye.com> That and DC move planning some more are on my plate this week...
2025-03-24 19:10:23 <@nirik:matrix.scrye.com> seems like there's been a lot of mailman alerts. Zlopez was looking into it some...
2025-03-24 19:10:54 <@nirik:matrix.scrye.com> and we had a bit of a scare friday/over the weekend on the fedora_koji volume. It got near 100% due to them snapmirroring it to rdu3
2025-03-24 19:11:01 <@james:fedora.im> Yeh, was planning on turning nftables on for everything in staging with the update/reboot of staging.
2025-03-24 19:11:07 <@nirik:matrix.scrye.com> but it's happy now.
2025-03-24 19:11:17 <@nirik:matrix.scrye.com> ok, sounds good to me.
2025-03-24 19:11:35 <@nirik:matrix.scrye.com> we can hold on doing any staging until we merge a pr to flip it over?
2025-03-24 19:13:26 <@smilner:fedora.im> Is that for James Antill? 
2025-03-24 19:13:49 <@nirik:matrix.scrye.com> the happy now was the fedora_koji volume. ;) The rest was for jame's nftables comment
2025-03-24 19:14:00 <@nirik:matrix.scrye.com> sorry for mixing things.
2025-03-24 19:14:06 <@james:fedora.im> yeh, I just pushed the ansible change and will open the PR in about 30s
2025-03-24 19:14:17 <@nirik:matrix.scrye.com> great!
2025-03-24 19:15:02 <@nirik:matrix.scrye.com> phsmoura: I'm planning to update our openshift clusters later... I doubt it will have any effect on those crashing pods we have seen, but just FYI in case it does.
2025-03-24 19:16:36 <@phsmoura:fedora.im> ok, I replied you on noc ... I can help with that
2025-03-24 19:16:36 <@nirik:matrix.scrye.com> I'd also like to discuss the AI scraper scourge at the meeting on thursday... just FYI in case you want to gather any thoughts or ideas on it before then
2025-03-24 19:17:13 <@james:fedora.im> PR made. I did want to ask about the rough time tomorrow. I know we usually do things at roughly the same time we'll start on Wed. ... that the plan this week?
2025-03-24 19:17:17 <@nirik:matrix.scrye.com> phsmoura: on stg updates? yeah... can get you added/up to speed over there after the meeting
2025-03-24 19:18:26 <@nirik:matrix.scrye.com> monday (staging only) and tuesday (things we can do without anyone noticing) we can do anytime we like pretty much. I tend to do them later in the day because then I am all caught up on emails, fires, etc... but if you want to do them eariler thats 100% fine with me.
2025-03-24 19:18:46 <@nirik:matrix.scrye.com> the wed ones need to be in the outage window as people may notice things there.
2025-03-24 19:19:53 <@james:fedora.im> yeh, okay. I'm fine with the randomness ... I might drop something into the noc channel too as I do things, so that people aren't confused.
2025-03-24 19:20:13 <@james:fedora.im> Obviously updating the hackmd doc. too
2025-03-24 19:21:32 <@nirik:matrix.scrye.com> yeah, +1 to also noting things in noc... just in case.
2025-03-24 19:21:38 <@nirik:matrix.scrye.com> and of course let me know if something blows up.
2025-03-24 19:21:54 <@nirik:matrix.scrye.com> I didn't have much else off hand
2025-03-24 19:22:13 <@james:fedora.im> As a last question from me on it ... I assume we aren't doing any moves to rhel9 this week?
2025-03-24 19:22:40 <@nirik:matrix.scrye.com> nope. I don't think we can move much else. let me check my tally.
2025-03-24 19:23:03 <@nirik:matrix.scrye.com> I was considering moving all the prod proxies to f41 on wed... might do that
2025-03-24 19:24:16 <@james:fedora.im> The big thing I thought was moving something to a different virt server so we could change something to rhel9 (batcave01 or log01)?
2025-03-24 19:24:46 <@james:fedora.im> But you hadn't said anything about it, so assumed not this week. And I'm not sure how big a problem migrating to a new virt server is
2025-03-24 19:25:07 <@nirik:matrix.scrye.com> So, there's still 2 things we could possibly move: value02... but we should move the stg version of it first...
2025-03-24 19:25:33 <@nirik:matrix.scrye.com> and log01... thats going to be a mess because we would need to move all the logs off it and reinstall and move them back... or archive them
2025-03-24 19:26:13 <@nirik:matrix.scrye.com> I think you are thinking of the thing the last outage or two back where I needed to move batcave01 and bastion01 vm's off vmhost-x86-01 so I could reinstall it. But I did that last outage, so it's done. ;)
2025-03-24 19:26:47 <@james:fedora.im> Ahh, Yeh it was probably that.
2025-03-24 19:28:05 <@nirik:matrix.scrye.com> The only other 8.10 rhel's left are: pagure (4), and rabbitmq (6) and copr vmhosts. pagure we can't do because it isn't packaged for rhel9... rabbitmq I am just going to defer to the datacenter move and install rhel9 in the new dc. copr vmhosts we could do sometime, but they might be easier in rdu3-cc once that move happens.
2025-03-24 19:29:21 <@nirik:matrix.scrye.com> log01 probibly would be best to defer also and just install the new one in rdu3 with rhel9
2025-03-24 19:29:43 <@james:fedora.im> Yeh, that sounds less painful.
2025-03-24 19:30:18 <@james:fedora.im> *looks for the ticket* ... Want me to update https://pagure.io/fedora-infrastructure/issue/10632 with the above info?
2025-03-24 19:30:39 <@james:fedora.im> Also we are at time ;)
2025-03-24 19:31:44 <@phsmoura:fedora.im> yes, we are... anything else we can continue on noc
2025-03-24 19:32:03 <@nirik:matrix.scrye.com> James Antill: sure, or I can...
2025-03-24 19:32:13 <@phsmoura:fedora.im> thanks everyone :)
2025-03-24 19:32:23 <@phsmoura:fedora.im> !endmeeting