<@phsmoura:fedora.im>
19:01:30
!startmeeting Fedora Infrastructure Ops Daily Standup Meeting
<@meetbot:fedora.im>
19:01:31
Meeting started at 2025-03-24 19:01:30 UTC
<@meetbot:fedora.im>
19:01:31
The Meeting name is 'Fedora Infrastructure Ops Daily Standup Meeting'
<@smilner:fedora.im>
19:01:33
!hi
<@phsmoura:fedora.im>
19:01:35
!info agenda is at https://board.net/p/fedora-infra-daily
<@phsmoura:fedora.im>
19:01:35
!info reminder: speak up if you want to work on a ticket!
<@phsmoura:fedora.im>
19:01:35
!chair nirik phsmoura james
<@phsmoura:fedora.im>
19:01:35
!meetingname fedora_infrastructure_ops_daily_standup_meeting
<@phsmoura:fedora.im>
19:01:35
!info meeting is 30 minutes MAX. At the end of 30, its stops
<@meetbot:fedora.im>
19:01:35
The Meeting Name is now fedora_infrastructure_ops_daily_standup_meeting
<@zodbot:fedora.im>
19:01:36
None (smilner)
<@phsmoura:fedora.im>
19:01:37
hello
<@nirik:matrix.scrye.com>
19:01:52
morning
<@james:fedora.im>
19:01:55
!hi
<@zodbot:fedora.im>
19:01:56
James Antill (james)
<@phsmoura:fedora.im>
19:02:38
!topic Tickets needing review
<@phsmoura:fedora.im>
19:02:38
!info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1
<@james:fedora.im>
19:02:51
Steve: If you edit: https://accounts.fedoraproject.org/user/smilner/ ... you'll get a name instead of None ;)
<@phsmoura:fedora.im>
19:03:08
!ticket 12457
<@zodbot:fedora.im>
19:03:08
**fedora-infrastructure #12457** (https://pagure.io/fedora-infrastructure/issue/12457):**User samcday cannot be added as co-maintainer**
<@zodbot:fedora.im>
19:03:08
<@zodbot:fedora.im>
19:03:08
● **Opened:** 6 hours ago by samcday
<@zodbot:fedora.im>
19:03:08
● **Last Updated:** 15 minutes ago
<@zodbot:fedora.im>
19:03:08
● **Assignee:** Not Assigned
<@smilner:fedora.im>
19:03:17
Fair :-) I'll do that!
<@nirik:matrix.scrye.com>
19:03:52
I think it's the 'private' setting.
<@nirik:matrix.scrye.com>
19:04:06
so, on this one, it seems like it might be solved, but we aren't sure... low low ops?
<@james:fedora.im>
19:04:12
low / low
<@james:fedora.im>
19:04:46
yeh, we both spent some time on it and then it maybe fixed itself ... or due to logouts. But seems good
<@nirik:matrix.scrye.com>
19:05:50
yeah, was puzzling, but perhaps the login fixed it...
<@phsmoura:fedora.im>
19:06:52
!ticket 12458
<@zodbot:fedora.im>
19:06:54
● **Assignee:** Not Assigned
<@zodbot:fedora.im>
19:06:54
**fedora-infrastructure #12458** (https://pagure.io/fedora-infrastructure/issue/12458):**Fedora Copr Outage - Updating to a March 2025 Version**
<@zodbot:fedora.im>
19:06:54
<@zodbot:fedora.im>
19:06:54
● **Opened:** 49 minutes ago by nikromen
<@zodbot:fedora.im>
19:06:54
● **Last Updated:** 47 minutes ago
<@nirik:matrix.scrye.com>
19:07:14
med med outage I guess... this is just tracking for the copr team
<@smilner:fedora.im>
19:07:19
Looks like an FYI?
<@smilner:fedora.im>
19:07:27
ah
<@nirik:matrix.scrye.com>
19:08:25
yeah, mostly so
<@phsmoura:fedora.im>
19:08:49
!topic Planning, Upcoming work and Open floor
<@nirik:matrix.scrye.com>
19:09:22
!info mass update/reboot cycle this week
<@nirik:matrix.scrye.com>
19:09:46
That and DC move planning some more are on my plate this week...
<@nirik:matrix.scrye.com>
19:10:23
seems like there's been a lot of mailman alerts. Zlopez was looking into it some...
<@nirik:matrix.scrye.com>
19:10:54
and we had a bit of a scare friday/over the weekend on the fedora_koji volume. It got near 100% due to them snapmirroring it to rdu3
<@james:fedora.im>
19:11:01
Yeh, was planning on turning nftables on for everything in staging with the update/reboot of staging.
<@nirik:matrix.scrye.com>
19:11:07
but it's happy now.
<@nirik:matrix.scrye.com>
19:11:17
ok, sounds good to me.
<@nirik:matrix.scrye.com>
19:11:35
we can hold on doing any staging until we merge a pr to flip it over?
<@smilner:fedora.im>
19:13:26
Is that for James Antill?
<@nirik:matrix.scrye.com>
19:13:49
the happy now was the fedora_koji volume. ;) The rest was for jame's nftables comment
<@nirik:matrix.scrye.com>
19:14:00
sorry for mixing things.
<@james:fedora.im>
19:14:06
yeh, I just pushed the ansible change and will open the PR in about 30s
<@nirik:matrix.scrye.com>
19:14:17
great!
<@nirik:matrix.scrye.com>
19:15:02
phsmoura: I'm planning to update our openshift clusters later... I doubt it will have any effect on those crashing pods we have seen, but just FYI in case it does.
<@phsmoura:fedora.im>
19:16:36
ok, I replied you on noc ... I can help with that
<@nirik:matrix.scrye.com>
19:16:36
I'd also like to discuss the AI scraper scourge at the meeting on thursday... just FYI in case you want to gather any thoughts or ideas on it before then
<@james:fedora.im>
19:17:13
PR made. I did want to ask about the rough time tomorrow. I know we usually do things at roughly the same time we'll start on Wed. ... that the plan this week?
<@nirik:matrix.scrye.com>
19:17:17
phsmoura: on stg updates? yeah... can get you added/up to speed over there after the meeting
<@nirik:matrix.scrye.com>
19:18:26
monday (staging only) and tuesday (things we can do without anyone noticing) we can do anytime we like pretty much. I tend to do them later in the day because then I am all caught up on emails, fires, etc... but if you want to do them eariler thats 100% fine with me.
<@nirik:matrix.scrye.com>
19:18:46
the wed ones need to be in the outage window as people may notice things there.
<@james:fedora.im>
19:19:53
yeh, okay. I'm fine with the randomness ... I might drop something into the noc channel too as I do things, so that people aren't confused.
<@james:fedora.im>
19:20:13
Obviously updating the hackmd doc. too
<@nirik:matrix.scrye.com>
19:21:32
yeah, +1 to also noting things in noc... just in case.
<@nirik:matrix.scrye.com>
19:21:38
and of course let me know if something blows up.
<@nirik:matrix.scrye.com>
19:21:54
I didn't have much else off hand
<@james:fedora.im>
19:22:13
As a last question from me on it ... I assume we aren't doing any moves to rhel9 this week?
<@nirik:matrix.scrye.com>
19:22:40
nope. I don't think we can move much else. let me check my tally.
<@nirik:matrix.scrye.com>
19:23:03
I was considering moving all the prod proxies to f41 on wed... might do that
<@james:fedora.im>
19:24:16
The big thing I thought was moving something to a different virt server so we could change something to rhel9 (batcave01 or log01)?
<@james:fedora.im>
19:24:46
But you hadn't said anything about it, so assumed not this week. And I'm not sure how big a problem migrating to a new virt server is
<@nirik:matrix.scrye.com>
19:25:07
So, there's still 2 things we could possibly move: value02... but we should move the stg version of it first...
<@nirik:matrix.scrye.com>
19:25:33
and log01... thats going to be a mess because we would need to move all the logs off it and reinstall and move them back... or archive them
<@nirik:matrix.scrye.com>
19:26:13
I think you are thinking of the thing the last outage or two back where I needed to move batcave01 and bastion01 vm's off vmhost-x86-01 so I could reinstall it. But I did that last outage, so it's done. ;)
<@james:fedora.im>
19:26:47
Ahh, Yeh it was probably that.
<@nirik:matrix.scrye.com>
19:28:05
The only other 8.10 rhel's left are: pagure (4), and rabbitmq (6) and copr vmhosts. pagure we can't do because it isn't packaged for rhel9... rabbitmq I am just going to defer to the datacenter move and install rhel9 in the new dc. copr vmhosts we could do sometime, but they might be easier in rdu3-cc once that move happens.
<@nirik:matrix.scrye.com>
19:29:21
log01 probibly would be best to defer also and just install the new one in rdu3 with rhel9
<@james:fedora.im>
19:29:43
Yeh, that sounds less painful.
<@james:fedora.im>
19:30:18
*looks for the ticket* ... Want me to update https://pagure.io/fedora-infrastructure/issue/10632 with the above info?
<@james:fedora.im>
19:30:39
Also we are at time ;)
<@phsmoura:fedora.im>
19:31:44
yes, we are... anything else we can continue on noc
<@nirik:matrix.scrye.com>
19:32:03
James Antill: sure, or I can...
<@phsmoura:fedora.im>
19:32:13
thanks everyone :)
<@phsmoura:fedora.im>
19:32:23
!endmeeting