2024-10-22 19:04:18 <@james:fedora.im> !startmeeting Fedora Infrastructure Ops Daily Standup Meeting

2024-10-22 19:04:19 <@meetbot:fedora.im> Meeting started at 2024-10-22 19:04:18 UTC
2024-10-22 19:04:19 <@meetbot:fedora.im> The Meeting name is 'Fedora Infrastructure Ops Daily Standup Meeting'
2024-10-22 19:04:22 <@james:fedora.im> !chair nirik smooge phsmoura
2024-10-22 19:04:25 <@james:fedora.im> !meetingname fedora_infrastructure_ops_daily_standup_meeting
2024-10-22 19:04:26 <@meetbot:fedora.im> The Meeting Name is now fedora_infrastructure_ops_daily_standup_meeting
2024-10-22 19:04:27 <@phsmoura:fedora.im> hello
2024-10-22 19:04:28 <@james:fedora.im> !info meeting is 30 minutes MAX. At the end of 30, its stops
2024-10-22 19:04:32 <@james:fedora.im> !info agenda is at https://board.net/p/fedora-infra-daily
2024-10-22 19:04:36 <@james:fedora.im> !info reminder: speak up if you want to work on a ticket!
2024-10-22 19:04:42 <@james:fedora.im> !hi
2024-10-22 19:04:43 <@zodbot:fedora.im> James Antill (james)
2024-10-22 19:04:47 <@james:fedora.im> !topic Tickets needing review
2024-10-22 19:04:54 <@james:fedora.im> !info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1
2024-10-22 19:05:15 <@james:fedora.im> !info https://pagure.io/releng/issues?status=Open
2024-10-22 19:05:28 <@nirik:matrix.scrye.com> morning
2024-10-22 19:05:41 <@james:fedora.im> .releng 12412
2024-10-22 19:05:46 <@james:fedora.im> !releng #12412
2024-10-22 19:05:54 <@james:fedora.im> !releng 12412
2024-10-22 19:05:54 <@zodbot:fedora.im> ● **Assignee:** patrikp
2024-10-22 19:05:54 <@zodbot:fedora.im> **releng #12412** (https://pagure.io/releng/issue/12412):** Unretirement request: rust-leb128**
2024-10-22 19:05:54 <@zodbot:fedora.im> 
2024-10-22 19:05:54 <@zodbot:fedora.im> ● **Opened:** 6 hours ago by decathorpe
2024-10-22 19:05:54 <@zodbot:fedora.im> ● **Last Updated:** 6 hours ago
2024-10-22 19:06:03 <@nirik:matrix.scrye.com> low low...
2024-10-22 19:06:20 <@james:fedora.im> !releng 12413
2024-10-22 19:06:21 <@zodbot:fedora.im> ● **Last Updated:** Never
2024-10-22 19:06:21 <@zodbot:fedora.im> **releng #12413** (https://pagure.io/releng/issue/12413):**Stalled EPEL package: adobe-source-code-pro-fonts**
2024-10-22 19:06:21 <@zodbot:fedora.im> 
2024-10-22 19:06:21 <@zodbot:fedora.im> ● **Opened:** 2 hours ago by dcavalca
2024-10-22 19:06:21 <@zodbot:fedora.im> ● **Assignee:** Not Assigned
2024-10-22 19:06:47 <@nirik:matrix.scrye.com> low low also
2024-10-22 19:07:16 <@james:fedora.im> !releng 12398
2024-10-22 19:07:20 <@zodbot:fedora.im> ● **Assignee:** Not Assigned
2024-10-22 19:07:20 <@zodbot:fedora.im> **releng #12398** (https://pagure.io/releng/issue/12398):**Make ppc64le Server_KVM image build not take two hours**
2024-10-22 19:07:20 <@zodbot:fedora.im> 
2024-10-22 19:07:20 <@zodbot:fedora.im> ● **Opened:** 5 days ago by adamwill
2024-10-22 19:07:20 <@zodbot:fedora.im> ● **Last Updated:** 20 hours ago
2024-10-22 19:07:37 <@nirik:matrix.scrye.com> med med I suppose? there's some attempts
2024-10-22 19:08:53 <@james:fedora.im> !topic Planning, Upcoming work and Open floor
2024-10-22 19:09:36 <@james:fedora.im> I was wondering ... when we (or more accurately, you ;) reinstall a machine ... do we delete or archive the old logs on log01?
2024-10-22 19:10:20 <@nirik:matrix.scrye.com> nope... we do archive off all the hosts logs yearly... but I havent' been deleting or doing anything with old logs.
2024-10-22 19:10:33 <@nirik:matrix.scrye.com> I'm a bit of a packrat sadly... I like keeping things. ;)
2024-10-22 19:11:01 <@nirik:matrix.scrye.com> but yeah, log01 has been alerting for space the last few days... I looked at it and found some builders with fs errors, but I am not sure thats the main reason.
2024-10-22 19:11:10 <@james:fedora.im> Yeh, but we could keep some of the older ones somewhere else ... like on a part of the netapp or something?
2024-10-22 19:11:23 <@nirik:matrix.scrye.com> so if others would like to look and see if they can find other problem applications/hosts that would be great
2024-10-22 19:11:30 <@nirik:matrix.scrye.com> sure, we could
2024-10-22 19:12:07 <@james:fedora.im> Had a quick look, and "proxy*" seemed like something near the top of the list ... but that's not new, or a bug, AFAICS.
2024-10-22 19:12:31 <@nirik:matrix.scrye.com> There's a lot of messages from fedocal and fmn... but not sure it's more than normal.
2024-10-22 19:12:39 <@nirik:matrix.scrye.com> it's kinda hard to sort through
2024-10-22 19:13:04 <@james:fedora.im> ```
2024-10-22 19:13:04 <@james:fedora.im> ``` 85G ┣━ proxy101.iad2.fedoraproject.org/2024
2024-10-22 19:13:04 <@james:fedora.im>      ┃ ┖┄ [165,787 file(s)]
2024-10-22 19:13:04 <@james:fedora.im>  85G ┣━ proxy04.vpn.fedoraproject.org/2024
2024-10-22 19:13:04 <@james:fedora.im>      ┃ ┖┄ [153,719 file(s)]
2024-10-22 19:13:04 <@james:fedora.im>  88G ┣━ proxy110.iad2.fedoraproject.org/2024
2024-10-22 19:13:04 <@james:fedora.im>      ┃ ┖┄ [165,647 file(s)]
2024-10-22 19:13:04 <@james:fedora.im> 164G ┣━ proxy10.iad2.fedoraproject.org/2024
2024-10-22 19:13:04 <@james:fedora.im>      ┃ ┖┄ [169,637 file(s)]
2024-10-22 19:13:04 <@james:fedora.im> 167G ┗━ proxy01.iad2.fedoraproject.org/2024
2024-10-22 19:13:27 <@james:fedora.im> Those are hosts > 75GB.
2024-10-22 19:13:59 <@nirik:matrix.scrye.com> yeah... I was looking at /var/log/merged/ as it seems like the daily messages.log gets really big until it's compressed, so it alerts and then recovers.
2024-10-22 19:14:16 <@nirik:matrix.scrye.com> It could be particular hosts tho perhaps yeah
2024-10-22 19:15:30 <@james:fedora.im> Yeh, merged is like 12GB ... and hosts is 1.5TB :)
2024-10-22 19:17:05 <@nirik:matrix.scrye.com> so yeah, probibly we should just look at archiving the last 6 months or something?
2024-10-22 19:17:13 <@nirik:matrix.scrye.com> the oldest 6 months
2024-10-22 19:17:54 <@james:fedora.im> Yeh, I think if we just archived Jan-Jul for proxies we'd be fine. But maybe easier to do it for everyone.
2024-10-22 19:18:27 <@nirik:matrix.scrye.com> can you open a ticket on it? we can figure out the best plan and do it... I don't know it's super urgent, but the nightly alerts are anoying. ;)
2024-10-22 19:18:45 <@james:fedora.im> Sure, med/med ?
2024-10-22 19:18:52 <@nirik:matrix.scrye.com> sure
2024-10-22 19:20:18 <@nirik:matrix.scrye.com> Anything else ?
2024-10-22 19:20:58 <@nirik:matrix.scrye.com> Hoping that the current RC will last and we will have release next week. ;)
2024-10-22 19:21:00 <@nirik:matrix.scrye.com> we will see.
2024-10-22 19:22:48 <@phsmoura:fedora.im> I have some questions but will sent in internal channel since the doc is internal
2024-10-22 19:23:04 <@nirik:matrix.scrye.com> ah yeah, I plan to put that somewhere today...
2024-10-22 19:23:41 <@nirik:matrix.scrye.com> will do that after lunch. ;)
2024-10-22 19:24:14 <@james:fedora.im> Also ... can you send me the iptables script in private
2024-10-22 19:24:28 <@nirik:matrix.scrye.com> yeah
2024-10-22 19:25:44 <@nirik:matrix.scrye.com> I don't think I have anything else...
2024-10-22 19:27:20 <@james:fedora.im> Me either.
2024-10-22 19:28:02 <@james:fedora.im> !endmeeting