2025-04-17 17:01:31 <@Zlopez:matrix.org> !startmeeting Infrastructure (2025-04-17)
2025-04-17 17:01:38 <@meetbot:fedora.im> Meeting started at 2025-04-17 17:01:31 UTC
2025-04-17 17:01:39 <@meetbot:fedora.im> The Meeting name is 'Infrastructure (2025-04-17)'
2025-04-17 17:02:14 <@Zlopez:matrix.org> Where are you @meetbot:fedora.im
2025-04-17 17:02:15 <@nirik:matrix.scrye.com> morning
2025-04-17 17:02:21 <@Zlopez:matrix.org> Where are you @meetbot:fedora.im?
2025-04-17 17:02:22 <@nirik:matrix.scrye.com> slow?
2025-04-17 17:02:26 <@phsmoura:fedora.im> hello
2025-04-17 17:02:30 <@Zlopez:matrix.org> Yes
2025-04-17 17:02:41 <@markrosenbaum:fedora.im> Lol, Meet Bot awol
2025-04-17 17:03:19 <@nirik:matrix.scrye.com> hum
2025-04-17 17:03:27 <@Zlopez:matrix.org> I'm waiting for it to react at start of the meeting
2025-04-17 17:03:49 <@nirik:matrix.scrye.com> morning.
2025-04-17 17:03:56 <@Zlopez:matrix.org> !chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp
2025-04-17 17:03:56 <@Zlopez:matrix.org> !topic ahoy
2025-04-17 17:03:56 <@Zlopez:matrix.org> !info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
2025-04-17 17:03:56 <@Zlopez:matrix.org> !info About our team: https://docs.fedoraproject.org/en-US/cle/
2025-04-17 17:03:56 <@Zlopez:matrix.org> !info Agenda is at: https://board.net/p/fedora-infra
2025-04-17 17:03:56 <@Zlopez:matrix.org> !meetingname infrastructure
2025-04-17 17:03:56 <@nirik:matrix.scrye.com> I am gonna go grab coffee... back in a sec
2025-04-17 17:04:28 <@meetbot:fedora.im> The Meeting Name is now infrastructure
2025-04-17 17:04:42 <@Zlopez:matrix.org> It takes forever for the bot to react, but it worked fine in the morning
2025-04-17 17:04:46 <@Zlopez:matrix.org> !hi
2025-04-17 17:04:54 <@Zlopez:matrix.org> Welcome everybody to today infra meeting
2025-04-17 17:04:57 <@zodbot:fedora.im> Michal Konecny (zlopez)
2025-04-17 17:05:31 <@markrosenbaum:fedora.im> !hi
2025-04-17 17:05:32 <@zodbot:fedora.im> Mark Rosenbaum (markrosenbaum)
2025-04-17 17:06:59 <@Zlopez:matrix.org> I'm trying to look at the bot logs, to see if there is a reason for the slowness
2025-04-17 17:07:10 <@smilner:fedora.im> !hi
2025-04-17 17:07:11 <@zodbot:fedora.im> None (smilner)
2025-04-17 17:07:15 <@nirik:matrix.scrye.com> there's that crypto thing... but I think it might be unrelated.
2025-04-17 17:07:37 <@Zlopez:matrix.org> That was what I found last time
2025-04-17 17:07:49 <@Zlopez:matrix.org> !info Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/
2025-04-17 17:07:49 <@Zlopez:matrix.org> !topic New folks introductions
2025-04-17 17:07:49 <@Zlopez:matrix.org> !info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
2025-04-17 17:08:40 <@Zlopez:matrix.org> Do we have anybody new around?
2025-04-17 17:09:19 <@markrosenbaum:fedora.im> Hmmm, over at join I thought we had some people say they were infra interested but I could be wrong
2025-04-17 17:10:36 <@Zlopez:matrix.org> I saw somebody today in #admin:fedoraproject.org
2025-04-17 17:10:48 <@Zlopez:matrix.org> But it seems they aren't around for this meeting
2025-04-17 17:11:09 <@Zlopez:matrix.org> So let's move on
2025-04-17 17:11:15 <@nirik:matrix.scrye.com> timezones... pesky things
2025-04-17 17:11:19 <@Zlopez:matrix.org> !info chair 2025-04-24 - ???
2025-04-17 17:11:19 <@Zlopez:matrix.org> !topic Next chair
2025-04-17 17:11:19 <@Zlopez:matrix.org> !info magic eight ball says:
2025-04-17 17:11:19 <@Zlopez:matrix.org> !info chair 2025-04-17 - @Zlopez
2025-04-17 17:11:55 <@Zlopez:matrix.org> The last meeting was only me and @nirik:matrix.scrye.com (who was on Go/No go meeting at the same time)
2025-04-17 17:12:05 <@Zlopez:matrix.org> So I took this one as well
2025-04-17 17:12:15 <@Zlopez:matrix.org> Do we have volunteer for next one?
2025-04-17 17:12:51 <@nirik:matrix.scrye.com> I suppose I can if no one else is able.
2025-04-17 17:13:15 <@phsmoura:fedora.im> I can take
2025-04-17 17:13:46 <@Zlopez:matrix.org> Sold!
2025-04-17 17:14:00 <@Zlopez:matrix.org> !info chair 2025-04-24 - phsmoura
2025-04-17 17:14:40 <@Zlopez:matrix.org> !info chair 2025-05-01 - ???
2025-04-17 17:15:22 <@Zlopez:matrix.org> Any volunteer for week after that?
2025-04-17 17:16:17 <@Zlopez:matrix.org> @nirik:matrix.scrye.com You are right with the maubot, the error doesn't seem to affect it, but the info line is actually showed in the log on error level, but no error specified
2025-04-17 17:16:32 <@Zlopez:matrix.org> `[2025-04-17 17:14:41,381] [ERROR@maubot.instance.meetbot-meetings] info !info chair 2025-05-01 - ???`
2025-04-17 17:17:04 <@nirik:matrix.scrye.com> I think that error is just that maubot doesn't understand users with multiple sessions (each one has it's own key).
2025-04-17 17:17:14 <@nirik:matrix.scrye.com> anyhow, no idea why it's slow.
2025-04-17 17:17:19 <@Zlopez:matrix.org> Not sure if that is actually an info message or error message
2025-04-17 17:17:33 <@Zlopez:matrix.org> Back to the meeting
2025-04-17 17:18:49 <@Zlopez:matrix.org> So no volunteer for the first meeting in May?
2025-04-17 17:19:41 <@Zlopez:matrix.org> Let's move on, we can occupy that next week
2025-04-17 17:20:06 <@Zlopez:matrix.org> !info F42 final freeze is over
2025-04-17 17:20:06 <@Zlopez:matrix.org> !topic announcements and information
2025-04-17 17:20:06 <@Zlopez:matrix.org> !info CLE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0815 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
2025-04-17 17:20:06 <@Zlopez:matrix.org> !info CLE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1900 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
2025-04-17 17:20:06 <@Zlopez:matrix.org> !info OpenID EOL in Fedora Infra is set to 20-05-2025
2025-04-17 17:21:06 <@Zlopez:matrix.org> OpenID EOL will be probably postponed, but we already got plenty of services migrated to OIDC, which is a win :-)
2025-04-17 17:21:24 <@nirik:matrix.scrye.com> yeah, progress!
2025-04-17 17:21:56 <@Zlopez:matrix.org> Anything else to announce?
2025-04-17 17:23:43 <@Zlopez:matrix.org> Me neither
2025-04-17 17:23:53 <@Zlopez:matrix.org> !topic Oncall
2025-04-17 17:23:53 <@Zlopez:matrix.org> !info @markrosenbaum  is on call from 2025-04-11 to 2025-04-17
2025-04-17 17:23:53 <@Zlopez:matrix.org> !info ??? is on call from 2025-04-18 to 2025-04-24
2025-04-17 17:23:53 <@Zlopez:matrix.org> !info https://docs.fedoraproject.org/en-US/infra/day_to_day_fedora/#_the_oncall_role_in_our_team
2025-04-17 17:24:10 <@Zlopez:matrix.org> Any volunteer for next oncall?
2025-04-17 17:24:54 <@nirik:matrix.scrye.com> I can take it
2025-04-17 17:24:58 <@markrosenbaum:fedora.im> Not much oncall wise
2025-04-17 17:25:18 <@markrosenbaum:fedora.im> Like this week, don’t think I got any pings
2025-04-17 17:25:31 <@Zlopez:matrix.org> It seems nobody else wants it, so it's yours @nirik:matrix.scrye.com
2025-04-17 17:25:50 <@Zlopez:matrix.org> !info @nirik is on call from 2025-04-18 to 2025-04-24
2025-04-17 17:25:57 <@nirik:matrix.scrye.com> alright
2025-04-17 17:26:38 <@Zlopez:matrix.org> !info ??? is on call from 2025-04-25 to 2025-05-01
2025-04-17 17:26:44 <@nirik:matrix.scrye.com> !oncall
2025-04-17 17:26:57 <@zodbot:fedora.im> ● @nirik:matrix.scrye.com (kevin) Current Time for them: 10:26 (US/Pacific)
2025-04-17 17:26:57 <@zodbot:fedora.im> The following people are oncall:
2025-04-17 17:26:57 <@zodbot:fedora.im> 
2025-04-17 17:26:57 <@zodbot:fedora.im> If they do not respond, please file a ticket (https://pagure.io/fedora-infrastructure/issues)
2025-04-17 17:27:04 <@Zlopez:matrix.org> Just to be sure, any volunteer for 2025-04-25 to 2025-05-01?
2025-04-17 17:27:14 <@Zlopez:matrix.org> It seem that even @zodbot is slow
2025-04-17 17:28:36 <@nirik:matrix.scrye.com> it's ok... I'm slow today as well...
2025-04-17 17:30:00 <@Zlopez:matrix.org> Doesn't seem that we have a volunteer, so let me move on
2025-04-17 17:30:05 <@Zlopez:matrix.org> !info Summary of last week: (from current oncall)
2025-04-17 17:30:07 <@markrosenbaum:fedora.im> Hmmmm, maybe I can take that 
2025-04-17 17:30:26 <@markrosenbaum:fedora.im> Yea basically nothing
2025-04-17 17:30:31 <@markrosenbaum:fedora.im> Don’t think o got any pings
2025-04-17 17:30:32 <@Zlopez:matrix.org> You can even next week :-)
2025-04-17 17:30:40 <@markrosenbaum:fedora.im> Don’t think I got any pings
2025-04-17 17:30:55 <@Zlopez:matrix.org> I didn't saw any as well
2025-04-17 17:31:05 <@Zlopez:matrix.org> Probably because of the freeze
2025-04-17 17:31:07 <@Zlopez:matrix.org> !topic Monitoring discussion [nirik]
2025-04-17 17:31:07 <@Zlopez:matrix.org> !info https://nagios.fedoraproject.org/nagios 
2025-04-17 17:31:07 <@Zlopez:matrix.org> !info Go over existing items and fix them
2025-04-17 17:31:19 <@smooge:fedora.im> most of the 'pings' I have seen was AI scraper related and answered quickly as such
2025-04-17 17:31:25 <@nirik:matrix.scrye.com> lets see...
2025-04-17 17:31:41 <@nirik:matrix.scrye.com> pretty much the same as last week. No changes off hand
2025-04-17 17:32:05 <@Zlopez:matrix.org> I'm still waiting for last review and then I should be able to resolve the datanommer checks
2025-04-17 17:32:19 <@nirik:matrix.scrye.com> that would be good.
2025-04-17 17:32:29 <@nirik:matrix.scrye.com> One of those we depend on is the one checking for updates pushes.
2025-04-17 17:32:41 <@nirik:matrix.scrye.com> and it's been broken a few times this week. ;(
2025-04-17 17:32:42 <@Zlopez:matrix.org> The review process is just taking so long
2025-04-17 17:33:05 <@nirik:matrix.scrye.com> yeah, sometimes it does. ;(
2025-04-17 17:33:18 <@Zlopez:matrix.org> For info, here is link for that https://bugzilla.redhat.com/show_bug.cgi?id=2346613
2025-04-17 17:34:52 <@nirik:matrix.scrye.com> I had a few topics for now or for open floor if we can leave some room.
2025-04-17 17:35:09 <@Zlopez:matrix.org> 1️⃣ Backlog refinement
2025-04-17 17:35:09 <@Zlopez:matrix.org> So what do we want to do now?
2025-04-17 17:35:09 <@Zlopez:matrix.org> 4️⃣ End early
2025-04-17 17:35:09 <@Zlopez:matrix.org> 3️⃣ Open Floor
2025-04-17 17:35:09 <@Zlopez:matrix.org> 2️⃣ Decide on learning topic
2025-04-17 17:35:20 <@Zlopez:matrix.org> Vote with corresponding emoji :-)
2025-04-17 17:35:29 <@nirik:matrix.scrye.com> fancy.
2025-04-17 17:36:26 <@Zlopez:matrix.org> I will wait a minute or two for people to have time to vote
2025-04-17 17:38:35 <@Zlopez:matrix.org> I see 3 votes for open floor, so let's continue with that
2025-04-17 17:38:43 <@Zlopez:matrix.org> !topic Open Floor
2025-04-17 17:39:04 <@Zlopez:matrix.org> @nirik:matrix.scrye.com You said you have something for open floor
2025-04-17 17:39:08 <@smooge:fedora.im> Nice floor you have there? What do you use to shine it?
2025-04-17 17:39:36 <@markrosenbaum:fedora.im> So nirik I haven’t had time to read blog at all, any thing new with the infra move?
2025-04-17 17:39:38 <@nirik:matrix.scrye.com> it's a floor wax and a dessert toping!
2025-04-17 17:39:46 <@nirik:matrix.scrye.com> ok, a few things:
2025-04-17 17:39:50 <@markrosenbaum:fedora.im> So nirik I haven’t found time to read blog recently, any thing new with the infra move?
2025-04-17 17:39:55 <@Zlopez:matrix.org> And cherry on top!
2025-04-17 17:40:24 <@nirik:matrix.scrye.com> Nothing much new on DC move. I have gotten a prelim setup of what the vlans/network will be. Waiting for access to mgmt interfaces on new hardware. Hopefully more news later today
2025-04-17 17:40:54 <@smooge:fedora.im> good luck
2025-04-17 17:40:56 <@nirik:matrix.scrye.com> With the release of 42 the other day, we need to upgrade our wiki again. ;( I can file a ticket to track that...
2025-04-17 17:41:13 <@markrosenbaum:fedora.im> Nice, nice, good luck with that
2025-04-17 17:41:54 <@nirik:matrix.scrye.com> The usual way is to build a bunch of rpms in our infra koji tags, redeploy a vm, run the migration and fix problems. If someone wanted to be fancy tho, we could perhaps look at moving it into openshift to make this less painful.
2025-04-17 17:42:20 <@Zlopez:matrix.org> I assume there would be more things that will need to be updated
2025-04-17 17:43:10 <@nirik:matrix.scrye.com> not much actually. There's bodhi-backend01, two proxies I somehow missed and the copr power9 box. I can upgrade all those pretty easily I think
2025-04-17 17:43:57 <@Zlopez:matrix.org> Not sure if the move will be less painful
2025-04-17 17:44:33 <@james:fedora.im> There are ansible playbooks to install?
2025-04-17 17:44:41 <@nirik:matrix.scrye.com> yes, it's all in ansible
2025-04-17 17:44:54 <@nirik:matrix.scrye.com> something always breaks tho. ;)
2025-04-17 17:44:58 <@james:fedora.im> So it's just the building of packages and rsync of data that is manual?
2025-04-17 17:45:35 <@nirik:matrix.scrye.com> the data is all in the database... so reinstall with new version and run some migrate thing that updates the existing db content to the new version
2025-04-17 17:45:42 <@nirik:matrix.scrye.com> no rsyncing should be needed.
2025-04-17 17:45:56 <@james:fedora.im> Ahh ... fancy builtin rsync ;)
2025-04-17 17:46:13 <@nirik:matrix.scrye.com> and of course in staging first to work out problems.
2025-04-17 17:46:56 <@nirik:matrix.scrye.com> anyhow, I'll file a ticket and try and find takers for it. :)
2025-04-17 17:47:24 <@nirik:matrix.scrye.com> random small thing: ipa01.stg was running low on disk the other day, so I looked at it.
2025-04-17 17:47:57 <@nirik:matrix.scrye.com> there's a logrotate config thats... not perhaps working as expected. ;)
2025-04-17 17:48:22 <@Zlopez:matrix.org> Does it generate too much logs?
2025-04-17 17:48:43 <@nirik:matrix.scrye.com> /var/log/dirsrv/slapd-STG-FEDORAPROJECT-ORG/access.20*
2025-04-17 17:48:43 <@nirik:matrix.scrye.com> # which can be done either here or globally in /etc/logrotate.conf.
2025-04-17 17:48:43 <@nirik:matrix.scrye.com> # Note that logs are not compressed unless "compress" is configured,
2025-04-17 17:48:43 <@nirik:matrix.scrye.com> # cat /etc/logrotate.d/slapd
2025-04-17 17:48:43 <@nirik:matrix.scrye.com> /var/log/dirsrv/slapd-STG-FEDORAPROJECT-ORG/errors.20*
2025-04-17 17:48:48 <@nirik:matrix.scrye.com> /var/log/dirsrv/slapd-STG-FEDORAPROJECT-ORG/errors.20*
2025-04-17 17:48:48 <@nirik:matrix.scrye.com> /var/log/dirsrv/slapd-STG-FEDORAPROJECT-ORG/access.20*
2025-04-17 17:48:48 <@nirik:matrix.scrye.com> # which can be done either here or globally in /etc/logrotate.conf.
2025-04-17 17:48:48 <@nirik:matrix.scrye.com> # Note that logs are not compressed unless "compress" is configured,
2025-04-17 17:48:48 <@nirik:matrix.scrye.com> ``` # cat /etc/logrotate.d/slapd
2025-04-17 17:48:58 <@nirik:matrix.scrye.com> the problem is... it compresses them.
2025-04-17 17:49:03 <@nirik:matrix.scrye.com> then it compresses them again
2025-04-17 17:49:05 <@nirik:matrix.scrye.com> and again
2025-04-17 17:49:07 <@nirik:matrix.scrye.com> and again
2025-04-17 17:49:21 <@nirik:matrix.scrye.com> errors.20250224-050925-20250309.gz-20250316.gz-20250323.gz-20250330.gz-20250413.gz
2025-04-17 17:49:35 <@nirik:matrix.scrye.com> anyhow, I deleted all the 2024 ones.
2025-04-17 17:49:38 <@james:fedora.im> Ahh ... needs to not match *.gz
2025-04-17 17:49:46 <@nirik:matrix.scrye.com> but we should fix that. It wasn't in ansible, so I am not sure where it came from
2025-04-17 17:49:51 <@Zlopez:matrix.org> I remember seeing that :-D
2025-04-17 17:50:05 <@smooge:fedora.im> at least it didn't compress the gz with bzip2 and then that with xz and finally zip
2025-04-17 17:50:07 <@Zlopez:matrix.org> And I'm not sure if I'm the one to blame
2025-04-17 17:50:31 <@nirik:matrix.scrye.com> yeah, I don't recall doing it, but it could have been me...
2025-04-17 17:50:34 <@Zlopez:matrix.org> That sounds awesome!
2025-04-17 17:50:48 <@james:fedora.im> Hacky fix: have 10 entries for each logfile: <blah>*0 ; <blah>*1 ; etc.
2025-04-17 17:51:14 <@nirik:matrix.scrye.com> Finally on ai scraper news...
2025-04-17 17:51:47 <@smooge:fedora.im> Zlopez: I accidently had that happen when we had at least two different log compress scripts running
2025-04-17 17:51:50 <@Zlopez:matrix.org> Let me add this to my todo, so I can look at it. Do we have a ticket as well?
2025-04-17 17:53:18 <@james:fedora.im> *reads logrotate docs* ... Ahh you can do: <blah>*[0-9]
2025-04-17 17:53:50 <@nirik:matrix.scrye.com> I have been having a lot more luck 403'ing things that they are abusing (as in many cases these aren't things normal users would care about). On pagure.io there's about 3-4 projects that are mirrors of other upstream projects (kernel, git, gdb) or forks of things like releng repo. They hit koji the last few days and I blocked fileinfo and buildrootinfo web endpoints. That seems to basically drop their load to nothing. Of course sometime soon they will pick something thats important and that real users need to access, so not sure what we do then. ;(
2025-04-17 17:54:24 <@nirik:matrix.scrye.com> no ticket (yet). I don't know why prod doesn't have it either tho, I think ipa is supposed to rotate those? so the answer may be rm ing it
2025-04-17 17:55:39 <@Zlopez:matrix.org> It's possible that ipa has that configured somewhere
2025-04-17 17:56:48 <@nirik:matrix.scrye.com> You would also think the scrapers would realize that something is a 403 and stop trying, but no... about 1 million-ish 403's served by koji today
2025-04-17 17:56:52 <@Zlopez:matrix.org> But I will be off till Wednesday, so no looking at it before that :-)
2025-04-17 17:57:34 <@nirik:matrix.scrye.com> I might have more, but that's all I can remember. ;)
2025-04-17 17:57:47 <@Zlopez:matrix.org> Maybe they are not the same botnet
2025-04-17 17:58:03 <@Zlopez:matrix.org> There is one thing I wanted to add as well
2025-04-17 17:59:41 <@Zlopez:matrix.org> I spent some time to improve scm_request_processor, so it should be more interactive and less annoying. Currently testing on staging, although encountered issue on dist-git on staging with that is not related to changes I did :/
2025-04-17 17:59:46 <@james:fedora.im> Is it worth collecting the 403 IPs and we could deploy that as a firewall set to drop packets?
2025-04-17 17:59:57 <@Zlopez:matrix.org> I spent some time to improve `scm_request_processor`, so it should be more interactive and less annoying. Currently testing on staging, although encountered issue on dist-git on staging with that is not related to changes I did :/
2025-04-17 18:00:42 <@Zlopez:matrix.org> So we are almost at the end of our time, thanks everybody for coming and see you next time :-)
2025-04-17 18:00:48 <@nirik:matrix.scrye.com> James: it's too vast. It's like 100,000 ips, mostly not in the same networks.
2025-04-17 18:00:50 <@Zlopez:matrix.org> !endmeeting