#meeting-3:fedoraproject.org log

<@Zlopez:matrix.org>

17:01:31

!startmeeting Infrastructure (2025-04-17)

<@meetbot:fedora.im>

17:01:38

Meeting started at 2025-04-17 17:01:31 UTC

<@meetbot:fedora.im>

17:01:39

The Meeting name is 'Infrastructure (2025-04-17)'

<@Zlopez:matrix.org>

17:02:14

Where are you @meetbot:fedora.im

<@nirik:matrix.scrye.com>

17:02:15

morning

<@Zlopez:matrix.org>

17:02:21

Where are you @meetbot:fedora.im?

<@nirik:matrix.scrye.com>

17:02:22

slow?

<@phsmoura:fedora.im>

17:02:26

hello

<@Zlopez:matrix.org>

17:02:30

Yes

<@markrosenbaum:fedora.im>

17:02:41

Lol, Meet Bot awol

<@nirik:matrix.scrye.com>

17:03:19

hum

<@Zlopez:matrix.org>

17:03:27

I'm waiting for it to react at start of the meeting

<@nirik:matrix.scrye.com>

17:03:49

morning.

<@Zlopez:matrix.org>

17:03:56

!chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp

<@Zlopez:matrix.org>

17:03:56

!topic ahoy

<@Zlopez:matrix.org>

17:03:56

!info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra

<@Zlopez:matrix.org>

17:03:56

!info About our team: https://docs.fedoraproject.org/en-US/cle/

<@Zlopez:matrix.org>

17:03:56

!info Agenda is at: https://board.net/p/fedora-infra

<@Zlopez:matrix.org>

17:03:56

!meetingname infrastructure

<@nirik:matrix.scrye.com>

17:03:56

I am gonna go grab coffee... back in a sec

<@meetbot:fedora.im>

17:04:28

The Meeting Name is now infrastructure

<@Zlopez:matrix.org>

17:04:42

It takes forever for the bot to react, but it worked fine in the morning

<@Zlopez:matrix.org>

17:04:46

!hi

<@Zlopez:matrix.org>

17:04:54

Welcome everybody to today infra meeting

<@zodbot:fedora.im>

17:04:57

Michal Konecny (zlopez)

<@markrosenbaum:fedora.im>

17:05:31

!hi

<@zodbot:fedora.im>

17:05:32

Mark Rosenbaum (markrosenbaum)

<@Zlopez:matrix.org>

17:06:59

I'm trying to look at the bot logs, to see if there is a reason for the slowness

<@smilner:fedora.im>

17:07:10

!hi

<@zodbot:fedora.im>

17:07:11

None (smilner)

<@nirik:matrix.scrye.com>

17:07:15

there's that crypto thing... but I think it might be unrelated.

<@Zlopez:matrix.org>

17:07:37

That was what I found last time

<@Zlopez:matrix.org>

17:07:49

!info Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/

<@Zlopez:matrix.org>

17:07:49

!topic New folks introductions

<@Zlopez:matrix.org>

17:07:49

!info This is a place where people who are interested in Fedora Infrastructure can introduce themselves

<@Zlopez:matrix.org>

17:08:40

Do we have anybody new around?

<@markrosenbaum:fedora.im>

17:09:19

Hmmm, over at join I thought we had some people say they were infra interested but I could be wrong

<@Zlopez:matrix.org>

17:10:36

I saw somebody today in #admin:fedoraproject.org

<@Zlopez:matrix.org>

17:10:48

But it seems they aren't around for this meeting

<@Zlopez:matrix.org>

17:11:09

So let's move on

<@nirik:matrix.scrye.com>

17:11:15

timezones... pesky things

<@Zlopez:matrix.org>

17:11:19

!info chair 2025-04-24 - ???

<@Zlopez:matrix.org>

17:11:19

!topic Next chair

<@Zlopez:matrix.org>

17:11:19

!info magic eight ball says:

<@Zlopez:matrix.org>

17:11:19

!info chair 2025-04-17 - @Zlopez

<@Zlopez:matrix.org>

17:11:55

The last meeting was only me and @nirik:matrix.scrye.com (who was on Go/No go meeting at the same time)

<@Zlopez:matrix.org>

17:12:05

So I took this one as well

<@Zlopez:matrix.org>

17:12:15

Do we have volunteer for next one?

<@nirik:matrix.scrye.com>

17:12:51

I suppose I can if no one else is able.

<@phsmoura:fedora.im>

17:13:15

I can take

<@Zlopez:matrix.org>

17:13:46

Sold!

<@Zlopez:matrix.org>

17:14:00

!info chair 2025-04-24 - phsmoura

<@Zlopez:matrix.org>

17:14:40

!info chair 2025-05-01 - ???

<@Zlopez:matrix.org>

17:15:22

Any volunteer for week after that?

<@Zlopez:matrix.org>

17:16:17

@nirik:matrix.scrye.com You are right with the maubot, the error doesn't seem to affect it, but the info line is actually showed in the log on error level, but no error specified

<@Zlopez:matrix.org>

17:16:32

`[2025-04-17 17:14:41,381] [ERROR@maubot.instance.meetbot-meetings] info !info chair 2025-05-01 - ???`

<@nirik:matrix.scrye.com>

17:17:04

I think that error is just that maubot doesn't understand users with multiple sessions (each one has it's own key).

<@nirik:matrix.scrye.com>

17:17:14

anyhow, no idea why it's slow.

<@Zlopez:matrix.org>

17:17:19

Not sure if that is actually an info message or error message

<@Zlopez:matrix.org>

17:17:33

Back to the meeting

<@Zlopez:matrix.org>

17:18:49

So no volunteer for the first meeting in May?

<@Zlopez:matrix.org>

17:19:41

Let's move on, we can occupy that next week

<@Zlopez:matrix.org>

17:20:06

!info F42 final freeze is over

<@Zlopez:matrix.org>

17:20:06

!topic announcements and information

<@Zlopez:matrix.org>

17:20:06

!info CLE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0815 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org

<@Zlopez:matrix.org>

17:20:06

!info CLE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1900 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org

<@Zlopez:matrix.org>

17:20:06

!info OpenID EOL in Fedora Infra is set to 20-05-2025

<@Zlopez:matrix.org>

17:21:06

OpenID EOL will be probably postponed, but we already got plenty of services migrated to OIDC, which is a win :-)

<@nirik:matrix.scrye.com>

17:21:24

yeah, progress!

<@Zlopez:matrix.org>

17:21:56

Anything else to announce?

<@Zlopez:matrix.org>

17:23:43

Me neither

<@Zlopez:matrix.org>

17:23:53

!topic Oncall

<@Zlopez:matrix.org>

17:23:53

!info @markrosenbaum is on call from 2025-04-11 to 2025-04-17

<@Zlopez:matrix.org>

17:23:53

!info ??? is on call from 2025-04-18 to 2025-04-24

<@Zlopez:matrix.org>

17:23:53

!info https://docs.fedoraproject.org/en-US/infra/day_to_day_fedora/#_the_oncall_role_in_our_team

<@Zlopez:matrix.org>

17:24:10

Any volunteer for next oncall?

<@nirik:matrix.scrye.com>

17:24:54

I can take it

<@markrosenbaum:fedora.im>

17:24:58

Not much oncall wise

<@markrosenbaum:fedora.im>

17:25:18

Like this week, don’t think I got any pings

<@Zlopez:matrix.org>

17:25:31

It seems nobody else wants it, so it's yours @nirik:matrix.scrye.com

<@Zlopez:matrix.org>

17:25:50

!info @nirik is on call from 2025-04-18 to 2025-04-24

<@nirik:matrix.scrye.com>

17:25:57

alright

<@Zlopez:matrix.org>

17:26:38

!info ??? is on call from 2025-04-25 to 2025-05-01

<@nirik:matrix.scrye.com>

17:26:44

!oncall

<@zodbot:fedora.im>

17:26:57

● @nirik:matrix.scrye.com (kevin) Current Time for them: 10:26 (US/Pacific)

<@zodbot:fedora.im>

17:26:57

The following people are oncall:

<@zodbot:fedora.im>

17:26:57

<@zodbot:fedora.im>

17:26:57

If they do not respond, please file a ticket (https://pagure.io/fedora-infrastructure/issues)

<@Zlopez:matrix.org>

17:27:04

Just to be sure, any volunteer for 2025-04-25 to 2025-05-01?

<@Zlopez:matrix.org>

17:27:14

It seem that even @zodbot is slow

<@nirik:matrix.scrye.com>

17:28:36

it's ok... I'm slow today as well...

<@Zlopez:matrix.org>

17:30:00

Doesn't seem that we have a volunteer, so let me move on

<@Zlopez:matrix.org>

17:30:05

!info Summary of last week: (from current oncall)

<@markrosenbaum:fedora.im>

17:30:07

Hmmmm, maybe I can take that

<@markrosenbaum:fedora.im>

17:30:26

Yea basically nothing

<@markrosenbaum:fedora.im>

17:30:31

Don’t think o got any pings

<@Zlopez:matrix.org>

17:30:32

You can even next week :-)

<@markrosenbaum:fedora.im>

17:30:40

Don’t think I got any pings

<@Zlopez:matrix.org>

17:30:55

I didn't saw any as well

<@Zlopez:matrix.org>

17:31:05

Probably because of the freeze

<@Zlopez:matrix.org>

17:31:07

!topic Monitoring discussion [nirik]

<@Zlopez:matrix.org>

17:31:07

!info https://nagios.fedoraproject.org/nagios

<@Zlopez:matrix.org>

17:31:07

!info Go over existing items and fix them

<@smooge:fedora.im>

17:31:19

most of the 'pings' I have seen was AI scraper related and answered quickly as such

<@nirik:matrix.scrye.com>

17:31:25

lets see...

<@nirik:matrix.scrye.com>

17:31:41

pretty much the same as last week. No changes off hand

<@Zlopez:matrix.org>

17:32:05

I'm still waiting for last review and then I should be able to resolve the datanommer checks

<@nirik:matrix.scrye.com>

17:32:19

that would be good.

<@nirik:matrix.scrye.com>

17:32:29

One of those we depend on is the one checking for updates pushes.

<@nirik:matrix.scrye.com>

17:32:41

and it's been broken a few times this week. ;(

<@Zlopez:matrix.org>

17:32:42

The review process is just taking so long

<@nirik:matrix.scrye.com>

17:33:05

yeah, sometimes it does. ;(

<@Zlopez:matrix.org>

17:33:18

For info, here is link for that https://bugzilla.redhat.com/show_bug.cgi?id=2346613

<@nirik:matrix.scrye.com>

17:34:52

I had a few topics for now or for open floor if we can leave some room.

<@Zlopez:matrix.org>

17:35:09

1️⃣ Backlog refinement

<@Zlopez:matrix.org>

17:35:09

So what do we want to do now?

<@Zlopez:matrix.org>

17:35:09

4️⃣ End early

<@Zlopez:matrix.org>

17:35:09

3️⃣ Open Floor

<@Zlopez:matrix.org>

17:35:09

2️⃣ Decide on learning topic

<@Zlopez:matrix.org>

17:35:20

Vote with corresponding emoji :-)

<@nirik:matrix.scrye.com>

17:35:29

fancy.

<@Zlopez:matrix.org>

17:36:26

I will wait a minute or two for people to have time to vote

<@Zlopez:matrix.org>

17:38:35

I see 3 votes for open floor, so let's continue with that

<@Zlopez:matrix.org>

17:38:43

!topic Open Floor

<@Zlopez:matrix.org>

17:39:04

@nirik:matrix.scrye.com You said you have something for open floor

<@smooge:fedora.im>

17:39:08

Nice floor you have there? What do you use to shine it?

<@markrosenbaum:fedora.im>

17:39:36

So nirik I haven’t had time to read blog at all, any thing new with the infra move?

<@nirik:matrix.scrye.com>

17:39:38

it's a floor wax and a dessert toping!

<@nirik:matrix.scrye.com>

17:39:46

ok, a few things:

<@markrosenbaum:fedora.im>

17:39:50

So nirik I haven’t found time to read blog recently, any thing new with the infra move?

<@Zlopez:matrix.org>

17:39:55

And cherry on top!

<@nirik:matrix.scrye.com>

17:40:24

Nothing much new on DC move. I have gotten a prelim setup of what the vlans/network will be. Waiting for access to mgmt interfaces on new hardware. Hopefully more news later today

<@smooge:fedora.im>

17:40:54

good luck

<@nirik:matrix.scrye.com>

17:40:56

With the release of 42 the other day, we need to upgrade our wiki again. ;( I can file a ticket to track that...

<@markrosenbaum:fedora.im>

17:41:13

Nice, nice, good luck with that

<@nirik:matrix.scrye.com>

17:41:54

The usual way is to build a bunch of rpms in our infra koji tags, redeploy a vm, run the migration and fix problems. If someone wanted to be fancy tho, we could perhaps look at moving it into openshift to make this less painful.

<@Zlopez:matrix.org>

17:42:20

I assume there would be more things that will need to be updated

<@nirik:matrix.scrye.com>

17:43:10

not much actually. There's bodhi-backend01, two proxies I somehow missed and the copr power9 box. I can upgrade all those pretty easily I think

<@Zlopez:matrix.org>

17:43:57

Not sure if the move will be less painful

<@james:fedora.im>

17:44:33

There are ansible playbooks to install?

<@nirik:matrix.scrye.com>

17:44:41

yes, it's all in ansible

<@nirik:matrix.scrye.com>

17:44:54

something always breaks tho. ;)

<@james:fedora.im>

17:44:58

So it's just the building of packages and rsync of data that is manual?

<@nirik:matrix.scrye.com>

17:45:35

the data is all in the database... so reinstall with new version and run some migrate thing that updates the existing db content to the new version

<@nirik:matrix.scrye.com>

17:45:42

no rsyncing should be needed.

<@james:fedora.im>

17:45:56

Ahh ... fancy builtin rsync ;)

<@nirik:matrix.scrye.com>

17:46:13

and of course in staging first to work out problems.

<@nirik:matrix.scrye.com>

17:46:56

anyhow, I'll file a ticket and try and find takers for it. :)

<@nirik:matrix.scrye.com>

17:47:24

random small thing: ipa01.stg was running low on disk the other day, so I looked at it.

<@nirik:matrix.scrye.com>

17:47:57

there's a logrotate config thats... not perhaps working as expected. ;)

<@Zlopez:matrix.org>

17:48:22

Does it generate too much logs?

<@nirik:matrix.scrye.com>

17:48:43

/var/log/dirsrv/slapd-STG-FEDORAPROJECT-ORG/access.20*

<@nirik:matrix.scrye.com>

17:48:43

# which can be done either here or globally in /etc/logrotate.conf.

<@nirik:matrix.scrye.com>

17:48:43

# Note that logs are not compressed unless "compress" is configured,

<@nirik:matrix.scrye.com>

17:48:43

# cat /etc/logrotate.d/slapd

<@nirik:matrix.scrye.com>

17:48:43

/var/log/dirsrv/slapd-STG-FEDORAPROJECT-ORG/errors.20*

<@nirik:matrix.scrye.com>

17:48:48

/var/log/dirsrv/slapd-STG-FEDORAPROJECT-ORG/errors.20*

<@nirik:matrix.scrye.com>

17:48:48

/var/log/dirsrv/slapd-STG-FEDORAPROJECT-ORG/access.20*

<@nirik:matrix.scrye.com>

17:48:48

# which can be done either here or globally in /etc/logrotate.conf.

<@nirik:matrix.scrye.com>

17:48:48

# Note that logs are not compressed unless "compress" is configured,

<@nirik:matrix.scrye.com>

17:48:48

``` # cat /etc/logrotate.d/slapd

<@nirik:matrix.scrye.com>

17:48:58

the problem is... it compresses them.

<@nirik:matrix.scrye.com>

17:49:03

then it compresses them again

<@nirik:matrix.scrye.com>

17:49:05

and again

<@nirik:matrix.scrye.com>

17:49:07

and again

<@nirik:matrix.scrye.com>

17:49:21

errors.20250224-050925-20250309.gz-20250316.gz-20250323.gz-20250330.gz-20250413.gz

<@nirik:matrix.scrye.com>

17:49:35

anyhow, I deleted all the 2024 ones.

<@james:fedora.im>

17:49:38

Ahh ... needs to not match *.gz

<@nirik:matrix.scrye.com>

17:49:46

but we should fix that. It wasn't in ansible, so I am not sure where it came from

<@Zlopez:matrix.org>

17:49:51

I remember seeing that :-D

<@smooge:fedora.im>

17:50:05

at least it didn't compress the gz with bzip2 and then that with xz and finally zip

<@Zlopez:matrix.org>

17:50:07

And I'm not sure if I'm the one to blame

<@nirik:matrix.scrye.com>

17:50:31

yeah, I don't recall doing it, but it could have been me...

<@Zlopez:matrix.org>

17:50:34

That sounds awesome!

<@james:fedora.im>

17:50:48

Hacky fix: have 10 entries for each logfile: <blah>*0 ; <blah>*1 ; etc.

<@nirik:matrix.scrye.com>

17:51:14

Finally on ai scraper news...

<@smooge:fedora.im>

17:51:47

Zlopez: I accidently had that happen when we had at least two different log compress scripts running

<@Zlopez:matrix.org>

17:51:50

Let me add this to my todo, so I can look at it. Do we have a ticket as well?

<@james:fedora.im>

17:53:18

*reads logrotate docs* ... Ahh you can do: <blah>*[0-9]

<@nirik:matrix.scrye.com>

17:53:50

I have been having a lot more luck 403'ing things that they are abusing (as in many cases these aren't things normal users would care about). On pagure.io there's about 3-4 projects that are mirrors of other upstream projects (kernel, git, gdb) or forks of things like releng repo. They hit koji the last few days and I blocked fileinfo and buildrootinfo web endpoints. That seems to basically drop their load to nothing. Of course sometime soon they will pick something thats important and that real users need to access, so not sure what we do then. ;(

<@nirik:matrix.scrye.com>

17:54:24

no ticket (yet). I don't know why prod doesn't have it either tho, I think ipa is supposed to rotate those? so the answer may be rm ing it

<@Zlopez:matrix.org>

17:55:39

It's possible that ipa has that configured somewhere

<@nirik:matrix.scrye.com>

17:56:48

You would also think the scrapers would realize that something is a 403 and stop trying, but no... about 1 million-ish 403's served by koji today

<@Zlopez:matrix.org>

17:56:52

But I will be off till Wednesday, so no looking at it before that :-)

<@nirik:matrix.scrye.com>

17:57:34

I might have more, but that's all I can remember. ;)

<@Zlopez:matrix.org>

17:57:47

Maybe they are not the same botnet

<@Zlopez:matrix.org>

17:58:03

There is one thing I wanted to add as well

<@Zlopez:matrix.org>

17:59:41

I spent some time to improve scm_request_processor, so it should be more interactive and less annoying. Currently testing on staging, although encountered issue on dist-git on staging with that is not related to changes I did :/

<@james:fedora.im>

17:59:46

Is it worth collecting the 403 IPs and we could deploy that as a firewall set to drop packets?

<@Zlopez:matrix.org>

17:59:57

I spent some time to improve `scm_request_processor`, so it should be more interactive and less annoying. Currently testing on staging, although encountered issue on dist-git on staging with that is not related to changes I did :/

<@Zlopez:matrix.org>

18:00:42

So we are almost at the end of our time, thanks everybody for coming and see you next time :-)

<@nirik:matrix.scrye.com>

18:00:48

James: it's too vast. It's like 100,000 ips, mostly not in the same networks.

<@Zlopez:matrix.org>

18:00:50

!endmeeting