#meeting-3:fedoraproject.org log

<@nirik:matrix.scrye.com>

17:00:05

!startmeeting Infrastructure (2025-04-03)

<@meetbot:fedora.im>

17:00:07

Meeting started at 2025-04-03 17:00:05 UTC

<@meetbot:fedora.im>

17:00:08

The Meeting name is 'Infrastructure (2025-04-03)'

<@nirik:matrix.scrye.com>

17:00:13

!info Agenda is at: https://board.net/p/fedora-infra

<@nirik:matrix.scrye.com>

17:00:13

!chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp

<@nirik:matrix.scrye.com>

17:00:13

!meetingname infrastructure

<@nirik:matrix.scrye.com>

17:00:13

!info About our team: https://docs.fedoraproject.org/en-US/cle/

<@nirik:matrix.scrye.com>

17:00:13

!info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra

<@nirik:matrix.scrye.com>

17:00:13

!topic ahoy

<@meetbot:fedora.im>

17:00:14

The Meeting Name is now infrastructure

<@nirik:matrix.scrye.com>

17:00:23

morning everyone!

<@markrosenbaum:fedora.im>

17:01:11

Afternoon!

<@Zlopez:matrix.org>

17:01:13

!hi

<@zodbot:fedora.im>

17:01:16

Michal Konecny (zlopez)

<@markrosenbaum:fedora.im>

17:01:23

!hi

<@zodbot:fedora.im>

17:01:23

Mark Rosenbaum (markrosenbaum)

<@phsmoura:fedora.im>

17:01:54

hello

<@nirik:matrix.scrye.com>

17:02:31

will wait another minute or two for folks to come in.

<@nirik:matrix.scrye.com>

17:04:00

!info Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/

<@nirik:matrix.scrye.com>

17:04:00

!info This is a place where people who are interested in Fedora Infrastructure can introduce themselves

<@nirik:matrix.scrye.com>

17:04:00

!topic New folks introductions

<@carlwgeorge:fedora.im>

17:04:02

!hi

<@zodbot:fedora.im>

17:04:03

Carl George (carlwgeorge) - he / him / his

<@nirik:matrix.scrye.com>

17:04:20

any new folks around today? This is a safe place to introduce yourself.

<@markrosenbaum:fedora.im>

17:04:41

Jae "Awa" J4

<@markrosenbaum:fedora.im>

17:05:31

Jae "Awa" J4, if you want to introduce yourself here you can

<@j:j4.lc>

17:06:15

Hey there, don't mind me, I'm new around here, manifested my interest in contributing to Fedora today

<@nirik:matrix.scrye.com>

17:06:35

well, welcome. :) Feel free to lurk around and ask questions.

<@nirik:matrix.scrye.com>

17:07:06

!topic Next chair

<@nirik:matrix.scrye.com>

17:07:06

!info chair 2025-04-10 - @Zlopez

<@nirik:matrix.scrye.com>

17:07:06

!info magic eight ball says:

<@nirik:matrix.scrye.com>

17:07:06

!info chair 2025-04-24 - ???

<@nirik:matrix.scrye.com>

17:07:06

!info chair 2025-04-17 - ???

<@nirik:matrix.scrye.com>

17:07:17

anyone want to claim the 17th or 24th?

<@nirik:matrix.scrye.com>

17:07:46

If not, we could wait for next week...

<@nirik:matrix.scrye.com>

17:09:11

ok then, lets move on to announcements...

<@nirik:matrix.scrye.com>

17:09:26

!topic announcements and information

<@nirik:matrix.scrye.com>

17:09:26

!info CLE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0815 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org

<@nirik:matrix.scrye.com>

17:09:26

!info CLE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1900 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org

<@nirik:matrix.scrye.com>

17:09:26

!info OpenID EOL in Fedora Infra is set to 20-05-2025

<@nirik:matrix.scrye.com>

17:09:34

!info we are in f42 final freeze now

<@nirik:matrix.scrye.com>

17:09:50

oh right, it doesn't like topic with the rest?

<@nirik:matrix.scrye.com>

17:09:56

oh, just slow

<@nirik:matrix.scrye.com>

17:10:12

any other announcements from anyone?

<@carlwgeorge:fedora.im>

17:10:54

Is this the best section for freeze break requests?

<@nirik:matrix.scrye.com>

17:11:33

well, we don't really have one for those, but sure! :)

<@carlwgeorge:fedora.im>

17:12:25

!link https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/thread/O2WU6RDJ255PGYUW6H5MQK7VUT4SV4TE/

<@nirik:matrix.scrye.com>

17:13:01

I'm probibly happy to +1 it, but I haven't gotten to that pass on email yet

<@carlwgeorge:fedora.im>

17:14:08

Cool, mainly wanted to bring it up for awareness. Will wait for that second +1

<@nirik:matrix.scrye.com>

17:14:15

should be pretty reasonable/easy to back out.

<@nirik:matrix.scrye.com>

17:14:22

will do so after the meeting. :)

<@Zlopez:matrix.org>

17:14:39

It looked straightforward, so I already acked it :-)

<@nirik:matrix.scrye.com>

17:15:09

oh, we should probibly adjust that openid announcement... if we have changed plans and are giving it more time...

<@Zlopez:matrix.org>

17:15:33

It's still in discussion right now

<@nirik:matrix.scrye.com>

17:15:41

ok, fair enough.

<@Zlopez:matrix.org>

17:16:07

But I will do another announcement once we decide to go with it

<@nirik:matrix.scrye.com>

17:16:29

<@nirik:matrix.scrye.com>

17:16:36

lets move on to oncall then...

<@nirik:matrix.scrye.com>

17:16:41

!info ??? is on call from 2025-04-18 to 2025-04-24

<@nirik:matrix.scrye.com>

17:16:41

!topic Oncall

<@nirik:matrix.scrye.com>

17:16:41

!info https://docs.fedoraproject.org/en-US/infra/day_to_day_fedora/#_the_oncall_role_in_our_team

<@nirik:matrix.scrye.com>

17:16:41

!info @nirik is on call from 2025-03-21 to 2025-03-27

<@nirik:matrix.scrye.com>

17:16:41

!info @markrosenbaum is on call from 2025-03-28 to 2025-04-03

<@nirik:matrix.scrye.com>

17:16:41

!info @Zlopez is on call from 2025-04-04 to 2025-04-10

<@nirik:matrix.scrye.com>

17:16:41

!info ??? is on call from 2025-04-11 to 2025-04-17

<@nirik:matrix.scrye.com>

17:16:50

oops. Deleting those old ones.

<@nirik:matrix.scrye.com>

17:17:13

anyone want to claim those last 2? otherwise we can wait for next week...

<@Zlopez:matrix.org>

17:17:22

Let me switch the oncall

<@markrosenbaum:fedora.im>

17:17:42

So only one on call pings this week

<@Zlopez:matrix.org>

17:17:51

!oncall

<@zodbot:fedora.im>

17:17:52

The following people are oncall:

<@zodbot:fedora.im>

17:17:52

<@zodbot:fedora.im>

17:17:52

● @Zlopez:matrix.org (zlopez) Current Time for them: 19:17 (Europe/Prague)

<@zodbot:fedora.im>

17:17:52

If they do not respond, please file a ticket (https://pagure.io/fedora-infrastructure/issues)

<@Zlopez:matrix.org>

17:17:57

Set :-)

<@markrosenbaum:fedora.im>

17:18:07

Was about Pagure and it resolved itself

<@markrosenbaum:fedora.im>

17:18:23

Also I can take 2025-04-11 to 2025-04-17

<@markrosenbaum:fedora.im>

17:18:33

Unless someone else wants it

<@nirik:matrix.scrye.com>

17:18:36

cool.

<@nirik:matrix.scrye.com>

17:18:59

updated.

<@nirik:matrix.scrye.com>

17:19:11

On to monitoring...

<@nirik:matrix.scrye.com>

17:19:13

!info https://nagios.fedoraproject.org/nagios

<@nirik:matrix.scrye.com>

17:19:13

!info Go over existing items and fix them

<@nirik:matrix.scrye.com>

17:19:13

!topic Monitoring discussion [nirik]

<@nirik:matrix.scrye.com>

17:19:30

pretty much exactly the same as last week.

<@nirik:matrix.scrye.com>

17:20:09

we continue to see mailman alerts when package updates announce list gets things... I think it's due to invalid addresses on that list that don't bounce right in a way for mailman to handle

<@nirik:matrix.scrye.com>

17:20:35

Thats it from nagios that I can think of.

<@Zlopez:matrix.org>

17:21:08

There was the issue with docs returning 404

<@nirik:matrix.scrye.com>

17:21:24

Gwmngilfen and Mark Rosenbaum did some work on zabbix staging in the last week... hopefully thats going well.

<@Zlopez:matrix.org>

17:21:46

It was fixed by me, but it seems that I fixed it only partly and @darknao:fedora.im did the rest today

<@nirik:matrix.scrye.com>

17:22:02

yeah, hung rsync's. :( Not sure what to do about that... I guess we could look at a timer to kill them if they don't finish in X time. Or a check to make sure the backend docs match the proxy versions

<@markrosenbaum:fedora.im>

17:22:10

Ehhhh, some cursed stuff but it seems ok now

<@markrosenbaum:fedora.im>

17:22:36

Again I just want to confirm, we're not affected by freeze right?

<@nirik:matrix.scrye.com>

17:23:11

zabbix servers are definitely not. If you want to change agents that are on frozen machines... thats best avoided. Staging is never frozen at all...

<@markrosenbaum:fedora.im>

17:23:26

Yeah not agents, just the zabbix server

<@Zlopez:matrix.org>

17:24:07

I saw it happen for first time

<@markrosenbaum:fedora.im>

17:24:17

I had previously asked a while back but Gwmngilfen was asking again on the latest PR

<@nirik:matrix.scrye.com>

17:24:41

the servers are not frozen. They are not important for us to release fedora 42. ;)

<@nirik:matrix.scrye.com>

17:25:29

However, i now notice they don't actually have a 'frozen: false' variable. We can fix that.

<@nirik:matrix.scrye.com>

17:25:46

any other monitoring news?

<@Zlopez:matrix.org>

17:26:25

Not from me

<@nirik:matrix.scrye.com>

17:27:03

I had a few quick topics before we look at backlog or the like...

<@nirik:matrix.scrye.com>

17:27:10

!topic flock 2025

<@nirik:matrix.scrye.com>

17:27:31

Just wanted to note that flock is coming up in a few months... if anyone wants to try and plan to be there

<@nirik:matrix.scrye.com>

17:27:40

https://fedoraproject.org/flock/2025/

<@x3mboy:fedora.im>

17:27:40

Hello team

<@x3mboy:fedora.im>

17:27:44

.hello2

<@x3mboy:fedora.im>

17:27:48

.hello

<@x3mboy:fedora.im>

17:27:53

!hello

<@nirik:matrix.scrye.com>

17:27:53

My infra workshop wasn't accepted. ;(

<@zodbot:fedora.im>

17:27:53

Eduard Lucena (x3mboy) - he / him / his

<@nirik:matrix.scrye.com>

17:28:00

hey x3mboy!

<@x3mboy:fedora.im>

17:28:13

!cookies nirik

<@Zlopez:matrix.org>

17:28:36

None of my 3 talks were accepted as well

<@nirik:matrix.scrye.com>

17:28:49

I think the schedule will be out later this week or early next

<@Zlopez:matrix.org>

17:28:52

From what I heard they had a record amount of proposals

<@markrosenbaum:fedora.im>

17:29:03

Our join talk was accepted but I can’t go :(

<@nirik:matrix.scrye.com>

17:29:15

In any case I am always happy to meet up with anyone face to face who's there... (I plan to be there)

<@nirik:matrix.scrye.com>

17:29:27

Mark Rosenbaum bummer. ;(

<@Zlopez:matrix.org>

17:29:33

Me too, the hallway track is always best :-)

<@nirik:matrix.scrye.com>

17:29:53

ok, just wanted to mention it... next...

<@nirik:matrix.scrye.com>

17:30:03

!topic Datacenter Move

<@nirik:matrix.scrye.com>

17:31:40

Things have been pretty quiet on the DC move front... but they are likely to start heating up soon. The current plan is to switch to the new DC the week of june 16th. So, we have to have things all lined up before then. In the next few weeks I hope to gain out of band access to our new hardware. Then it's bootstrapping up things there, etc... I am sure I will need lots of help from everyone.

<@nirik:matrix.scrye.com>

17:32:05

I have a hackmd doc with plans (which I need to get back to updating): https://hackmd.io/54xmtW6IQoKNKnbRXySxSg?edit

<@Zlopez:matrix.org>

17:32:12

Let me know if you need any help with that

<@nirik:matrix.scrye.com>

17:32:16

if anyone has comments/questions, let me know

<@nirik:matrix.scrye.com>

17:33:42

ok, what shall we do with our 27min left?

<@nirik:matrix.scrye.com>

17:33:47

backlog refinement

<@nirik:matrix.scrye.com>

17:33:53

some learning topic

<@nirik:matrix.scrye.com>

17:34:00

end early and get some time back

<@nirik:matrix.scrye.com>

17:34:12

(feel free to vote with 👍 )

<@markrosenbaum:fedora.im>

17:34:25

Same here

<@nirik:matrix.scrye.com>

17:34:57

Definitely will do!

<@nirik:matrix.scrye.com>

17:36:00

ok, we can do backlog then? Zlopez you want to run it? or want me to?

<@Zlopez:matrix.org>

17:36:09

I can run it

<@Zlopez:matrix.org>

17:36:35

!topic Fedora Infra backlog refinement

<@Zlopez:matrix.org>

17:36:35

!info Refine oldest tickets on Fedora Infra tracker

<@Zlopez:matrix.org>

17:36:35

!link https://pagure.io/fedora-infrastructure/issues?status=Open&order_key=last_updated&order=asc

<@Zlopez:matrix.org>

17:37:03

!ticket 11884

<@zodbot:fedora.im>

17:37:04

● **Opened:** 12 months ago by svashisht

<@zodbot:fedora.im>

17:37:04

● **Assignee:** Not Assigned

<@zodbot:fedora.im>

17:37:04

● **Last Updated:** 6 months ago

<@zodbot:fedora.im>

17:37:04

<@zodbot:fedora.im>

17:37:04

**fedora-infrastructure #11884** (https://pagure.io/fedora-infrastructure/issue/11884):**RFE: Add a role for deploying through OpenShift deployment**

<@nirik:matrix.scrye.com>

17:37:17

This just never got anyone willing to drive it...

<@nirik:matrix.scrye.com>

17:37:24

I think it's still a good idea.

<@Zlopez:matrix.org>

17:38:15

Right now only owner of the openshift project can do that, if I remember it correctly

<@nirik:matrix.scrye.com>

17:38:47

appowners manually yeah, but if there was a playbook anyone who can run the playbook could do so.

<@Zlopez:matrix.org>

17:38:58

That was the point

<@nirik:matrix.scrye.com>

17:39:10

and... we have playbooks like the bodhi one.

<@nirik:matrix.scrye.com>

17:39:34

it copies over all the templates, etc... but then at the end it scales down pods and back up and starts a build...

<@nirik:matrix.scrye.com>

17:40:14

could we do something a bit more clever here:

<@Zlopez:matrix.org>

17:40:22

We only need something for scaling pods down and back up

<@nirik:matrix.scrye.com>

17:40:31

have a manual playbook for every app, but mostly it just includes a generic one that they all use

<@nirik:matrix.scrye.com>

17:40:55

well, no, we need sometimes to start builds/rollout latest deployment.

<@nirik:matrix.scrye.com>

17:41:07

if you just scale down and up again it's still using the same version, not any new one...

<@nirik:matrix.scrye.com>

17:41:27

(as far as I recall)

<@Zlopez:matrix.org>

17:41:59

The build is usually part of normal playbook

<@nirik:matrix.scrye.com>

17:42:04

also, note that all our old apps that use deploymentconfig are slightly different that our newer stuff thats converted to deployment

<@Zlopez:matrix.org>

17:42:35

I need to check how the deployment one looks

<@nirik:matrix.scrye.com>

17:42:51

for 37/60 of them, yeah.

<@Zlopez:matrix.org>

17:42:54

I'm still mostly working with deploymentconfigs

<@Zlopez:matrix.org>

17:43:30

But doing something more general that could be shared by other playbooks sounds great

<@nirik:matrix.scrye.com>

17:43:56

I don't like that the playbook starts a build... it seems like to me the playbook should just handle the config... and then build/deployment rollout is decided seperately... but perhaps I am overcomplicating it.

<@nirik:matrix.scrye.com>

17:44:21

a build should rollout after it finishes.

<@nirik:matrix.scrye.com>

17:44:46

so, the openscanhub playbook doesn't do a build.

<@Zlopez:matrix.org>

17:44:50

That could be configured in build config

<@nirik:matrix.scrye.com>

17:45:02

so, just adding that for now would 'fix' this issue I guess.

<@Zlopez:matrix.org>

17:46:03

But it would still be neat to have something for just build and rollout

<@nirik:matrix.scrye.com>

17:47:50

I guess... but when would you want to build the new thing and not roll it out?

<@Zlopez:matrix.org>

17:48:58

We should probably write some of the OpenShift project best practices, so people know how the OpenShift project should look

<@nirik:matrix.scrye.com>

17:49:21

yeah, dkirwan did write up an initial version:

<@Zlopez:matrix.org>

17:49:32

I always have deployment triggered when image is changed, which means new build will automatically deploy new version

<@nirik:matrix.scrye.com>

17:49:43

https://docs.fedoraproject.org/en-US/infra/developer_guide/openshift/

<@nirik:matrix.scrye.com>

17:50:04

sorry, this one:

<@nirik:matrix.scrye.com>

17:50:30

https://docs.fedoraproject.org/en-US/infra/developer_guide/openshift_bestpractices/

<@nirik:matrix.scrye.com>

17:50:38

copy and paste was doing weird things there. ;(

<@Zlopez:matrix.org>

17:50:47

Nice

<@nirik:matrix.scrye.com>

17:51:25

anyhow, not sure what to do with this ticket and we have been talking about it for a while. ;)

<@nirik:matrix.scrye.com>

17:51:47

close for now? leave and try and find someone to work on playbooks? discuss more in a discussion thread or something?

<@Zlopez:matrix.org>

17:52:07

Let me just update it with what we talked about and go to next one

<@nirik:matrix.scrye.com>

17:52:18

<@Zlopez:matrix.org>

17:52:23

I still think this is a nice feature to have

<@Zlopez:matrix.org>

17:52:42

!ticket 11958

<@zodbot:fedora.im>

17:52:45

<@zodbot:fedora.im>

17:52:45

**fedora-infrastructure #11958** (https://pagure.io/fedora-infrastructure/issue/11958):**Add fedora-l10n pagure group as an admin to the fedora-l10n-docs namespace projects**

<@zodbot:fedora.im>

17:52:45

● **Assignee:** Not Assigned

<@zodbot:fedora.im>

17:52:45

● **Last Updated:** 6 months ago

<@zodbot:fedora.im>

17:52:45

● **Opened:** 11 months ago by peartown

<@nirik:matrix.scrye.com>

17:53:07

ugh, this one fell to the bottom of the pile

<@nirik:matrix.scrye.com>

17:53:41

I guess someone just needs to do this.

<@Zlopez:matrix.org>

17:53:51

I think it's a good easy ticket for anyone with admin rights for pagure

<@nirik:matrix.scrye.com>

17:54:04

thats 52 projects

<@Zlopez:matrix.org>

17:54:14

As it's not difficult just plenty manual work

<@x3mboy:fedora.im>

17:54:47

Is that something that I can do with my current permissions?

<@Zlopez:matrix.org>

17:55:12

Let me check

<@nirik:matrix.scrye.com>

17:55:17

I would think something could be done via the api? clicking on 52 projects seems error prone

<@nirik:matrix.scrye.com>

17:55:35

likely not, but you could make a script that uses the api and hand it to someone to run?

<@Zlopez:matrix.org>

17:55:58

That could work as well :-)

<@x3mboy:fedora.im>

17:55:59

Ok, I'll check it

<@nirik:matrix.scrye.com>

17:56:41

at least I hope it could be done via the api...

<@Zlopez:matrix.org>

17:56:46

Looking at your permissions you can't, but you can still write the script using API :-)

<@nirik:matrix.scrye.com>

17:56:53

thanks x3mboy... that would be helpful.

<@Zlopez:matrix.org>

17:57:30

@x3mboy:fedora.im Do you want to be assigned to this ticket?

<@x3mboy:fedora.im>

17:57:56

Sure

<@nirik:matrix.scrye.com>

17:58:17

we are getting low on time, lets hit open floor...

<@nirik:matrix.scrye.com>

17:58:21

!topic Open Floor

<@nirik:matrix.scrye.com>

17:58:29

Just a few bits from me:

<@Zlopez:matrix.org>

17:58:56

@x3mboy:fedora.im Done, have fun :-)

<@nirik:matrix.scrye.com>

17:59:18

There was a bit of a security scare with atop over the weekend. So to be paranoid I uninstalled it. Turned out to be not much of a issue, but I decided to replace atop with btop on our machines. So, btop should be there if you need it.

<@Zlopez:matrix.org>

17:59:51

The btop is great, I didn't even know it existed :-)

<@Zlopez:matrix.org>

18:00:10

I just saw it in command history when doing something on some of the machines :-)

<@nirik:matrix.scrye.com>

18:00:40

I also did some more blocking on pagure.io yesterday. There was a copy of the linux kernel that was 4-5years old labeled 'test' from someone, but all the crawlers were hitting it really hard. So, I gathered the ips (140,000 or so), collapsed them into /24's and created a 'spammers' ipset matching those nets. It seems to have helped a fair bit.

<@zodbot:fedora.im>

18:01:11

zlopez has already given cookies to kevin during the F41 timeframe

<@nirik:matrix.scrye.com>

18:01:35

we still do need a longer term solution, but hopefully this will tide us over for a bit at least

<@nirik:matrix.scrye.com>

18:01:47

2679116 160575818 REJECT tcp -- * * 0.0.0.0/0 0.0.0.0/0 match-set spammers src reject-with icmp-port-unreachable

<@nirik:matrix.scrye.com>

18:01:47

pkts bytes target prot opt in out source destination

<@nirik:matrix.scrye.com>

18:02:16

I figure anyone going to that project was likely a crawler.

<@Zlopez:matrix.org>

18:02:33

I need to look how does one work with ipsets

<@nirik:matrix.scrye.com>

18:02:54

I could do a learning/talk on it next week? or sometime else?

<@nirik:matrix.scrye.com>

18:03:08

anyhow, thats all I had...

<@nirik:matrix.scrye.com>

18:03:20

Thanks for coming everyone!

<@nirik:matrix.scrye.com>

18:03:23

!endmeeting