#meeting-3:fedoraproject.org log

<@nirik:matrix.scrye.com>

17:00:08

!startmeeting Infrastructure (2025-01-16)

<@meetbot:fedora.im>

17:00:10

Meeting started at 2025-01-16 17:00:08 UTC

<@meetbot:fedora.im>

17:00:11

The Meeting name is 'Infrastructure (2025-01-16)'

<@nirik:matrix.scrye.com>

17:00:18

!meetingname infrastructure

<@nirik:matrix.scrye.com>

17:00:18

!chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp

<@nirik:matrix.scrye.com>

17:00:18

!info Agenda is at: https://board.net/p/fedora-infra

<@nirik:matrix.scrye.com>

17:00:18

!info About our team: https://docs.fedoraproject.org/en-US/cle/

<@nirik:matrix.scrye.com>

17:00:18

!info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra

<@nirik:matrix.scrye.com>

17:00:18

!topic ahoy

<@meetbot:fedora.im>

17:00:20

The Meeting Name is now infrastructure

<@humaton:fedora.im>

17:00:58

!hi

<@zodbot:fedora.im>

17:00:59

Tomáš Hrčka (humaton) - he / him / his

<@nirik:matrix.scrye.com>

17:01:16

morning all.

<@humaton:fedora.im>

17:01:59

is it jsut the 2 of us :D

<@humaton:fedora.im>

17:02:13

is it just the 2 of us :D

<@nirik:matrix.scrye.com>

17:02:19

So far... I guess we can wait a few more and see who else makes it. ;)

<@humaton:fedora.im>

17:03:01

how is the winter over there?

<@humaton:fedora.im>

17:03:28

we have morning fog, during day snow and freeze over the night

<@nirik:matrix.scrye.com>

17:03:46

odd. No snow at all (just one day it snowed, but never stuck)... but yeah, last week we have had 'freezing fog'...

<@humaton:fedora.im>

17:04:14

oh yeah that happens more this winter than I have ever seen

<@humaton:fedora.im>

17:04:42

Fog than sudden -3C and it sticks to everything

<@nirik:matrix.scrye.com>

17:04:43

not much rain lately either, which is weird. winter is out wet time here. :)

<@nirik:matrix.scrye.com>

17:07:31

well, shall we go ahead with the meeting? or just cancel today since it's only us two. ;)

<@humaton:fedora.im>

17:08:12

well I have just one thing to ponder, and that is how are we going to deploy forgejo

<@humaton:fedora.im>

17:08:43

we have AWX template from rocky guys and kubernetes helm from upstream

<@nirik:matrix.scrye.com>

17:09:18

well, I havent had a chance to look much, but I wonder if we couldn't just deploy like we do all our other openshift apps?

<@nirik:matrix.scrye.com>

17:09:33

via ansible, etc...

<@humaton:fedora.im>

17:09:41

well it will be the most complex openshift app we have

<@nirik:matrix.scrye.com>

17:09:41

but again, I don't know how complex it is.

<@humaton:fedora.im>

17:09:55

So I would say maybe we want some more sophisticated deployment

<@humaton:fedora.im>

17:10:35

I think we should aim for HA deployment https://code.forgejo.org/forgejo-helm/forgejo-helm/src/branch/main/docs/ha-setup.md

<@james:fedora.im>

17:10:42

I'm here, but slow as usual

<@nirik:matrix.scrye.com>

17:10:44

!topic forejo deployment

<@james:fedora.im>

17:10:49

I think Carl is still on vacation

<@nirik:matrix.scrye.com>

17:11:33

yeah, I would love a HA setup allowing us to update/etc without downtime...

<@humaton:fedora.im>

17:11:37

Running it all in openshift including DB will drop sing point of failure in the PGSQL

<@humaton:fedora.im>

17:12:10

If you have time look at the helm thing. I dont want to use it but its nice inspiration

<@nirik:matrix.scrye.com>

17:12:26

I'll note there they say to use a cloud db serice. ;(

<@humaton:fedora.im>

17:12:27

having ansible operator doing that would be lovely...

<@nirik:matrix.scrye.com>

17:12:37

yeah, that would be nice.

<@nirik:matrix.scrye.com>

17:13:11

we could also do something like we did when we deployed osbs... have a thing in our ansible that checks out and runs the other complex thing (helm, whatever)

<@humaton:fedora.im>

17:14:17

I dont want to introduce yet another deployment method so avoiding the helm thing

<@nirik:matrix.scrye.com>

17:14:29

we may have to try several things and see what works best. I don't think we should be afraid to try and scrap something that doesn't work out

<@humaton:fedora.im>

17:14:59

the helm is extra easy to run harder to modify

<@humaton:fedora.im>

17:15:27

I played with it in comushift and it was as easy as helm run ...

<@humaton:fedora.im>

17:15:54

But we will need our images, I would like to avoid using the upstream images because they use debian

<@humaton:fedora.im>

17:16:28

so plugiing CI and image build together, pushing things to quay/maybe Konflux?/

<@humaton:fedora.im>

17:16:43

and deploying using operator

<@james:fedora.im>

17:17:31

When you say modify ... do you mean configuration or upgrades, or something more like "if you want to not use their specific version of postgresql you will have problems"

<@Zlopez:matrix.org>

17:17:52

!hi

<@zodbot:fedora.im>

17:17:53

Michal Konecny (zlopez)

<@humaton:fedora.im>

17:17:54

James Antill: the later

<@humaton:fedora.im>

17:18:33

configuration an UI modifications are done during imagebuild

<@nirik:matrix.scrye.com>

17:18:43

image building pipeline is one part, then deployment/config. If we build images right perhaps the deployment won't need that much complexity ?

<@humaton:fedora.im>

17:19:30

oh yeah image definitions and builds are the first thing we need to do

<@nirik:matrix.scrye.com>

17:20:47

there's some packaging attempts on copr I see: https://copr.fedorainfracloud.org/coprs/fulltext/?fulltext=forgejo

<@nirik:matrix.scrye.com>

17:21:26

oh, duh, it's already in

<@nirik:matrix.scrye.com>

17:21:44

oh, no, thats just the client lib

<@nirik:matrix.scrye.com>

17:21:47

https://bugzilla.redhat.com/show_bug.cgi?id=2268288

<@nirik:matrix.scrye.com>

17:22:16

anyhow. yeah, making a rhel/fedora image that has everything and works is probibly a good first step and can be done by anyone locally

<@Zlopez:matrix.org>

17:22:58

That doesn't sound too difficult

<@humaton:fedora.im>

17:23:15

well having the app is just small step

<@humaton:fedora.im>

17:23:27

there is redis there is DB there is minio

<@humaton:fedora.im>

17:23:43

btw iff you are unfamiliar with minio I recomend checking out

<@humaton:fedora.im>

17:23:57

its local s3 compatible free sw

<@Zlopez:matrix.org>

17:24:14

How much could we use from current test deployment?

<@humaton:fedora.im>

17:24:22

nothing

<@humaton:fedora.im>

17:24:30

the test deloyemnt should be nuked already

<@humaton:fedora.im>

17:24:47

and it was the default from upstream without any real production features

<@nirik:matrix.scrye.com>

17:24:52

FWIW, openshift has s3 compatible storage also...

<@humaton:fedora.im>

17:24:59

kind of devel deployment for hacking

<@Zlopez:matrix.org>

17:25:10

<@humaton:fedora.im>

17:25:45

oh minio is in openshift

<@nirik:matrix.scrye.com>

17:28:04

still a lot to investigate. ;)

<@humaton:fedora.im>

17:28:06

SO I am thinking about creating new project under fedora-infra group in pagure to track the deployment/migration

<@Zlopez:matrix.org>

17:28:47

<@humaton:fedora.im>

17:29:01

So far I have few tickets in mind: 1 OCI artifacts definitions, Explore Konflux to build images, Make the operator

<@humaton:fedora.im>

17:29:13

So far I have few tickets in mind: OCI artifacts definitions, Explore Konflux to build images, Make the operator

<@humaton:fedora.im>

17:29:40

there will be more granularity in it but those topics are the first totackle

<@nirik:matrix.scrye.com>

17:29:42

sounds like a good start.

<@humaton:fedora.im>

17:29:49

there will be more granularity in it but those topics are the first to tackle

<@humaton:fedora.im>

17:30:35

That is all from the top of my head

<@nirik:matrix.scrye.com>

17:31:19

should we go thru the rest of the meeting now that more folks are here?

<@nirik:matrix.scrye.com>

17:31:53

lets go ahead then...

<@nirik:matrix.scrye.com>

17:31:56

!info chair 2025-01-30 - ???

<@nirik:matrix.scrye.com>

17:31:56

!topic Next chair

<@nirik:matrix.scrye.com>

17:31:56

!info chair 2025-01-23 - Yaash

<@nirik:matrix.scrye.com>

17:31:56

!info magic eight ball says:

<@nirik:matrix.scrye.com>

17:32:02

anyone want chair for the 30th?

<@humaton:fedora.im>

17:32:21

I will be on my way to fosdem

<@nirik:matrix.scrye.com>

17:32:58

I guess we can decide next week anyhow. ;)

<@nirik:matrix.scrye.com>

17:33:03

!topic announcements and information

<@nirik:matrix.scrye.com>

17:33:03

!info CLE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org

<@nirik:matrix.scrye.com>

17:33:03

!info CLE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org

<@nirik:matrix.scrye.com>

17:33:07

any other announcements?

<@Zlopez:matrix.org>

17:34:47

!info release-monitoring.org v2.0.0 is now available

<@nirik:matrix.scrye.com>

17:34:56

ok, moving along then...

<@nirik:matrix.scrye.com>

17:35:00

!info https://docs.fedoraproject.org/en-US/infra/day_to_day_fedora/#_the_oncall_role_in_our_team

<@nirik:matrix.scrye.com>

17:35:00

!info ??? is on call from 2025-01-23 to 2025-01-30

<@nirik:matrix.scrye.com>

17:35:00

!info zlopez is on call from 2025-01-16 to 2025-01-23

<@nirik:matrix.scrye.com>

17:35:00

!topic Oncall

<@Zlopez:matrix.org>

17:35:18

Let me switch the oncall

<@nirik:matrix.scrye.com>

17:35:32

anyone want oncall that second week? If not, we can decide next week

<@Zlopez:matrix.org>

17:36:18

!oncall

<@zodbot:fedora.im>

17:36:19

● @Zlopez:matrix.org (zlopez) Current Time for them: 18:36 (Europe/Prague)

<@zodbot:fedora.im>

17:36:19

If they do not respond, please file a ticket (https://pagure.io/fedora-infrastructure/issues)

<@zodbot:fedora.im>

17:36:19

The following people are oncall:

<@zodbot:fedora.im>

17:36:19

<@Zlopez:matrix.org>

17:36:25

Set

<@nirik:matrix.scrye.com>

17:36:26

cool

<@nirik:matrix.scrye.com>

17:36:30

!info Go over existing items and fix them

<@nirik:matrix.scrye.com>

17:36:30

!topic Monitoring discussion [nirik]

<@nirik:matrix.scrye.com>

17:36:30

!info https://nagios.fedoraproject.org/nagios

<@nirik:matrix.scrye.com>

17:37:09

we have a builder I need to reinstall, the logdetective02 still showing up and...

<@nirik:matrix.scrye.com>

17:37:23

there's a rabbitmq queue that I don't think we use anymore, but it's alerting on it:

<@nirik:matrix.scrye.com>

17:37:27

RABBITMQ_QUEUE CRITICAL - Not Found : http://localhost:15672/api/queues/%2Fpubsub/toddlers-distgit-commit-processor

<@Zlopez:matrix.org>

17:37:48

That one is no longer running

<@nirik:matrix.scrye.com>

17:37:49

I think that moved to a poddlers queue or something? but we need to confirm and remove monitoring for it.

<@nirik:matrix.scrye.com>

17:37:58

yeah, it's alerting because it's not there...

<@Zlopez:matrix.org>

17:38:25

Do we have ticket for that?

<@nirik:matrix.scrye.com>

17:39:01

nope, it happened over the break I think. I can remove the monitoring, but I just wanted to make sure it was right that the queue no longer exists.

<@nirik:matrix.scrye.com>

17:39:38

so I can do that then if you are sure it's supposed to be gone

<@Zlopez:matrix.org>

17:39:41

Everything should be now moved to poddlers

<@nirik:matrix.scrye.com>

17:40:01

yeah, I wasn't sure why the queue name changed tho... but fine. ;)

<@Zlopez:matrix.org>

17:40:02

Better confirm with @abompard:fedora.im He did the move

<@nirik:matrix.scrye.com>

17:40:39

The last alert is disk space on bastion01. This is caused by https://pagure.io/fedora-infrastructure/issue/12361

<@nirik:matrix.scrye.com>

17:41:20

anything else on monitoring?

<@nirik:matrix.scrye.com>

17:43:08

ok, shall we just go to open floor?

<@nirik:matrix.scrye.com>

17:43:12

or triage some old tickets?

<@Zlopez:matrix.org>

17:43:24

I would go with open floor

<@nirik:matrix.scrye.com>

17:43:46

!topic open floor

<@nirik:matrix.scrye.com>

17:43:56

any other discussion items?

<@Zlopez:matrix.org>

17:43:57

I'm wondering if https://pagure.io/fedora-infrastructure/issue/12361 isn't caused by something that happened during mailman deployment

<@nirik:matrix.scrye.com>

17:44:29

shouldn't be... it's a completely different system

<@abompard:fedora.im>

17:44:41

Yeah you can remove the monitoring, it's for a disabled toddler

<@nirik:matrix.scrye.com>

17:45:06

hey Aurélien B! ok.

<@abompard:fedora.im>

17:45:16

hey :-)

<@nirik:matrix.scrye.com>

17:45:23

and to answer your query from eariler, no, I cannot login to accounts with my email address. ;(

<@Zlopez:matrix.org>

17:45:29

Didn't @nphilipp:fedora.im worked on that one?

<@nirik:matrix.scrye.com>

17:45:38

"Unauthorized: bad credentials."

<@nirik:matrix.scrye.com>

17:45:55

he worked on the ipa/ansible deployment... so perhaps he can see whats happening

<@nirik:matrix.scrye.com>

17:46:48

basically all users are there (even if they cannot login) so postfix sees them as users and the ones without a fedoraproject.org alias just get locally delivered email. ;( (at least from what I could see from a quick glance)

<@Zlopez:matrix.org>

17:48:02

I meant the distgit_commit_processor

<@Zlopez:matrix.org>

17:48:15

I meant the distgit\_commit\_processor

<@Zlopez:matrix.org>

17:48:27

I meant the distgit-commit-processor

<@nirik:matrix.scrye.com>

17:48:29

hum, not sure... I suppose possibly?

<@Zlopez:matrix.org>

17:48:51

I remember it was related to one open ticket

<@Zlopez:matrix.org>

17:49:29

I think this one https://pagure.io/fedora-infrastructure/issue/11641

<@Zlopez:matrix.org>

17:50:05

But I see that last update was 9 months ago https://pagure.io/fedora-infra/toddlers/issue/183

<@Zlopez:matrix.org>

17:50:14

So probably it was forgotten

<@nirik:matrix.scrye.com>

17:50:15

well, I don't see how that would affect this... but yeah

<@abompard:fedora.im>

17:50:28

Ah, right, it works in ipsilon but not in noggin

<@nirik:matrix.scrye.com>

17:50:30

if you could ping him on it that would be great. There's a PR related too.

<@Zlopez:matrix.org>

17:51:14

This is for the rabbitmq queue and disabled toddler, it's not related to the authentication problem

<@nirik:matrix.scrye.com>

17:51:17

https://pagure.io/fedora-infra/ansible/pull-request/2251

<@nirik:matrix.scrye.com>

17:51:37

ah, sorry, mixing things. ;)

<@Zlopez:matrix.org>

17:51:52

I'm probably just causing confusion :-D

<@Zlopez:matrix.org>

17:52:37

Should it work to login with e-mail? I always use username

<@nirik:matrix.scrye.com>

17:52:40

anyhow, we can work on all these out of meeting. ;) anything else for the meeting, or shall we close?

<@nirik:matrix.scrye.com>

17:53:00

it should...we did have it working everywhere with email at one point.

<@Zlopez:matrix.org>

17:53:24

I probably never tried it

<@nirik:matrix.scrye.com>

17:53:32

we have been getting a flow of users saying they cannot login recently due to this... they are using email...

<@Zlopez:matrix.org>

17:53:51

Which mail it's using the fedoraproject.org one?

<@Zlopez:matrix.org>

17:54:15

Or the user e-mail?

<@nirik:matrix.scrye.com>

17:55:15

The email associated with the account.

<@Zlopez:matrix.org>

17:55:55

Doesn't work for me as well

<@Zlopez:matrix.org>

17:56:44

Need to use username instead

<@nirik:matrix.scrye.com>

17:56:59

ok, lets continue over in #noc:fedoraproject.org and #admin:fedoraproject.org then.

<@nirik:matrix.scrye.com>

17:57:04

Thanks for coming everyone

<@nirik:matrix.scrye.com>

17:57:10

!endmeeting