#meeting-3:fedoraproject.org log

<@Zlopez:matrix.org>

16:59:36

!startmeeting Infrastructure (2024-12-12)

<@meetbot:fedora.im>

16:59:38

Meeting started at 2024-12-12 16:59:36 UTC

<@meetbot:fedora.im>

16:59:38

The Meeting name is 'Infrastructure (2024-12-12)'

<@Zlopez:matrix.org>

16:59:48

!info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra

<@Zlopez:matrix.org>

16:59:48

!info About our team: https://docs.fedoraproject.org/en-US/cpe/

<@Zlopez:matrix.org>

16:59:48

!topic ahoy

<@Zlopez:matrix.org>

16:59:48

!chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp

<@Zlopez:matrix.org>

16:59:48

!meetingname infrastructure

<@Zlopez:matrix.org>

16:59:48

!info Agenda is at: https://board.net/p/fedora-infra

<@meetbot:fedora.im>

16:59:49

The Meeting Name is now infrastructure

<@Zlopez:matrix.org>

16:59:56

!hi

<@zodbot:fedora.im>

16:59:57

Michal Konecny (zlopez)

<@lenkaseg:matrix.org>

17:00:07

!hi

<@Zlopez:matrix.org>

17:00:08

Welcome everyone

<@zodbot:fedora.im>

17:00:09

No Fedora Accounts users have the @lenkaseg:matrix.org Matrix Account defined

<@thisisyaash:fedora.im>

17:01:17

!hi

<@zodbot:fedora.im>

17:01:18

Yashwanth Rathakrishnan (thisisyaash)

<@james:fedora.im>

17:01:22

!hi

<@zodbot:fedora.im>

17:01:23

James Antill (james)

<@nirik:matrix.scrye.com>

17:01:26

morning

<@Zlopez:matrix.org>

17:01:43

How is everyone?

<@nirik:matrix.scrye.com>

17:03:00

tired and annoyed that things are broken... otherwise fine.

<@humaton:fedora.im>

17:03:04

!hi

<@zodbot:fedora.im>

17:03:05

Tomáš Hrčka (humaton) - he / him / his

<@thisisyaash:fedora.im>

17:03:12

fantastic :D

<@Zlopez:matrix.org>

17:03:44

@nirik:matrix.scrye.com I noticed few of them :-/

<@Zlopez:matrix.org>

17:04:05

I'm trying to fix what I messed in the latest changes for release-monitoring

<@Zlopez:matrix.org>

17:04:36

As this is my last day in work for this year, I want to finish what I started

<@zardian:matrix.org>

17:05:04

!hi

<@Zlopez:matrix.org>

17:05:06

Let's continue to your first topic

<@zodbot:fedora.im>

17:05:07

Aman Singh (zardian)

<@Zlopez:matrix.org>

17:05:16

!topic New folks introductions

<@Zlopez:matrix.org>

17:05:16

!info This is a place where people who are interested in Fedora Infrastructure can introduce themselves

<@Zlopez:matrix.org>

17:05:16

!info Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/

<@Zlopez:matrix.org>

17:05:51

Anybody new here?

<@zardian:matrix.org>

17:07:08

Not new here, but able to catch back after a long time.

<@seddik:fedora.im>

17:07:12

!hi

<@zodbot:fedora.im>

17:07:13

seddik alaouiismaili (seddik)

<@Zlopez:matrix.org>

17:07:22

@zardian:matrix.org Welcome back

<@zardian:matrix.org>

17:07:42

Thanks, Hello everyone!!

<@Zlopez:matrix.org>

17:08:14

Let's look at who will chair the next meeting :-)

<@Zlopez:matrix.org>

17:08:15

!info magic eight ball says:

<@Zlopez:matrix.org>

17:08:15

!info chair 2025-01-16 - ???

<@Zlopez:matrix.org>

17:08:15

!info chair 2025-01-09 - phsmoura

<@Zlopez:matrix.org>

17:08:15

!topic Next chair

<@nirik:matrix.scrye.com>

17:08:15

welcome

<@seddik:fedora.im>

17:08:52

Can take 16

<@Zlopez:matrix.org>

17:09:15

The next three weeks are cancelled as people are on holidays mostly

<@Zlopez:matrix.org>

17:09:22

@seddik:fedora.im It's yours

<@Zlopez:matrix.org>

17:09:31

!info chair 2025-01-16 - saibug

<@Zlopez:matrix.org>

17:10:02

!info chair 2025-01-23 - ???

<@Zlopez:matrix.org>

17:10:30

Does anybody wants to go for 23th January

<@Zlopez:matrix.org>

17:10:51

We don't need to fill it up now, just asking for volunteer in advance

<@thisisyaash:fedora.im>

17:11:08

i will take it

<@Zlopez:matrix.org>

17:11:46

And sold :-)

<@Zlopez:matrix.org>

17:11:59

!info chair 2025-01-23 - Yaash

<@Zlopez:matrix.org>

17:12:49

I think we are covered for now :-)

<@Zlopez:matrix.org>

17:13:05

!info CLE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org

<@Zlopez:matrix.org>

17:13:05

!info CLE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org

<@Zlopez:matrix.org>

17:13:05

!topic announcements and information

<@Zlopez:matrix.org>

17:13:15

Any other announcements?

<@Zlopez:matrix.org>

17:13:42

!info Holidays incoming, people could be unavailable for next few weeks

<@Zlopez:matrix.org>

17:14:19

!info Infrastructure meeting for 19th, 26th December and 2nd January is CANCELLED

<@Zlopez:matrix.org>

17:14:28

Did I miss something?

<@nirik:matrix.scrye.com>

17:14:43

!info happy holidays to everyone!

<@seddik:fedora.im>

17:14:52

LGTM

<@Zlopez:matrix.org>

17:16:10

Let's move to next topic

<@Zlopez:matrix.org>

17:16:20

!info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/

<@Zlopez:matrix.org>

17:16:20

!info nirik is on call from 2024-12-06 to 2024-12-12

<@Zlopez:matrix.org>

17:16:20

!info https://fedoraproject.org/wiki/Infrastructure/Oncall

<@Zlopez:matrix.org>

17:16:20

!info ??? is on call from 2024-12-13 to 2024-12-19

<@Zlopez:matrix.org>

17:16:20

!topic Oncall

<@Zlopez:matrix.org>

17:16:36

Do we want to have oncall during holidays?

<@Zlopez:matrix.org>

17:16:50

Or should we just ask people to fill tickets?

<@nirik:matrix.scrye.com>

17:17:18

right, lets just unset it and have people file tickets

<@Zlopez:matrix.org>

17:17:57

And set the next one at the next meeting

<@nirik:matrix.scrye.com>

17:17:59

Also, if anyone is working on something, do keep in mind that you want to be carefull not to pull others back from holidays

<@thisisyaash:fedora.im>

17:18:34

quick question, what does this 'on call' means? :)

<@nirik:matrix.scrye.com>

17:18:58

https://docs.fedoraproject.org/en-US/infra/day_to_day_fedora/#_the_oncall_role_in_our_team

<@Zlopez:matrix.org>

17:18:58

!info NOBODY is on call from 2024-12-13 to 2025-01-09

<@Zlopez:matrix.org>

17:19:48

Should we remove the wiki page for oncall from info?

<@offdutypirate:fedora.im>

17:19:59

The link https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/ in the above notes in not valid.

<@Zlopez:matrix.org>

17:20:12

Let me fix that

<@Zlopez:matrix.org>

17:20:27

!info https://docs.fedoraproject.org/en-US/infra/day_to_day_fedora/#_the_oncall_role_in_our_team

<@nirik:matrix.scrye.com>

17:20:57

yeah, we should nuke that cpe one and wiki one. ;)

<@Zlopez:matrix.org>

17:21:00

Changed it in agenda document

<@thisisyaash:fedora.im>

17:21:46

can i make a PR updating it with current doc page?

<@thisisyaash:fedora.im>

17:21:46

actually, i found that infra/ansible repo README.md in pagure still uses wiki

<@thisisyaash:fedora.im>

17:22:02

actually, i found that infra/ansible repo's README.md in pagure still uses wiki

<@thisisyaash:fedora.im>

17:22:02

can i make a PR updating it with current doc page?

<@Zlopez:matrix.org>

17:22:09

Definitely

<@thisisyaash:fedora.im>

17:22:48

okay then

<@Zlopez:matrix.org>

17:23:10

We are usually not good at keeping READMEs and docs up to date

<@nirik:matrix.scrye.com>

17:24:16

it's hard when there's a flood of higher pri work. ;(

<@thisisyaash:fedora.im>

17:24:18

yup, but it gives more opportunity for new contributors tho XD

<@Zlopez:matrix.org>

17:24:26

As we have oncall "covered" let us look at the next topic

<@Zlopez:matrix.org>

17:24:35

!info Summary of last week: (from current oncall)

<@Zlopez:matrix.org>

17:24:59

@nirik:matrix.scrye.com That is true

<@Zlopez:matrix.org>

17:25:25

@nirik:matrix.scrye.com You were the one on oncall, so the floor is yours for now

<@nirik:matrix.scrye.com>

17:25:48

I don't know of any oncall alerts... or at least I don't recall any

<@Zlopez:matrix.org>

17:26:34

I didn't saw anything as well

<@Zlopez:matrix.org>

17:27:37

!info Go over existing items and fix them

<@Zlopez:matrix.org>

17:27:37

!topic Monitoring discussion [nirik]

<@Zlopez:matrix.org>

17:27:37

!info https://nagios.fedoraproject.org/nagios

<@nirik:matrix.scrye.com>

17:27:59

lets see...

<@Zlopez:matrix.org>

17:28:00

I see only the logdetective in nagios

<@nirik:matrix.scrye.com>

17:28:12

we are down to just that one alert. ;)

<@nirik:matrix.scrye.com>

17:28:33

we have been told we don't need to monitor that instance. It's often off.

<@Zlopez:matrix.org>

17:28:34

That is really good before the holidays :-)

<@nirik:matrix.scrye.com>

17:28:51

but I was unable so far to get nagios to stop doing so. ;( If someone wants to look into it, feel free.

<@seddik:fedora.im>

17:29:08

I can take a look

<@Zlopez:matrix.org>

17:29:29

I thought it would be simple to remove a machine

<@seddik:fedora.im>

17:29:43

From monitoring??

<@nirik:matrix.scrye.com>

17:29:51

there's some variables to get it to not monitor, and I set them, but perhaps I missed one.

<@nirik:matrix.scrye.com>

17:30:21

nagios assembles it's config from inventory with ansible templates. So default is to monitor

<@nirik:matrix.scrye.com>

17:31:02

saibug: if you want to look that would be excellent. ;)

<@Zlopez:matrix.org>

17:33:18

Let's see what the agenda has next for us

<@nirik:matrix.scrye.com>

17:33:45

we could touch on current broken things? ;)

<@Zlopez:matrix.org>

17:33:55

That is a good idea

<@Zlopez:matrix.org>

17:34:29

I saw that @abompard:fedora.im already has some fix in mind for elections

<@Zlopez:matrix.org>

17:34:59

I'm not sure how is the situation about the koji-listener

<@nirik:matrix.scrye.com>

17:35:05

Yeah, I am sorry about breaking that. ;( I am not sure if we need to ask if current elections need to be extended given people cannot currently vote

<@nirik:matrix.scrye.com>

17:35:27

any idea when elections fix might land?

<@Zlopez:matrix.org>

17:35:32

I wanted to vote today as well :-/

<@nirik:matrix.scrye.com>

17:35:46

I thought another option might be pinning the thing that changed and broke it to an older version?

<@Zlopez:matrix.org>

17:36:06

I'm not sure if @abompard:fedora.im is here today and I only saw that in the ticket

<@Zlopez:matrix.org>

17:36:18

!link https://pagure.io/fedora-infrastructure/issue/12330

<@nirik:matrix.scrye.com>

17:36:22

I talked with him eariler in noc...

<@abompard:fedora.im>

17:36:39

Just finishing the fixes! :-)

<@nirik:matrix.scrye.com>

17:36:45

yeah, so... hey!

<@nirik:matrix.scrye.com>

17:37:12

I was going to suggest pinning the flask-wtforms too... but if you have a fix thats probibly better

<@abompard:fedora.im>

17:37:16

I think I got it to work, I still have to commit those in elections, and then it'll need quite a bit of work to fix the unit tests

<@nirik:matrix.scrye.com>

17:37:50

I think if we can just bandaid it for now to get it working and fix it properly later that would be fine

<@nirik:matrix.scrye.com>

17:38:18

BTW, auth also broke on it, but I managed to stumble my way to a fix on that

<@abompard:fedora.im>

17:38:18

yeah that's the plan, the bandaid should be in place right now

<@abompard:fedora.im>

17:38:54

oh yeah when looking at the code I was wondering how auth could work at all 😁

<@nirik:matrix.scrye.com>

17:39:29

yeah... flask-oidc upgraded a bunch and it only was working with the old one...

<@nirik:matrix.scrye.com>

17:39:35

but anyhow... so people can vote again now

<@Zlopez:matrix.org>

17:40:05

@abompard:fedora.im I thought the same when I saw the ansible-lint job for ansible :-D

<@Zlopez:matrix.org>

17:40:26

And it didn't worked, it just silently passed without checking anything

<@zodbot:fedora.im>

17:40:32

kevin gave a cookie to abompard. They now have 118 cookies, 1 of which were obtained in the Fedora 41 release cycle

<@nirik:matrix.scrye.com>

17:41:13

so, that just leaves the koji_sync_listener problem. I recreated the queue, but need to have a message that should get to it happen to see if that helps any.

<@zodbot:fedora.im>

17:41:25

zlopez gave a cookie to abompard. They now have 119 cookies, 2 of which were obtained in the Fedora 41 release cycle

<@nirik:matrix.scrye.com>

17:41:30

and... it just did

<@zodbot:fedora.im>

17:41:43

zlopez gave a cookie to kevin. They now have 705 cookies, 13 of which were obtained in the Fedora 41 release cycle

<@zodbot:fedora.im>

17:42:22

abompard gave a cookie to kevin. They now have 706 cookies, 14 of which were obtained in the Fedora 41 release cycle

<@nirik:matrix.scrye.com>

17:42:26

so... hurray? but why do queues get in this state, and how can we detect them?

<@abompard:fedora.im>

17:42:49

yeah I don't know, maybe we should recreate all the bindings (without deleting the queues)

<@Zlopez:matrix.org>

17:42:55

I thought that we are monitoring the queues

<@abompard:fedora.im>

17:43:11

rabbitmq reported the correct bindings in the UI

<@nirik:matrix.scrye.com>

17:43:12

we are.. the queues exist fine and look right, but don't get any messages they should

<@abompard:fedora.im>

17:43:17

it was just not actually bound

<@nirik:matrix.scrye.com>

17:43:38

Aurélien B: oh, do you know what the batcave_test_watcher queue is for?

<@abompard:fedora.im>

17:43:55

hmm, nothing from the top of my mind

<@abompard:fedora.im>

17:44:00

sounds testish

<@nirik:matrix.scrye.com>

17:44:40

yeah, it's got no consumers and... 79,000 messages and growing

<@Zlopez:matrix.org>

17:45:11

A big one

<@nirik:matrix.scrye.com>

17:45:28

the biggest one.

<@nirik:matrix.scrye.com>

17:45:31

in prod anyhow

<@Zlopez:matrix.org>

17:45:40

Let me switch to open floor as we are talking about everything now :-)

<@Zlopez:matrix.org>

17:45:42

!topic Open Floor

<@nirik:matrix.scrye.com>

17:45:59

sure. ;)

<@seddik:fedora.im>

17:47:05

Question hehe

<@seddik:fedora.im>

17:47:05

Who will be at FOSDEM ?

<@Zlopez:matrix.org>

17:47:13

I forgot to mention that the staging instance of Anitya now supports OIDC https://stg.release-monitoring.org/login

<@Zlopez:matrix.org>

17:47:32

I will do a proper release when I'm back from holidays

<@Zlopez:matrix.org>

17:48:08

@seddik:fedora.im I don't plan to

<@nirik:matrix.scrye.com>

17:48:35

I'm not planning to either.

<@nirik:matrix.scrye.com>

17:49:03

However, it's sounding like flock might be around the time of devconf...

<@Zlopez:matrix.org>

17:49:04

But I didn't planned it last year as well and I was there :-D

<@Zlopez:matrix.org>

17:49:20

We will see

<@seddik:fedora.im>

17:49:29

See you there hhh

<@humaton:fedora.im>

17:49:58

I will be there

<@Zlopez:matrix.org>

17:49:58

The Flock is great :-)

<@seddik:fedora.im>

17:50:28

Some times I have trouble finding subjects to work on..

<@seddik:fedora.im>

17:50:28

Another question

<@seddik:fedora.im>

17:51:17

Lot of issues, but those issues need update from people..

<@nirik:matrix.scrye.com>

17:51:47

yeah, there's a lot of old things where people aren't answering us very well. ;(

<@seddik:fedora.im>

17:52:22

<@nirik:matrix.scrye.com>

17:52:41

saibug: I'll try and finish those last 2 apps for memcached and then we can retire memcached01 and close that out

<@nirik:matrix.scrye.com>

17:52:54

(if you want to do a retire/removal pr for 01)

<@seddik:fedora.im>

17:53:21

Yes I'll look at this later..

<@nirik:matrix.scrye.com>

17:53:35

thanks

<@nirik:matrix.scrye.com>

17:54:17

oh, David Kirwan pushed a initial openshift best practices guide:

<@nirik:matrix.scrye.com>

17:54:19

https://docs.fedoraproject.org/en-US/infra/developer_guide/openshift/

<@humaton:fedora.im>

17:54:34

saibug: if you have a list of tickets of interest I dont mind going over and pinging people for updates

<@Zlopez:matrix.org>

17:54:35

@dkirwan:fedora.im++

<@nirik:matrix.scrye.com>

17:54:35

we can suggest changes in pr's or discussion/list/whatever

<@zodbot:fedora.im>

17:54:37

zlopez gave a cookie to dkirwan. They now have 9 cookies, 1 of which were obtained in the Fedora 41 release cycle

<@nirik:matrix.scrye.com>

17:55:04

https://docs.fedoraproject.org/en-US/infra/developer_guide/openshift_bestpractices/ I guess is the doc

<@nirik:matrix.scrye.com>

17:57:44

I don't think I have much else...

<@nirik:matrix.scrye.com>

17:58:04

I will be on pto after tomorrow, but will still be around somewhat

<@Zlopez:matrix.org>

17:59:05

I will be on PTO from tomorrow, I'm not sure how much I will be reacting here

<@Zlopez:matrix.org>

17:59:56

I see that we are already at the end of our time

<@Zlopez:matrix.org>

18:00:00

Thanks for coming everyone

<@Zlopez:matrix.org>

18:00:08

!endmeeting