#meeting-3:fedoraproject.org log

<@seddik:fedora.im>

17:00:17

!startmeeting Infrastructure (2025-02-27)

<@meetbot:fedora.im>

17:00:18

Meeting started at 2025-02-27 17:00:17 UTC

<@meetbot:fedora.im>

17:00:18

The Meeting name is 'Infrastructure (2025-02-27)'

<@seddik:fedora.im>

17:00:27

!meetingname infrastructure

<@meetbot:fedora.im>

17:00:28

The Meeting Name is now infrastructure

<@seddik:fedora.im>

17:00:36

!info About our team: https://docs.fedoraproject.org/en-US/cle/

<@seddik:fedora.im>

17:00:42

!topic ahoy

<@Zlopez:matrix.org>

17:00:55

!hi

<@seddik:fedora.im>

17:00:55

!info Agenda is at: https://board.net/p/fedora-infra

<@smilner:fedora.im>

17:00:57

!hi

<@zodbot:fedora.im>

17:00:57

Michal Konecny (zlopez)

<@zodbot:fedora.im>

17:00:58

None (smilner)

<@ralph:fedora.im>

17:01:01

!hi

<@seddik:fedora.im>

17:01:01

!info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra

<@zodbot:fedora.im>

17:01:02

Ralph Bean (ralph)

<@seddik:fedora.im>

17:01:20

!hi

<@seddik:fedora.im>

17:01:20

<@nirik:matrix.scrye.com>

17:01:44

morning

<@seddik:fedora.im>

17:01:44

morning everyone

<@seddik:fedora.im>

17:02:21

let's wait a bit

<@seddik:fedora.im>

17:02:35

ppl can join :)

<@seddik:fedora.im>

17:05:13

ok ..

<@seddik:fedora.im>

17:05:20

!info This is a place where people who are interested in Fedora Infrastructure can introduce themselves

<@seddik:fedora.im>

17:05:26

!info Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted

<@seddik:fedora.im>

17:05:32

!topic New folks introductions

<@nphilipp:fedora.im>

17:05:36

!hi

<@zodbot:fedora.im>

17:05:37

Nils Philippsen (nphilipp) - he / him / his

<@seddik:fedora.im>

17:05:51

Anybody new here today ??

<@ralph:fedora.im>

17:06:44

I used to attend a long time ago, but I'm back today! I'm Ralph and I'm working on konflux.

<@nirik:matrix.scrye.com>

17:07:13

Welcome (back) Ralph Bean. :)

<@Zlopez:matrix.org>

17:07:20

Welcome back!

<@seddik:fedora.im>

17:07:33

welcome backkk!

<@seddik:fedora.im>

17:07:48

let's move to chair topic

<@seddik:fedora.im>

17:07:59

!topic Next chair

<@seddik:fedora.im>

17:08:16

!info chair 2025-03-20 - ??

<@seddik:fedora.im>

17:08:16

!info chair 2025-02-27 - @saibug

<@seddik:fedora.im>

17:08:16

!info chair 2025-03-06 - @zlopez

<@seddik:fedora.im>

17:08:16

!info chair 2025-03-13 - @saibug

<@seddik:fedora.im>

17:08:38

hum any volunteer for 20 ??

<@seddik:fedora.im>

17:09:51

hum any volunteer for 20th??

<@phsmoura:fedora.im>

17:11:39

I can take it

<@seddik:fedora.im>

17:12:20

ok sold :)

<@seddik:fedora.im>

17:12:37

!info chair 2025-03-20 - @phsmoura

<@seddik:fedora.im>

17:13:01

nice we are covered for next weeks ;)

<@seddik:fedora.im>

17:13:16

!topic announcements and information

<@seddik:fedora.im>

17:13:23

!info CLE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org

<@seddik:fedora.im>

17:13:23

!info CLE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org

<@nirik:matrix.scrye.com>

17:13:40

!info tomorrow is a Red Hat day of learning

<@seddik:fedora.im>

17:14:01

anything that should be announced ???

<@smooge:fedora.im>

17:14:10

I didn't know tomorrow was a day of Learning

<@smooge:fedora.im>

17:14:17

thanks

<@Zlopez:matrix.org>

17:14:25

!info Fedora 42 Beta Go/No Go meeting will happen on 6th March

<@Zlopez:matrix.org>

17:14:57

!info We are in F42 Beta freeze

<@ralph:fedora.im>

17:15:18

!info I created a konflux channel at https://matrix.to/#/#konflux:fedora.im open to anyone.

<@seddik:fedora.im>

17:17:39

ok let's move to the next topic

<@seddik:fedora.im>

17:17:42

!topic Oncall

<@seddik:fedora.im>

17:18:11

!info ??? is on call from 2025-02-27 to 2025-03-06

<@Zlopez:matrix.org>

17:18:38

Let me take that

<@seddik:fedora.im>

17:18:38

do we have any volunteers for this week ??

<@seddik:fedora.im>

17:19:39

ok nice

<@seddik:fedora.im>

17:19:39

!info Zlopez is on call from 2025-02-27 to 2025-03-06

<@seddik:fedora.im>

17:20:22

so let's look at the summary of last week

<@Zlopez:matrix.org>

17:20:44

!oncall

<@zodbot:fedora.im>

17:20:46

<@zodbot:fedora.im>

17:20:46

If they do not respond, please file a ticket (https://pagure.io/fedora-infrastructure/issues)

<@zodbot:fedora.im>

17:20:46

● @Zlopez:matrix.org (zlopez) Current Time for them: 18:20 (Europe/Prague)

<@zodbot:fedora.im>

17:20:46

The following people are oncall:

<@Zlopez:matrix.org>

17:20:52

Everything set

<@seddik:fedora.im>

17:21:06

https://pagure.io/releng/issue/12612

<@seddik:fedora.im>

17:21:06

I only got this ping for this

<@seddik:fedora.im>

17:21:06

<@Zlopez:matrix.org>

17:21:54

It is partially resolved

<@Zlopez:matrix.org>

17:22:13

The quick fix is in place, but the PR needs to be open for more permanent solution

<@nirik:matrix.scrye.com>

17:22:27

well, for an extra layer, but yeah

<@seddik:fedora.im>

17:22:38

yes I saw the message on channel releng

<@seddik:fedora.im>

17:23:25

nirik thanks for resolving bot issue (for oncall ping)

<@nirik:matrix.scrye.com>

17:23:50

yeah, sorry that was missed in the last meeting.

<@seddik:fedora.im>

17:24:26

and now ... moving to monitoring discussion :)

<@seddik:fedora.im>

17:24:30

!topic Monitoring discussion [nirik]

<@nirik:matrix.scrye.com>

17:24:50

so, lets see...

<@nirik:matrix.scrye.com>

17:25:03

2 things pending:

<@nirik:matrix.scrye.com>

17:25:52

* There's a down power9 machine. I have a ticket filed to get them to power cycle it completely. Hopefully that brings it back, if not more digging required.

<@nirik:matrix.scrye.com>

17:26:24

* There's some checks we used to run on busgateway01, and with it being retired we need to move them to another machine. I think Zlopez was working on that

<@nirik:matrix.scrye.com>

17:26:41

* Oh, and logdetective02 is still there. ;)

<@Zlopez:matrix.org>

17:27:23

Waiting currently for review of the packages needed for that

<@nirik:matrix.scrye.com>

17:27:42

cool

<@nirik:matrix.scrye.com>

17:27:48

thats all I think, we can move on...

<@seddik:fedora.im>

17:28:08

<@seddik:fedora.im>

17:28:18

!topic check blocked tickets

<@Zlopez:matrix.org>

17:28:26

I added some topics for discussion

<@seddik:fedora.im>

17:28:32

!link https://pagure.io/fedora-infrastructure/issues?status=Open&tags=blocked&priority=0&close_status=

<@seddik:fedora.im>

17:28:58

!info Check if any blocked ticket

<@Zlopez:matrix.org>

17:29:23

Let's wait for them to open floor

<@seddik:fedora.im>

17:31:02

so let's go over few tickets or no ??

<@Zlopez:matrix.org>

17:31:11

We had some progress on OIDC

<@Zlopez:matrix.org>

17:31:21

The COPR is now done

<@nirik:matrix.scrye.com>

17:31:43

but we found another one right? blockerbugs?

<@Zlopez:matrix.org>

17:32:29

Yes, but we are not blocked on this anymore

<@Zlopez:matrix.org>

17:32:36

I will remove the blocked label

<@seddik:fedora.im>

17:32:51

ok thanks

<@seddik:fedora.im>

17:33:18

hum let's move to open floor topic

<@seddik:fedora.im>

17:33:20

!topic Open Floor

<@Zlopez:matrix.org>

17:33:37

Ok, so I have two things I wanted to discuss here

<@Zlopez:matrix.org>

17:33:56

!info Sunset of OpenID

<@Zlopez:matrix.org>

17:34:20

Most of the apps are moved already to OIDC

<@nirik:matrix.scrye.com>

17:34:38

well, would be good to have blockerbugs keep working right?

<@Zlopez:matrix.org>

17:35:10

I propose to announce some sunset date and then disable the openID

<@Zlopez:matrix.org>

17:35:24

We need to get the blockerbugs fixed first that is true

<@nirik:matrix.scrye.com>

17:35:55

I think it might be also nice to identify any users we can easily identify and try and notify them...

<@nirik:matrix.scrye.com>

17:36:01

but sure, we could set a date.

<@Zlopez:matrix.org>

17:36:58

It should be possible to just grep logs and find anybody who is reaching /openid endpoint

<@nirik:matrix.scrye.com>

17:37:08

There seem to be some lists using it.

<@nirik:matrix.scrye.com>

17:37:10

yeah

<@Zlopez:matrix.org>

17:37:30

lists? like mailman?

<@nirik:matrix.scrye.com>

17:38:20

yes, but not ours. ;)

<@nirik:matrix.scrye.com>

17:38:42

https://listat.jyps.fi seems to be using it, for example

<@nirik:matrix.scrye.com>

17:39:22

or lists.openldap.org

<@nirik:matrix.scrye.com>

17:39:23

etc

<@Zlopez:matrix.org>

17:39:46

Ok, I will update the OIDC ticket with the next plan, setting the date is still a good way forward at least it will set some deadline for the projects

<@Zlopez:matrix.org>

17:40:05

But first let's identify them

<@nirik:matrix.scrye.com>

17:40:08

yeah

<@Zlopez:matrix.org>

17:41:19

That was the first topic, the second one is more organizational

<@Zlopez:matrix.org>

17:41:31

!info Infra work handover between timezones

<@Zlopez:matrix.org>

17:42:32

I was thinking about this for some time, writing some summary at the end of the day to share with people in different timezone what happened

<@Zlopez:matrix.org>

17:43:12

Do you think that is a good idea?

<@nirik:matrix.scrye.com>

17:43:29

I'm happy to try things... but isn't there usually that information in our matrix channels?

<@Zlopez:matrix.org>

17:43:58

Sometimes it is getting lost in the chat, but it's there

<@nirik:matrix.scrye.com>

17:44:03

(and/or tickets I guess)

<@Zlopez:matrix.org>

17:44:57

Not everything we fix is in tickets, at least I'm not opening ticket for every alert I'm fixing

<@nirik:matrix.scrye.com>

17:45:24

What sort of things would this be for? Important stuff that needs continung or just day to day things or ?

<@smooge:fedora.im>

17:46:08

Generally if you are doing something in chat.. it might be helpful to use a "Form" to post a regular thing. Element search is not great from what I can tell but a visual "++++++++++ Report for YYYY-MM-DD X shift +++++++++++++" might be easy to find in a scroll up

<@nirik:matrix.scrye.com>

17:46:44

well, we have a flood of information, I am not sure adding more really helps the issue. ;) But if there's specific things we could try and highlight them somehow

<@smooge:fedora.im>

17:47:40

* People reported login problems on ZZZZ at XXYY UTC. Fixed itself before ticket was open but came back later

<@smooge:fedora.im>

17:47:40

* Got reports at XXYY UTC about s390 problems. Ticket XXXX is probably high priority

<@Zlopez:matrix.org>

17:47:47

Mostly the important things like "Lost connection for machine XYZ - fixed" "Investigating authentication failure - ipsilon returns 500 - didn't found the root issue" etc.

<@nirik:matrix.scrye.com>

17:48:54

well, I try and say those things in noc channel... but I am not sure that kind of thing would stand out much...

<@Zlopez:matrix.org>

17:49:05

I don't want to burden folks with more work and adding that to more spaces, but few bullet points would be helpful

<@Zlopez:matrix.org>

17:49:29

I'm trying to read that every day, but sometimes it's flooded with alerts

<@nirik:matrix.scrye.com>

17:49:33

I can try and add/highlight more, but... someone needs to pay attention to it. ;)

<@nirik:matrix.scrye.com>

17:50:00

so, I was talking with Gwmngilfen the other day about that... perhaps we should move those to a bot channel?

<@smooge:fedora.im>

17:50:07

but my suggestion is only useful if it contains a couple of things that people need to get their job done

<@nirik:matrix.scrye.com>

17:50:10

but I think they are useful to have.

<@smooge:fedora.im>

17:50:45

#nocbot ?

<@Zlopez:matrix.org>

17:50:52

I'm not against that

<@gwmngilfen:fedora.im>

17:51:22

NGL, I mute the not when I want to catch up

<@nirik:matrix.scrye.com>

17:51:24

the problem might be there that people ignore them, but then again they ignore them now. ;)

<@gwmngilfen:fedora.im>

17:51:33

NGL, I mute the bot when I want to catch up

<@nirik:matrix.scrye.com>

17:51:40

I'm fine moving it off to see if to helps folks.

<@Zlopez:matrix.org>

17:51:50

We need to find the information that is actually useful to share, but this will maybe take some trials and errors

<@gwmngilfen:fedora.im>

17:52:03

Then I unmute after :)

<@nirik:matrix.scrye.com>

17:52:31

zabbix we already have in it's own channel(s)... but I muted them... too much

<@smooge:fedora.im>

17:52:47

oh i didn't know there was a zabbix bot channel

<@nirik:matrix.scrye.com>

17:53:31

there's a prod and a stg one..

<@nirik:matrix.scrye.com>

17:53:37

#noc-zabbix-testing:fedora.im

<@nirik:matrix.scrye.com>

17:53:51

and... looking for the stg one

<@gwmngilfen:fedora.im>

17:54:01

Ah, I was looking for that earlier, thanks :)

<@smooge:fedora.im>

17:54:11

thanks

<@nirik:matrix.scrye.com>

17:54:22

#fedora-zodbot:fedora.im but it seems busted right now. ;)

<@nirik:matrix.scrye.com>

17:55:11

anyhow... we can try and add some kind of 'handoff' notice if there's ongoing things? or do you think matrix isn't good for this?

<@Zlopez:matrix.org>

17:55:35

I'm for using that channel just for updates, we will see if that will be actually helpful

<@nirik:matrix.scrye.com>

17:55:57

and moving the bot?

<@Zlopez:matrix.org>

17:56:13

Or create a new one

<@nirik:matrix.scrye.com>

17:57:20

a new room just for handoffs? I fear that would either just not be looked at/used much, or people would start discussing things in it...

<@Zlopez:matrix.org>

17:58:26

Or we can just use some keyword and share it in noc or fedora-infra room

<@Zlopez:matrix.org>

17:59:05

Not sure how good are the matrix clients with highlighting keywords

<@nirik:matrix.scrye.com>

17:59:24

almost any of them can let you add words to highlight

<@nirik:matrix.scrye.com>

17:59:33

(at least that I have seen)

<@seddik:fedora.im>

18:00:20

We can continue on another room ??

<@seddik:fedora.im>

18:00:33

Thanks for being here today

<@nirik:matrix.scrye.com>

18:00:35

yeah we are out of time. ;) continue in #noc ? ;)

<@Zlopez:matrix.org>

18:00:47

We can

<@seddik:fedora.im>

18:01:05

!endmeeting