#meeting-3:fedoraproject.org log

<@Zlopez:matrix.org>

17:00:19

!startmeeting Infrastructure (2025-05-15)

<@meetbot:fedora.im>

17:00:21

Meeting started at 2025-05-15 17:00:19 UTC

<@meetbot:fedora.im>

17:00:21

The Meeting name is 'Infrastructure (2025-05-15)'

<@Zlopez:matrix.org>

17:00:33

!topic aloha

<@Zlopez:matrix.org>

17:00:33

!chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp

<@Zlopez:matrix.org>

17:00:33

!info About our team: https://docs.fedoraproject.org/en-US/cle/

<@Zlopez:matrix.org>

17:00:33

!info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra

<@Zlopez:matrix.org>

17:00:33

!meetingname infrastructure

<@Zlopez:matrix.org>

17:00:33

!info Agenda is at: https://board.net/p/fedora-infra

<@meetbot:fedora.im>

17:00:36

The Meeting Name is now infrastructure

<@Zlopez:matrix.org>

17:00:52

!hi

<@zodbot:fedora.im>

17:00:57

Michal Konecny (zlopez)

<@Zlopez:matrix.org>

17:01:29

Welcome everyone on this week Infrastructure meeting

<@nirik:matrix.scrye.com>

17:01:48

morning

<@nirik:matrix.scrye.com>

17:01:57

ahoy!

<@james:fedora.im>

17:02:31

Hey

<@Zlopez:matrix.org>

17:04:12

We have a pirate here today :-D

<@Zlopez:matrix.org>

17:04:15

Ahoy!

<@nirik:matrix.scrye.com>

17:04:38

🏴‍☠️

<@Zlopez:matrix.org>

17:05:39

It seems that there wouldn't be much folks today

<@Zlopez:matrix.org>

17:06:21

For those who are here, welcome aboard this pirate ship!

<@Zlopez:matrix.org>

17:06:54

Let's see if we have some landlovers that want to get wet a little

<@Zlopez:matrix.org>

17:06:55

!topic New folks introductions

<@Zlopez:matrix.org>

17:06:55

!info Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/

<@Zlopez:matrix.org>

17:06:55

!info This is a place where people who are interested in Fedora Infrastructure can introduce themselves

<@Zlopez:matrix.org>

17:09:28

I don't see any around, maybe they are too afraid

<@markrosenbaum:fedora.im>

17:09:36

!hi

<@zodbot:fedora.im>

17:09:37

Mark Rosenbaum (markrosenbaum)

<@markrosenbaum:fedora.im>

17:09:44

Sorry I’m a bit late

<@nirik:matrix.scrye.com>

17:10:18

never too late to come aboard.

<@markrosenbaum:fedora.im>

17:10:32

Pirates today I see

<@Zlopez:matrix.org>

17:10:52

Let's look at who wants to take the helm of this ship in upcoming weeks

<@Zlopez:matrix.org>

17:10:53

!topic Next chair

<@Zlopez:matrix.org>

17:10:53

!info chair 2025-05-15 - zlopez

<@Zlopez:matrix.org>

17:10:53

!info chair 2025-05-22 - James

<@Zlopez:matrix.org>

17:10:53

!info chair 2025-05-29 - ???

<@Zlopez:matrix.org>

17:10:53

!info magic eight ball says:

<@Zlopez:matrix.org>

17:11:24

James it is for next week

<@Zlopez:matrix.org>

17:11:42

Who would like to take this ship for a sail on 2025-05-29

<@Zlopez:matrix.org>

17:11:47

<@Zlopez:matrix.org>

17:13:58

Anybody else?

<@Zlopez:matrix.org>

17:14:17

We can keep the helm free till next week

<@markrosenbaum:fedora.im>

17:14:40

I could only ever take it if it were easier

<@markrosenbaum:fedora.im>

17:15:02

Like exactly an hour before we regularly do it

<@smilner:fedora.im>

17:15:08

👋 (oops, lost track of time)

<@markrosenbaum:fedora.im>

17:15:12

Also I’ve never tried before

<@markrosenbaum:fedora.im>

17:15:21

So🤷‍♂️

<@Zlopez:matrix.org>

17:15:30

It's not hard, you just follow the agenda

<@Zlopez:matrix.org>

17:15:35

https://board.net/p/fedora-infra

<@Zlopez:matrix.org>

17:15:55

But I understand that it could be too late for some

<@nirik:matrix.scrye.com>

17:16:39

We could move an hour eariler... thats fine with me... but moving things is always hard

<@Zlopez:matrix.org>

17:17:07

It will switch again once the DST kicks in

<@nirik:matrix.scrye.com>

17:18:33

right, DST messes up everything... always

<@Zlopez:matrix.org>

17:19:31

From what I heard it should be the last shift here next time

<@nirik:matrix.scrye.com>

17:20:11

they have been saying that for years. ;)

<@Zlopez:matrix.org>

17:21:10

I know, this is why I'm saying "From what I heard" :-D

<@Zlopez:matrix.org>

17:21:25

OK, let's move to the next deck, sailors!

<@Zlopez:matrix.org>

17:21:37

!info OpenID EOL in Fedora Infra is TBD as we changed the approach. See https://communityblog.fedoraproject.org/end-of-openid-authentication-in-fedora-account-system-may-update/

<@Zlopez:matrix.org>

17:21:37

!topic announcements and information

<@Zlopez:matrix.org>

17:21:37

!info CLE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0815 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org

<@Zlopez:matrix.org>

17:21:37

!info CLE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1900 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org

<@nirik:matrix.scrye.com>

17:22:12

!info 2025-05-23 is a Red Hat recharge day. Many folks will be away

<@nirik:matrix.scrye.com>

17:22:29

!info 2025-05-26 is a holiday in the US

<@nirik:matrix.scrye.com>

17:24:18

thats all I had I think

<@Zlopez:matrix.org>

17:24:55

Anything else to put on the board?

<@Zlopez:matrix.org>

17:27:02

Let's move to next deck then

<@Zlopez:matrix.org>

17:27:11

!info ??? is on call from 2025-05-15 to 2025-05-21

<@Zlopez:matrix.org>

17:27:11

!topic Oncall

<@Zlopez:matrix.org>

17:27:11

!info https://docs.fedoraproject.org/en-US/infra/day_to_day_fedora/#_the_oncall_role_in_our_team

<@Zlopez:matrix.org>

17:27:11

!info Mark Rosenbaum is on call from 2025-05-08 to 2025-05-14

<@Zlopez:matrix.org>

17:27:37

Any volunteer to watch for any danger?

<@markrosenbaum:fedora.im>

17:28:26

If none others I can

<@Zlopez:matrix.org>

17:30:58

It's yours

<@Zlopez:matrix.org>

17:31:15

But you are already on this week

<@Zlopez:matrix.org>

17:31:21

Let me take it instead

<@Zlopez:matrix.org>

17:31:29

!info zlopez is on call from 2025-05-15 to 2025-05-21

<@Zlopez:matrix.org>

17:32:24

!info ??? is on call from 2025-05-22 to 2025-05-29

<@Zlopez:matrix.org>

17:32:35

Do we have a volunteer for week after that?

<@nirik:matrix.scrye.com>

17:33:00

I'm always around anyhow, I guess I could.

<@Zlopez:matrix.org>

17:33:30

Let me add you then

<@Zlopez:matrix.org>

17:33:43

!info nirik is on call from 2025-05-22 to 2025-05-29

<@Zlopez:matrix.org>

17:34:21

!info Summary of last week: (from current oncall)

<@Zlopez:matrix.org>

17:34:32

Mark Rosenbaum: The deck is yours

<@Zlopez:matrix.org>

17:35:13

!oncall

<@zodbot:fedora.im>

17:35:15

The following people are oncall:

<@zodbot:fedora.im>

17:35:15

<@zodbot:fedora.im>

17:35:15

● @Zlopez:matrix.org (zlopez) Current Time for them: 19:35 (Europe/Prague)

<@zodbot:fedora.im>

17:35:15

If they do not respond, please file a ticket (https://pagure.io/fedora-infrastructure/issues)

<@Zlopez:matrix.org>

17:35:21

Oncall set

<@Zlopez:matrix.org>

17:37:26

Mark Rosenbaum: Did you saw any oncall ping when you were on call?

<@markrosenbaum:fedora.im>

17:38:12

Oop, sorry didn’t realize it was my turn

<@markrosenbaum:fedora.im>

17:38:13

Nope

<@markrosenbaum:fedora.im>

17:38:14

No pings

<@Zlopez:matrix.org>

17:39:08

In that case let's see how the system on our ship are faring

<@Zlopez:matrix.org>

17:39:09

!topic Monitoring discussion [nirik]

<@Zlopez:matrix.org>

17:39:09

!info Go over existing items and fix them

<@Zlopez:matrix.org>

17:39:09

!info https://nagios.fedoraproject.org/nagios

<@nirik:matrix.scrye.com>

17:39:45

so, back to the way it's been for a while: logdetective02 showing, db-koji01 low on disk (but this should be fixed by moving it to rdu3) and those datagrepper checks

<@Zlopez:matrix.org>

17:40:42

I need to find some time to finish the package review, I got one request to add something to spec, but didn't get to that yet

<@nirik:matrix.scrye.com>

17:40:50

There were some anoying alerts from proxy35/37, but somehow it turned out that crond was not running there. I am not sure why at all...

<@nirik:matrix.scrye.com>

17:41:08

fair enough

<@nirik:matrix.scrye.com>

17:43:31

thats all I can see on monitoring off hand

<@Zlopez:matrix.org>

17:44:19

OK, let's see what is waiting us on the lower deck

<@Zlopez:matrix.org>

17:44:44

Backlog refinement

<@nirik:matrix.scrye.com>

17:44:59

I can also give a short datacenter move update...

<@Zlopez:matrix.org>

17:45:05

Discussion about learning topic

<@Zlopez:matrix.org>

17:45:12

Or we can skip to open floor

<@Zlopez:matrix.org>

17:45:42

From what I see it would be best to skip to open floor and give the work to nirik

<@Zlopez:matrix.org>

17:45:49

!topic Open Floor

<@nirik:matrix.scrye.com>

17:48:00

ok, so a quick datacenter move update. We got access to the mgmt on the new hardware. Gwmngilfen and I (mostly him) have been configuring them and doing firmware updates on them. Thats mostly done now on all of them. We are still waiting on network for the actual machines. Thats hopefully very soon. I have setup dns for rdu3 and pushed it out. We are a week or two behind in our schedule, so it may be that we move the switcharoo week out some, but not yet decided. Hopefully we learn more about the network soon to decide that.

<@nirik:matrix.scrye.com>

17:48:56

Once network comes up, I plan to somewhat manually install a bastion01.rdu3 vm, then setup ansible to gateway via that to all the machines there, and then setup a noc01.rdu3, ns01.rdu3 and a few more. From that we can then install other things.

<@nirik:matrix.scrye.com>

17:51:06

Should know more about any schedule changes after this week. :)

<@nirik:matrix.scrye.com>

17:51:15

any questions?

<@nirik:matrix.scrye.com>

17:52:02

Zlopez: I might see if you have cycles to install a ipa01.rdu3 and replicate it to our existing cluster once enough things are up to do that.

<@Zlopez:matrix.org>

17:52:44

I should find some

<@nirik:matrix.scrye.com>

17:53:37

Thats all I think

<@Zlopez:matrix.org>

17:53:40

Let me know when that's actual

<@Zlopez:matrix.org>

17:55:09

I would like some help with the dist-git pushing over https, I think I setup everything correctly, but I'm still getting authentication failure when trying to push the changes

<@Zlopez:matrix.org>

17:55:25

After that I will continue on datagrepper checks

<@nirik:matrix.scrye.com>

17:57:14

so, I wonder on that...

<@Zlopez:matrix.org>

17:57:17

Could there be something in dist-git configuration preventing that?

<@Zlopez:matrix.org>

17:57:50

I should have the correct ACL on token I'm using for authentication and the user is admin of the repository

<@nirik:matrix.scrye.com>

17:57:52

well, regular pagure https auth does not work on src... we have it setup to use oauth tokens (see fedpkg code to get those tokens)

<@nirik:matrix.scrye.com>

17:58:06

so I wonder if thats interfering?

<@Zlopez:matrix.org>

17:58:42

I will look at the fedpkg code and see how does it working

<@Zlopez:matrix.org>

17:59:02

As it's hard to troubleshoot why you are rejected

<@nirik:matrix.scrye.com>

17:59:06

yeah, thats all I can think of that might be affecting it.

<@nirik:matrix.scrye.com>

17:59:42

but admin tokens work for other things... so... not sure.

<@Zlopez:matrix.org>

18:00:03

At least something I can try as I'm running out of ideas

<@Zlopez:matrix.org>

18:00:17

But only over API, not direct git operations

<@nirik:matrix.scrye.com>

18:00:48

ah yeah...

<@Zlopez:matrix.org>

18:01:16

We are at the end of our sail today. So please everybody leave the boat and see you on next sail :-)

<@Zlopez:matrix.org>

18:01:23

!endmeeting