<@lenkaseg:fedora.im>
16:00:06
!startmeeting Infrastructure (2024-04-04)
<@meetbot:fedora.im>
16:00:07
Meeting started at 2024-04-04 16:00:06 UTC
<@meetbot:fedora.im>
16:00:07
The Meeting name is 'Infrastructure (2024-04-04)'
<@lenkaseg:fedora.im>
16:00:14
!meetingname infrastructure
<@lenkaseg:fedora.im>
16:00:20
!chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp
<@lenkaseg:fedora.im>
16:00:32
!info Agenda is at: https://board.net/p/fedora-infra
<@lenkaseg:fedora.im>
16:00:39
About our team: https://docs.fedoraproject.org/en-US/cpe/
<@lenkaseg:fedora.im>
16:00:45
Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
<@lenkaseg:fedora.im>
16:00:50
!topic namaste
<@nirik:matrix.scrye.com>
16:00:54
morning 🌄
<@lenkaseg:fedora.im>
16:01:00
Hello everybody!
<@leo:fedora.im>
16:01:02
hey!
<@pcreech:matrix.org>
16:01:03
!hello pcreech17
<@zodbot:fedora.im>
16:01:05
Patrick Creech (pcreech17)
<@pcreech:matrix.org>
16:01:06
mornin!
<@leo:fedora.im>
16:01:13
hope everyone is doing well
<@Zlopez:matrix.org>
16:01:44
!hi
<@zodbot:fedora.im>
16:01:46
Michal Konecny (zlopez)
<@lenkaseg:fedora.im>
16:02:03
!topic New folks introductions
<@lenkaseg:fedora.im>
16:02:10
Do we have any new folks here?
<@lenkaseg:fedora.im>
16:02:28
!info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/
<@lenkaseg:fedora.im>
16:03:24
Only old folks it seems
<@lenkaseg:fedora.im>
16:03:35
!topic Next chair
<@aheath1992:matrix.org>
16:03:46
!hi
<@zodbot:fedora.im>
16:03:48
Andrew Heath (aheath1992)
<@lenkaseg:fedora.im>
16:03:56
Hi aheath1992 !
<@nirik:matrix.scrye.com>
16:04:02
Hey! who you calling old? Oh wait, I _am_ old. ;)
<@freedisch:matrix.org>
16:04:13
for Next chair✋
<@leo:fedora.im>
16:04:24
well... young at heart. :D
<@lenkaseg:fedora.im>
16:04:44
nirik fedora infra veterans let's say
<@lenkaseg:fedora.im>
16:05:04
Freedisch Thibaut Batale: Thanks for volunteering!
<@lenkaseg:fedora.im>
16:05:15
!info magic eight ball says:
<@lenkaseg:fedora.im>
16:05:19
!info chair 2024-04-11 zlopez
<@lenkaseg:fedora.im>
16:05:31
!info chair 2024-04-18 Freedisch Thibaut Batale
<@lenkaseg:fedora.im>
16:07:07
ok, we have chairfolks for next 14 days, that's good
<@lenkaseg:fedora.im>
16:07:11
!topic announcements and information
<@lenkaseg:fedora.im>
16:07:38
!info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going
<@lenkaseg:fedora.im>
16:07:45
through tickets at 0800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
<@lenkaseg:fedora.im>
16:07:52
!info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
<@pcreech:matrix.org>
16:08:01
Old is in the eye of the beholder....
<@lenkaseg:fedora.im>
16:08:02
!info F40 final freeze started 2024-04-02
<@lenkaseg:fedora.im>
16:08:16
!info we are out of F40 Beta freeze, F40 Beta has been released!
<@leo:fedora.im>
16:08:44
hooray!
<@lenkaseg:fedora.im>
16:09:08
Do we have some more announcements?
<@nirik:matrix.scrye.com>
16:10:09
I'd like to ask folks to read and reply on:
<@nirik:matrix.scrye.com>
16:10:23
!info please provide feedback on https://discussion.fedoraproject.org/t/planning-for-infra-and-releng-hackfest-at-flock-2024/110244
<@nirik:matrix.scrye.com>
16:10:39
to plan a hackfest at flock later this year. ;)
<@lenkaseg:fedora.im>
16:10:51
Yup, I read it
<@lenkaseg:fedora.im>
16:11:05
Good ideas there
<@Zlopez:matrix.org>
16:11:14
I have it open, didn't get to it yet
<@nirik:matrix.scrye.com>
16:11:42
Cool. Just wanted to note it... feedback welcome/desired. ;)
<@lenkaseg:fedora.im>
16:13:13
let's move on to oncall!
<@lenkaseg:fedora.im>
16:13:25
!topic Oncall
<@lenkaseg:fedora.im>
16:13:29
!info https://fedoraproject.org/wiki/Infrastructure/Oncall
<@lenkaseg:fedora.im>
16:13:34
https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
<@lenkaseg:fedora.im>
16:13:40
##!oncall add <fas_name>
<@lenkaseg:fedora.im>
16:13:48
!info nirik is on call from 2024-03-28 to 2024-04-04
<@lenkaseg:fedora.im>
16:14:09
Any volunteer for 2024-04-05 to 2024-04-11?
<@leo:fedora.im>
16:14:14
i'll take it
<@lenkaseg:fedora.im>
16:14:20
sold!
<@lenkaseg:fedora.im>
16:14:30
!info leo is on call from 2024-04-05 to 2024-04-11
<@lenkaseg:fedora.im>
16:14:45
!info Summary of last week: (from current oncall)
<@lenkaseg:fedora.im>
16:14:52
nirik, floor's yours!
<@nirik:matrix.scrye.com>
16:15:33
I had one oncall ping... about fedora review service not working because (it was thought) bugzilla messages were stuck... but they were not...
<@nirik:matrix.scrye.com>
16:15:43
I am not sure if the review side got sorted or not.
<@nirik:matrix.scrye.com>
16:15:48
That was all
<@lenkaseg:fedora.im>
16:16:27
Alright, thanks!
<@lenkaseg:fedora.im>
16:16:32
!topic Monitoring discussion [nirik]
<@lenkaseg:fedora.im>
16:16:45
Over to nirik again :)
<@nirik:matrix.scrye.com>
16:17:27
lets see...
<@nirik:matrix.scrye.com>
16:17:56
we have one machine down (because it was replaced with a new machine and they haven't setup the network ports yet on it)
<@nirik:matrix.scrye.com>
16:18:14
proxy14 has been misbehaving. ;(
<@nirik:matrix.scrye.com>
16:18:32
would be good to figure out whats happening there.
<@nirik:matrix.scrye.com>
16:19:00
Otherwise we have db01 and db-datanommer02 alerting on disk space as they have been for a while.
<@nirik:matrix.scrye.com>
16:19:26
I am going to move them to rhel9 and more space after freeze, because I am waiting on a new virthost in iad2 to put them on being racked.
<@nirik:matrix.scrye.com>
16:19:52
I think thats pretty much it. Things are looking better than they have in a while. ;)
<@lenkaseg:fedora.im>
16:20:17
Thanks for the updates!
<@lenkaseg:fedora.im>
16:20:47
I don't see any learning topic
<@lenkaseg:fedora.im>
16:20:51
Thas means
<@lenkaseg:fedora.im>
16:20:55
!topic Revisit blocked tickets
<@lenkaseg:fedora.im>
16:21:01
!info Check if any blocked ticket is unblocked
<@lenkaseg:fedora.im>
16:21:11
!link https://pagure.io/fedora-infrastructure/issues?status=Open&tags=blocked&priority=0&close_status=
<@Zlopez:matrix.org>
16:21:15
We did that last week, so I think we can end up early
<@Zlopez:matrix.org>
16:21:40
Or we can talk about anything
<@nirik:matrix.scrye.com>
16:21:43
Yeah, or... perhaps we could get a quick status on the rhel7 eol tickets?
<@lenkaseg:fedora.im>
16:21:48
Zlopez: you mean learning topic or blocked?
<@leo:fedora.im>
16:21:54
blocked
<@Zlopez:matrix.org>
16:22:16
nirik: Good idea
<@lenkaseg:fedora.im>
16:22:58
https://pagure.io/fedora-infrastructure/issues?status=Open&search_pattern=rhel7&priority=0&close_status=
<@Zlopez:matrix.org>
16:23:17
So we have the metaticket
<@Zlopez:matrix.org>
16:23:27
!ticket 11815
<@zodbot:fedora.im>
16:23:29
**fedora-infrastructure #11815** (https://pagure.io/fedora-infrastructure/issue/11815):**rhel7 eol**
● **Opened:** 4 weeks ago by zlopez
● **Last Updated:** 4 weeks ago
● **Assignee:** zlopez
<@Zlopez:matrix.org>
16:24:09
Some things are already closed: OSBS, kerneltest, os-control01, cloud-noc01
<@Zlopez:matrix.org>
16:24:47
There is progress being done on Badges, PDC, mirrormanager, Fedimg and mailman
<@Zlopez:matrix.org>
16:25:43
We still have few services using fedmsg instead of fedora-messaging and there should be outreachy project starting for replacement of github2fedmsg
<@nirik:matrix.scrye.com>
16:25:47
on mailman, you said you were doing a mailman3-fedmsg-plugin... is that really fedmsg? or moving to fedora-messaging?
<@Zlopez:matrix.org>
16:26:04
It's fedora-messaging, just the name is the same
<@nirik:matrix.scrye.com>
16:26:13
ah, ok.
<@Zlopez:matrix.org>
16:26:48
I will start looking in authentication tomorrow as it currently fails when trying to login with Fedora
<@Zlopez:matrix.org>
16:27:23
And I forgot to mention that MBS will go away with F38
<@nirik:matrix.scrye.com>
16:27:23
Yeah, I don't recall if it was openid or what... would be good to do oidc.
<@Zlopez:matrix.org>
16:27:36
It is openid
<@Zlopez:matrix.org>
16:27:50
It's using this library https://gitlab.com/mailman/django-mailman3
<@nirik:matrix.scrye.com>
16:27:58
ah... ;(
<@Zlopez:matrix.org>
16:28:26
I will discuss this with Aurélien B tomorrow as he is the one who added fedora provider
<@Zlopez:matrix.org>
16:30:21
So what remains in the list for rhel7 eol is
* db-fas01
* sundries servers
* Fedora people
<@nirik:matrix.scrye.com>
16:30:37
So, the first two I am planning on doing after freeze.
<@nirik:matrix.scrye.com>
16:31:05
the last one we need to start moving people to the new planet... I can look at what we need to communicate there.
<@nirik:matrix.scrye.com>
16:32:07
that will entail: asking people to check for issues with it, then asking contibutors to update their settings in accounts, then setting a 'old planet goes away on X date' date.
<@Zlopez:matrix.org>
16:32:37
Does this mean that we are running pluto now?
<@nirik:matrix.scrye.com>
16:32:52
yes, pluto is running in openshift.
<@nirik:matrix.scrye.com>
16:32:58
https://planet.apps.ocp.fedoraproject.org/
<@nirik:matrix.scrye.com>
16:33:21
Kudos to phsmoura on it. ;)
<@Zlopez:matrix.org>
16:33:36
I see the ticket is still open for the update
<@nirik:matrix.scrye.com>
16:33:43
I'd also like to tell people we will only do https blogs. There's little excuse for http these days now
<@Zlopez:matrix.org>
16:33:43
!ticket 10383
<@zodbot:fedora.im>
16:33:46
**fedora-infrastructure #10383** (https://pagure.io/fedora-infrastructure/issue/10383):**Upgrade Venus to Pluto**
● **Opened:** 2 years ago by nerdsville
● **Last Updated:** a week ago
● **Assignee:** phsmoura
<@nirik:matrix.scrye.com>
16:33:52
yes, it's not done.
<@nirik:matrix.scrye.com>
16:33:59
we need to do the hard social part. ;)
<@nirik:matrix.scrye.com>
16:34:06
"that will entail: asking people to check for issues with it, then asking contibutors to update their settings in accounts, then setting a 'old planet goes away on X date' date."
<@Zlopez:matrix.org>
16:34:24
OK, so let's keep the ticket open
<@nirik:matrix.scrye.com>
16:34:24
if we just kill the old one now I think people will be mad that we didn't notify about it.
<@nirik:matrix.scrye.com>
16:34:48
but perhaps we don't need a ton of time... we can discuss that.
<@nirik:matrix.scrye.com>
16:35:27
after that, I can redo people as rhel9 easily. I already did deploy a rhel9 one before I saw planet needed moving...
<@nirik:matrix.scrye.com>
16:35:48
but venus (the current planet) is python2... and very very dead.
<@Zlopez:matrix.org>
16:36:00
It seems we are on the good track for RHEL7 EOL
<@nirik:matrix.scrye.com>
16:36:25
yeah, looking not too bad...
<@Zlopez:matrix.org>
16:37:08
Anything else we want to discuss?
<@nirik:matrix.scrye.com>
16:37:13
I had one more small item (not rhel7)...
<@lenkaseg:fedora.im>
16:37:49
Go on nirik
<@nirik:matrix.scrye.com>
16:38:23
So, package retirements were broken for a while... from March 4th to April 1st. ;(
<@nirik:matrix.scrye.com>
16:38:41
This was fallout from disabling the git hook on src.fedoraproject.org.
<@nirik:matrix.scrye.com>
16:38:58
I re-enabled it, but if you talk to Aurélien B tomorrow, you might mention it to him...
<@nirik:matrix.scrye.com>
16:39:20
The toddler needs info in that hook currently... I don't know if the pagure hook will have enough info.
<@nirik:matrix.scrye.com>
16:39:31
https://pagure.io/releng/issue/12028 is the releng issue on it.
<@Zlopez:matrix.org>
16:40:11
It should contain the info now
<@nirik:matrix.scrye.com>
16:41:16
the toddler looks for 'files changed' and I don't think it has that.
<@nirik:matrix.scrye.com>
16:41:19
but I could be wrong
<@Zlopez:matrix.org>
16:41:44
I think it was disabled, because the same message was sent twice, but as we found out, that it wasn't quite same
<@nirik:matrix.scrye.com>
16:42:03
right, there's a git hook and a pagure hook...
<@nirik:matrix.scrye.com>
16:42:17
we thought we could disable the git hook, but... not until we fix retirements apparently. ;)
<@Zlopez:matrix.org>
16:42:25
The packit needed the info as well
<@nirik:matrix.scrye.com>
16:42:37
yeah, they were fixed...
<@nirik:matrix.scrye.com>
16:43:02
It might just be some bug in the toddler. It wasn't throwing any error, but it wasn't doing anything either.
<@Zlopez:matrix.org>
16:43:38
Looking at the ticket, I see that I was pinged there, I must have missed that :-/
<@Zlopez:matrix.org>
16:44:12
nils is currently working on toddlers, I will let him know
<@nirik:matrix.scrye.com>
16:44:30
cool. yeah, we just need to sort it out...
<@nphilipp:fedora.im>
16:45:38
There’s a message sent out by fedora-messaging-git-hook and one sent out by Pagure directly
<@nirik:matrix.scrye.com>
16:45:53
yep
<@nphilipp:fedora.im>
16:45:58
Different schemas of course 😂
<@nphilipp:fedora.im>
16:47:21
Right now, not if you want mails being sent to scm-commits (eventually), as the Pagure message doesn’t have the required information
<@Zlopez:matrix.org>
16:47:22
And now nils know :-D
<@nirik:matrix.scrye.com>
16:47:58
perhaps we could make the pagure one better?
<@nirik:matrix.scrye.com>
16:48:06
and yeah, this looks like a schema problem.
<@Zlopez:matrix.org>
16:48:17
I think that is already done, just not deployed yet
<@nphilipp:fedora.im>
16:48:18
Not sure if I know anything about that other toddler, I’m happy that mine seems to do what it should (according to unit tests 😁)
<@nirik:matrix.scrye.com>
16:48:49
I don't think the pagure messages have this: if "dead.package" not in msg["commit"]["stats"]["files"]:
<@nirik:matrix.scrye.com>
16:48:56
anyhow.
<@nphilipp:fedora.im>
16:49:03
I talked with Aurélien B before and there are existing consumers for both message topics (and schemas)
<@nphilipp:fedora.im>
16:49:13
So nixing either one will be trouble
<@nirik:matrix.scrye.com>
16:49:38
we thought we got all the ones for the src.fedoraproject.org git hook, but we were wrong. :)
<@nirik:matrix.scrye.com>
16:49:47
and I know! Lets make a 3rd one!
<@Zlopez:matrix.org>
16:50:03
Now let's make 2 more!
<@nphilipp:fedora.im>
16:50:05
Yeah, the git hook msg has this
<@nirik:matrix.scrye.com>
16:50:36
right, which is why the retirement toddler broke. ;(
<@nphilipp:fedora.im>
16:50:43
We have the option of introducing new versions to message schemas too, just in case THINGS ARE TOO SIMPLE
<@Zlopez:matrix.org>
16:51:43
That is because you don't want to broke the old ones just deprecate them
<@Zlopez:matrix.org>
16:52:11
I already needed that for Anitya messaging schema
<@nirik:matrix.scrye.com>
16:52:21
I'd love for the pagure messages to have more data for gatwaying to matrix too...
<@nphilipp:fedora.im>
16:52:31
The Pagure message has "changed_files" in the body…
<@nphilipp:fedora.im>
16:53:06
If only people would notice deprecation before you eventually remove things 🤷
<@Zlopez:matrix.org>
16:53:34
It's usually just a warning, so most people ignore it
<@Zlopez:matrix.org>
16:54:07
Solving every warning you see is not really productive
<@Zlopez:matrix.org>
16:54:52
And not sure if you even notice that when running consumer
<@nphilipp:fedora.im>
16:55:09
`-Werror` or equivalent should be default for developer envs 😛
<@nphilipp:fedora.im>
16:55:18
Otherwise nothing will happen
<@nphilipp:fedora.im>
16:55:32
(IMO, no need to debate 😂)
<@Zlopez:matrix.org>
16:55:55
I agree with everything :-D
<@Zlopez:matrix.org>
16:57:24
I think we are nearing the end of this meeting
<@lenkaseg:fedora.im>
16:58:18
Do we have something more to discuss in 3 minutes or let's end the meeting?
<@lenkaseg:fedora.im>
16:59:31
I added a topic RHEL7 EOL to the meeting template. No need to run it every week, just as a reminder to stay on track
<@lenkaseg:fedora.im>
16:59:53
Thanks everyone for joining today!
<@lenkaseg:fedora.im>
16:59:58
!endmeeting