2023-11-16 16:00:04
<@Zlopez:matrix.org>
!startmeeting Infrastructure (2023-11-16)
!meetingname infrastructure
2023-11-16 16:00:07
<@meetbot:fedora.im>
Meeting started at 2023-11-16 16:00:04 UTC
2023-11-16 16:00:08
<@meetbot:fedora.im>
The Meeting name is 'Infrastructure (2023-11-16) !meetingname infrastructure'
2023-11-16 16:00:21
<@seddik:fedora.im>
hello 👋
2023-11-16 16:00:25
<@Zlopez:matrix.org>
!meetingname infrastructure
2023-11-16 16:00:29
<@nirik:matrix.scrye.com>
morning. 👋
2023-11-16 16:00:36
<@Zlopez:matrix.org>
!chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp
2023-11-16 16:00:42
<@pcreech:matrix.org>
hello
2023-11-16 16:00:45
<@Zlopez:matrix.org>
!info Agenda is at: https://board.net/p/fedora-infra
About our team: https://docs.fedoraproject.org/en-US/cpe/
Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
2023-11-16 16:00:51
<@Zlopez:matrix.org>
!topic namaste
2023-11-16 16:01:02
<@Zlopez:matrix.org>
!hi
2023-11-16 16:01:04
<@zodbot:fedora.im>
Michal Konecny (zlopez)
2023-11-16 16:01:34
<@Zlopez:matrix.org>
This is the first time we are trying to run the meeting on matrix
2023-11-16 16:01:49
<@aheath1992:matrix.org>
hi
2023-11-16 16:02:06
<@Zlopez:matrix.org>
Let's see if it goes well
2023-11-16 16:02:16
<@seddik:fedora.im>
!hi
2023-11-16 16:02:17
<@zodbot:fedora.im>
seddik alaouiismaili (seddik)
2023-11-16 16:03:22
<@Zlopez:matrix.org>
Let's wait a minute for everyone to join first :-)
2023-11-16 16:04:00
<@nirik:matrix.scrye.com>
🎶 musical interlude 🎵
2023-11-16 16:04:29
<@Zlopez:matrix.org>
How is everyone doing?
2023-11-16 16:04:58
<@Zlopez:matrix.org>
Here is already dark outside
2023-11-16 16:05:18
<@aheath1992:matrix.org>
Zlopez: not to bad, lot of meeting this morning
2023-11-16 16:05:36
<@nirik:matrix.scrye.com>
Just getting light here. ;)
2023-11-16 16:06:05
<@jrichardson:matrix.org>
just getting dark here :)
2023-11-16 16:06:08
<@leo:fedora.im>
hey all
2023-11-16 16:06:17
<@leo:fedora.im>
!hello
2023-11-16 16:06:18
<@zodbot:fedora.im>
Leo Puvilland (leo) - he / him / his
2023-11-16 16:06:27
<@Zlopez:matrix.org>
Let's see if the matrix zodbot handles multiple lines at once
2023-11-16 16:06:33
<@Zlopez:matrix.org>
!topic New folks introductions
!info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
2023-11-16 16:07:28
<@Zlopez:matrix.org>
I'm not sure, there is not any feedback for changing the topic :-D
2023-11-16 16:07:37
<@Zlopez:matrix.org>
Do we have anybody new here?
2023-11-16 16:07:40
<@leo:fedora.im>
i mean there’s a check mark
2023-11-16 16:07:50
<@leo:fedora.im>
not sure what that means though :)
2023-11-16 16:07:54
<@nirik:matrix.scrye.com>
I think that is just for in the minutes... it doesn't change the channel topic any
2023-11-16 16:08:22
<@Zlopez:matrix.org>
Yeah, but I don't know if the topic doesn't consume everything, but we will see
2023-11-16 16:08:42
<@nirik:matrix.scrye.com>
we shall do... science!
2023-11-16 16:09:45
<@Zlopez:matrix.org>
Ok, it seems that nobody new is here today
2023-11-16 16:09:52
<@Zlopez:matrix.org>
Let's go to next one
2023-11-16 16:09:56
<@Zlopez:matrix.org>
!topic Next chair
2023-11-16 16:10:06
<@Zlopez:matrix.org>
!info magic eight ball says:
!info chair 2023-11-16 zlopez
!info chair 2023-11-23 zlopez
2023-11-16 16:10:28
<@Zlopez:matrix.org>
!info chair 2023-11-30 ???
2023-11-16 16:10:42
<@Zlopez:matrix.org>
Hm, I'm there twice :-D
2023-11-16 16:10:59
<@leo:fedora.im>
huh
2023-11-16 16:11:01
<@nirik:matrix.scrye.com>
I think many people won't be around next week (it's thanksgiving week in the US)
2023-11-16 16:11:10
<@Zlopez:matrix.org>
Do we want to cancel the thanksgiving one?
2023-11-16 16:11:37
<@nirik:matrix.scrye.com>
I'd say yes, but if people want to meet, go for it... but I won't be here. :)
2023-11-16 16:12:16
<@Zlopez:matrix.org>
Let's cancel it then, I will add this to announcements
2023-11-16 16:13:33
<@Zlopez:matrix.org>
!info chair 2023-11-23 Canceled
2023-11-16 16:13:52
<@Zlopez:matrix.org>
Who wants to chair on 2023-11-30?
2023-11-16 16:14:30
<@nirik:matrix.scrye.com>
I can if no one else...
2023-11-16 16:14:37
<@lenkaseg:fedora.im>
I can too
2023-11-16 16:14:52
<@leo:fedora.im>
i would but it’s really early for me
2023-11-16 16:15:25
<@Zlopez:matrix.org>
lenkaseg: sold!
2023-11-16 16:15:50
<@Zlopez:matrix.org>
!info chair 2023-11-30 lenkaseg
2023-11-16 16:16:52
<@Zlopez:matrix.org>
Do we also have volunteer for 2023-12-07?
2023-11-16 16:17:38
<@nirik:matrix.scrye.com>
I could take that one? I'm likely to be out later in december... have a bunch of PTO to use. ;)
2023-11-16 16:18:02
<@Zlopez:matrix.org>
It's yours nirik
2023-11-16 16:18:08
<@pcreech:matrix.org>
I'll .............
2023-11-16 16:18:12
<@Zlopez:matrix.org>
!info chair 2023-12-07 nirik
2023-11-16 16:18:44
<@Zlopez:matrix.org>
I think we are OK for chairs
2023-11-16 16:18:58
<@Zlopez:matrix.org>
!topic announcements and information
!info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0830 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
!info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
!info Meeting is cancelled on 2023-11-23 because of Thanksgiving
2023-11-16 16:19:14
<@nirik:matrix.scrye.com>
!info nirik out next week for thanksgiving
2023-11-16 16:19:27
<@Zlopez:matrix.org>
Do we have anything else to announce?
2023-11-16 16:20:56
<@nirik:matrix.scrye.com>
oh...
2023-11-16 16:21:11
<@nirik:matrix.scrye.com>
!info nuancier, fedmsg-irc and old fmn are all retired now.
2023-11-16 16:21:38
<@nirik:matrix.scrye.com>
osbs will need to wait more for the last of the f38 flatpaks to move
2023-11-16 16:21:50
<@Zlopez:matrix.org>
Don't we have some outage planned?
2023-11-16 16:22:00
<@nirik:matrix.scrye.com>
it was yesterday. ;)
2023-11-16 16:22:15
<@leo:fedora.im>
how’d it go)
2023-11-16 16:22:20
<@leo:fedora.im>
how’d it go?
2023-11-16 16:22:46
<@nirik:matrix.scrye.com>
long, but went pretty well.... I didn't get all the reinstalls done I was hoping for, but thats ok.
2023-11-16 16:22:56
<@Zlopez:matrix.org>
So the ticket 11626 could be closed?
2023-11-16 16:23:01
<@Zlopez:matrix.org>
!ticket 11626
2023-11-16 16:23:03
<@zodbot:fedora.im>
fedora-infrastructure #11626 (https://pagure.io/fedora-infrastructure/issue/11626): Planned Outage - pagure.io network switch updates - 2023-11-17 13:00 UTC
2023-11-16 16:23:40
<@Zlopez:matrix.org>
It seems to be planned for tomorrow
2023-11-16 16:24:01
<@nirik:matrix.scrye.com>
oh no thats a different one.
2023-11-16 16:24:04
<@nirik:matrix.scrye.com>
thats tomorrow.
2023-11-16 16:24:09
<@leo:fedora.im>
outage party
2023-11-16 16:24:19
<@nirik:matrix.scrye.com>
but we don't have to do anything, just notify people.
2023-11-16 16:24:25
<@nirik:matrix.scrye.com>
I will mail it out in a bit.
2023-11-16 16:24:29
<@Zlopez:matrix.org>
OK
2023-11-16 16:24:54
<@Zlopez:matrix.org>
In this case let's go to our favorite topic oncall :-)
2023-11-16 16:25:02
<@Zlopez:matrix.org>
!topic Oncall
!info https://fedoraproject.org/wiki/Infrastructure/Oncall
https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
2023-11-16 16:25:14
<@Zlopez:matrix.org>
!info pcreech is on call from 2023-11-10 to 2023-11-16
!info zlopez is on call from 2023-11-17 to 2023-11-23
!info zlopez is on call from 2023-11-24 to 2023-11-30
!info ??? is on call from 2023-12-01 to 2023-12-07
2023-11-16 16:25:35
<@Zlopez:matrix.org>
As the meeting is cancelled next week I will stay on oncall till the next one
2023-11-16 16:25:49
<@Zlopez:matrix.org>
Who wants the first week in December
2023-11-16 16:25:51
<@leo:fedora.im>
i’ll take next on call
2023-11-16 16:25:52
<@Zlopez:matrix.org>
?
2023-11-16 16:25:58
<@Zlopez:matrix.org>
Sold!
2023-11-16 16:26:11
<@Zlopez:matrix.org>
!info Leo is on call from 2023-12-01 to 2023-12-07
2023-11-16 16:27:02
<@Zlopez:matrix.org>
Let's see if this is working
2023-11-16 16:27:06
<@Zlopez:matrix.org>
!oncalltakeeu
2023-11-16 16:27:18
<@nirik:matrix.scrye.com>
nope. ;) I think the oncall stuff still needs work
2023-11-16 16:27:24
<@leo:fedora.im>
i think the commands are different
2023-11-16 16:27:42
<@pcreech:matrix.org>
And there's an out of sync issue between irc and matrix as well as to who is oncall... :D (foudn that one out)
2023-11-16 16:27:43
<@leo:fedora.im>
though it mentions a control room
2023-11-16 16:28:11
<@Zlopez:matrix.org>
pcreech: The bot is different, so the information depends on where you ask
2023-11-16 16:28:15
<@nirik:matrix.scrye.com>
!oncall add zlopez
2023-11-16 16:28:16
<@zodbot:fedora.im>
Sorry, adding to the oncall list can only be done from the controlroom
2023-11-16 16:28:34
<@Zlopez:matrix.org>
Thanks nirik
2023-11-16 16:28:38
<@leo:fedora.im>
can you not self assign anymore
2023-11-16 16:28:55
<@nirik:matrix.scrye.com>
not at the moment... the matrix bots are under heavy development. ;)
2023-11-16 16:29:02
<@nirik:matrix.scrye.com>
!oncall
2023-11-16 16:29:03
<@zodbot:fedora.im>
The following people are oncall:
● @darknao:fedora.im (darknao) Current Time for them: 17:29 (Europe/Paris)
● @Zlopez:matrix.org (zlopez) Current Time for them: 17:29 (Europe/Prague)
If they do not respond, please file a ticket (https://pagure.io/fedora-infrastructure/issues)
2023-11-16 16:29:03
<@leo:fedora.im>
right ok
2023-11-16 16:29:12
<@nirik:matrix.scrye.com>
ha. ok.
2023-11-16 16:29:18
<@pcreech:matrix.org>
And since i'm getting pinged a lot at the moment for dayjob, quick summary: I was pinged once, overnight, couple days ago. https://pagure.io/releng/issue/11780 was created and was handled
2023-11-16 16:29:19
<@leo:fedora.im>
wow two oncalls
2023-11-16 16:29:32
<@nirik:matrix.scrye.com>
!oncall
2023-11-16 16:29:32
<@zodbot:fedora.im>
The following people are oncall:
● @Zlopez:matrix.org (zlopez) Current Time for them: 17:29 (Europe/Prague)
If they do not respond, please file a ticket (https://pagure.io/fedora-infrastructure/issues)
2023-11-16 16:29:38
<@Zlopez:matrix.org>
!info Summary of last week: (from current oncall )
2023-11-16 16:29:50
<@nirik:matrix.scrye.com>
thanks pcreech
2023-11-16 16:29:50
<@pcreech:matrix.org>
I'll repost, for the log: And since i'm getting pinged a lot at the moment for dayjob, quick summary: I was pinged once, overnight, couple days ago. https://pagure.io/releng/issue/11780 was created and was handled
2023-11-16 16:29:56
<@pcreech:matrix.org>
dangit... stoopid matrix
2023-11-16 16:30:04
<@pcreech:matrix.org>
And since i'm getting pinged a lot at the moment for dayjob, quick summary: I was pinged once, overnight, couple days ago. https://pagure.io/releng/issue/11780 was created and was handled
2023-11-16 16:30:10
<@pcreech:matrix.org>
reposted for the log
2023-11-16 16:30:55
<@Zlopez:matrix.org>
No problem, this is a demo run of meeting on matrix :-D
2023-11-16 16:31:11
<@leo:fedora.im>
going well so far
2023-11-16 16:31:26
<@pcreech:matrix.org>
My biggest gripe is, at least in the nheko client, hitting the up arrow actually edits the entry instead of reposting
2023-11-16 16:31:40
<@nirik:matrix.scrye.com>
yeah, I don't like that either. ;(
2023-11-16 16:31:42
<@pcreech:matrix.org>
of matrix in general that is
2023-11-16 16:32:15
<@Zlopez:matrix.org>
pcreech: The element does the same
2023-11-16 16:33:15
<@Zlopez:matrix.org>
!topic Monitoring discussion [nirik]
!info https://nagios.fedoraproject.org/nagios
Go over existing out items and fix
2023-11-16 16:34:10
<@Zlopez:matrix.org>
Hopefully the cats didn't take him away yet :-)
2023-11-16 16:34:22
<@leo:fedora.im>
stolen away
2023-11-16 16:35:59
<@Zlopez:matrix.org>
Do we have jednorozec here?
2023-11-16 16:37:43
<@Zlopez:matrix.org>
It doesn't seem so, so let us skip the learning topic for today and move it to next meeting
2023-11-16 16:39:26
<@Zlopez:matrix.org>
We could either wait for nirik or continue with backlog refinement
2023-11-16 16:40:00
<@aheath1992:matrix.org>
I say backlog and we can come back to monitoring
2023-11-16 16:40:28
<@leo:fedora.im>
by the time we're through backlog they should be back
2023-11-16 16:41:27
<@Zlopez:matrix.org>
!topic Revisit blocked tickets
!info Check if any blocked ticket is unblocked
!link https://pagure.io/fedora-infrastructure/issues?status=Open&tags=blocked&priority=0&close_status=
2023-11-16 16:42:04
<@Zlopez:matrix.org>
!ticket 11338
2023-11-16 16:42:05
<@zodbot:fedora.im>
fedora-infrastructure #11338 (https://pagure.io/fedora-infrastructure/issue/11338): Please upgrade mailman for lists.fedoraproject.org to fix ARC signing
2023-11-16 16:42:39
<@Zlopez:matrix.org>
This is blocked by update of mailman in fedora infrastructure and it seems we are closed
2023-11-16 16:42:48
<@Zlopez:matrix.org>
s/closed/close
2023-11-16 16:42:52
<@nirik:matrix.scrye.com>
sheesh, tried to hurry and everything went slow. ;)
2023-11-16 16:43:33
<@nirik:matrix.scrye.com>
should have said I was stepping away, sorry about that.
2023-11-16 16:43:56
<@Zlopez:matrix.org>
No issue, we went forward in the meantime
2023-11-16 16:44:09
<@Zlopez:matrix.org>
!ticket 10241
2023-11-16 16:44:11
<@zodbot:fedora.im>
fedora-infrastructure #10241 (https://pagure.io/fedora-infrastructure/issue/10241): Port apps to OIDC
2023-11-16 16:44:38
<@Zlopez:matrix.org>
By decommissioning nuancier and removing PDC we are closer to finish this ticket
2023-11-16 16:44:54
<@nirik:matrix.scrye.com>
yep
2023-11-16 16:45:32
<@Zlopez:matrix.org>
After that only Pagure and Mailman are remaining
2023-11-16 16:46:03
<@Zlopez:matrix.org>
And those are being ported to RHEL 9, which should solve it
2023-11-16 16:46:11
<@nirik:matrix.scrye.com>
mailman is very close. I intend to try and stand up a mailman01.stg after thanksgiving if I can get the cycles.
2023-11-16 16:47:57
<@Zlopez:matrix.org>
I think the other ones are still blocked
2023-11-16 16:48:26
<@seddik:fedora.im>
!ticket 11560
2023-11-16 16:48:27
<@zodbot:fedora.im>
fedora-infrastructure #11560 (https://pagure.io/fedora-infrastructure/issue/11560): countme log gathering script improvment/cleanup
2023-11-16 16:48:40
<@seddik:fedora.im>
i will work on it to add allowing rule
2023-11-16 16:48:47
<@seddik:fedora.im>
FYI ;)
2023-11-16 16:48:48
<@Zlopez:matrix.org>
Let's switch to open floor
2023-11-16 16:48:54
<@Zlopez:matrix.org>
!topic Open Floor
2023-11-16 16:49:01
<@nirik:matrix.scrye.com>
thanks saibug!
2023-11-16 16:49:24
<@Zlopez:matrix.org>
I today deployed the ipa02.stg with RHEL9
2023-11-16 16:49:31
<@Zlopez:matrix.org>
!ticket 10358
2023-11-16 16:49:32
<@zodbot:fedora.im>
fedora-infrastructure #10358 (https://pagure.io/fedora-infrastructure/issue/10358): IPA backup fails on ipa03 and ipa02.stg
2023-11-16 16:49:45
<@nirik:matrix.scrye.com>
great news. ;)
2023-11-16 16:50:04
<@nirik:matrix.scrye.com>
question: did the changes you made to the playbook affect prod any? or it's still ok to run playbook against prod?
2023-11-16 16:50:30
<@Zlopez:matrix.org>
It seems that the IPA is working as it should, it took me most of today to get the playbook running
2023-11-16 16:51:34
<@Zlopez:matrix.org>
I only changed how the replication_status is obtained as the /etc/ipa/default is already created in ipa/client role
2023-11-16 16:51:42
<@Zlopez:matrix.org>
It's now trying to run `ipactl status` and if the rc is greater than 0 it means that the IPA server is not running yet
2023-11-16 16:53:02
<@Zlopez:matrix.org>
Otherwise the changes have conditions for RHEL9
2023-11-16 16:53:08
<@nirik:matrix.scrye.com>
great.
2023-11-16 16:53:20
<@nirik:matrix.scrye.com>
so we need to do ipa01.stg and all stg will be completed?
2023-11-16 16:53:25
<@nirik:matrix.scrye.com>
and then move on to prod?
2023-11-16 16:53:40
<@Zlopez:matrix.org>
If the ipa02.stg will have some issues I made backups on vmhost-x86-02.stg
2023-11-16 16:53:59
<@Zlopez:matrix.org>
Yes, the ipa01.stg is the last one
2023-11-16 16:54:38
<@Zlopez:matrix.org>
We will probably need to first set the ipa02.stg as primary, replicate it and then switch it back to ipa01.stg
2023-11-16 16:55:04
<@Zlopez:matrix.org>
I need to check how this is done
2023-11-16 16:55:06
<@nirik:matrix.scrye.com>
I didn't think there was any primary, all of them should be primary no?
2023-11-16 16:55:38
<@Zlopez:matrix.org>
There is some server set that contains info about the replicas
2023-11-16 16:55:57
<@Zlopez:matrix.org>
Let me check how it is called in IPA
2023-11-16 16:56:10
<@nirik:matrix.scrye.com>
I guess we point to ipa01.stg in dns.
2023-11-16 16:56:27
<@nirik:matrix.scrye.com>
but in prod we point to all 3
2023-11-16 16:56:38
<@Zlopez:matrix.org>
Yes, it is mention in ipa-replica-install as well
2023-11-16 16:57:33
<@nirik:matrix.scrye.com>
I guess in stg replication topology is ipa03.stg < - > ipa01.stg < - > ipa02.stg
2023-11-16 16:57:35
<@Zlopez:matrix.org>
Maybe it will be enough to just change the server for replication in ipa-replica-install command
2023-11-16 16:57:49
<@nirik:matrix.scrye.com>
so if we take 01 out it would make 02/03 not able to replicate
2023-11-16 16:58:22
<@Zlopez:matrix.org>
They should be already replicated from 01, but let me check the RHEL9 upgrade guide
2023-11-16 16:58:43
<@nirik:matrix.scrye.com>
in prod we have them in a ring...
2023-11-16 16:59:04
<@nirik:matrix.scrye.com>
ie, 01 to 02/03, 02 to 01/03, 03 to 02/01
2023-11-16 16:59:27
<@nirik:matrix.scrye.com>
you can see in https://id.fedoraproject.org/ipa/ui/#/p/topology-graph (not sure if that requires perms or if anyone can see it. ;)
2023-11-16 16:59:57
<@nirik:matrix.scrye.com>
and stg: https://id.stg.fedoraproject.org/ipa/ui/#/p/topology-graph
2023-11-16 17:00:15
<@nirik:matrix.scrye.com>
so I think we just need to adjust stg to replicate around and then we could reinstall 01 anytime
2023-11-16 17:00:39
<@Zlopez:matrix.org>
I'm not sure how this is done in prod
2023-11-16 17:00:58
<@nirik:matrix.scrye.com>
I think there's commands to adjust it.
2023-11-16 17:01:07
<@Zlopez:matrix.org>
And we are at the end of our time
2023-11-16 17:01:12
<@nirik:matrix.scrye.com>
yeah.
2023-11-16 17:01:21
<@Zlopez:matrix.org>
!endmeeting