<@Zlopez:matrix.org>
17:00:07
!startmeeting Infrastructure (2025-01-30)
<@meetbot:fedora.im>
17:00:09
Meeting started at 2025-02-06 17:00:07 UTC
<@meetbot:fedora.im>
17:00:09
The Meeting name is 'Infrastructure (2025-01-30)'
<@Zlopez:matrix.org>
17:00:15
!info Agenda is at: https://board.net/p/fedora-infra
<@Zlopez:matrix.org>
17:00:15
!topic ahoy
<@Zlopez:matrix.org>
17:00:15
!info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
<@Zlopez:matrix.org>
17:00:15
!info About our team: https://docs.fedoraproject.org/en-US/cle/
<@Zlopez:matrix.org>
17:00:15
!meetingname infrastructure
<@Zlopez:matrix.org>
17:00:15
!chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp
<@meetbot:fedora.im>
17:00:16
The Meeting Name is now infrastructure
<@Zlopez:matrix.org>
17:00:24
!hi
<@zodbot:fedora.im>
17:00:26
Michal Konecny (zlopez)
<@Zlopez:matrix.org>
17:00:36
Welcome everyone to Fedora Infra weekly meeting :-)
<@nirik:matrix.scrye.com>
17:00:40
morning
<@james:fedora.im>
17:01:03
!hi
<@zodbot:fedora.im>
17:01:04
James Antill (james)
<@smilner:fedora.im>
17:01:59
!hi
<@zodbot:fedora.im>
17:02:00
None (smilner)
<@Zlopez:matrix.org>
17:03:37
!info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
<@Zlopez:matrix.org>
17:03:37
!info Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/
<@Zlopez:matrix.org>
17:03:37
!topic New folks introductions
<@Zlopez:matrix.org>
17:03:43
Anybody new around?
<@x3mboy:fedora.im>
17:04:12
Yes and no
<@x3mboy:fedora.im>
17:04:13
Hello
<@x3mboy:fedora.im>
17:04:15
!hello
<@zodbot:fedora.im>
17:04:16
Eduard Lucena (x3mboy) - he / him / his
<@x3mboy:fedora.im>
17:04:20
My name is Eduard Lucena
<@nirik:matrix.scrye.com>
17:04:26
Hey x3mboy! Welcome back
<@Zlopez:matrix.org>
17:04:29
Hi @x3mboy
<@x3mboy:fedora.im>
17:04:33
I've started twice in the team, I'm my third attempt
<@Zlopez:matrix.org>
17:04:34
Welcome back!
<@smilner:fedora.im>
17:04:44
Welcome x3mboy!
<@Zlopez:matrix.org>
17:05:28
Still going back that is good :-)
<@x3mboy:fedora.im>
17:06:20
I built my first RPM, that was something new
<@james:fedora.im>
17:06:53
Uh oh, gateway drug
<@nirik:matrix.scrye.com>
17:07:05
ha. ;) packaging can be fun (and also sometimes frustrating)
<@x3mboy:fedora.im>
17:07:08
To the people who don't know me, I worked mostly in people people stuff in Fedora, but in my job I'm a SysAdmin and SRE now
<@x3mboy:fedora.im>
17:07:43
Bash, Python, Kubernetes, Docker and Jenkins are my daily food
<@nirik:matrix.scrye.com>
17:08:15
we have many of those. ;)
<@Zlopez:matrix.org>
17:08:31
And more :-)
<@seddik:fedora.im>
17:08:42
!hi
<@zodbot:fedora.im>
17:08:43
seddik alaouiismaili (seddik)
<@nirik:matrix.scrye.com>
17:08:45
x3mboy: let me (re) add you to the apprentice group
<@Zlopez:matrix.org>
17:09:08
Hi @seddik:fedora.im
<@nirik:matrix.scrye.com>
17:09:15
Cool. Today it's a š„³
<@seddik:fedora.im>
17:09:26
Hey folks
<@Zlopez:matrix.org>
17:10:03
Plenty of people here :-)
<@nirik:matrix.scrye.com>
17:10:05
hey saibug and pcreech
<@thisisyaash:fedora.im>
17:10:24
!hi
<@zodbot:fedora.im>
17:10:26
Yashwanth Rathakrishnan (thisisyaash)
<@nphilipp:fedora.im>
17:10:36
!hi
<@zodbot:fedora.im>
17:10:37
Nils Philippsen (nphilipp) - he / him / his
<@nphilipp:fedora.im>
17:10:56
(Iām only half here š)
<@nirik:matrix.scrye.com>
17:11:02
The stars have aligned and lots of folks can make it today. ;)
<@Zlopez:matrix.org>
17:11:38
Let's move to next topic
<@Zlopez:matrix.org>
17:11:50
!topic Next chair
<@Zlopez:matrix.org>
17:11:50
!info magic eight ball says:
<@Zlopez:matrix.org>
17:11:50
!info chair 2025-02-06 - Zlopez
<@Zlopez:matrix.org>
17:11:50
!info chair 2025-02-13 - phsmoura
<@Zlopez:matrix.org>
17:11:50
!info chair 2025-02-20 - ???
<@Zlopez:matrix.org>
17:11:50
!info chair 2025-02-27 - ???
<@Zlopez:matrix.org>
17:12:11
Any volunteers for the upcoming weeks?
<@thisisyaash:fedora.im>
17:12:33
I can take it for the 20th
<@Zlopez:matrix.org>
17:12:40
Sold!
<@Zlopez:matrix.org>
17:12:52
!info chair 2025-02-20 - @thisisyaash:fedora.im
<@pcreech:matrix.org>
17:13:03
Let me rebuild some consistency first, then i'll start jumping back on the horse :D
<@seddik:fedora.im>
17:13:24
Next for me
<@seddik:fedora.im>
17:13:26
Yeah
<@Zlopez:matrix.org>
17:13:40
And sold to the highest bidder!
<@Zlopez:matrix.org>
17:13:52
!info chair 2025-02-27 - @seddik:fedora.im
<@seddik:fedora.im>
17:14:19
Nice :)
<@Zlopez:matrix.org>
17:14:22
I think we are pretty covered for next few weeks
<@Zlopez:matrix.org>
17:14:34
!topic announcements and information
<@Zlopez:matrix.org>
17:14:34
!info github2fedmsg sunset date was changed to 13rd February! fedmsg will go away with it!
<@Zlopez:matrix.org>
17:14:34
!info CLE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
<@Zlopez:matrix.org>
17:14:34
!info CLE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
<@Zlopez:matrix.org>
17:14:53
Anything else that should be announced?
<@seddik:fedora.im>
17:15:57
Zlopez congratulations for github2fedmsg decommissioning šš
<@nirik:matrix.scrye.com>
17:16:38
!info f42 has branched off
<@Zlopez:matrix.org>
17:17:06
@seddik:fedora.im We had to postpone the date, but it's done on staging and nothing break
<@nirik:matrix.scrye.com>
17:17:18
(that we know of. ;)
<@Zlopez:matrix.org>
17:17:24
Otherwise it would be already done this week
<@Zlopez:matrix.org>
17:17:44
@nirik:matrix.scrye.com We will see next week :-D
<@nirik:matrix.scrye.com>
17:18:09
yep
<@Zlopez:matrix.org>
17:20:12
Let's move on
<@Zlopez:matrix.org>
17:20:24
!info https://docs.fedoraproject.org/en-US/infra/day_to_day_fedora/#_the_oncall_role_in_our_team
<@Zlopez:matrix.org>
17:20:24
!topic Oncall
<@Zlopez:matrix.org>
17:20:24
!info ??? is on call from 2025-02-20 to 2025-02-27
<@Zlopez:matrix.org>
17:20:24
!info lenkaseg is on call from 2025-01-30 to 2025-02-06
<@Zlopez:matrix.org>
17:20:24
!info ??? is on call from 2025-02-13 to 2025-02-20
<@Zlopez:matrix.org>
17:20:24
!info zlopez is on call from 2025-02-06 to 2025-02-13
<@Zlopez:matrix.org>
17:20:48
I'm oncall now! Fear me! HAHAHAHA
<@nirik:matrix.scrye.com>
17:20:57
ha.
<@Zlopez:matrix.org>
17:21:43
!oncall
<@zodbot:fedora.im>
17:21:44
ā @Zlopez:matrix.org (zlopez) Current Time for them: 18:21 (Europe/Prague)
<@zodbot:fedora.im>
17:21:44
If they do not respond, please file a ticket (https://pagure.io/fedora-infrastructure/issues)
<@zodbot:fedora.im>
17:21:44
<@zodbot:fedora.im>
17:21:44
The following people are oncall:
<@Zlopez:matrix.org>
17:21:53
And here is the proof
<@Zlopez:matrix.org>
17:22:17
Anybody wants to try it for upcoming weeks?
<@seddik:fedora.im>
17:23:01
From 20 to 27
<@seddik:fedora.im>
17:23:07
O can take
<@Zlopez:matrix.org>
17:23:21
It's yours
<@seddik:fedora.im>
17:23:22
I can take it
<@Zlopez:matrix.org>
17:23:32
!info @saibug is on call from 2025-02-20 to 2025-02-27
<@Zlopez:matrix.org>
17:23:53
Anybody wants next week?
<@nirik:matrix.scrye.com>
17:24:07
I can take it if no one else wants it
<@pcreech:matrix.org>
17:24:34
Thow me up there, why not :D
<@Zlopez:matrix.org>
17:24:43
OK :-D
<@nirik:matrix.scrye.com>
17:24:57
thanks saibug and pcreech
<@Zlopez:matrix.org>
17:24:59
!info @pcreech:matrix.org is on call from 2025-02-13 to 2025-02-20
<@Zlopez:matrix.org>
17:25:22
We are pretty covered for next few weeks :-)
<@Zlopez:matrix.org>
17:25:43
!info Summary of last week: (from current oncall)
<@Zlopez:matrix.org>
17:25:52
Is @lenkaseg:fedora.im here?
<@Zlopez:matrix.org>
17:27:55
It doesn't seem so, I remember seeing one or two pings this week
<@smilner:fedora.im>
17:27:57
Letting her know about the ping just in case
<@Zlopez:matrix.org>
17:29:02
Let's skip it for now
<@smilner:fedora.im>
17:29:04
I sent her a ping but it's also pretty late local time
<@Zlopez:matrix.org>
17:29:24
It is
<@Zlopez:matrix.org>
17:29:26
!info https://nagios.fedoraproject.org/nagios
<@Zlopez:matrix.org>
17:29:26
!info Go over existing items and fix them
<@Zlopez:matrix.org>
17:29:26
!topic Monitoring discussion [nirik]
<@nirik:matrix.scrye.com>
17:29:34
So, two items here...
<@Zlopez:matrix.org>
17:29:50
There is only the logdetective in nagios right now
<@Zlopez:matrix.org>
17:29:59
I see only one :-/
<@nirik:matrix.scrye.com>
17:30:03
1. we still have logdetective02 listed, despite trying to get nagios to not monitor it. ;( If someone would like to poke at that again that would be great
<@seddik:fedora.im>
17:30:40
Ohh I think i put a pr for that..
<@seddik:fedora.im>
17:30:44
No?
<@nirik:matrix.scrye.com>
17:30:49
2. There's still a daily alert from mailman around the time the updates pushes go out (so possibly bounces from the package-announce list?) I think phsmoura was gonna look more, but I don't know if he found anything yet or would like more eyes on it?
<@Zlopez:matrix.org>
17:30:53
I don't even understand why it's still being monitored as it's not in our inventory
<@nirik:matrix.scrye.com>
17:30:58
saibug: yep, it didn't work tho. ;(
<@nirik:matrix.scrye.com>
17:31:18
it is in our inventory
<@seddik:fedora.im>
17:31:35
Hum maybe reload nrpe server?
<@Zlopez:matrix.org>
17:31:35
That explains it than
<@nirik:matrix.scrye.com>
17:31:47
it's just often down, so we don't want to monitor it... but nagios doesn't seem to want to let go. ;)
<@Zlopez:matrix.org>
17:32:19
I saw few issues with bvmhost-p09-03, the one with faulty disk
<@nirik:matrix.scrye.com>
17:32:43
hostgroups/all.cfg and hosts/aws-hosts.cfg still lists it on noc01:/etc/nagios/...
<@phsmoura:fedora.im>
17:32:44
yep, couldnt find anything so far, but I intend to go back to it after finishing to update those proxies
<@Zlopez:matrix.org>
17:32:52
The last one was today, I needed to recreate one VM from scratch as reboot didn't work
<@nirik:matrix.scrye.com>
17:33:53
yeah. ;( I need to get that disk replaced, but I keep not getting it. Perhaps I can pass it off to someone else (probibly rh only as it requires paying for a new drive, arranging datacenter replacement, etc)
<@smilner:fedora.im>
17:34:43
Is there anyone willing and able to help with the above?
<@Zlopez:matrix.org>
17:34:58
I thought that @dkirwan:fedora.im is already working on this kind of issues
<@nirik:matrix.scrye.com>
17:35:39
yeah, perhaps he can... ;) We have a ticket, can ping him there?
<@nirik:matrix.scrye.com>
17:37:02
updated the ticket. ;)
<@nirik:matrix.scrye.com>
17:37:19
thats it monitoring wise I think?
<@Zlopez:matrix.org>
17:37:32
It looks like that
<@Zlopez:matrix.org>
17:37:48
We have plenty of folks around
<@nirik:matrix.scrye.com>
17:37:57
I had a couple of general topics if we have some time?
<@Zlopez:matrix.org>
17:37:59
Does anybody want to talk about something here?
<@Zlopez:matrix.org>
17:38:15
Let me skip to open floor and we can do it there
<@Zlopez:matrix.org>
17:38:27
!topic Open Floor
<@nirik:matrix.scrye.com>
17:38:32
sure...
<@nirik:matrix.scrye.com>
17:38:57
so first, there's a ansible handler issue going on. I guess I should file a ticket on it (so many new tickets. ;(
<@nirik:matrix.scrye.com>
17:39:13
ansible-lint for some reason wants you to use a capital letter on names...
<@Zlopez:matrix.org>
17:39:18
We are still under 70 :-)
<@nirik:matrix.scrye.com>
17:39:24
ie, name: foobar is bad, but name: Foobar is good
<@nirik:matrix.scrye.com>
17:39:39
ryanlerch renamed a bunch of handlers to use cap letter.
<@Zlopez:matrix.org>
17:39:42
Do you want to disable this check?
<@Zlopez:matrix.org>
17:39:52
Or rename them
<@nirik:matrix.scrye.com>
17:39:56
but... you can't just change the name in the handler, the way ansible calls them is with the name
<@nirik:matrix.scrye.com>
17:40:06
so every place that has the lower case name is broken
<@nirik:matrix.scrye.com>
17:40:18
I have fixed various ones, but it's getting annoying.
<@nirik:matrix.scrye.com>
17:40:36
So, we should either: tell ansible-lint we don't care, or just fix everything.
<@Zlopez:matrix.org>
17:40:51
Both are doable
<@nirik:matrix.scrye.com>
17:40:55
either way it's anoying to change because it's hard to find all the handler calls with a grep or anything
<@Zlopez:matrix.org>
17:41:44
I would assume that grep will help, I can look into that
<@nirik:matrix.scrye.com>
17:41:48
I don't care too much which way, but I just want it fixed. ;)
<@nirik:matrix.scrye.com>
17:41:58
in many playbooks theres:
<@nirik:matrix.scrye.com>
17:42:13
- restart stunnel
<@nirik:matrix.scrye.com>
17:42:13
notify:
<@nirik:matrix.scrye.com>
17:42:13
- reload systemd
<@nirik:matrix.scrye.com>
17:42:42
which I mean you can get with grep -A2, but what if there's 3 or 4 or more?
<@nirik:matrix.scrye.com>
17:42:59
anyhow, will file a ticket and hope someone will fix it. ;)
<@seddik:fedora.im>
17:43:21
Yes
<@Zlopez:matrix.org>
17:43:25
I will do it in e-macs it should be quick
<@Zlopez:matrix.org>
17:43:54
Not quick, but easy, it will probably take time
<@nirik:matrix.scrye.com>
17:44:17
There may be some way with yq...
<@nirik:matrix.scrye.com>
17:44:35
but yeah... not sure.
<@Zlopez:matrix.org>
17:44:39
Hm, my TODO list is growing today :-D
<@nirik:matrix.scrye.com>
17:44:57
it may be ryan is willing to poke at it more too... or some others. ;)
<@seddik:fedora.im>
17:45:09
Feel a ticket, and can take a look also..
<@Zlopez:matrix.org>
17:45:13
There is a replace feature in emacs you can probably run on whole project, I will look into it
<@nirik:matrix.scrye.com>
17:45:48
will do.
<@seddik:fedora.im>
17:45:53
File a ticket, and can take a look also..
<@nirik:matrix.scrye.com>
17:46:16
ok, second item: I filed https://pagure.io/fedora-infrastructure/issue/12390 a bit ago... a top level tracking ticket for our datacenter move later this year.
<@nirik:matrix.scrye.com>
17:46:52
One thing related to this is that we probibly want to try and finish off things that are 'in flight' right now and then not change stuff to dramatically when we are going to move.
<@nirik:matrix.scrye.com>
17:47:11
Some things would be nice to fix/get done before that too.
<@nirik:matrix.scrye.com>
17:47:46
And there's going to be a lot of work needed by a lot of folks on this. :) I'm working on a more detailed plan, which also will need help/input.
<@Zlopez:matrix.org>
17:48:28
That would be great
<@james:fedora.im>
17:48:46
So I tried with yq, and got pretty close ... but yq likes to reformat everything when it changes anything ... so I think that's a no go.
<@nirik:matrix.scrye.com>
17:48:52
Things in flight I mean: move to quay.io, rabbitmq cluster upgrades, possibly try and accelerate zabbix so we only use it in new dc, etc
<@nirik:matrix.scrye.com>
17:49:22
James Antill :( bummer. yeah, I used yq before and it did indeed reformat everything.
<@nirik:matrix.scrye.com>
17:50:16
also, another thing that occurrs to me is the wiki. Currently it's on f40... there were some issues with mediawiki packaging in f41+... and f40 will EOL right around the datacenter move time, so it woud be good to look into that.
<@nirik:matrix.scrye.com>
17:51:09
Anyhow, just wanted to get everyone thinking about it...
<@nirik:matrix.scrye.com>
17:51:34
and if there's questions I can do my best to answer them, or ask/find out.
<@Zlopez:matrix.org>
17:51:50
Is the wiki still used?
<@nirik:matrix.scrye.com>
17:52:39
yes. QA uses it for test results. It's 100% needed.
<@nirik:matrix.scrye.com>
17:53:02
or we would need an alternative, but... thats a ton of work/workflow disruption.
<@nirik:matrix.scrye.com>
17:53:31
They did look at some TCMS not long ago, but none of them really work for what we are doing. ;(
<@Zlopez:matrix.org>
17:54:40
Understood
<@nirik:matrix.scrye.com>
17:55:41
https://pagure.io/releng/issue/12229 for background on the f41+ problem...
<@nirik:matrix.scrye.com>
17:55:59
I am not sure if the solution I came up with works (I don't know php at all, just wildly guessing)
<@Zlopez:matrix.org>
17:56:49
I don't know much about PHP either :-/
<@nirik:matrix.scrye.com>
17:57:19
related: proxy09 is caching the wiki wrong... I couldn't see anything obvious, more š would be great:
<@nirik:matrix.scrye.com>
17:57:44
https://pagure.io/fedora-infrastructure/issue/12385
<@Zlopez:matrix.org>
17:58:43
I saw another user reporting that today in fedora infra matrix channel
<@Zlopez:matrix.org>
17:59:03
It's strange that is happening only on one proxy
<@nirik:matrix.scrye.com>
17:59:28
yeah. I don't see why.
<@x3mboy:fedora.im>
18:00:00
Yeah, it was me
<@x3mboy:fedora.im>
18:00:01
Sorry
<@x3mboy:fedora.im>
18:00:16
I was looking old stuff and it happened to be broken
<@Zlopez:matrix.org>
18:00:22
No problem
<@Zlopez:matrix.org>
18:00:32
We are at the end of our time
<@Zlopez:matrix.org>
18:00:37
So let's end it here
<@Zlopez:matrix.org>
18:00:42
!endmeeting