2025-02-06 17:00:07 <@Zlopez:matrix.org> !startmeeting Infrastructure (2025-01-30)
2025-02-06 17:00:09 <@meetbot:fedora.im> Meeting started at 2025-02-06 17:00:07 UTC
2025-02-06 17:00:09 <@meetbot:fedora.im> The Meeting name is 'Infrastructure (2025-01-30)'
2025-02-06 17:00:15 <@Zlopez:matrix.org> !info Agenda is at: https://board.net/p/fedora-infra
2025-02-06 17:00:15 <@Zlopez:matrix.org> !topic ahoy
2025-02-06 17:00:15 <@Zlopez:matrix.org> !info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
2025-02-06 17:00:15 <@Zlopez:matrix.org> !info About our team: https://docs.fedoraproject.org/en-US/cle/
2025-02-06 17:00:15 <@Zlopez:matrix.org> !meetingname infrastructure
2025-02-06 17:00:15 <@Zlopez:matrix.org> !chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp
2025-02-06 17:00:16 <@meetbot:fedora.im> The Meeting Name is now infrastructure
2025-02-06 17:00:24 <@Zlopez:matrix.org> !hi
2025-02-06 17:00:26 <@zodbot:fedora.im> Michal Konecny (zlopez)
2025-02-06 17:00:36 <@Zlopez:matrix.org> Welcome everyone to Fedora Infra weekly meeting :-)
2025-02-06 17:00:40 <@nirik:matrix.scrye.com> morning
2025-02-06 17:01:03 <@james:fedora.im> !hi
2025-02-06 17:01:04 <@zodbot:fedora.im> James Antill (james)
2025-02-06 17:01:59 <@smilner:fedora.im> !hi
2025-02-06 17:02:00 <@zodbot:fedora.im> None (smilner)
2025-02-06 17:03:37 <@Zlopez:matrix.org> !info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
2025-02-06 17:03:37 <@Zlopez:matrix.org> !info Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/
2025-02-06 17:03:37 <@Zlopez:matrix.org> !topic New folks introductions
2025-02-06 17:03:43 <@Zlopez:matrix.org> Anybody new around?
2025-02-06 17:04:12 <@x3mboy:fedora.im> Yes and no
2025-02-06 17:04:13 <@x3mboy:fedora.im> Hello
2025-02-06 17:04:15 <@x3mboy:fedora.im> !hello
2025-02-06 17:04:16 <@zodbot:fedora.im> Eduard Lucena (x3mboy) - he / him / his
2025-02-06 17:04:20 <@x3mboy:fedora.im> My name is Eduard Lucena
2025-02-06 17:04:26 <@nirik:matrix.scrye.com> Hey x3mboy! Welcome back
2025-02-06 17:04:29 <@Zlopez:matrix.org> Hi @x3mboy
2025-02-06 17:04:33 <@x3mboy:fedora.im> I've started twice in the team, I'm my third attempt
2025-02-06 17:04:34 <@Zlopez:matrix.org> Welcome back!
2025-02-06 17:04:44 <@smilner:fedora.im> Welcome x3mboy!
2025-02-06 17:05:28 <@Zlopez:matrix.org> Still going back that is good :-)
2025-02-06 17:06:20 <@x3mboy:fedora.im> I built my first RPM, that was something new
2025-02-06 17:06:53 <@james:fedora.im> Uh oh, gateway drug
2025-02-06 17:07:05 <@nirik:matrix.scrye.com> ha. ;) packaging can be fun (and also sometimes frustrating)
2025-02-06 17:07:08 <@x3mboy:fedora.im> To the people who don't know me, I worked mostly in people people stuff in Fedora, but in my job I'm a SysAdmin and SRE now
2025-02-06 17:07:43 <@x3mboy:fedora.im> Bash, Python, Kubernetes, Docker and Jenkins are my daily food
2025-02-06 17:08:15 <@nirik:matrix.scrye.com> we have many of those. ;)
2025-02-06 17:08:31 <@Zlopez:matrix.org> And more :-)
2025-02-06 17:08:42 <@seddik:fedora.im> !hi
2025-02-06 17:08:43 <@zodbot:fedora.im> seddik alaouiismaili (seddik)
2025-02-06 17:08:45 <@nirik:matrix.scrye.com> x3mboy: let me (re) add you to the apprentice group
2025-02-06 17:09:08 <@Zlopez:matrix.org> Hi @seddik:fedora.im
2025-02-06 17:09:15 <@nirik:matrix.scrye.com> Cool. Today it's a 🥳
2025-02-06 17:09:26 <@seddik:fedora.im> Hey folks 
2025-02-06 17:10:03 <@Zlopez:matrix.org> Plenty of people here :-)
2025-02-06 17:10:05 <@nirik:matrix.scrye.com> hey saibug and pcreech
2025-02-06 17:10:24 <@thisisyaash:fedora.im> !hi
2025-02-06 17:10:26 <@zodbot:fedora.im> Yashwanth Rathakrishnan (thisisyaash)
2025-02-06 17:10:36 <@nphilipp:fedora.im> !hi
2025-02-06 17:10:37 <@zodbot:fedora.im> Nils Philippsen (nphilipp) - he / him / his
2025-02-06 17:10:56 <@nphilipp:fedora.im> (I’m only half here 😉)
2025-02-06 17:11:02 <@nirik:matrix.scrye.com> The stars have aligned and lots of folks can make it today. ;)
2025-02-06 17:11:38 <@Zlopez:matrix.org> Let's move to next topic
2025-02-06 17:11:50 <@Zlopez:matrix.org> !topic Next chair
2025-02-06 17:11:50 <@Zlopez:matrix.org> !info magic eight ball says:
2025-02-06 17:11:50 <@Zlopez:matrix.org> !info chair 2025-02-06 - Zlopez
2025-02-06 17:11:50 <@Zlopez:matrix.org> !info chair 2025-02-13 - phsmoura
2025-02-06 17:11:50 <@Zlopez:matrix.org> !info chair 2025-02-20 - ???
2025-02-06 17:11:50 <@Zlopez:matrix.org> !info chair 2025-02-27 - ???
2025-02-06 17:12:11 <@Zlopez:matrix.org> Any volunteers for the upcoming weeks?
2025-02-06 17:12:33 <@thisisyaash:fedora.im> I can take it for the 20th
2025-02-06 17:12:40 <@Zlopez:matrix.org> Sold!
2025-02-06 17:12:52 <@Zlopez:matrix.org> !info chair 2025-02-20 - @thisisyaash:fedora.im
2025-02-06 17:13:03 <@pcreech:matrix.org> Let me rebuild some consistency first, then i'll start jumping back on the horse :D 
2025-02-06 17:13:24 <@seddik:fedora.im> Next for me 
2025-02-06 17:13:26 <@seddik:fedora.im> Yeah
2025-02-06 17:13:40 <@Zlopez:matrix.org> And sold to the highest bidder!
2025-02-06 17:13:52 <@Zlopez:matrix.org> !info chair 2025-02-27 - @seddik:fedora.im
2025-02-06 17:14:19 <@seddik:fedora.im> Nice :) 
2025-02-06 17:14:22 <@Zlopez:matrix.org> I think we are pretty covered for next few weeks
2025-02-06 17:14:34 <@Zlopez:matrix.org> !topic announcements and information
2025-02-06 17:14:34 <@Zlopez:matrix.org> !info github2fedmsg sunset date was changed to 13rd February! fedmsg will go away with it!
2025-02-06 17:14:34 <@Zlopez:matrix.org> !info CLE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
2025-02-06 17:14:34 <@Zlopez:matrix.org> !info CLE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
2025-02-06 17:14:53 <@Zlopez:matrix.org> Anything else that should be announced?
2025-02-06 17:15:57 <@seddik:fedora.im> Zlopez congratulations for github2fedmsg decommissioning 🎉🎉
2025-02-06 17:16:38 <@nirik:matrix.scrye.com> !info f42 has branched off
2025-02-06 17:17:06 <@Zlopez:matrix.org> @seddik:fedora.im We had to postpone the date, but it's done on staging and nothing break
2025-02-06 17:17:18 <@nirik:matrix.scrye.com> (that we know of. ;)
2025-02-06 17:17:24 <@Zlopez:matrix.org> Otherwise it would be already done this week
2025-02-06 17:17:44 <@Zlopez:matrix.org> @nirik:matrix.scrye.com We will see next week :-D
2025-02-06 17:18:09 <@nirik:matrix.scrye.com> yep
2025-02-06 17:20:12 <@Zlopez:matrix.org> Let's move on
2025-02-06 17:20:24 <@Zlopez:matrix.org> !info https://docs.fedoraproject.org/en-US/infra/day_to_day_fedora/#_the_oncall_role_in_our_team
2025-02-06 17:20:24 <@Zlopez:matrix.org> !topic Oncall
2025-02-06 17:20:24 <@Zlopez:matrix.org> !info ??? is on call from 2025-02-20 to 2025-02-27
2025-02-06 17:20:24 <@Zlopez:matrix.org> !info lenkaseg is on call from 2025-01-30 to 2025-02-06
2025-02-06 17:20:24 <@Zlopez:matrix.org> !info ??? is on call from 2025-02-13 to 2025-02-20
2025-02-06 17:20:24 <@Zlopez:matrix.org> !info zlopez is on call from 2025-02-06 to 2025-02-13
2025-02-06 17:20:48 <@Zlopez:matrix.org> I'm oncall now! Fear me! HAHAHAHA
2025-02-06 17:20:57 <@nirik:matrix.scrye.com> ha.
2025-02-06 17:21:43 <@Zlopez:matrix.org> !oncall
2025-02-06 17:21:44 <@zodbot:fedora.im> ● @Zlopez:matrix.org (zlopez) Current Time for them: 18:21 (Europe/Prague)
2025-02-06 17:21:44 <@zodbot:fedora.im> If they do not respond, please file a ticket (https://pagure.io/fedora-infrastructure/issues)
2025-02-06 17:21:44 <@zodbot:fedora.im> 
2025-02-06 17:21:44 <@zodbot:fedora.im> The following people are oncall:
2025-02-06 17:21:53 <@Zlopez:matrix.org> And here is the proof
2025-02-06 17:22:17 <@Zlopez:matrix.org> Anybody wants to try it for upcoming weeks?
2025-02-06 17:23:01 <@seddik:fedora.im> From 20 to 27 
2025-02-06 17:23:07 <@seddik:fedora.im> O can take 
2025-02-06 17:23:21 <@Zlopez:matrix.org> It's yours
2025-02-06 17:23:22 <@seddik:fedora.im> I can take it 
2025-02-06 17:23:32 <@Zlopez:matrix.org> !info @saibug is on call from 2025-02-20 to 2025-02-27
2025-02-06 17:23:53 <@Zlopez:matrix.org> Anybody wants next week?
2025-02-06 17:24:07 <@nirik:matrix.scrye.com> I can take it if no one else wants it
2025-02-06 17:24:34 <@pcreech:matrix.org> Thow me up there, why not :D 
2025-02-06 17:24:43 <@Zlopez:matrix.org> OK :-D
2025-02-06 17:24:57 <@nirik:matrix.scrye.com> thanks saibug and pcreech
2025-02-06 17:24:59 <@Zlopez:matrix.org> !info @pcreech:matrix.org is on call from 2025-02-13 to 2025-02-20
2025-02-06 17:25:22 <@Zlopez:matrix.org> We are pretty covered for next few weeks :-)
2025-02-06 17:25:43 <@Zlopez:matrix.org> !info Summary of last week: (from current oncall)
2025-02-06 17:25:52 <@Zlopez:matrix.org> Is @lenkaseg:fedora.im here?
2025-02-06 17:27:55 <@Zlopez:matrix.org> It doesn't seem so, I remember seeing one or two pings this week
2025-02-06 17:27:57 <@smilner:fedora.im> Letting her know about the ping just in case
2025-02-06 17:29:02 <@Zlopez:matrix.org> Let's skip it for now
2025-02-06 17:29:04 <@smilner:fedora.im> I sent her a ping but it's also pretty late local time
2025-02-06 17:29:24 <@Zlopez:matrix.org> It is
2025-02-06 17:29:26 <@Zlopez:matrix.org> !info https://nagios.fedoraproject.org/nagios 
2025-02-06 17:29:26 <@Zlopez:matrix.org> !info Go over existing items and fix them
2025-02-06 17:29:26 <@Zlopez:matrix.org> !topic Monitoring discussion [nirik]
2025-02-06 17:29:34 <@nirik:matrix.scrye.com> So, two items here...
2025-02-06 17:29:50 <@Zlopez:matrix.org> There is only the logdetective in nagios right now
2025-02-06 17:29:59 <@Zlopez:matrix.org> I see only one :-/
2025-02-06 17:30:03 <@nirik:matrix.scrye.com> 1. we still have logdetective02 listed, despite trying to get nagios to not monitor it. ;( If someone would like to poke at that again that would be great
2025-02-06 17:30:40 <@seddik:fedora.im> Ohh I think i put a pr for that.. 
2025-02-06 17:30:44 <@seddik:fedora.im> No? 
2025-02-06 17:30:49 <@nirik:matrix.scrye.com> 2. There's still a daily alert from mailman around the time the updates pushes go out (so possibly bounces from the package-announce list?) I think phsmoura was gonna look more, but I don't know if he found anything yet or would like more eyes on it?
2025-02-06 17:30:53 <@Zlopez:matrix.org> I don't even understand why it's still being monitored as it's not in our inventory
2025-02-06 17:30:58 <@nirik:matrix.scrye.com> saibug: yep, it didn't work tho. ;(
2025-02-06 17:31:18 <@nirik:matrix.scrye.com> it is in our inventory
2025-02-06 17:31:35 <@seddik:fedora.im> Hum maybe reload nrpe server? 
2025-02-06 17:31:35 <@Zlopez:matrix.org> That explains it than
2025-02-06 17:31:47 <@nirik:matrix.scrye.com> it's just often down, so we don't want to monitor it... but nagios doesn't seem to want to let go. ;)
2025-02-06 17:32:19 <@Zlopez:matrix.org> I saw few issues with bvmhost-p09-03, the one with faulty disk
2025-02-06 17:32:43 <@nirik:matrix.scrye.com> hostgroups/all.cfg and hosts/aws-hosts.cfg still lists it on noc01:/etc/nagios/...
2025-02-06 17:32:44 <@phsmoura:fedora.im> yep, couldnt find anything so far, but I intend to go back to it after finishing to update those proxies
2025-02-06 17:32:52 <@Zlopez:matrix.org> The last one was today, I needed to recreate one VM from scratch as reboot didn't work
2025-02-06 17:33:53 <@nirik:matrix.scrye.com> yeah. ;( I need to get that disk replaced, but I keep not getting it. Perhaps I can pass it off to someone else (probibly rh only as it requires paying for a new drive, arranging datacenter replacement, etc)
2025-02-06 17:34:43 <@smilner:fedora.im> Is there anyone willing and able to help with the above?
2025-02-06 17:34:58 <@Zlopez:matrix.org> I thought that @dkirwan:fedora.im is already working on this kind of issues
2025-02-06 17:35:39 <@nirik:matrix.scrye.com> yeah, perhaps he can... ;) We have a ticket, can ping him there?
2025-02-06 17:37:02 <@nirik:matrix.scrye.com> updated the ticket. ;)
2025-02-06 17:37:19 <@nirik:matrix.scrye.com> thats it monitoring wise I think?
2025-02-06 17:37:32 <@Zlopez:matrix.org> It looks like that
2025-02-06 17:37:48 <@Zlopez:matrix.org> We have plenty of folks around
2025-02-06 17:37:57 <@nirik:matrix.scrye.com> I had a couple of general topics if we have some time?
2025-02-06 17:37:59 <@Zlopez:matrix.org> Does anybody want to talk about something here?
2025-02-06 17:38:15 <@Zlopez:matrix.org> Let me skip to open floor and we can do it there
2025-02-06 17:38:27 <@Zlopez:matrix.org> !topic Open Floor
2025-02-06 17:38:32 <@nirik:matrix.scrye.com> sure...
2025-02-06 17:38:57 <@nirik:matrix.scrye.com> so first, there's a ansible handler issue going on. I guess I should file a ticket on it (so many new tickets. ;(
2025-02-06 17:39:13 <@nirik:matrix.scrye.com> ansible-lint for some reason wants you to use a capital letter on names...
2025-02-06 17:39:18 <@Zlopez:matrix.org> We are still under 70 :-)
2025-02-06 17:39:24 <@nirik:matrix.scrye.com> ie, name: foobar is bad, but name: Foobar is good
2025-02-06 17:39:39 <@nirik:matrix.scrye.com> ryanlerch renamed a bunch of handlers to use cap letter.
2025-02-06 17:39:42 <@Zlopez:matrix.org> Do you want to disable this check?
2025-02-06 17:39:52 <@Zlopez:matrix.org> Or rename them
2025-02-06 17:39:56 <@nirik:matrix.scrye.com> but... you can't just change the name in the handler, the way ansible calls them is with the name
2025-02-06 17:40:06 <@nirik:matrix.scrye.com> so every place that has the lower case name is broken
2025-02-06 17:40:18 <@nirik:matrix.scrye.com> I have fixed various ones, but it's getting annoying.
2025-02-06 17:40:36 <@nirik:matrix.scrye.com> So, we should either: tell ansible-lint we don't care, or just fix everything.
2025-02-06 17:40:51 <@Zlopez:matrix.org> Both are doable
2025-02-06 17:40:55 <@nirik:matrix.scrye.com> either way it's anoying to change because it's hard to find all the handler calls with a grep or anything
2025-02-06 17:41:44 <@Zlopez:matrix.org> I would assume that grep will help, I can look into that
2025-02-06 17:41:48 <@nirik:matrix.scrye.com> I don't care too much which way, but I just want it fixed. ;)
2025-02-06 17:41:58 <@nirik:matrix.scrye.com> in many playbooks theres:
2025-02-06 17:42:13 <@nirik:matrix.scrye.com>   - restart stunnel
2025-02-06 17:42:13 <@nirik:matrix.scrye.com> notify:
2025-02-06 17:42:13 <@nirik:matrix.scrye.com>   - reload systemd
2025-02-06 17:42:42 <@nirik:matrix.scrye.com> which I mean you can get with grep -A2, but what if there's 3 or 4 or more?
2025-02-06 17:42:59 <@nirik:matrix.scrye.com> anyhow, will file a ticket and hope someone will fix it. ;)
2025-02-06 17:43:21 <@seddik:fedora.im> Yes
2025-02-06 17:43:25 <@Zlopez:matrix.org> I will do it in e-macs it should be quick
2025-02-06 17:43:54 <@Zlopez:matrix.org> Not quick, but easy, it will probably take time
2025-02-06 17:44:17 <@nirik:matrix.scrye.com> There may be some way with yq...
2025-02-06 17:44:35 <@nirik:matrix.scrye.com> but yeah... not sure.
2025-02-06 17:44:39 <@Zlopez:matrix.org> Hm, my TODO list is growing today :-D
2025-02-06 17:44:57 <@nirik:matrix.scrye.com> it may be ryan is willing to poke at it more too... or some others. ;)
2025-02-06 17:45:09 <@seddik:fedora.im> Feel a ticket, and can take a look also.. 
2025-02-06 17:45:13 <@Zlopez:matrix.org> There is a replace feature in emacs you can probably run on whole project, I will look into it
2025-02-06 17:45:48 <@nirik:matrix.scrye.com> will do.
2025-02-06 17:45:53 <@seddik:fedora.im> File a ticket, and can take a look also.. 
2025-02-06 17:46:16 <@nirik:matrix.scrye.com> ok, second item: I filed https://pagure.io/fedora-infrastructure/issue/12390 a bit ago... a top level tracking ticket for our datacenter move later this year.
2025-02-06 17:46:52 <@nirik:matrix.scrye.com> One thing related to this is that we probibly want to try and finish off things that are 'in flight' right now and then not change stuff to dramatically when we are going to move.
2025-02-06 17:47:11 <@nirik:matrix.scrye.com> Some things would be nice to fix/get done before that too.
2025-02-06 17:47:46 <@nirik:matrix.scrye.com> And there's going to be a lot of work needed by a lot of folks on this. :) I'm working on a more detailed plan, which also will need help/input.
2025-02-06 17:48:28 <@Zlopez:matrix.org> That would be great
2025-02-06 17:48:46 <@james:fedora.im> So I tried with yq, and got pretty close ... but yq likes to reformat everything when it changes anything ... so I think that's a no go.
2025-02-06 17:48:52 <@nirik:matrix.scrye.com> Things in flight I mean: move to quay.io, rabbitmq cluster upgrades, possibly try and accelerate zabbix so we only use it in new dc, etc
2025-02-06 17:49:22 <@nirik:matrix.scrye.com> James Antill :( bummer. yeah, I used yq before and it did indeed reformat everything.
2025-02-06 17:50:16 <@nirik:matrix.scrye.com> also, another thing that occurrs to me is the wiki. Currently it's on f40... there were some issues with mediawiki packaging in f41+... and f40 will EOL right around the datacenter move time, so it woud be good to look into that.
2025-02-06 17:51:09 <@nirik:matrix.scrye.com> Anyhow, just wanted to get everyone thinking about it...
2025-02-06 17:51:34 <@nirik:matrix.scrye.com> and if there's questions I can do my best to answer them, or ask/find out.
2025-02-06 17:51:50 <@Zlopez:matrix.org> Is the wiki still used?
2025-02-06 17:52:39 <@nirik:matrix.scrye.com> yes. QA uses it for test results. It's 100% needed.
2025-02-06 17:53:02 <@nirik:matrix.scrye.com> or we would need an alternative, but... thats a ton of work/workflow disruption.
2025-02-06 17:53:31 <@nirik:matrix.scrye.com> They did look at some TCMS not long ago, but none of them really work for what we are doing. ;(
2025-02-06 17:54:40 <@Zlopez:matrix.org> Understood
2025-02-06 17:55:41 <@nirik:matrix.scrye.com> https://pagure.io/releng/issue/12229 for background on the f41+ problem...
2025-02-06 17:55:59 <@nirik:matrix.scrye.com> I am not sure if the solution I came up with works (I don't know php at all, just wildly guessing)
2025-02-06 17:56:49 <@Zlopez:matrix.org> I don't know much about PHP either :-/
2025-02-06 17:57:19 <@nirik:matrix.scrye.com> related: proxy09 is caching the wiki wrong... I couldn't see anything obvious, more 👀 would be great:
2025-02-06 17:57:44 <@nirik:matrix.scrye.com> https://pagure.io/fedora-infrastructure/issue/12385
2025-02-06 17:58:43 <@Zlopez:matrix.org> I saw another user reporting that today in fedora infra matrix channel
2025-02-06 17:59:03 <@Zlopez:matrix.org> It's strange that is happening only on one proxy
2025-02-06 17:59:28 <@nirik:matrix.scrye.com> yeah. I don't see why.
2025-02-06 18:00:00 <@x3mboy:fedora.im> Yeah, it was me
2025-02-06 18:00:01 <@x3mboy:fedora.im> Sorry
2025-02-06 18:00:16 <@x3mboy:fedora.im> I was looking old stuff and it happened to be broken
2025-02-06 18:00:22 <@Zlopez:matrix.org> No problem
2025-02-06 18:00:32 <@Zlopez:matrix.org> We are at the end of our time
2025-02-06 18:00:37 <@Zlopez:matrix.org> So let's end it here
2025-02-06 18:00:42 <@Zlopez:matrix.org> !endmeeting