2025-02-27 17:00:17 <@seddik:fedora.im> !startmeeting Infrastructure (2025-02-27)
2025-02-27 17:00:18 <@meetbot:fedora.im> Meeting started at 2025-02-27 17:00:17 UTC
2025-02-27 17:00:18 <@meetbot:fedora.im> The Meeting name is 'Infrastructure (2025-02-27)'
2025-02-27 17:00:27 <@seddik:fedora.im> !meetingname infrastructure
2025-02-27 17:00:28 <@meetbot:fedora.im> The Meeting Name is now infrastructure
2025-02-27 17:00:36 <@seddik:fedora.im> !info About our team: https://docs.fedoraproject.org/en-US/cle/
2025-02-27 17:00:42 <@seddik:fedora.im> !topic ahoy
2025-02-27 17:00:55 <@Zlopez:matrix.org> !hi
2025-02-27 17:00:55 <@seddik:fedora.im> !info Agenda is at: https://board.net/p/fedora-infra
2025-02-27 17:00:57 <@smilner:fedora.im> !hi
2025-02-27 17:00:57 <@zodbot:fedora.im> Michal Konecny (zlopez)
2025-02-27 17:00:58 <@zodbot:fedora.im> None (smilner)
2025-02-27 17:01:01 <@ralph:fedora.im> !hi
2025-02-27 17:01:01 <@seddik:fedora.im> !info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
2025-02-27 17:01:02 <@zodbot:fedora.im> Ralph Bean (ralph)
2025-02-27 17:01:20 <@seddik:fedora.im> !hi
2025-02-27 17:01:20 <@seddik:fedora.im> 
2025-02-27 17:01:44 <@nirik:matrix.scrye.com> morning
2025-02-27 17:01:44 <@seddik:fedora.im> morning everyone 
2025-02-27 17:02:21 <@seddik:fedora.im> let's wait a bit 
2025-02-27 17:02:35 <@seddik:fedora.im> ppl can join :)
2025-02-27 17:05:13 <@seddik:fedora.im> ok .. 
2025-02-27 17:05:20 <@seddik:fedora.im> !info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
2025-02-27 17:05:26 <@seddik:fedora.im> !info Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted
2025-02-27 17:05:32 <@seddik:fedora.im> !topic New folks introductions
2025-02-27 17:05:36 <@nphilipp:fedora.im> !hi
2025-02-27 17:05:37 <@zodbot:fedora.im> Nils Philippsen (nphilipp) - he / him / his
2025-02-27 17:05:51 <@seddik:fedora.im> Anybody new here today ?? 
2025-02-27 17:06:44 <@ralph:fedora.im> I used to attend a long time ago, but I'm back today! I'm Ralph and I'm working on konflux.
2025-02-27 17:07:13 <@nirik:matrix.scrye.com> Welcome (back) Ralph Bean. :)
2025-02-27 17:07:20 <@Zlopez:matrix.org> Welcome back!
2025-02-27 17:07:33 <@seddik:fedora.im> welcome backkk! 
2025-02-27 17:07:48 <@seddik:fedora.im> let's move to chair topic 
2025-02-27 17:07:59 <@seddik:fedora.im> !topic Next chair
2025-02-27 17:08:16 <@seddik:fedora.im> !info chair 2025-03-20 - ??
2025-02-27 17:08:16 <@seddik:fedora.im> !info chair 2025-02-27 - @saibug
2025-02-27 17:08:16 <@seddik:fedora.im> !info chair 2025-03-06 - @zlopez
2025-02-27 17:08:16 <@seddik:fedora.im> !info chair 2025-03-13 - @saibug
2025-02-27 17:08:38 <@seddik:fedora.im> hum any volunteer for 20 ?? 
2025-02-27 17:09:51 <@seddik:fedora.im> hum any volunteer for 20th?? 
2025-02-27 17:11:39 <@phsmoura:fedora.im> I can take it
2025-02-27 17:12:20 <@seddik:fedora.im> ok sold :) 
2025-02-27 17:12:37 <@seddik:fedora.im> !info chair 2025-03-20 - @phsmoura
2025-02-27 17:13:01 <@seddik:fedora.im> nice we are covered for next weeks ;) 
2025-02-27 17:13:16 <@seddik:fedora.im> !topic announcements and information
2025-02-27 17:13:23 <@seddik:fedora.im> !info CLE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
2025-02-27 17:13:23 <@seddik:fedora.im> !info CLE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
2025-02-27 17:13:40 <@nirik:matrix.scrye.com> !info tomorrow is a Red Hat day of learning
2025-02-27 17:14:01 <@seddik:fedora.im> anything that should be announced  ??? 
2025-02-27 17:14:10 <@smooge:fedora.im> I didn't know tomorrow was a day of Learning
2025-02-27 17:14:17 <@smooge:fedora.im> thanks
2025-02-27 17:14:25 <@Zlopez:matrix.org> !info Fedora 42 Beta Go/No Go meeting will happen on 6th March
2025-02-27 17:14:57 <@Zlopez:matrix.org> !info We are in F42 Beta freeze
2025-02-27 17:15:18 <@ralph:fedora.im> !info I created a konflux channel at https://matrix.to/#/#konflux:fedora.im open to anyone.
2025-02-27 17:17:39 <@seddik:fedora.im> ok let's move to the next topic 
2025-02-27 17:17:42 <@seddik:fedora.im> !topic Oncall
2025-02-27 17:18:11 <@seddik:fedora.im> !info ??? is on call from 2025-02-27 to 2025-03-06 
2025-02-27 17:18:38 <@Zlopez:matrix.org> Let me take that
2025-02-27 17:18:38 <@seddik:fedora.im> do we have any volunteers  for this week ?? 
2025-02-27 17:19:39 <@seddik:fedora.im> ok nice
2025-02-27 17:19:39 <@seddik:fedora.im> !info Zlopez is on call from 2025-02-27 to 2025-03-06
2025-02-27 17:20:22 <@seddik:fedora.im> so let's look at the summary of last week 
2025-02-27 17:20:44 <@Zlopez:matrix.org> !oncall
2025-02-27 17:20:46 <@zodbot:fedora.im> 
2025-02-27 17:20:46 <@zodbot:fedora.im> If they do not respond, please file a ticket (https://pagure.io/fedora-infrastructure/issues)
2025-02-27 17:20:46 <@zodbot:fedora.im> ● @Zlopez:matrix.org (zlopez) Current Time for them: 18:20 (Europe/Prague)
2025-02-27 17:20:46 <@zodbot:fedora.im> The following people are oncall:
2025-02-27 17:20:52 <@Zlopez:matrix.org> Everything set
2025-02-27 17:21:06 <@seddik:fedora.im> https://pagure.io/releng/issue/12612
2025-02-27 17:21:06 <@seddik:fedora.im> I only got this ping for this 
2025-02-27 17:21:06 <@seddik:fedora.im> 
2025-02-27 17:21:54 <@Zlopez:matrix.org> It is partially resolved
2025-02-27 17:22:13 <@Zlopez:matrix.org> The quick fix is in place, but the PR needs to be open for more permanent solution
2025-02-27 17:22:27 <@nirik:matrix.scrye.com> well, for an extra layer, but yeah
2025-02-27 17:22:38 <@seddik:fedora.im> yes I saw the message on channel releng
2025-02-27 17:23:25 <@seddik:fedora.im> nirik thanks for  resolving bot issue (for oncall ping) 
2025-02-27 17:23:50 <@nirik:matrix.scrye.com> yeah, sorry that was missed in the last meeting.
2025-02-27 17:24:26 <@seddik:fedora.im> and now ... moving to monitoring discussion :) 
2025-02-27 17:24:30 <@seddik:fedora.im> !topic Monitoring discussion [nirik]
2025-02-27 17:24:50 <@nirik:matrix.scrye.com> so, lets see...
2025-02-27 17:25:03 <@nirik:matrix.scrye.com> 2 things pending:
2025-02-27 17:25:52 <@nirik:matrix.scrye.com> * There's a down power9 machine. I have a ticket filed to get them to power cycle it completely. Hopefully that brings it back, if not more digging required.
2025-02-27 17:26:24 <@nirik:matrix.scrye.com> * There's some checks we used to run on busgateway01, and with it being retired we need to move them to another machine. I think Zlopez was working on that
2025-02-27 17:26:41 <@nirik:matrix.scrye.com> * Oh, and logdetective02 is still there. ;)
2025-02-27 17:27:23 <@Zlopez:matrix.org> Waiting currently for review of the packages needed for that
2025-02-27 17:27:42 <@nirik:matrix.scrye.com> cool
2025-02-27 17:27:48 <@nirik:matrix.scrye.com> thats all I think, we can move on...
2025-02-27 17:28:08 <@seddik:fedora.im> ok 
2025-02-27 17:28:18 <@seddik:fedora.im> !topic check blocked tickets
2025-02-27 17:28:26 <@Zlopez:matrix.org> I added some topics for discussion
2025-02-27 17:28:32 <@seddik:fedora.im> !link https://pagure.io/fedora-infrastructure/issues?status=Open&tags=blocked&priority=0&close_status=
2025-02-27 17:28:58 <@seddik:fedora.im> !info Check if any blocked ticket
2025-02-27 17:29:23 <@Zlopez:matrix.org> Let's wait for them to open floor
2025-02-27 17:31:02 <@seddik:fedora.im> so let's go over few tickets or no ?? 
2025-02-27 17:31:11 <@Zlopez:matrix.org> We had some progress on OIDC
2025-02-27 17:31:21 <@Zlopez:matrix.org> The COPR is now done
2025-02-27 17:31:43 <@nirik:matrix.scrye.com> but we found another one right? blockerbugs?
2025-02-27 17:32:29 <@Zlopez:matrix.org> Yes, but we are not blocked on this anymore
2025-02-27 17:32:36 <@Zlopez:matrix.org> I will remove the blocked label
2025-02-27 17:32:51 <@seddik:fedora.im> ok thanks 
2025-02-27 17:33:18 <@seddik:fedora.im> hum let's move to open floor topic 
2025-02-27 17:33:20 <@seddik:fedora.im> !topic Open Floor
2025-02-27 17:33:37 <@Zlopez:matrix.org> Ok, so I have two things I wanted to discuss here
2025-02-27 17:33:56 <@Zlopez:matrix.org> !info Sunset of OpenID
2025-02-27 17:34:20 <@Zlopez:matrix.org> Most of the apps are moved already to OIDC
2025-02-27 17:34:38 <@nirik:matrix.scrye.com> well, would be good to have blockerbugs keep working right?
2025-02-27 17:35:10 <@Zlopez:matrix.org> I propose to announce some sunset date and then disable the openID
2025-02-27 17:35:24 <@Zlopez:matrix.org> We need to get the blockerbugs fixed first that is true
2025-02-27 17:35:55 <@nirik:matrix.scrye.com> I think it might be also nice to identify any users we can easily identify and try and notify them...
2025-02-27 17:36:01 <@nirik:matrix.scrye.com> but sure, we could set a date.
2025-02-27 17:36:58 <@Zlopez:matrix.org> It should be possible to just grep logs and find anybody who is reaching /openid endpoint
2025-02-27 17:37:08 <@nirik:matrix.scrye.com> There seem to be some lists using it.
2025-02-27 17:37:10 <@nirik:matrix.scrye.com> yeah
2025-02-27 17:37:30 <@Zlopez:matrix.org> lists? like mailman?
2025-02-27 17:38:20 <@nirik:matrix.scrye.com> yes, but not ours. ;)
2025-02-27 17:38:42 <@nirik:matrix.scrye.com> https://listat.jyps.fi seems to be using it, for example
2025-02-27 17:39:22 <@nirik:matrix.scrye.com> or lists.openldap.org
2025-02-27 17:39:23 <@nirik:matrix.scrye.com> etc
2025-02-27 17:39:46 <@Zlopez:matrix.org> Ok, I will update the OIDC ticket with the next plan, setting the date is still a good way forward at least it will set some deadline for the projects
2025-02-27 17:40:05 <@Zlopez:matrix.org> But first let's identify them
2025-02-27 17:40:08 <@nirik:matrix.scrye.com> yeah
2025-02-27 17:41:19 <@Zlopez:matrix.org> That was the first topic, the second one is more organizational
2025-02-27 17:41:31 <@Zlopez:matrix.org> !info Infra work handover between timezones
2025-02-27 17:42:32 <@Zlopez:matrix.org> I was thinking about this for some time, writing some summary at the end of the day to share with people in different timezone what happened
2025-02-27 17:43:12 <@Zlopez:matrix.org> Do you think that is a good idea?
2025-02-27 17:43:29 <@nirik:matrix.scrye.com> I'm happy to try things... but isn't there usually that information in our matrix channels?
2025-02-27 17:43:58 <@Zlopez:matrix.org> Sometimes it is getting lost in the chat, but it's there
2025-02-27 17:44:03 <@nirik:matrix.scrye.com> (and/or tickets I guess)
2025-02-27 17:44:57 <@Zlopez:matrix.org> Not everything we fix is in tickets, at least I'm not opening ticket for every alert I'm fixing
2025-02-27 17:45:24 <@nirik:matrix.scrye.com> What sort of things would this be for? Important stuff that needs continung or just day to day things or ?
2025-02-27 17:46:08 <@smooge:fedora.im> Generally if you are doing something in chat.. it might be helpful to use a "Form" to post a regular thing. Element search is not great from what I can tell but a visual "++++++++++ Report for YYYY-MM-DD X shift +++++++++++++" might be easy to find in a scroll up
2025-02-27 17:46:44 <@nirik:matrix.scrye.com> well, we have a flood of information, I am not sure adding more really helps the issue. ;) But if there's specific things we could try and highlight them somehow
2025-02-27 17:47:40 <@smooge:fedora.im> * People reported login problems on ZZZZ at XXYY UTC. Fixed itself before ticket was open but came back later
2025-02-27 17:47:40 <@smooge:fedora.im> * Got reports at XXYY UTC about s390 problems. Ticket XXXX is probably high priority
2025-02-27 17:47:47 <@Zlopez:matrix.org> Mostly the important things like "Lost connection for machine XYZ - fixed" "Investigating authentication failure - ipsilon returns 500 - didn't found the root issue" etc.
2025-02-27 17:48:54 <@nirik:matrix.scrye.com> well, I try and say those things in noc channel... but I am not sure that kind of thing would stand out much...
2025-02-27 17:49:05 <@Zlopez:matrix.org> I don't want to burden folks with more work and adding that to more spaces, but few bullet points would be helpful
2025-02-27 17:49:29 <@Zlopez:matrix.org> I'm trying to read that every day, but sometimes it's flooded with alerts
2025-02-27 17:49:33 <@nirik:matrix.scrye.com> I can try and add/highlight more, but... someone needs to pay attention to it. ;)
2025-02-27 17:50:00 <@nirik:matrix.scrye.com> so, I was talking with Gwmngilfen the other day about that... perhaps we should move those to a bot channel?
2025-02-27 17:50:07 <@smooge:fedora.im> but my suggestion is only useful if it contains a couple of things that people need to get their job done
2025-02-27 17:50:10 <@nirik:matrix.scrye.com> but I think they are useful to have.
2025-02-27 17:50:45 <@smooge:fedora.im> #nocbot ?
2025-02-27 17:50:52 <@Zlopez:matrix.org> I'm not against that
2025-02-27 17:51:22 <@gwmngilfen:fedora.im> NGL, I mute the not when I want to catch up
2025-02-27 17:51:24 <@nirik:matrix.scrye.com> the problem might be there that people ignore them, but then again they ignore them now. ;)
2025-02-27 17:51:33 <@gwmngilfen:fedora.im> NGL, I mute the bot when I want to catch up
2025-02-27 17:51:40 <@nirik:matrix.scrye.com> I'm fine moving it off to see if to helps folks.
2025-02-27 17:51:50 <@Zlopez:matrix.org> We need to find the information that is actually useful to share, but this will maybe take some trials and errors
2025-02-27 17:52:03 <@gwmngilfen:fedora.im> Then I unmute after :)
2025-02-27 17:52:31 <@nirik:matrix.scrye.com> zabbix we already have in it's own channel(s)... but I muted them... too much
2025-02-27 17:52:47 <@smooge:fedora.im> oh i didn't know there was a zabbix bot channel
2025-02-27 17:53:31 <@nirik:matrix.scrye.com> there's a prod and a stg one..
2025-02-27 17:53:37 <@nirik:matrix.scrye.com> #noc-zabbix-testing:fedora.im
2025-02-27 17:53:51 <@nirik:matrix.scrye.com> and... looking for the stg one
2025-02-27 17:54:01 <@gwmngilfen:fedora.im> Ah, I was looking for that earlier, thanks :)
2025-02-27 17:54:11 <@smooge:fedora.im> thanks
2025-02-27 17:54:22 <@nirik:matrix.scrye.com> #fedora-zodbot:fedora.im but it seems busted right now. ;)
2025-02-27 17:55:11 <@nirik:matrix.scrye.com> anyhow... we can try and add some kind of 'handoff' notice if there's ongoing things? or do you think matrix isn't good for this?
2025-02-27 17:55:35 <@Zlopez:matrix.org> I'm for using that channel just for updates, we will see if that will be actually helpful
2025-02-27 17:55:57 <@nirik:matrix.scrye.com> and moving the bot?
2025-02-27 17:56:13 <@Zlopez:matrix.org> Or create a new one
2025-02-27 17:57:20 <@nirik:matrix.scrye.com> a new room just for handoffs? I fear that would either just not be looked at/used much, or people would start discussing things in it...
2025-02-27 17:58:26 <@Zlopez:matrix.org> Or we can just use some keyword and share it in noc or fedora-infra room
2025-02-27 17:59:05 <@Zlopez:matrix.org> Not sure how good are the matrix clients with highlighting keywords
2025-02-27 17:59:24 <@nirik:matrix.scrye.com> almost any of them can let you add words to highlight
2025-02-27 17:59:33 <@nirik:matrix.scrye.com> (at least that I have seen)
2025-02-27 18:00:20 <@seddik:fedora.im> We can continue on another room ??
2025-02-27 18:00:33 <@seddik:fedora.im> Thanks for being here today 
2025-02-27 18:00:35 <@nirik:matrix.scrye.com> yeah we are out of time. ;) continue in #noc ? ;)
2025-02-27 18:00:47 <@Zlopez:matrix.org> We can
2025-02-27 18:01:05 <@seddik:fedora.im> !endmeeting