2025-05-15 17:00:19 <@Zlopez:matrix.org> !startmeeting Infrastructure (2025-05-15)
2025-05-15 17:00:21 <@meetbot:fedora.im> Meeting started at 2025-05-15 17:00:19 UTC
2025-05-15 17:00:21 <@meetbot:fedora.im> The Meeting name is 'Infrastructure (2025-05-15)'
2025-05-15 17:00:33 <@Zlopez:matrix.org> !topic aloha
2025-05-15 17:00:33 <@Zlopez:matrix.org> !chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp
2025-05-15 17:00:33 <@Zlopez:matrix.org> !info About our team: https://docs.fedoraproject.org/en-US/cle/
2025-05-15 17:00:33 <@Zlopez:matrix.org> !info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
2025-05-15 17:00:33 <@Zlopez:matrix.org> !meetingname infrastructure
2025-05-15 17:00:33 <@Zlopez:matrix.org> !info Agenda is at: https://board.net/p/fedora-infra
2025-05-15 17:00:36 <@meetbot:fedora.im> The Meeting Name is now infrastructure
2025-05-15 17:00:52 <@Zlopez:matrix.org> !hi
2025-05-15 17:00:57 <@zodbot:fedora.im> Michal Konecny (zlopez)
2025-05-15 17:01:29 <@Zlopez:matrix.org> Welcome everyone on this week Infrastructure meeting
2025-05-15 17:01:48 <@nirik:matrix.scrye.com> morning
2025-05-15 17:01:57 <@nirik:matrix.scrye.com> ahoy!
2025-05-15 17:02:31 <@james:fedora.im> Hey
2025-05-15 17:04:12 <@Zlopez:matrix.org> We have a pirate here today :-D
2025-05-15 17:04:15 <@Zlopez:matrix.org> Ahoy!
2025-05-15 17:04:38 <@nirik:matrix.scrye.com> 🏴‍☠️
2025-05-15 17:05:39 <@Zlopez:matrix.org> It seems that there wouldn't be much folks today
2025-05-15 17:06:21 <@Zlopez:matrix.org> For those who are here, welcome aboard this pirate ship!
2025-05-15 17:06:54 <@Zlopez:matrix.org> Let's see if we have some landlovers that want to get wet a little
2025-05-15 17:06:55 <@Zlopez:matrix.org> !topic New folks introductions
2025-05-15 17:06:55 <@Zlopez:matrix.org> !info Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/
2025-05-15 17:06:55 <@Zlopez:matrix.org> !info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
2025-05-15 17:09:28 <@Zlopez:matrix.org> I don't see any around, maybe they are too afraid
2025-05-15 17:09:36 <@markrosenbaum:fedora.im> !hi
2025-05-15 17:09:37 <@zodbot:fedora.im> Mark Rosenbaum (markrosenbaum)
2025-05-15 17:09:44 <@markrosenbaum:fedora.im> Sorry I’m a bit late
2025-05-15 17:10:18 <@nirik:matrix.scrye.com> never too late to come aboard.
2025-05-15 17:10:32 <@markrosenbaum:fedora.im> Pirates today I see
2025-05-15 17:10:52 <@Zlopez:matrix.org> Let's look at who wants to take the helm of this ship in upcoming weeks
2025-05-15 17:10:53 <@Zlopez:matrix.org> !topic Next chair
2025-05-15 17:10:53 <@Zlopez:matrix.org> !info chair 2025-05-15 - zlopez
2025-05-15 17:10:53 <@Zlopez:matrix.org> !info chair 2025-05-22 - James
2025-05-15 17:10:53 <@Zlopez:matrix.org> !info chair 2025-05-29 - ???
2025-05-15 17:10:53 <@Zlopez:matrix.org> !info magic eight ball says:
2025-05-15 17:11:24 <@Zlopez:matrix.org> James it is for next week
2025-05-15 17:11:42 <@Zlopez:matrix.org> Who would like to take this ship for a sail on 2025-05-29
2025-05-15 17:11:47 <@Zlopez:matrix.org> ?
2025-05-15 17:13:58 <@Zlopez:matrix.org> Anybody else?
2025-05-15 17:14:17 <@Zlopez:matrix.org> We can keep the helm free till next week
2025-05-15 17:14:40 <@markrosenbaum:fedora.im> I could only ever take it if it were easier
2025-05-15 17:15:02 <@markrosenbaum:fedora.im> Like exactly an hour before we regularly do it 
2025-05-15 17:15:08 <@smilner:fedora.im> 👋 (oops, lost track of time)
2025-05-15 17:15:12 <@markrosenbaum:fedora.im> Also I’ve never tried before
2025-05-15 17:15:21 <@markrosenbaum:fedora.im> So🤷‍♂️
2025-05-15 17:15:30 <@Zlopez:matrix.org> It's not hard, you just follow the agenda
2025-05-15 17:15:35 <@Zlopez:matrix.org> https://board.net/p/fedora-infra
2025-05-15 17:15:55 <@Zlopez:matrix.org> But I understand that it could be too late for some
2025-05-15 17:16:39 <@nirik:matrix.scrye.com> We could move an hour eariler... thats fine with me... but moving things is always hard
2025-05-15 17:17:07 <@Zlopez:matrix.org> It will switch again once the DST kicks in
2025-05-15 17:18:33 <@nirik:matrix.scrye.com> right, DST messes up everything... always
2025-05-15 17:19:31 <@Zlopez:matrix.org> From what I heard it should be the last shift here next time
2025-05-15 17:20:11 <@nirik:matrix.scrye.com> they have been saying that for years. ;)
2025-05-15 17:21:10 <@Zlopez:matrix.org> I know, this is why I'm saying "From what I heard" :-D
2025-05-15 17:21:25 <@Zlopez:matrix.org> OK, let's move to the next deck, sailors!
2025-05-15 17:21:37 <@Zlopez:matrix.org> !info OpenID EOL in Fedora Infra is TBD as we changed the approach. See https://communityblog.fedoraproject.org/end-of-openid-authentication-in-fedora-account-system-may-update/
2025-05-15 17:21:37 <@Zlopez:matrix.org> !topic announcements and information
2025-05-15 17:21:37 <@Zlopez:matrix.org> !info CLE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0815 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
2025-05-15 17:21:37 <@Zlopez:matrix.org> !info CLE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1900 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
2025-05-15 17:22:12 <@nirik:matrix.scrye.com> !info 2025-05-23 is a Red Hat recharge day. Many folks will be away
2025-05-15 17:22:29 <@nirik:matrix.scrye.com> !info 2025-05-26 is a holiday in the US
2025-05-15 17:24:18 <@nirik:matrix.scrye.com> thats all I had I think
2025-05-15 17:24:55 <@Zlopez:matrix.org> Anything else to put on the board?
2025-05-15 17:27:02 <@Zlopez:matrix.org> Let's move to next deck then
2025-05-15 17:27:11 <@Zlopez:matrix.org> !info ??? is on call from 2025-05-15 to 2025-05-21
2025-05-15 17:27:11 <@Zlopez:matrix.org> !topic Oncall
2025-05-15 17:27:11 <@Zlopez:matrix.org> !info https://docs.fedoraproject.org/en-US/infra/day_to_day_fedora/#_the_oncall_role_in_our_team
2025-05-15 17:27:11 <@Zlopez:matrix.org> !info Mark Rosenbaum is on call from 2025-05-08 to 2025-05-14
2025-05-15 17:27:37 <@Zlopez:matrix.org> Any volunteer to watch for any danger?
2025-05-15 17:28:26 <@markrosenbaum:fedora.im> If none others I can
2025-05-15 17:30:58 <@Zlopez:matrix.org> It's yours
2025-05-15 17:31:15 <@Zlopez:matrix.org> But you are already on this week
2025-05-15 17:31:21 <@Zlopez:matrix.org> Let me take it instead
2025-05-15 17:31:29 <@Zlopez:matrix.org> !info zlopez is on call from 2025-05-15 to 2025-05-21
2025-05-15 17:32:24 <@Zlopez:matrix.org> !info ??? is on call from 2025-05-22 to 2025-05-29
2025-05-15 17:32:35 <@Zlopez:matrix.org> Do we have a volunteer for week after that?
2025-05-15 17:33:00 <@nirik:matrix.scrye.com> I'm always around anyhow, I guess I could.
2025-05-15 17:33:30 <@Zlopez:matrix.org> Let me add you then
2025-05-15 17:33:43 <@Zlopez:matrix.org> !info nirik is on call from 2025-05-22 to 2025-05-29
2025-05-15 17:34:21 <@Zlopez:matrix.org> !info Summary of last week: (from current oncall)
2025-05-15 17:34:32 <@Zlopez:matrix.org> Mark Rosenbaum: The deck is yours
2025-05-15 17:35:13 <@Zlopez:matrix.org> !oncall
2025-05-15 17:35:15 <@zodbot:fedora.im> The following people are oncall:
2025-05-15 17:35:15 <@zodbot:fedora.im> 
2025-05-15 17:35:15 <@zodbot:fedora.im> ● @Zlopez:matrix.org (zlopez) Current Time for them: 19:35 (Europe/Prague)
2025-05-15 17:35:15 <@zodbot:fedora.im> If they do not respond, please file a ticket (https://pagure.io/fedora-infrastructure/issues)
2025-05-15 17:35:21 <@Zlopez:matrix.org> Oncall set
2025-05-15 17:37:26 <@Zlopez:matrix.org> Mark Rosenbaum: Did you saw any oncall ping when you were on call?
2025-05-15 17:38:12 <@markrosenbaum:fedora.im> Oop, sorry didn’t realize it was my turn
2025-05-15 17:38:13 <@markrosenbaum:fedora.im> Nope
2025-05-15 17:38:14 <@markrosenbaum:fedora.im> No pings
2025-05-15 17:39:08 <@Zlopez:matrix.org> In that case let's see how the system on our ship are faring
2025-05-15 17:39:09 <@Zlopez:matrix.org> !topic Monitoring discussion [nirik]
2025-05-15 17:39:09 <@Zlopez:matrix.org> !info Go over existing items and fix them
2025-05-15 17:39:09 <@Zlopez:matrix.org> !info https://nagios.fedoraproject.org/nagios 
2025-05-15 17:39:45 <@nirik:matrix.scrye.com> so, back to the way it's been for a while: logdetective02 showing, db-koji01 low on disk (but this should be fixed by moving it to rdu3) and those datagrepper checks
2025-05-15 17:40:42 <@Zlopez:matrix.org> I need to find some time to finish the package review, I got one request to add something to spec, but didn't get to that yet
2025-05-15 17:40:50 <@nirik:matrix.scrye.com> There were some anoying alerts from proxy35/37, but somehow it turned out that crond was not running there. I am not sure why at all...
2025-05-15 17:41:08 <@nirik:matrix.scrye.com> fair enough
2025-05-15 17:43:31 <@nirik:matrix.scrye.com> thats all I can see on monitoring off hand
2025-05-15 17:44:19 <@Zlopez:matrix.org> OK, let's see what is waiting us on the lower deck
2025-05-15 17:44:44 <@Zlopez:matrix.org> Backlog refinement
2025-05-15 17:44:59 <@nirik:matrix.scrye.com> I can also give a short datacenter move update...
2025-05-15 17:45:05 <@Zlopez:matrix.org> Discussion about learning topic
2025-05-15 17:45:12 <@Zlopez:matrix.org> Or we can skip to open floor
2025-05-15 17:45:42 <@Zlopez:matrix.org> From what I see it would be best to skip to open floor and give the work to nirik 
2025-05-15 17:45:49 <@Zlopez:matrix.org> !topic Open Floor
2025-05-15 17:48:00 <@nirik:matrix.scrye.com> ok, so a quick datacenter move update. We got access to the mgmt on the new hardware. Gwmngilfen and I (mostly him) have been configuring them and doing firmware updates on them. Thats mostly done now on all of them. We are still waiting on network for the actual machines. Thats hopefully very soon. I have setup dns for rdu3 and pushed it out. We are a week or two behind in our schedule, so it may be that we move the switcharoo week out some, but not yet decided. Hopefully we learn more about the network soon to decide that.
2025-05-15 17:48:56 <@nirik:matrix.scrye.com> Once network comes up, I plan to somewhat manually install a bastion01.rdu3 vm, then setup ansible to gateway via that to all the machines there, and then setup a noc01.rdu3, ns01.rdu3 and a few more. From that we can then install other things.
2025-05-15 17:51:06 <@nirik:matrix.scrye.com> Should know more about any schedule changes after this week. :)
2025-05-15 17:51:15 <@nirik:matrix.scrye.com> any questions?
2025-05-15 17:52:02 <@nirik:matrix.scrye.com> Zlopez: I might see if you have cycles to install a ipa01.rdu3 and replicate it to our existing cluster once enough things are up to do that.
2025-05-15 17:52:44 <@Zlopez:matrix.org> I should find some
2025-05-15 17:53:37 <@nirik:matrix.scrye.com> Thats all I think
2025-05-15 17:53:40 <@Zlopez:matrix.org> Let me know when that's actual
2025-05-15 17:55:09 <@Zlopez:matrix.org> I would like some help with the dist-git pushing over https, I think I setup everything correctly, but I'm still getting authentication failure when trying to push the changes
2025-05-15 17:55:25 <@Zlopez:matrix.org> After that I will continue on datagrepper checks
2025-05-15 17:57:14 <@nirik:matrix.scrye.com> so, I wonder on that...
2025-05-15 17:57:17 <@Zlopez:matrix.org> Could there be something in dist-git configuration preventing that?
2025-05-15 17:57:50 <@Zlopez:matrix.org> I should have the correct ACL on token I'm using for authentication and the user is admin of the repository
2025-05-15 17:57:52 <@nirik:matrix.scrye.com> well, regular pagure https auth does not work on src... we have it setup to use oauth tokens (see fedpkg code to get those tokens)
2025-05-15 17:58:06 <@nirik:matrix.scrye.com> so I wonder if thats interfering?
2025-05-15 17:58:42 <@Zlopez:matrix.org> I will look at the fedpkg code and see how does it working
2025-05-15 17:59:02 <@Zlopez:matrix.org> As it's hard to troubleshoot why you are rejected
2025-05-15 17:59:06 <@nirik:matrix.scrye.com> yeah, thats all I can think of that might be affecting it.
2025-05-15 17:59:42 <@nirik:matrix.scrye.com> but admin tokens work for other things... so... not sure.
2025-05-15 18:00:03 <@Zlopez:matrix.org> At least something I can try as I'm running out of ideas
2025-05-15 18:00:17 <@Zlopez:matrix.org> But only over API, not direct git operations
2025-05-15 18:00:48 <@nirik:matrix.scrye.com> ah yeah...
2025-05-15 18:01:16 <@Zlopez:matrix.org> We are at the end of our sail today. So please everybody leave the boat and see you on next sail :-)
2025-05-15 18:01:23 <@Zlopez:matrix.org> !endmeeting