2024-11-07 16:00:06 <@nirik:matrix.scrye.com> !startmeeting Infrastructure (2024-11-07)
2024-11-07 16:00:08 <@meetbot:fedora.im> Meeting started at 2024-11-07 16:00:06 UTC
2024-11-07 16:00:08 <@meetbot:fedora.im> The Meeting name is 'Infrastructure (2024-11-07)'
2024-11-07 16:00:12 <@nirik:matrix.scrye.com> !info About our team: https://docs.fedoraproject.org/en-US/cpe/
2024-11-07 16:00:12 <@nirik:matrix.scrye.com> !meetingname infrastructure
2024-11-07 16:00:12 <@nirik:matrix.scrye.com> !chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp
2024-11-07 16:00:12 <@nirik:matrix.scrye.com> !info Agenda is at: https://board.net/p/fedora-infra
2024-11-07 16:00:12 <@nirik:matrix.scrye.com> !topic ahoy
2024-11-07 16:00:12 <@nirik:matrix.scrye.com> !info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
2024-11-07 16:00:12 <@meetbot:fedora.im> The Meeting Name is now infrastructure
2024-11-07 16:01:35 <@seddik:fedora.im> !hi 
2024-11-07 16:01:36 <@zodbot:fedora.im> seddik alaouiismaili (seddik)
2024-11-07 16:01:37 <@nirik:matrix.scrye.com> any folks around for a meeting?
2024-11-07 16:01:53 <@amedvede:fedora.im> yop, hello everyone
2024-11-07 16:01:54 <@Zlopez:matrix.org> !hi
2024-11-07 16:01:59 <@zodbot:fedora.im> Michal Konecny (zlopez)
2024-11-07 16:02:16 <@amedvede:fedora.im> !hi
2024-11-07 16:02:17 <@zodbot:fedora.im> Anton Medvedev (amedvede)
2024-11-07 16:03:16 <@nirik:matrix.scrye.com> morning everyone
2024-11-07 16:03:28 <@nirik:matrix.scrye.com> will wait a minute or two more for more folks to arrive
2024-11-07 16:03:57 <@nphilipp:fedora.im> !hi
2024-11-07 16:03:58 <@zodbot:fedora.im> Nils Philippsen (nphilipp) - he / him / his
2024-11-07 16:05:27 <@nirik:matrix.scrye.com> ok, lets go ahead then. ;)
2024-11-07 16:05:38 <@nirik:matrix.scrye.com> !topic New folks introductions
2024-11-07 16:05:38 <@nirik:matrix.scrye.com> !info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
2024-11-07 16:05:38 <@nirik:matrix.scrye.com> !info Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/
2024-11-07 16:05:45 <@nirik:matrix.scrye.com> anyone new like to introduce themselves?
2024-11-07 16:06:55 <@nirik:matrix.scrye.com> seems not... lets move on then. :)
2024-11-07 16:06:57 <@nirik:matrix.scrye.com> !info magic eight ball says:
2024-11-07 16:06:57 <@nirik:matrix.scrye.com> !topic Next chair
2024-11-07 16:06:57 <@nirik:matrix.scrye.com> !info chair 2024-11-07 - nirik
2024-11-07 16:06:57 <@nirik:matrix.scrye.com> !info chair 2024-11-14 - saibug
2024-11-07 16:06:57 <@nirik:matrix.scrye.com> !info chair 2024-11-21 - ?
2024-11-07 16:07:05 <@nirik:matrix.scrye.com> anyone want to handle the chair on the 21st?
2024-11-07 16:07:20 <@amedvede:fedora.im> i do
2024-11-07 16:08:34 <@nirik:matrix.scrye.com> Excellent. Thanks!
2024-11-07 16:08:48 <@nirik:matrix.scrye.com> !topic announcements and information
2024-11-07 16:08:48 <@nirik:matrix.scrye.com> !info CLE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
2024-11-07 16:08:48 <@nirik:matrix.scrye.com> !info CLE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
2024-11-07 16:08:58 <@nirik:matrix.scrye.com> Any other announcements from anyone?
2024-11-07 16:09:11 <@nirik:matrix.scrye.com> Not sure it's worth an announcement but:
2024-11-07 16:09:25 <@nirik:matrix.scrye.com> !info noc01 was reinstalled with rhel9 yesterday as was secondary01
2024-11-07 16:10:45 <@nirik:matrix.scrye.com> alrightly, moving along...
2024-11-07 16:10:54 <@nirik:matrix.scrye.com> !info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
2024-11-07 16:10:54 <@nirik:matrix.scrye.com> !info https://fedoraproject.org/wiki/Infrastructure/Oncall
2024-11-07 16:10:54 <@nirik:matrix.scrye.com> !info ? is on call from 2024-11-15 to 2024-11-21
2024-11-07 16:10:54 <@nirik:matrix.scrye.com> !info nirik is on call from 2024-11-08 to 2024-11-14
2024-11-07 16:10:54 <@nirik:matrix.scrye.com> !info zlopez is on call from 2024-11-01 to 2024-11-07
2024-11-07 16:10:54 <@nirik:matrix.scrye.com> !topic Oncall
2024-11-07 16:11:33 <@nirik:matrix.scrye.com> anyone want to take it for the following week?
2024-11-07 16:11:54 <@Zlopez:matrix.org> I can
2024-11-07 16:12:04 <@nirik:matrix.scrye.com> cool. Thanks
2024-11-07 16:12:21 <@nirik:matrix.scrye.com> !topic Monitoring discussion [nirik]
2024-11-07 16:12:21 <@nirik:matrix.scrye.com> !info Go over existing items and fix them
2024-11-07 16:12:21 <@nirik:matrix.scrye.com> !info https://nagios.fedoraproject.org/nagios 
2024-11-07 16:12:53 <@nirik:matrix.scrye.com> So, we have 4 down hosts. 3 of them are aarch64 machines I am moving to openqa. Thats waiting on an internal ticket to move them to the qa vlan...
2024-11-07 16:13:14 <@nirik:matrix.scrye.com> the last one is logdetective02. I guess it's blocking icmp...
2024-11-07 16:13:22 <@nirik:matrix.scrye.com> (its an aws instance)
2024-11-07 16:13:31 <@nirik:matrix.scrye.com> otherwise we are nice and green.
2024-11-07 16:13:57 <@Zlopez:matrix.org> That is great to hear :-)
2024-11-07 16:14:14 <@nirik:matrix.scrye.com> I'll try and look or ask logdetective folks about that one.
2024-11-07 16:14:34 <@nirik:matrix.scrye.com> On the zabbix side, James Antill was going to look at upgrading our staging instance.
2024-11-07 16:14:51 <@james:fedora.im> Note: Upgrading it to the version in EPEL
2024-11-07 16:15:16 <@nirik:matrix.scrye.com> yeah. well, and newer than the one we have now...
2024-11-07 16:15:27 <@nirik:matrix.scrye.com> the prod one seems less noisy. ;)
2024-11-07 16:15:45 <@james:fedora.im> Yeh, I keep hitting the warnings that come up that we don't care about with a hammer
2024-11-07 16:16:05 <@Zlopez:matrix.org> Banhammer?
2024-11-07 16:16:13 <@nirik:matrix.scrye.com> is that in some template? or just in the database? or ?
2024-11-07 16:16:17 <@james:fedora.im> I turn them off, so kind of :)
2024-11-07 16:16:49 <@nirik:matrix.scrye.com> The only ones I see this morning have been high cpu on proxy01/10...which, yeah, they get high cpu...
2024-11-07 16:16:54 <@james:fedora.im> Just on the website (which I assume it saves in it's DB) ... AFAIK that's the only way.
2024-11-07 16:17:29 <@nirik:matrix.scrye.com> I thought there was some way to adjust the templates... but I don't know any details.
2024-11-07 16:17:41 <@Zlopez:matrix.org> Do you need some special permissions for zabbix.fedoraproject.org?
2024-11-07 16:17:43 <@nirik:matrix.scrye.com> fabian might have some pointers.
2024-11-07 16:17:48 <@Zlopez:matrix.org> I wasn't able to see any host there
2024-11-07 16:18:44 <@nirik:matrix.scrye.com> but you can login?
2024-11-07 16:18:48 <@james:fedora.im> You login with SSO?
2024-11-07 16:18:53 <@Zlopez:matrix.org> Yes
2024-11-07 16:19:37 <@nirik:matrix.scrye.com> I don't recall if you needed to be added to anything... would have to dig around
2024-11-07 16:19:48 <@smilner:fedora.im> For some reason I think I remember there being a need to "enable the account" for the moment within Zabbix, but I could be wrong on that.
2024-11-07 16:20:02 <@smilner:fedora.im> For some reason I think I remember there being a need to "enable the account" for the moment within Zabbix, but I could be wrong on that. IE: allow the SSO account through.
2024-11-07 16:20:29 <@nirik:matrix.scrye.com> huh, you are an admin... or at least have the admin 'role'
2024-11-07 16:20:48 <@nirik:matrix.scrye.com> anyhow we could look at it more out of band...
2024-11-07 16:21:13 <@james:fedora.im> I see you have "admin role" and I have "super admin role"
2024-11-07 16:21:15 <@nirik:matrix.scrye.com> I wonder if we shouldn't try and setup some milestones for zabbix, so we know what to work toward better?
2024-11-07 16:21:56 <@nirik:matrix.scrye.com> man, there's some old problems/alerts.
2024-11-07 16:22:16 <@james:fedora.im> Zlopez: I just changed your role to the super one ... see if that helps.
2024-11-07 16:22:27 <@Zlopez:matrix.org> Now I can see it
2024-11-07 16:22:35 <@nirik:matrix.scrye.com> anyhow, upgrading to the latest is a good step so we can poke at the most recent version
2024-11-07 16:22:45 <@Zlopez:matrix.org> Number of hosts: 100
2024-11-07 16:23:04 <@james:fedora.im> Yeh, that's what I see
2024-11-07 16:23:45 <@nirik:matrix.scrye.com> yeah, we need to get everything added too.
2024-11-07 16:24:21 <@nirik:matrix.scrye.com> ok, anything more here, or shall we move on?
2024-11-07 16:25:49 <@nirik:matrix.scrye.com> So, next I have a topic... would it be possible to move the meeting 1 hour later? Or is that going to be too late for folks?
2024-11-07 16:26:04 <@nirik:matrix.scrye.com> DST is anoying
2024-11-07 16:26:37 <@james:fedora.im> I would like it at a time that isn't the same as FPC ... but I'm also aware you are 3 hours behind.
2024-11-07 16:26:47 <@Zlopez:matrix.org> I don't have any issue with that
2024-11-07 16:27:01 <@nirik:matrix.scrye.com> is FPC in an hour?
2024-11-07 16:27:13 <@seddik:fedora.im> me too
2024-11-07 16:27:36 <@james:fedora.im> Also FWIW you can decide the meeting as in east/west coast time ... and then fedcal automatically moves it with DST (for you ;)
2024-11-07 16:27:52 <@james:fedora.im> yeh, FPC is 12 noon east coast
2024-11-07 16:28:37 <@nirik:matrix.scrye.com> yeah. Well, an hour later would be much better for me, but thats FPC time, so... not sure.
2024-11-07 16:29:13 <@amedvede:fedora.im> yeah, it can be moved imho, so the time of releng meet and this one will be exactly the same, it's better for planning a schedule
2024-11-07 16:29:18 <@james:fedora.im> Yeh, don't not move it just for me ... I can still attend in an hour (although a bit distracted). I can only really run it if it's 2 hours or todays time.
2024-11-07 16:30:21 <@nirik:matrix.scrye.com> ok, lets try moving it next week and if we come up with other ideas we can move it again.
2024-11-07 16:30:33 <@nirik:matrix.scrye.com> !info meeting next week will be at 17UTC
2024-11-07 16:30:43 <@nirik:matrix.scrye.com> So, what shall we do next:
2024-11-07 16:30:49 <@nirik:matrix.scrye.com> * Backlog
2024-11-07 16:30:57 <@nirik:matrix.scrye.com> * learning topic of some kind
2024-11-07 16:31:06 <@nirik:matrix.scrye.com> * open floor so nirik can go feed cats
2024-11-07 16:31:15 <@nirik:matrix.scrye.com> vote with emoji? ;)
2024-11-07 16:31:24 <@Zlopez:matrix.org> I don't remember us going through oncall summary
2024-11-07 16:31:34 <@nirik:matrix.scrye.com> oh, whoops. I missed it.
2024-11-07 16:31:38 <@Zlopez:matrix.org> But I didn't noticed any ping this week
2024-11-07 16:31:43 <@nirik:matrix.scrye.com> ha, ok.
2024-11-07 16:32:33 <@nirik:matrix.scrye.com> ok, lets open floor it.
2024-11-07 16:32:41 <@nirik:matrix.scrye.com> !topic open Floor
2024-11-07 16:32:53 <@nirik:matrix.scrye.com> anyone have any topics? or shall we get a bit of time back
2024-11-07 16:32:59 <@Zlopez:matrix.org> So I'm still looking at https://pagure.io/fedora-infrastructure/issue/12260
2024-11-07 16:33:22 <@nirik:matrix.scrye.com> any luck
2024-11-07 16:33:23 <@nirik:matrix.scrye.com> ?
2024-11-07 16:33:32 <@amedvede:fedora.im> it's not fair that i don't have a cat to feed((
2024-11-07 16:33:37 <@Zlopez:matrix.org> And yesterday I found out that this is happening mostly during 0300-0600 UTC
2024-11-07 16:34:00 <@Zlopez:matrix.org> Which corresponds to releng scripts time running
2024-11-07 16:34:10 <@nirik:matrix.scrye.com> Anton Medvedev: get a virtual one! ;) 🐈
2024-11-07 16:34:28 <@Zlopez:matrix.org> After that I found this on releng-cron mailing list https://lists.fedoraproject.org/archives/list/releng-cron@lists.fedoraproject.org/thread/FNG6AIF4TKZB52XC3RIJ4XJSKS7HOT64/
2024-11-07 16:35:11 <@Zlopez:matrix.org> So it's possible that something in git repositories is somewhat corrupted, maybe permissions or some metadata, which is probably causing both coredumps and 503s
2024-11-07 16:35:31 <@nirik:matrix.scrye.com> yeah, but no.
2024-11-07 16:35:39 <@nirik:matrix.scrye.com> I have seen that for a while. thats staging.
2024-11-07 16:36:09 <@Zlopez:matrix.org> Oh, I didn't noticed that it's staging
2024-11-07 16:36:34 <@nirik:matrix.scrye.com> unless the script is hitting prod by mistake... but it should be firewalled off.
2024-11-07 16:37:08 <@Zlopez:matrix.org> Hm, I saw root@pkgs01, so I thought that is production
2024-11-07 16:37:24 <@nirik:matrix.scrye.com> oddly, I also don't see the prod output... is it going to /dev/null perhaps
2024-11-07 16:38:28 <@Zlopez:matrix.org> Maybe, but if this is not the issue I need to go back to drawing board
2024-11-07 16:38:36 <@nirik:matrix.scrye.com> yeah, or it's getting no output.
2024-11-07 16:38:44 <@nirik:matrix.scrye.com> no stderr anyhow.
2024-11-07 16:38:55 <@nirik:matrix.scrye.com> but the load could be causing it if it's making too many requests...
2024-11-07 16:39:07 <@Zlopez:matrix.org> I can try to run it manually and see what happens
2024-11-07 16:39:26 <@Zlopez:matrix.org> It's basically going through git repositories directly on the machine
2024-11-07 16:39:43 <@Zlopez:matrix.org> The load could be one of the issues
2024-11-07 16:39:58 <@Zlopez:matrix.org> But this doesn't explain the coredumps :-/
2024-11-07 16:40:18 <@nirik:matrix.scrye.com> yeah
2024-11-07 16:40:25 <@nirik:matrix.scrye.com> it's a puzzler.
2024-11-07 16:40:40 <@nirik:matrix.scrye.com> I think it's been happening much less lately... but no idea why
2024-11-07 16:40:51 <@Zlopez:matrix.org> It is, plenty of dead ends when trying to investigate
2024-11-07 16:41:08 <@Zlopez:matrix.org> The Segfault is happening every few minutes
2024-11-07 16:41:26 <@Zlopez:matrix.org> But it's always only one thread at a time, so it's not that visible
2024-11-07 16:41:42 <@nirik:matrix.scrye.com> I think (but have no proof) that the segfault is end finishing of sending a fedora-messaging message)
2024-11-07 16:42:14 <@nirik:matrix.scrye.com> ok, I'm gonna go feed cats... can someone else close the meeting when done? ;)
2024-11-07 16:42:15 <@Zlopez:matrix.org> But I didn't saw anything like that on pagure.io
2024-11-07 16:42:27 <@Zlopez:matrix.org> I can do it :-)
2024-11-07 16:42:27 <@nirik:matrix.scrye.com> yeah, true. ;(
2024-11-07 16:44:06 <@Zlopez:matrix.org> Anything else we want to discuss here?
2024-11-07 16:45:02 <@seddik:fedora.im> nothing for me 
2024-11-07 16:45:18 <@seddik:fedora.im> working on PR for https://pagure.io/fedora-infrastructure/issue/12262
2024-11-07 16:46:50 <@Zlopez:matrix.org> Anything interesting happening recently?
2024-11-07 16:47:16 <@Zlopez:matrix.org> I'm currently found an interesting kernel bug that is from time to time freezing my GPU
2024-11-07 16:47:52 <@Zlopez:matrix.org> It's like playing the russian roulete, you don't know when your desktop will freeze :-D
2024-11-07 16:48:12 <@smilner:fedora.im> Oof! Is it already reported?
2024-11-07 16:48:12 <@Zlopez:matrix.org> Will it happen with this click, or the next one :-D
2024-11-07 16:48:19 <@Zlopez:matrix.org> I reported it
2024-11-07 16:48:38 <@Zlopez:matrix.org> It seems to be specific to my exact integrated GPU
2024-11-07 16:49:45 <@Zlopez:matrix.org> https://bugzilla.redhat.com/show_bug.cgi?id=2324340
2024-11-07 16:51:04 <@Zlopez:matrix.org> Originally I thought that is a GNOME issue, but it happens even after switching to KDE
2024-11-07 16:53:06 <@Zlopez:matrix.org> We have 8 minutes left, anything else to share?
2024-11-07 16:53:28 <@seddik:fedora.im> i3 spin <3
2024-11-07 16:54:25 <@Zlopez:matrix.org> I'm more an Atomic desktop user :-)
2024-11-07 16:54:34 <@Zlopez:matrix.org> Right now running on Kinoite
2024-11-07 16:55:36 <@Zlopez:matrix.org> I must say that KDE 6 is really nice and polished experience
2024-11-07 16:55:51 <@nirik:matrix.scrye.com> I'm on kde right now... aside the focus bug it's nice.
2024-11-07 16:56:56 <@Zlopez:matrix.org> I didn't experienced any focus bug
2024-11-07 16:57:06 <@Zlopez:matrix.org> Maybe I didn't noticed
2024-11-07 16:57:51 <@nirik:matrix.scrye.com> It may be particular to my setup... make at least 2 virtual desktops, set task switcher to show from all desktops, put a konsole (or whatever) in each one and alt tab between them.
2024-11-07 16:58:23 <@james:fedora.im> FWIW, I use cinnamon ... and it's also very nice, and integrates with a couple of gnome apps. I like to use a bit better
2024-11-07 16:58:58 <@Zlopez:matrix.org> @nirik:matrix.scrye.com Now I can see what you are talking about
2024-11-07 16:59:21 <@Zlopez:matrix.org> I usually switch windows by Meta+W and choosing one, so I didn't noticed that
2024-11-07 17:00:04 <@Zlopez:matrix.org> We are at the end of our time, so let me close the meeting
2024-11-07 17:00:09 <@Zlopez:matrix.org> !endmeeting