<@nirik:matrix.scrye.com>
16:00:06
!startmeeting Infrastructure (2024-11-07)
<@meetbot:fedora.im>
16:00:08
Meeting started at 2024-11-07 16:00:06 UTC
<@meetbot:fedora.im>
16:00:08
The Meeting name is 'Infrastructure (2024-11-07)'
<@nirik:matrix.scrye.com>
16:00:12
!info About our team: https://docs.fedoraproject.org/en-US/cpe/
<@nirik:matrix.scrye.com>
16:00:12
!meetingname infrastructure
<@nirik:matrix.scrye.com>
16:00:12
!chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp
<@nirik:matrix.scrye.com>
16:00:12
!info Agenda is at: https://board.net/p/fedora-infra
<@nirik:matrix.scrye.com>
16:00:12
!topic ahoy
<@nirik:matrix.scrye.com>
16:00:12
!info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
<@meetbot:fedora.im>
16:00:12
The Meeting Name is now infrastructure
<@seddik:fedora.im>
16:01:35
!hi
<@zodbot:fedora.im>
16:01:36
seddik alaouiismaili (seddik)
<@nirik:matrix.scrye.com>
16:01:37
any folks around for a meeting?
<@amedvede:fedora.im>
16:01:53
yop, hello everyone
<@Zlopez:matrix.org>
16:01:54
!hi
<@zodbot:fedora.im>
16:01:59
Michal Konecny (zlopez)
<@amedvede:fedora.im>
16:02:16
!hi
<@zodbot:fedora.im>
16:02:17
Anton Medvedev (amedvede)
<@nirik:matrix.scrye.com>
16:03:16
morning everyone
<@nirik:matrix.scrye.com>
16:03:28
will wait a minute or two more for more folks to arrive
<@nphilipp:fedora.im>
16:03:57
!hi
<@zodbot:fedora.im>
16:03:58
Nils Philippsen (nphilipp) - he / him / his
<@nirik:matrix.scrye.com>
16:05:27
ok, lets go ahead then. ;)
<@nirik:matrix.scrye.com>
16:05:38
!topic New folks introductions
<@nirik:matrix.scrye.com>
16:05:38
!info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
<@nirik:matrix.scrye.com>
16:05:38
!info Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/
<@nirik:matrix.scrye.com>
16:05:45
anyone new like to introduce themselves?
<@nirik:matrix.scrye.com>
16:06:55
seems not... lets move on then. :)
<@nirik:matrix.scrye.com>
16:06:57
!info magic eight ball says:
<@nirik:matrix.scrye.com>
16:06:57
!topic Next chair
<@nirik:matrix.scrye.com>
16:06:57
!info chair 2024-11-07 - nirik
<@nirik:matrix.scrye.com>
16:06:57
!info chair 2024-11-14 - saibug
<@nirik:matrix.scrye.com>
16:06:57
!info chair 2024-11-21 - ?
<@nirik:matrix.scrye.com>
16:07:05
anyone want to handle the chair on the 21st?
<@amedvede:fedora.im>
16:07:20
i do
<@nirik:matrix.scrye.com>
16:08:34
Excellent. Thanks!
<@nirik:matrix.scrye.com>
16:08:48
!topic announcements and information
<@nirik:matrix.scrye.com>
16:08:48
!info CLE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
<@nirik:matrix.scrye.com>
16:08:48
!info CLE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
<@nirik:matrix.scrye.com>
16:08:58
Any other announcements from anyone?
<@nirik:matrix.scrye.com>
16:09:11
Not sure it's worth an announcement but:
<@nirik:matrix.scrye.com>
16:09:25
!info noc01 was reinstalled with rhel9 yesterday as was secondary01
<@nirik:matrix.scrye.com>
16:10:45
alrightly, moving along...
<@nirik:matrix.scrye.com>
16:10:54
!info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
<@nirik:matrix.scrye.com>
16:10:54
!info https://fedoraproject.org/wiki/Infrastructure/Oncall
<@nirik:matrix.scrye.com>
16:10:54
!info ? is on call from 2024-11-15 to 2024-11-21
<@nirik:matrix.scrye.com>
16:10:54
!info nirik is on call from 2024-11-08 to 2024-11-14
<@nirik:matrix.scrye.com>
16:10:54
!info zlopez is on call from 2024-11-01 to 2024-11-07
<@nirik:matrix.scrye.com>
16:10:54
!topic Oncall
<@nirik:matrix.scrye.com>
16:11:33
anyone want to take it for the following week?
<@Zlopez:matrix.org>
16:11:54
I can
<@nirik:matrix.scrye.com>
16:12:04
cool. Thanks
<@nirik:matrix.scrye.com>
16:12:21
!topic Monitoring discussion [nirik]
<@nirik:matrix.scrye.com>
16:12:21
!info Go over existing items and fix them
<@nirik:matrix.scrye.com>
16:12:21
!info https://nagios.fedoraproject.org/nagios
<@nirik:matrix.scrye.com>
16:12:53
So, we have 4 down hosts. 3 of them are aarch64 machines I am moving to openqa. Thats waiting on an internal ticket to move them to the qa vlan...
<@nirik:matrix.scrye.com>
16:13:14
the last one is logdetective02. I guess it's blocking icmp...
<@nirik:matrix.scrye.com>
16:13:22
(its an aws instance)
<@nirik:matrix.scrye.com>
16:13:31
otherwise we are nice and green.
<@Zlopez:matrix.org>
16:13:57
That is great to hear :-)
<@nirik:matrix.scrye.com>
16:14:14
I'll try and look or ask logdetective folks about that one.
<@nirik:matrix.scrye.com>
16:14:34
On the zabbix side, James Antill was going to look at upgrading our staging instance.
<@james:fedora.im>
16:14:51
Note: Upgrading it to the version in EPEL
<@nirik:matrix.scrye.com>
16:15:16
yeah. well, and newer than the one we have now...
<@nirik:matrix.scrye.com>
16:15:27
the prod one seems less noisy. ;)
<@james:fedora.im>
16:15:45
Yeh, I keep hitting the warnings that come up that we don't care about with a hammer
<@Zlopez:matrix.org>
16:16:05
Banhammer?
<@nirik:matrix.scrye.com>
16:16:13
is that in some template? or just in the database? or ?
<@james:fedora.im>
16:16:17
I turn them off, so kind of :)
<@nirik:matrix.scrye.com>
16:16:49
The only ones I see this morning have been high cpu on proxy01/10...which, yeah, they get high cpu...
<@james:fedora.im>
16:16:54
Just on the website (which I assume it saves in it's DB) ... AFAIK that's the only way.
<@nirik:matrix.scrye.com>
16:17:29
I thought there was some way to adjust the templates... but I don't know any details.
<@Zlopez:matrix.org>
16:17:41
Do you need some special permissions for zabbix.fedoraproject.org?
<@nirik:matrix.scrye.com>
16:17:43
fabian might have some pointers.
<@Zlopez:matrix.org>
16:17:48
I wasn't able to see any host there
<@nirik:matrix.scrye.com>
16:18:44
but you can login?
<@james:fedora.im>
16:18:48
You login with SSO?
<@Zlopez:matrix.org>
16:18:53
Yes
<@nirik:matrix.scrye.com>
16:19:37
I don't recall if you needed to be added to anything... would have to dig around
<@smilner:fedora.im>
16:19:48
For some reason I think I remember there being a need to "enable the account" for the moment within Zabbix, but I could be wrong on that.
<@smilner:fedora.im>
16:20:02
For some reason I think I remember there being a need to "enable the account" for the moment within Zabbix, but I could be wrong on that. IE: allow the SSO account through.
<@nirik:matrix.scrye.com>
16:20:29
huh, you are an admin... or at least have the admin 'role'
<@nirik:matrix.scrye.com>
16:20:48
anyhow we could look at it more out of band...
<@james:fedora.im>
16:21:13
I see you have "admin role" and I have "super admin role"
<@nirik:matrix.scrye.com>
16:21:15
I wonder if we shouldn't try and setup some milestones for zabbix, so we know what to work toward better?
<@nirik:matrix.scrye.com>
16:21:56
man, there's some old problems/alerts.
<@james:fedora.im>
16:22:16
Zlopez: I just changed your role to the super one ... see if that helps.
<@Zlopez:matrix.org>
16:22:27
Now I can see it
<@nirik:matrix.scrye.com>
16:22:35
anyhow, upgrading to the latest is a good step so we can poke at the most recent version
<@Zlopez:matrix.org>
16:22:45
Number of hosts: 100
<@james:fedora.im>
16:23:04
Yeh, that's what I see
<@nirik:matrix.scrye.com>
16:23:45
yeah, we need to get everything added too.
<@nirik:matrix.scrye.com>
16:24:21
ok, anything more here, or shall we move on?
<@nirik:matrix.scrye.com>
16:25:49
So, next I have a topic... would it be possible to move the meeting 1 hour later? Or is that going to be too late for folks?
<@nirik:matrix.scrye.com>
16:26:04
DST is anoying
<@james:fedora.im>
16:26:37
I would like it at a time that isn't the same as FPC ... but I'm also aware you are 3 hours behind.
<@Zlopez:matrix.org>
16:26:47
I don't have any issue with that
<@nirik:matrix.scrye.com>
16:27:01
is FPC in an hour?
<@seddik:fedora.im>
16:27:13
me too
<@james:fedora.im>
16:27:36
Also FWIW you can decide the meeting as in east/west coast time ... and then fedcal automatically moves it with DST (for you ;)
<@james:fedora.im>
16:27:52
yeh, FPC is 12 noon east coast
<@nirik:matrix.scrye.com>
16:28:37
yeah. Well, an hour later would be much better for me, but thats FPC time, so... not sure.
<@amedvede:fedora.im>
16:29:13
yeah, it can be moved imho, so the time of releng meet and this one will be exactly the same, it's better for planning a schedule
<@james:fedora.im>
16:29:18
Yeh, don't not move it just for me ... I can still attend in an hour (although a bit distracted). I can only really run it if it's 2 hours or todays time.
<@nirik:matrix.scrye.com>
16:30:21
ok, lets try moving it next week and if we come up with other ideas we can move it again.
<@nirik:matrix.scrye.com>
16:30:33
!info meeting next week will be at 17UTC
<@nirik:matrix.scrye.com>
16:30:43
So, what shall we do next:
<@nirik:matrix.scrye.com>
16:30:49
* Backlog
<@nirik:matrix.scrye.com>
16:30:57
* learning topic of some kind
<@nirik:matrix.scrye.com>
16:31:06
* open floor so nirik can go feed cats
<@nirik:matrix.scrye.com>
16:31:15
vote with emoji? ;)
<@Zlopez:matrix.org>
16:31:24
I don't remember us going through oncall summary
<@nirik:matrix.scrye.com>
16:31:34
oh, whoops. I missed it.
<@Zlopez:matrix.org>
16:31:38
But I didn't noticed any ping this week
<@nirik:matrix.scrye.com>
16:31:43
ha, ok.
<@nirik:matrix.scrye.com>
16:32:33
ok, lets open floor it.
<@nirik:matrix.scrye.com>
16:32:41
!topic open Floor
<@nirik:matrix.scrye.com>
16:32:53
anyone have any topics? or shall we get a bit of time back
<@Zlopez:matrix.org>
16:32:59
So I'm still looking at https://pagure.io/fedora-infrastructure/issue/12260
<@nirik:matrix.scrye.com>
16:33:22
any luck
<@nirik:matrix.scrye.com>
16:33:23
?
<@amedvede:fedora.im>
16:33:32
it's not fair that i don't have a cat to feed((
<@Zlopez:matrix.org>
16:33:37
And yesterday I found out that this is happening mostly during 0300-0600 UTC
<@Zlopez:matrix.org>
16:34:00
Which corresponds to releng scripts time running
<@nirik:matrix.scrye.com>
16:34:10
Anton Medvedev: get a virtual one! ;) 🐈
<@Zlopez:matrix.org>
16:34:28
After that I found this on releng-cron mailing list https://lists.fedoraproject.org/archives/list/releng-cron@lists.fedoraproject.org/thread/FNG6AIF4TKZB52XC3RIJ4XJSKS7HOT64/
<@Zlopez:matrix.org>
16:35:11
So it's possible that something in git repositories is somewhat corrupted, maybe permissions or some metadata, which is probably causing both coredumps and 503s
<@nirik:matrix.scrye.com>
16:35:31
yeah, but no.
<@nirik:matrix.scrye.com>
16:35:39
I have seen that for a while. thats staging.
<@Zlopez:matrix.org>
16:36:09
Oh, I didn't noticed that it's staging
<@nirik:matrix.scrye.com>
16:36:34
unless the script is hitting prod by mistake... but it should be firewalled off.
<@Zlopez:matrix.org>
16:37:08
Hm, I saw root@pkgs01, so I thought that is production
<@nirik:matrix.scrye.com>
16:37:24
oddly, I also don't see the prod output... is it going to /dev/null perhaps
<@Zlopez:matrix.org>
16:38:28
Maybe, but if this is not the issue I need to go back to drawing board
<@nirik:matrix.scrye.com>
16:38:36
yeah, or it's getting no output.
<@nirik:matrix.scrye.com>
16:38:44
no stderr anyhow.
<@nirik:matrix.scrye.com>
16:38:55
but the load could be causing it if it's making too many requests...
<@Zlopez:matrix.org>
16:39:07
I can try to run it manually and see what happens
<@Zlopez:matrix.org>
16:39:26
It's basically going through git repositories directly on the machine
<@Zlopez:matrix.org>
16:39:43
The load could be one of the issues
<@Zlopez:matrix.org>
16:39:58
But this doesn't explain the coredumps :-/
<@nirik:matrix.scrye.com>
16:40:18
yeah
<@nirik:matrix.scrye.com>
16:40:25
it's a puzzler.
<@nirik:matrix.scrye.com>
16:40:40
I think it's been happening much less lately... but no idea why
<@Zlopez:matrix.org>
16:40:51
It is, plenty of dead ends when trying to investigate
<@Zlopez:matrix.org>
16:41:08
The Segfault is happening every few minutes
<@Zlopez:matrix.org>
16:41:26
But it's always only one thread at a time, so it's not that visible
<@nirik:matrix.scrye.com>
16:41:42
I think (but have no proof) that the segfault is end finishing of sending a fedora-messaging message)
<@nirik:matrix.scrye.com>
16:42:14
ok, I'm gonna go feed cats... can someone else close the meeting when done? ;)
<@Zlopez:matrix.org>
16:42:15
But I didn't saw anything like that on pagure.io
<@Zlopez:matrix.org>
16:42:27
I can do it :-)
<@nirik:matrix.scrye.com>
16:42:27
yeah, true. ;(
<@Zlopez:matrix.org>
16:44:06
Anything else we want to discuss here?
<@seddik:fedora.im>
16:45:02
nothing for me
<@seddik:fedora.im>
16:45:18
working on PR for https://pagure.io/fedora-infrastructure/issue/12262
<@Zlopez:matrix.org>
16:46:50
Anything interesting happening recently?
<@Zlopez:matrix.org>
16:47:16
I'm currently found an interesting kernel bug that is from time to time freezing my GPU
<@Zlopez:matrix.org>
16:47:52
It's like playing the russian roulete, you don't know when your desktop will freeze :-D
<@smilner:fedora.im>
16:48:12
Oof! Is it already reported?
<@Zlopez:matrix.org>
16:48:12
Will it happen with this click, or the next one :-D
<@Zlopez:matrix.org>
16:48:19
I reported it
<@Zlopez:matrix.org>
16:48:38
It seems to be specific to my exact integrated GPU
<@Zlopez:matrix.org>
16:49:45
https://bugzilla.redhat.com/show_bug.cgi?id=2324340
<@Zlopez:matrix.org>
16:51:04
Originally I thought that is a GNOME issue, but it happens even after switching to KDE
<@Zlopez:matrix.org>
16:53:06
We have 8 minutes left, anything else to share?
<@seddik:fedora.im>
16:53:28
i3 spin <3
<@Zlopez:matrix.org>
16:54:25
I'm more an Atomic desktop user :-)
<@Zlopez:matrix.org>
16:54:34
Right now running on Kinoite
<@Zlopez:matrix.org>
16:55:36
I must say that KDE 6 is really nice and polished experience
<@nirik:matrix.scrye.com>
16:55:51
I'm on kde right now... aside the focus bug it's nice.
<@Zlopez:matrix.org>
16:56:56
I didn't experienced any focus bug
<@Zlopez:matrix.org>
16:57:06
Maybe I didn't noticed
<@nirik:matrix.scrye.com>
16:57:51
It may be particular to my setup... make at least 2 virtual desktops, set task switcher to show from all desktops, put a konsole (or whatever) in each one and alt tab between them.
<@james:fedora.im>
16:58:23
FWIW, I use cinnamon ... and it's also very nice, and integrates with a couple of gnome apps. I like to use a bit better
<@Zlopez:matrix.org>
16:58:58
@nirik:matrix.scrye.com Now I can see what you are talking about
<@Zlopez:matrix.org>
16:59:21
I usually switch windows by Meta+W and choosing one, so I didn't noticed that
<@Zlopez:matrix.org>
17:00:04
We are at the end of our time, so let me close the meeting
<@Zlopez:matrix.org>
17:00:09
!endmeeting