<@Zlopez:matrix.org>
16:00:06
!startmeeting Infrastructure (2024-06-06)
<@meetbot:fedora.im>
16:00:07
Meeting started at 2024-06-06 16:00:06 UTC
<@meetbot:fedora.im>
16:00:07
The Meeting name is 'Infrastructure (2024-06-06)'
<@Zlopez:matrix.org>
16:00:14
!meetingname infrastructure
<@meetbot:fedora.im>
16:00:16
The Meeting Name is now infrastructure
<@nirik:matrix.scrye.com>
16:00:21
morning
<@Zlopez:matrix.org>
16:00:24
!chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp
<@Zlopez:matrix.org>
16:00:24
!info Agenda is at: https://board.net/p/fedora-infra
<@Zlopez:matrix.org>
16:00:24
!info About our team: https://docs.fedoraproject.org/en-US/cpe/
<@Zlopez:matrix.org>
16:00:24
!info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
<@Zlopez:matrix.org>
16:00:24
!topic namaste
<@Zlopez:matrix.org>
16:00:36
!hi
<@zodbot:fedora.im>
16:00:40
Michal Konecny (zlopez)
<@smilner:fedora.im>
16:00:53
!hi
<@zodbot:fedora.im>
16:00:54
None (smilner)
<@Zlopez:matrix.org>
16:00:56
Welcome everyone to today's infra meeting
<@Zlopez:matrix.org>
16:01:15
I am your host for today
<@patrikp:matrix.org>
16:02:02
Hello.
<@Zlopez:matrix.org>
16:02:08
How is everybody this week?
<@smilner:fedora.im>
16:02:44
Pretty good! How about you?
<@nirik:matrix.scrye.com>
16:02:44
not too bad here...
<@Zlopez:matrix.org>
16:03:26
Actually I'm finally finishing some of the tasks I had open for long time, so good :-)
<@pcreech:matrix.org>
16:03:46
.hello pcreech17
<@nirik:matrix.scrye.com>
16:04:35
yeah, actually getting to do todo's is nice.
<@Zlopez:matrix.org>
16:04:53
Still adding new things, but it's getting better
<@Zlopez:matrix.org>
16:05:46
Someday I will probably get to backlog as well :-D
<@humaton:fedora.im>
16:05:49
howdy
<@Zlopez:matrix.org>
16:06:33
Let's continue to next topic to see if anybody new is here today
<@Zlopez:matrix.org>
16:06:38
!topic New folks introductions
<@Zlopez:matrix.org>
16:06:38
!info Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/
<@Zlopez:matrix.org>
16:06:38
!info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
<@Zlopez:matrix.org>
16:07:50
So anybody new in here today?
<@Zlopez:matrix.org>
16:09:53
Hm, it doesn't seem so
<@Zlopez:matrix.org>
16:10:05
Let's look at the choice of chairs then
<@Zlopez:matrix.org>
16:10:10
!info chair 2024-06-06 zlopez
<@Zlopez:matrix.org>
16:10:10
!topic Next chair
<@Zlopez:matrix.org>
16:10:10
!info magic eight ball says:
<@Zlopez:matrix.org>
16:10:10
!info chair 2024-06-13 Paul Maconi
<@Zlopez:matrix.org>
16:10:10
!info chair 2024-06-20 ???
<@Zlopez:matrix.org>
16:10:10
!info chair 2024-06-27 ???
<@Zlopez:matrix.org>
16:10:28
Hm, who is Paul Maconi?
<@Zlopez:matrix.org>
16:11:38
I can see, it's aggraxis
<@nirik:matrix.scrye.com>
16:12:16
I could possibly do the 27th. I will be not here on the 20th
<@Zlopez:matrix.org>
16:12:29
So anybody want's to have a new comfortable chair for the 20th or 27th?
<@Zlopez:matrix.org>
16:12:51
nirik once
<@Zlopez:matrix.org>
16:12:56
nirik twice
<@Zlopez:matrix.org>
16:13:01
and sold!
<@Zlopez:matrix.org>
16:13:14
nirik is a new owner of 27th chair
<@pcreech:matrix.org>
16:13:14
I can perhaps jump back in on the 20th
<@lenkaseg:matrix.org>
16:13:17
I can take 20th
<@Zlopez:matrix.org>
16:13:27
Congratulations!
<@pcreech:matrix.org>
16:13:39
You can take it lenkaseg :D
<@Zlopez:matrix.org>
16:13:57
OK, so lenkaseg is the owner of 20th chair :-)
<@Zlopez:matrix.org>
16:14:13
!info chair 2024-06-27 nirik
<@Zlopez:matrix.org>
16:14:13
!info chair 2024-06-20 lenkaseg
<@Zlopez:matrix.org>
16:14:36
pcreech: Do you want the 2024-07-04?
<@pcreech:matrix.org>
16:14:41
Sure!
<@nirik:matrix.scrye.com>
16:14:55
note that that might be a holiday in the us... ;)
<@pcreech:matrix.org>
16:15:04
actually, what nirik just said
<@pcreech:matrix.org>
16:15:08
i think this year we're going out of town for it
<@Zlopez:matrix.org>
16:15:23
Do we want to skip the meeting then?
<@nirik:matrix.scrye.com>
16:16:49
we could yeah.
<@Zlopez:matrix.org>
16:17:20
So in this case pcreech do you want the next one after that?
<@pcreech:matrix.org>
16:17:27
Yup
<@Zlopez:matrix.org>
16:17:35
!info chair 2024-07-11 pcreech
<@pcreech:matrix.org>
16:17:36
I should be much more reliable then :D
<@Zlopez:matrix.org>
16:17:45
:-D
<@Zlopez:matrix.org>
16:18:18
We have plenty of chairs occupied for following weeks, so let's go to next part of today meeting
<@Zlopez:matrix.org>
16:18:24
!topic announcements and information
<@Zlopez:matrix.org>
16:18:24
!info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
<@Zlopez:matrix.org>
16:18:24
!info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
<@Zlopez:matrix.org>
16:18:24
!info Fedora infra meeting is cancelled for 2024-07-04 due to US Holidays
<@Zlopez:matrix.org>
16:18:37
Anything else to announce?
<@nirik:matrix.scrye.com>
16:18:49
!info nirik is out all next week.
<@nirik:matrix.scrye.com>
16:19:08
I'll probibly poke in a few times, but don't plan to be around too much. ;)
<@patrikp:matrix.org>
16:19:31
DevConf is next week too.
<@Zlopez:matrix.org>
16:20:41
!info Next week 13th-16th June 2024 is DevConf.CZ plenty of folks will not be available
<@Zlopez:matrix.org>
16:21:45
I wonder if we have anything else to announce
<@Zlopez:matrix.org>
16:23:29
IT seems that we don't, so next on our plate is oncall
<@Zlopez:matrix.org>
16:23:40
!info https://fedoraproject.org/wiki/Infrastructure/Oncall
<@Zlopez:matrix.org>
16:23:40
!topic Oncall
<@Zlopez:matrix.org>
16:23:40
!info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
<@Zlopez:matrix.org>
16:23:47
!info lenkaseg is on call from 2024-05-31 to 2024-06-06
<@Zlopez:matrix.org>
16:23:47
!info ??? is on call from 2024-06-07 to 2024-06-13
<@Zlopez:matrix.org>
16:23:47
!info zlopez is on call from 2024-06-14 to 2024-06-20
<@Zlopez:matrix.org>
16:23:47
!info ??? is on call from 2024-06-21 to 2024-06-27
<@Zlopez:matrix.org>
16:24:07
So we are missing the oncall for next week
<@humaton:fedora.im>
16:24:24
I will have laptop on me during devconf
<@Zlopez:matrix.org>
16:24:24
Who is not going to DevConf.CZ?
<@humaton:fedora.im>
16:24:32
but if there is anybody else...
<@nirik:matrix.scrye.com>
16:24:39
I suppose we could just unset it? ask people to file tickets?
<@Zlopez:matrix.org>
16:24:49
That is option too
<@humaton:fedora.im>
16:25:06
we can have on call on booth at devconf :D
<@Zlopez:matrix.org>
16:25:38
We should have a telephone there as well, so it's a proper on call :-D
<@nirik:matrix.scrye.com>
16:25:39
š
<@humaton:fedora.im>
16:25:59
hm I have recently up[cycled bunch of old cisco phones
<@humaton:fedora.im>
16:26:17
dont make me bring one
<@Zlopez:matrix.org>
16:26:40
I'm not sure if anybody would actually call on it :-D
<@nirik:matrix.scrye.com>
16:26:40
I think it would be best to just not set anyone
<@Zlopez:matrix.org>
16:26:51
!info nobody is on call from 2024-06-07 to 2024-06-13
<@nirik:matrix.scrye.com>
16:27:07
but if folks could check tickets from time to time...
<@humaton:fedora.im>
16:27:10
well it can dial into meetings
<@humaton:fedora.im>
16:27:26
I can do that at the conference
<@humaton:fedora.im>
16:27:37
check the tickets that is
<@Zlopez:matrix.org>
16:27:55
We still have 2024-06-21 to 2024-06-27 free if anybody wants it
<@humaton:fedora.im>
16:28:34
its summer solstice celebrations for me at that week so I will be offline
<@nirik:matrix.scrye.com>
16:29:03
I'm out on the 20th getting a molar pulled, so not sure I will be up for much on the 21st...
<@nirik:matrix.scrye.com>
16:29:23
but I guess I could take it for the later part.
<@lenkaseg:matrix.org>
16:30:18
I might be on PTO towards 27th... but not sure yet... so if nobody takes it I guess i could
<@Zlopez:matrix.org>
16:30:37
Let's keep it free for now, no need to occupy it today
<@Zlopez:matrix.org>
16:31:38
So let's go to next topic
<@Zlopez:matrix.org>
16:31:45
!info Summary of last week: (from current oncall)
<@lenkaseg:matrix.org>
16:32:02
That was me
<@lenkaseg:matrix.org>
16:32:29
There was one call on Friday I think
<@lenkaseg:matrix.org>
16:32:59
Because of anitya down
<@lenkaseg:matrix.org>
16:33:14
I pointed the person to the (wrong) repo
<@lenkaseg:matrix.org>
16:33:43
Should be infrastructure, not anitya repo
<@Zlopez:matrix.org>
16:33:44
I think that was caused by not enough resources for the pod
<@lenkaseg:matrix.org>
16:33:51
But it got solved
<@lenkaseg:matrix.org>
16:34:18
That's all! No pings the rest of the week
<@nirik:matrix.scrye.com>
16:36:13
yeah, mirrormanager crawler was sucking up all the memory on a compute node and causing other things on it to get oom killed.
<@nirik:matrix.scrye.com>
16:36:34
we should set some limits on all our openshift apps someday. ;)
<@aggraxis:fedora.im>
16:36:57
64kb go. lol
<@nirik:matrix.scrye.com>
16:37:14
640k should be enough for any app!
<@Zlopez:matrix.org>
16:38:32
Who needs more?
<@Zlopez:matrix.org>
16:39:45
Let's continue with next topic
<@Zlopez:matrix.org>
16:39:53
!info Go over existing out items and fix
<@Zlopez:matrix.org>
16:39:53
!topic Monitoring discussion [nirik]
<@Zlopez:matrix.org>
16:39:53
!info https://nagios.fedoraproject.org/nagios
<@Zlopez:matrix.org>
16:40:09
I noticed that proxy14 is still misbehaves
<@nirik:matrix.scrye.com>
16:40:49
so, yeah... I am not sure whats going on with it now.
<@Zlopez:matrix.org>
16:40:58
And there was some issue with toddlers this week, which should be solved
<@nirik:matrix.scrye.com>
16:41:00
it's hard to debug since it happens in the middle of the night for me. ;)
<@nirik:matrix.scrye.com>
16:42:05
we have 2 down machines: vmhost-x86-copr01 (there's a ticket on it and David Kirwan is working on getting it fixed), and buildhw-x86-16... I guess I should file a ticket on it. It won't power on.
<@Zlopez:matrix.org>
16:42:06
And it happens only for few minutes and then it's back on
<@dkirwan:fedora.im>
16:42:59
Waiting on response from teh RH techs, need to "reseat an OCP card" before Dell tech support will continue on with troubleshoot
<@nirik:matrix.scrye.com>
16:43:20
I think actually we dropped warnetee on those, so I guess buildhw-x86-16 is just EOL. ;)
<@nirik:matrix.scrye.com>
16:44:13
other than the 2 down machines, everything is green.
<@Zlopez:matrix.org>
16:44:32
David Kirwan: How does it look with the zabbix monitoring
<@Zlopez:matrix.org>
16:44:35
?
<@dkirwan:fedora.im>
16:45:12
umm so at a natural point where I need more feedback, ive addressed all the last feedback nirik gave, need to get some eyes on it to see if this is ok
<@nirik:matrix.scrye.com>
16:45:33
hum. I can't seem to login to it... at least with SSO...
<@dkirwan:fedora.im>
16:46:03
ehh ok let me check and get back to you on that one.. after reinstall, might have misconfigured that
<@nirik:matrix.scrye.com>
16:46:04
guest works...
<@nirik:matrix.scrye.com>
16:46:34
no worries. I can try and do another round of looking at it after that? we should probibly start looking at it more. ;)
<@Zlopez:matrix.org>
16:47:29
https://zabbix.fedoraproject.org/
<@dkirwan:fedora.im>
16:48:04
nirik: ok lol i put your username as "kevin" š¤¦āāļø
<@nirik:matrix.scrye.com>
16:48:15
perhaps we could look at dumping it's alerts into a seperate channel... would let us see what it's doing more visibily...
<@Zlopez:matrix.org>
16:48:28
I'm able to log in without issue
<@nirik:matrix.scrye.com>
16:48:38
well, my fas account is kevin...
<@Zlopez:matrix.org>
16:48:41
Hm, it doesn't report any issue
<@leo:fedora.im>
16:48:42
hey
<@aggraxis:fedora.im>
16:48:47
heh oh yeah it hates my guts, but I'm probably not supposed to be in it anyways heh. Nice that it touched back to the FAS login though
<@leo:fedora.im>
16:48:48
iām able to log in too
<@nirik:matrix.scrye.com>
16:49:16
I get a dialog with: "Session initialization error."
<@nirik:matrix.scrye.com>
16:49:23
but we can debug outside the meeting.
<@Zlopez:matrix.org>
16:50:27
I think we can just switch to Open floor and continue debugging this
<@Zlopez:matrix.org>
16:50:36
We have 10 minutes left anyway
<@nirik:matrix.scrye.com>
16:51:01
I had a quick thing or two...
<@leo:fedora.im>
16:51:23
hm, it used to work for me but doesn't work anymore
<@Zlopez:matrix.org>
16:51:23
Should I switch to open floor?
<@nirik:matrix.scrye.com>
16:51:24
first, I'm gonna try and work on sundries migration later today...
<@leo:fedora.im>
16:51:34
ah wait never mind i can login
<@Zlopez:matrix.org>
16:51:51
!topic Open Floor
<@nirik:matrix.scrye.com>
16:52:09
second, I'm gonna ask ryan if he is willing to do the wiki upgrade next week... we can't do it now due to test days. ;(
<@nirik:matrix.scrye.com>
16:52:42
thats it... just fyi...
<@Zlopez:matrix.org>
16:54:07
I will continue working on the mailman, created a PR for postfix, hopefully it's correct
<@Zlopez:matrix.org>
16:54:15
https://pagure.io/fedora-infra/ansible/pull-request/2055#
<@Zlopez:matrix.org>
16:54:36
I updated the sundries ticket as well, so you can see where I ended
<@nirik:matrix.scrye.com>
16:55:07
yeah, can review later
<@Zlopez:matrix.org>
16:55:44
David Kirwan: I can see 60 problems mentioned in the dashboard, but according to severity there are 0
<@Zlopez:matrix.org>
16:55:56
On zabbix
<@dkirwan:fedora.im>
16:56:16
hmm Might need to look at the default permissions given to users which can log in..
<@dkirwan:fedora.im>
16:56:23
could be that you dont have enough permissions to see.
<@Zlopez:matrix.org>
16:56:56
I can still see this `Number of triggers (enabled/disabled [problem/ok]) 3697 3697 / 0 [60 / 3637]`
<@leo:fedora.im>
16:57:15
yeah, i see that too but if i actually look at hosts i see nothing.
<@nirik:matrix.scrye.com>
16:57:21
Ideally it would be like nagios... guest / anyone can see all, but only sysadmin-noc or whoever can ack/change anything
<@leo:fedora.im>
16:57:39
yeah, i see that too but if i actually look at hosts i see nothing. (am in -noc)
<@dkirwan:fedora.im>
16:58:30
doesn't seem to be that way at least in the default permission roles, so will need to look at that
<@dkirwan:fedora.im>
16:59:52
actually one thing you might take a look at nirik can you try login to console.openshift.com using the fedora account and update the updates channel that communishift is running on ?
<@nirik:matrix.scrye.com>
17:00:23
sure... can look.
<@dkirwan:fedora.im>
17:00:32
for some reason im unable to at the moment, i ahve a ticket open to try get that sorted
<@dkirwan:fedora.im>
17:00:41
the version its running will soon be unsupported
<@nirik:matrix.scrye.com>
17:00:42
I think since thats a managed one we have to 'ack' major updates?
<@Zlopez:matrix.org>
17:00:46
We are at the end of our time
<@nirik:matrix.scrye.com>
17:00:54
but will go look.
<@Zlopez:matrix.org>
17:01:02
Feel free to continue with the discussion outside the meeting :-)
<@dkirwan:fedora.im>
17:01:07
yeah, but ack in that console.openshfit it seems ;D
<@Zlopez:matrix.org>
17:01:08
!endmeeting