<@nirik:matrix.scrye.com>
16:00:22
!startmeeting Infrastructure (2024-03-07)
<@meetbot:fedora.im>
16:00:24
Meeting started at 2024-03-07 16:00:22 UTC
<@meetbot:fedora.im>
16:00:24
The Meeting name is 'Infrastructure (2024-03-07)'
<@nirik:matrix.scrye.com>
16:00:29
!meetingname infrastructure
<@nirik:matrix.scrye.com>
16:00:35
!info Agenda is at: https://board.net/p/fedora-infra
<@nirik:matrix.scrye.com>
16:00:42
About our team: https://docs.fedoraproject.org/en-US/cpe/
<@nirik:matrix.scrye.com>
16:00:46
Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
<@nirik:matrix.scrye.com>
16:00:53
!topic namaste
<@leo:fedora.im>
16:00:53
!hi
<@zodbot:fedora.im>
16:00:55
Leo Puvilland (leo) - he / him / his
<@seddik:fedora.im>
16:01:01
!hi
<@dtometzki:fedora.im>
16:01:02
hi
<@zodbot:fedora.im>
16:01:02
seddik alaouiismaili (seddik)
<@nirik:matrix.scrye.com>
16:01:04
Morning everyone
<@nirik:matrix.scrye.com>
16:01:58
!topic New folks introductions
<@nirik:matrix.scrye.com>
16:02:03
!info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
<@nirik:matrix.scrye.com>
16:02:08
Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/
<@nirik:matrix.scrye.com>
16:02:13
Any new folks today?
<@Zlopez:matrix.org>
16:03:04
!hi
<@zodbot:fedora.im>
16:03:06
Michal Konecny (zlopez)
<@Zlopez:matrix.org>
16:03:21
How is everyone today?
<@leo:fedora.im>
16:03:35
good how are you
<@jsteffan:fedora.im>
16:03:39
!hi
<@zodbot:fedora.im>
16:03:40
Jonathan Steffan (jsteffan)
<@dtometzki:fedora.im>
16:03:42
fine thanks and you ?
<@nirik:matrix.scrye.com>
16:03:52
To early to tell for me. ;)
<@nirik:matrix.scrye.com>
16:04:19
I guess lets move on to...
<@nirik:matrix.scrye.com>
16:04:21
!topic Next chair
<@nirik:matrix.scrye.com>
16:04:27
!info magic eight ball says:
<@nirik:matrix.scrye.com>
16:04:32
!info chair 2024-03-14 dtometzki
<@nirik:matrix.scrye.com>
16:04:36
!info chair 2024-03-21 ???
<@nirik:matrix.scrye.com>
16:04:41
!info chair 2024-03-28 ???
<@nirik:matrix.scrye.com>
16:04:50
Anyone want either of those ?
<@dtometzki:fedora.im>
16:04:52
can i exchange my chair with anyone ?
<@nirik:matrix.scrye.com>
16:05:03
sure!
<@dtometzki:fedora.im>
16:05:05
i am not available next week
<@Zlopez:matrix.org>
16:05:11
leo, dtometzki: Great, thanks for asking :-)
<@Zlopez:matrix.org>
16:05:25
I can take it
<@nirik:matrix.scrye.com>
16:05:26
dtometzki: want to move to one of the later ones? or just change next week for now?
<@nirik:matrix.scrye.com>
16:05:36
thanks Zlopez
<@dtometzki:fedora.im>
16:05:48
move one or two weeks later
<@seddik:fedora.im>
16:05:51
doing good, what about you??
<@nirik:matrix.scrye.com>
16:06:32
So, 2024-03-14 zlopez and 2024-03-21 dtometzki ?
<@Zlopez:matrix.org>
16:06:38
Yes
<@dtometzki:fedora.im>
16:06:46
perfekt
<@nirik:matrix.scrye.com>
16:06:52
thanks!
<@nirik:matrix.scrye.com>
16:06:59
!topic announcements and information
<@nirik:matrix.scrye.com>
16:07:05
!info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0830 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
<@nirik:matrix.scrye.com>
16:07:09
!info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
<@nirik:matrix.scrye.com>
16:07:15
!info we are in F40 Beta freeze
<@Zlopez:matrix.org>
16:07:28
!info Communishift is now publicly available https://communityblog.fedoraproject.org/communishift-now-available/
<@nirik:matrix.scrye.com>
16:07:40
!info nirik is out on pto next monday (2024-03-11)
<@nirik:matrix.scrye.com>
16:08:08
!info F40 beta is 'no go' for the early target date next week. ;(
<@Zlopez:matrix.org>
16:08:18
!info Outreachy started this week, you can see plenty of new contributors around
<@nirik:matrix.scrye.com>
16:08:44
yeah, lots of activity!
<@leo:fedora.im>
16:09:05
hooray!
<@nirik:matrix.scrye.com>
16:09:20
Any other announcements or information?
<@Zlopez:matrix.org>
16:09:32
I'm not able to even respond to everything :-D
<@nirik:matrix.scrye.com>
16:10:46
!topic Oncall
<@nirik:matrix.scrye.com>
16:10:52
!info https://fedoraproject.org/wiki/Infrastructure/Oncall
<@nirik:matrix.scrye.com>
16:11:26
!info leo is on call from 2024-03-01 to 2024-03-07
<@nirik:matrix.scrye.com>
16:11:38
!info patrikp is on call from 2024-03-07 to 2024-03-14
<@nirik:matrix.scrye.com>
16:12:09
!info ? is on call from 2024-03-15 to 2024-03-28
<@nirik:matrix.scrye.com>
16:12:32
Anyone want that last slot? or we can just save it for next time?
<@nirik:matrix.scrye.com>
16:13:02
!info Summary of last week: (from current oncall)
<@nirik:matrix.scrye.com>
16:13:13
leo: any oncall pings?
<@leo:fedora.im>
16:13:15
i don’t think there was a single on call ping, very quiet.
<@dtometzki:fedora.im>
16:13:27
I can took it from 03-21 -03-28
<@Zlopez:matrix.org>
16:13:42
Isn't the last slot for 2 weeks?
<@dtometzki:fedora.im>
16:14:10
yes i think it is a copy paste error
<@nirik:matrix.scrye.com>
16:14:52
oops.
<@nirik:matrix.scrye.com>
16:15:05
Sorry, had a cat jumping in my face (he wants breakfast).
<@nirik:matrix.scrye.com>
16:16:02
fixed...
<@nirik:matrix.scrye.com>
16:16:15
!topic Monitoring discussion [nirik]
<@nirik:matrix.scrye.com>
16:16:27
!info https://nagios.fedoraproject.org/nagios
<@dtometzki:fedora.im>
16:16:32
you are added me ?
<@nirik:matrix.scrye.com>
16:17:03
yes, 2024-03-15 to 2024-03-21
<@dtometzki:fedora.im>
16:17:18
Its ok too
<@nphilipp:fedora.im>
16:18:16
!hi
<@zodbot:fedora.im>
16:18:17
Nils Philippsen (nphilipp) - he / him / his
<@nirik:matrix.scrye.com>
16:18:25
so, nothing to new on nagios. We still have a copr hypervisor down. We tried a bunch of things onsite to bring it back up, but it didn't work. Hopefully we can get them to send a tech to replace CPU/MB now.
<@nirik:matrix.scrye.com>
16:18:46
db-datanommer02 is low on disk, but I need to look at moving it to a rhel9 one anyhow.
<@leo:fedora.im>
16:18:46
fingers crossed
<@seddik:fedora.im>
16:19:06
and bvmhost ??
<@nirik:matrix.scrye.com>
16:19:20
which one?
<@seddik:fedora.im>
16:19:27
bvmhost-p09-02.iad2.fedoraproject.org
<@seddik:fedora.im>
16:19:49
we have some critocal alerts
<@nirik:matrix.scrye.com>
16:19:52
oh yeah, so all those are alerting on max processes.
<@nirik:matrix.scrye.com>
16:20:05
The newer kernels just do a lot more for some reason.
<@nirik:matrix.scrye.com>
16:20:11
we should adjust nagios to not alert on it.
<@seddik:fedora.im>
16:20:27
increase thresholds ?
<@seddik:fedora.im>
16:20:33
for example ?
<@nirik:matrix.scrye.com>
16:20:52
Yeah... we should look back in nagios history, see the highest it got and raise them above that. ;)
<@nirik:matrix.scrye.com>
16:20:59
if you want to look at doing that, that would be great!
<@seddik:fedora.im>
16:21:32
ok , w'll add this in my todo-list ;)
<@nirik:matrix.scrye.com>
16:21:46
awesome. Thanks.
<@seddik:fedora.im>
16:21:56
i have a question hehe
<@seddik:fedora.im>
16:22:07
about oncall
<@seddik:fedora.im>
16:22:10
process
<@nirik:matrix.scrye.com>
16:22:16
So, thats it on nagios I think... we did backlog last time I think? but we don't have a scheduled learning topic... what do folks want to do today?
<@nirik:matrix.scrye.com>
16:22:26
sure. ask away.
<@seddik:fedora.im>
16:22:42
so , what things does the oncall person handle for example ??
<@leo:fedora.im>
16:23:31
> The oncall person should answer pings for the team and determine if the request is urgent enough to interrupt someone for, or if they can process the request they can do so, or they can direct that a ticket be created.
> The oncall person can triage tickets (move from the 'needs review' state to another state) as their time permits.
> The oncall person can answer alerts / notifications as their availability permits. Ideally they would ack or fix alerts before they go to pagers on the second alert.
<@leo:fedora.im>
16:23:38
from the wiki :)
<@nirik:matrix.scrye.com>
16:24:14
yeah. that. ;)
<@Zlopez:matrix.org>
16:24:33
There is another question, how do you ACK alert?
<@nirik:matrix.scrye.com>
16:24:46
In nagios, via the web interface.
<@seddik:fedora.im>
16:24:48
yes thanks
<@seddik:fedora.im>
16:25:05
but it requires privileges ?
<@leo:fedora.im>
16:25:09
nagios alerts, yeahs
<@leo:fedora.im>
16:25:12
nagios alerts, yeah
<@nirik:matrix.scrye.com>
16:25:18
There should be a 'service commands' or 'host commands' sidebar when you drill down to an alert.
<@Zlopez:matrix.org>
16:25:45
nirik: I probably don't have privileges, I can't do much in frontend and didn't found a way to login :-D
<@leo:fedora.im>
16:25:57
i don’t think we can do that… though i’ve never tried. hmm
<@leo:fedora.im>
16:26:14
Zlopez: if you have kerberos then it logs you in automatically
<@nirik:matrix.scrye.com>
16:26:17
Ah yeah, there is a list in a nagios config
<@nirik:matrix.scrye.com>
16:26:31
I thought we had changed it to groups, but I guess it can't handle that.
<@leo:fedora.im>
16:26:41
in theory… we could have the on call person dynamically added to that list? hmm
<@nirik:matrix.scrye.com>
16:26:43
we should make sure all oncall taking people are in the list.
<@leo:fedora.im>
16:26:53
ah yeah
<@nirik:matrix.scrye.com>
16:27:21
It's in roles/nagios_server/templates/nagios/configs/cgi.cfg.j2
<@nirik:matrix.scrye.com>
16:27:44
for zabbix, I am not sure. You can ack things there too, but not sure what permissions it needs.
<@Zlopez:matrix.org>
16:27:54
The kerberos is somewhat clunky in my browser, I wasn't able to get it running. I will probably try that again
<@leo:fedora.im>
16:28:06
in zabbix we can’t login ;(
<@leo:fedora.im>
16:28:11
only guest access iirc
<@leo:fedora.im>
16:28:16
even if you’re in -box
<@leo:fedora.im>
16:28:19
even if you’re in -noc
<@nirik:matrix.scrye.com>
16:28:38
Oh? I thought that was sorted out a while back...
<@leo:fedora.im>
16:29:46
yeah, i remember dkirwan saying it had a manual list of users
<@nirik:matrix.scrye.com>
16:29:54
can you try again? but yeah, we should fix things if it still has some issue
<@nirik:matrix.scrye.com>
16:30:07
yeah, but there's a script I thought that synced sysadmin-noc to it/
<@nirik:matrix.scrye.com>
16:30:08
?
<@nirik:matrix.scrye.com>
16:30:20
but I could be misremembering.
<@leo:fedora.im>
16:30:42
will try hold on
<@Zlopez:matrix.org>
16:31:10
What is the zabbix URL?
<@nirik:matrix.scrye.com>
16:31:21
prod: https://zabbix.fedoraproject.org/
<@seddik:fedora.im>
16:31:55
I think we can login with fas account , right ?
<@leo:fedora.im>
16:32:02
`You are not logged in
Incorrect user name or password or account is temporarily blocked.`
<@leo:fedora.im>
16:32:11
and i am in -noc, so yeah doesn't work :(
<@leo:fedora.im>
16:32:18
(via FAS)
<@Zlopez:matrix.org>
16:32:24
I'm able to log in
<@nirik:matrix.scrye.com>
16:32:59
ok, we should ask David Kirwan what the current plan is there then.
<@leo:fedora.im>
16:33:06
maybe it requires sysadmin-main?
<@nirik:matrix.scrye.com>
16:33:47
ah, it's a playbook perhaps?
<@darknao:fedora.im>
16:34:00
you need an account created on zabbix first
<@leo:fedora.im>
16:34:10
yeah, that's what i thought.
<@Zlopez:matrix.org>
16:34:42
Somebody probably created that for me already
<@leo:fedora.im>
16:34:55
i definitely don't have one created
<@nirik:matrix.scrye.com>
16:35:12
yes, but there's was supposed to be a sync. Anyhow, we can sort it out of band. ;)
<@nirik:matrix.scrye.com>
16:35:58
also, zabbix 7.0 is supposed to be out really soon... hopefully we can move to that before too long.
<@nirik:matrix.scrye.com>
16:36:24
ok, shall we just do open floor and close then?
<@Zlopez:matrix.org>
16:37:05
+1
<@nirik:matrix.scrye.com>
16:37:09
!topic Open Floor
<@nirik:matrix.scrye.com>
16:37:20
Anyone have anything for open floor?
<@dtometzki:fedora.im>
16:37:49
nop
<@Zlopez:matrix.org>
16:38:00
I have one thing for open floor, I did a meta ticket for all the RHEL 7 EOL work, so if anybody wants to help with anything look here https://pagure.io/fedora-infrastructure/issue/11815
<@nirik:matrix.scrye.com>
16:38:09
Oh, I forgot to mention, I will be out tomorrow morning too... I have a dentist appointment. ;(
<@leo:fedora.im>
16:38:30
still searching for the thing that differs on fedora infrastructure instead of my local one… hm.
<@dtometzki:fedora.im>
16:38:34
ouch
<@Zlopez:matrix.org>
16:39:10
I read that you tried to fix the haproxy issue
<@nirik:matrix.scrye.com>
16:40:57
I'm not sure what could be different/why apache isn't working there. :(
<@leo:fedora.im>
16:41:12
ah, me? yeah.
<@leo:fedora.im>
16:41:38
is it really the worst if we do the punch through NAT with a static port though? was the biggest issue that it isn’t static? or was it a general security issue
<@nirik:matrix.scrye.com>
16:42:31
well, it's just a pain, because we have to get them to do it (which could be a while till they get to it) and then if the port ever changes we would have to do it all over again...
<@nirik:matrix.scrye.com>
16:42:59
if we have to I suppose we could... but this should work via apache. ;(
<@leo:fedora.im>
16:43:02
yeah… that’s what i thought. we could make the port static but it’s probably better to figure out apache
<@nirik:matrix.scrye.com>
16:43:36
all I can now think of is that it's a rhel9/fedora39 difference in apache? perhaps try a f39 test vm?
<@leo:fedora.im>
16:43:53
will try that, yeah.
<@leo:fedora.im>
16:44:14
there's somehow a 302 happening somewhere between me, apache and OCP.
<@leo:fedora.im>
16:44:21
there's somehow a 302 happening somewhere between me, apache and OCP. i have no idea how, or where, or why.
<@leo:fedora.im>
16:44:27
there's somehow a 302 happening somewhere between me, apache and OCP. i have no idea how, or where, or why. more investigation needed :(
<@nirik:matrix.scrye.com>
16:44:40
thanks for coming everyone!
<@nirik:matrix.scrye.com>
16:44:43
!endmeeting