2024-03-07 16:00:22 <@nirik:matrix.scrye.com> !startmeeting Infrastructure (2024-03-07)
2024-03-07 16:00:24 <@meetbot:fedora.im> Meeting started at 2024-03-07 16:00:22 UTC
2024-03-07 16:00:24 <@meetbot:fedora.im> The Meeting name is 'Infrastructure (2024-03-07)'
2024-03-07 16:00:29 <@nirik:matrix.scrye.com> !meetingname infrastructure
2024-03-07 16:00:35 <@nirik:matrix.scrye.com> !info Agenda is at: https://board.net/p/fedora-infra
2024-03-07 16:00:42 <@nirik:matrix.scrye.com> About our team: https://docs.fedoraproject.org/en-US/cpe/
2024-03-07 16:00:46 <@nirik:matrix.scrye.com> Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
2024-03-07 16:00:53 <@nirik:matrix.scrye.com> !topic namaste
2024-03-07 16:00:53 <@leo:fedora.im> !hi
2024-03-07 16:00:55 <@zodbot:fedora.im> Leo Puvilland (leo) - he / him / his
2024-03-07 16:01:01 <@seddik:fedora.im> !hi 
2024-03-07 16:01:02 <@dtometzki:fedora.im> hi
2024-03-07 16:01:02 <@zodbot:fedora.im> seddik alaouiismaili (seddik)
2024-03-07 16:01:04 <@nirik:matrix.scrye.com> Morning everyone
2024-03-07 16:01:58 <@nirik:matrix.scrye.com> !topic New folks introductions
2024-03-07 16:02:03 <@nirik:matrix.scrye.com> !info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
2024-03-07 16:02:08 <@nirik:matrix.scrye.com> Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/
2024-03-07 16:02:13 <@nirik:matrix.scrye.com> Any new folks today?
2024-03-07 16:03:04 <@Zlopez:matrix.org> !hi
2024-03-07 16:03:06 <@zodbot:fedora.im> Michal Konecny (zlopez)
2024-03-07 16:03:21 <@Zlopez:matrix.org> How is everyone today?
2024-03-07 16:03:35 <@leo:fedora.im> good how are you
2024-03-07 16:03:39 <@jsteffan:fedora.im> !hi
2024-03-07 16:03:40 <@zodbot:fedora.im> Jonathan Steffan (jsteffan)
2024-03-07 16:03:42 <@dtometzki:fedora.im> fine thanks and you ?
2024-03-07 16:03:52 <@nirik:matrix.scrye.com> To early to tell for me. ;)
2024-03-07 16:04:19 <@nirik:matrix.scrye.com> I guess lets move on to...
2024-03-07 16:04:21 <@nirik:matrix.scrye.com> !topic Next chair
2024-03-07 16:04:27 <@nirik:matrix.scrye.com> !info magic eight ball says:
2024-03-07 16:04:32 <@nirik:matrix.scrye.com> !info chair 2024-03-14 dtometzki
2024-03-07 16:04:36 <@nirik:matrix.scrye.com> !info chair 2024-03-21 ???
2024-03-07 16:04:41 <@nirik:matrix.scrye.com> !info chair 2024-03-28 ???
2024-03-07 16:04:50 <@nirik:matrix.scrye.com> Anyone want either of those ?
2024-03-07 16:04:52 <@dtometzki:fedora.im> can i exchange my chair with anyone ?
2024-03-07 16:05:03 <@nirik:matrix.scrye.com> sure!
2024-03-07 16:05:05 <@dtometzki:fedora.im> i am not available next week
2024-03-07 16:05:11 <@Zlopez:matrix.org> leo, dtometzki: Great, thanks for asking :-)
2024-03-07 16:05:25 <@Zlopez:matrix.org> I can take it
2024-03-07 16:05:26 <@nirik:matrix.scrye.com> dtometzki: want to move to one of the later ones? or just change next week for now?
2024-03-07 16:05:36 <@nirik:matrix.scrye.com> thanks Zlopez
2024-03-07 16:05:48 <@dtometzki:fedora.im> move one or two weeks later
2024-03-07 16:05:51 <@seddik:fedora.im> doing good, what about you??  
2024-03-07 16:06:32 <@nirik:matrix.scrye.com> So, 2024-03-14 zlopez and 2024-03-21 dtometzki ?
2024-03-07 16:06:38 <@Zlopez:matrix.org> Yes
2024-03-07 16:06:46 <@dtometzki:fedora.im> perfekt
2024-03-07 16:06:52 <@nirik:matrix.scrye.com> thanks!
2024-03-07 16:06:59 <@nirik:matrix.scrye.com> !topic announcements and information
2024-03-07 16:07:05 <@nirik:matrix.scrye.com> !info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0830 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
2024-03-07 16:07:09 <@nirik:matrix.scrye.com> !info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
2024-03-07 16:07:15 <@nirik:matrix.scrye.com> !info we are in F40 Beta freeze
2024-03-07 16:07:28 <@Zlopez:matrix.org> !info Communishift is now publicly available https://communityblog.fedoraproject.org/communishift-now-available/
2024-03-07 16:07:40 <@nirik:matrix.scrye.com> !info nirik is out on pto next monday (2024-03-11)
2024-03-07 16:08:08 <@nirik:matrix.scrye.com> !info F40 beta is 'no go' for the early target date next week. ;(
2024-03-07 16:08:18 <@Zlopez:matrix.org> !info Outreachy started this week, you can see plenty of new contributors around
2024-03-07 16:08:44 <@nirik:matrix.scrye.com> yeah, lots of activity!
2024-03-07 16:09:05 <@leo:fedora.im> hooray!
2024-03-07 16:09:20 <@nirik:matrix.scrye.com> Any other announcements or information?
2024-03-07 16:09:32 <@Zlopez:matrix.org> I'm not able to even respond to everything :-D
2024-03-07 16:10:46 <@nirik:matrix.scrye.com> !topic Oncall
2024-03-07 16:10:52 <@nirik:matrix.scrye.com> !info https://fedoraproject.org/wiki/Infrastructure/Oncall
2024-03-07 16:11:26 <@nirik:matrix.scrye.com> !info leo is on call from 2024-03-01 to 2024-03-07
2024-03-07 16:11:38 <@nirik:matrix.scrye.com> !info patrikp is on call from 2024-03-07 to 2024-03-14
2024-03-07 16:12:09 <@nirik:matrix.scrye.com> !info ? is on call from 2024-03-15 to 2024-03-28
2024-03-07 16:12:32 <@nirik:matrix.scrye.com> Anyone want that last slot? or we can just save it for next time?
2024-03-07 16:13:02 <@nirik:matrix.scrye.com> !info Summary of last week: (from current oncall)
2024-03-07 16:13:13 <@nirik:matrix.scrye.com> leo: any oncall pings?
2024-03-07 16:13:15 <@leo:fedora.im> i don’t think there was a single on call ping, very quiet.
2024-03-07 16:13:27 <@dtometzki:fedora.im> I can took it from 03-21 -03-28
2024-03-07 16:13:42 <@Zlopez:matrix.org> Isn't the last slot for 2 weeks?
2024-03-07 16:14:10 <@dtometzki:fedora.im> yes i think it is a copy paste error
2024-03-07 16:14:52 <@nirik:matrix.scrye.com> oops.
2024-03-07 16:15:05 <@nirik:matrix.scrye.com> Sorry, had a cat jumping in my face (he wants breakfast).
2024-03-07 16:16:02 <@nirik:matrix.scrye.com> fixed...
2024-03-07 16:16:15 <@nirik:matrix.scrye.com> !topic Monitoring discussion [nirik]
2024-03-07 16:16:27 <@nirik:matrix.scrye.com> !info https://nagios.fedoraproject.org/nagios
2024-03-07 16:16:32 <@dtometzki:fedora.im> you are added me ?
2024-03-07 16:17:03 <@nirik:matrix.scrye.com> yes, 2024-03-15 to 2024-03-21
2024-03-07 16:17:18 <@dtometzki:fedora.im> Its ok too
2024-03-07 16:18:16 <@nphilipp:fedora.im> !hi
2024-03-07 16:18:17 <@zodbot:fedora.im> Nils Philippsen (nphilipp) - he / him / his
2024-03-07 16:18:25 <@nirik:matrix.scrye.com> so, nothing to new on nagios. We still have a copr hypervisor down. We tried a bunch of things onsite to bring it back up, but it didn't work. Hopefully we can get them to send a tech to replace CPU/MB now.
2024-03-07 16:18:46 <@nirik:matrix.scrye.com> db-datanommer02 is low on disk, but I need to look at moving it to a rhel9 one anyhow.
2024-03-07 16:18:46 <@leo:fedora.im> fingers crossed
2024-03-07 16:19:06 <@seddik:fedora.im> and bvmhost ?? 
2024-03-07 16:19:20 <@nirik:matrix.scrye.com> which one?
2024-03-07 16:19:27 <@seddik:fedora.im> bvmhost-p09-02.iad2.fedoraproject.org
2024-03-07 16:19:49 <@seddik:fedora.im> we have some critocal alerts 
2024-03-07 16:19:52 <@nirik:matrix.scrye.com> oh yeah, so all those are alerting on max processes.
2024-03-07 16:20:05 <@nirik:matrix.scrye.com> The newer kernels just do a lot more for some reason.
2024-03-07 16:20:11 <@nirik:matrix.scrye.com> we should adjust nagios to not alert on it.
2024-03-07 16:20:27 <@seddik:fedora.im> increase thresholds ? 
2024-03-07 16:20:33 <@seddik:fedora.im> for example ? 
2024-03-07 16:20:52 <@nirik:matrix.scrye.com> Yeah... we should look back in nagios history, see the highest it got and raise them above that. ;)
2024-03-07 16:20:59 <@nirik:matrix.scrye.com> if you want to look at doing that, that would be great!
2024-03-07 16:21:32 <@seddik:fedora.im> ok , w'll add this in my todo-list ;) 
2024-03-07 16:21:46 <@nirik:matrix.scrye.com> awesome. Thanks.
2024-03-07 16:21:56 <@seddik:fedora.im> i have a question hehe
2024-03-07 16:22:07 <@seddik:fedora.im> about oncall 
2024-03-07 16:22:10 <@seddik:fedora.im> process
2024-03-07 16:22:16 <@nirik:matrix.scrye.com> So, thats it on nagios I think... we did backlog last time I think? but we don't have a scheduled learning topic... what do folks want to do today?
2024-03-07 16:22:26 <@nirik:matrix.scrye.com> sure. ask away.
2024-03-07 16:22:42 <@seddik:fedora.im> so , what things does the oncall person handle for example ?? 
2024-03-07 16:23:31 <@leo:fedora.im> > The oncall person should answer pings for the team and determine if the request is urgent enough to interrupt someone for, or if they can process the request they can do so, or they can direct that a ticket be created.

> The oncall person can triage tickets (move from the 'needs review' state to another state) as their time permits.

> The oncall person can answer alerts / notifications as their availability permits. Ideally they would ack or fix alerts before they go to pagers on the second alert.
2024-03-07 16:23:38 <@leo:fedora.im> from the wiki :)
2024-03-07 16:24:14 <@nirik:matrix.scrye.com> yeah. that. ;)
2024-03-07 16:24:33 <@Zlopez:matrix.org> There is another question, how do you ACK alert?
2024-03-07 16:24:46 <@nirik:matrix.scrye.com> In nagios, via the web interface.
2024-03-07 16:24:48 <@seddik:fedora.im> yes thanks 
2024-03-07 16:25:05 <@seddik:fedora.im> but it requires privileges ? 
2024-03-07 16:25:09 <@leo:fedora.im> nagios alerts, yeahs
2024-03-07 16:25:12 <@leo:fedora.im> nagios alerts, yeah
2024-03-07 16:25:18 <@nirik:matrix.scrye.com> There should be a 'service commands' or 'host commands' sidebar when you drill down to an alert.
2024-03-07 16:25:45 <@Zlopez:matrix.org> nirik: I probably don't have privileges, I can't do much in frontend and didn't found a way to login :-D
2024-03-07 16:25:57 <@leo:fedora.im> i don’t think we can do that… though i’ve never tried. hmm
2024-03-07 16:26:14 <@leo:fedora.im> Zlopez: if you have kerberos then it logs you in automatically
2024-03-07 16:26:17 <@nirik:matrix.scrye.com> Ah yeah, there is a list in a nagios config
2024-03-07 16:26:31 <@nirik:matrix.scrye.com> I thought we had changed it to groups, but I guess it can't handle that.
2024-03-07 16:26:41 <@leo:fedora.im> in theory… we could have the on call person dynamically added to that list? hmm
2024-03-07 16:26:43 <@nirik:matrix.scrye.com> we should make sure all oncall taking people are in the list.
2024-03-07 16:26:53 <@leo:fedora.im> ah yeah
2024-03-07 16:27:21 <@nirik:matrix.scrye.com> It's in roles/nagios_server/templates/nagios/configs/cgi.cfg.j2
2024-03-07 16:27:44 <@nirik:matrix.scrye.com> for zabbix, I am not sure. You can ack things there too, but not sure what permissions it needs.
2024-03-07 16:27:54 <@Zlopez:matrix.org> The kerberos is somewhat clunky in my browser, I wasn't able to get it running. I will probably try that again
2024-03-07 16:28:06 <@leo:fedora.im> in zabbix we can’t login ;(
2024-03-07 16:28:11 <@leo:fedora.im> only guest access iirc 
2024-03-07 16:28:16 <@leo:fedora.im> even if you’re in -box
2024-03-07 16:28:19 <@leo:fedora.im> even if you’re in -noc
2024-03-07 16:28:38 <@nirik:matrix.scrye.com> Oh? I thought that was sorted out a while back...
2024-03-07 16:29:46 <@leo:fedora.im> yeah, i remember dkirwan saying it had a manual list of users
2024-03-07 16:29:54 <@nirik:matrix.scrye.com> can you try again? but yeah, we should fix things if it still has some issue
2024-03-07 16:30:07 <@nirik:matrix.scrye.com> yeah, but there's a script I thought that synced sysadmin-noc to it/
2024-03-07 16:30:08 <@nirik:matrix.scrye.com> ?
2024-03-07 16:30:20 <@nirik:matrix.scrye.com> but I could be misremembering.
2024-03-07 16:30:42 <@leo:fedora.im> will try hold on
2024-03-07 16:31:10 <@Zlopez:matrix.org> What is the zabbix URL?
2024-03-07 16:31:21 <@nirik:matrix.scrye.com> prod: https://zabbix.fedoraproject.org/
2024-03-07 16:31:55 <@seddik:fedora.im> I think we can login with fas account , right ? 
2024-03-07 16:32:02 <@leo:fedora.im> `You are not logged in

    Incorrect user name or password or account is temporarily blocked.`
2024-03-07 16:32:11 <@leo:fedora.im> and i am in -noc, so yeah doesn't work :(
2024-03-07 16:32:18 <@leo:fedora.im> (via FAS)
2024-03-07 16:32:24 <@Zlopez:matrix.org> I'm able to log in
2024-03-07 16:32:59 <@nirik:matrix.scrye.com> ok, we should ask David Kirwan what the current plan is there then.
2024-03-07 16:33:06 <@leo:fedora.im> maybe it requires sysadmin-main?
2024-03-07 16:33:47 <@nirik:matrix.scrye.com> ah, it's a playbook perhaps?
2024-03-07 16:34:00 <@darknao:fedora.im> you need an account created on zabbix first
2024-03-07 16:34:10 <@leo:fedora.im> yeah, that's what i thought.
2024-03-07 16:34:42 <@Zlopez:matrix.org> Somebody probably created that for me already
2024-03-07 16:34:55 <@leo:fedora.im> i definitely don't have one created
2024-03-07 16:35:12 <@nirik:matrix.scrye.com> yes, but there's was supposed to be a sync. Anyhow, we can sort it out of band. ;)
2024-03-07 16:35:58 <@nirik:matrix.scrye.com> also, zabbix 7.0 is supposed to be out really soon... hopefully we can move to that before too long.
2024-03-07 16:36:24 <@nirik:matrix.scrye.com> ok, shall we just do open floor and close then?
2024-03-07 16:37:05 <@Zlopez:matrix.org> +1
2024-03-07 16:37:09 <@nirik:matrix.scrye.com> !topic Open Floor
2024-03-07 16:37:20 <@nirik:matrix.scrye.com> Anyone have anything for open floor?
2024-03-07 16:37:49 <@dtometzki:fedora.im> nop
2024-03-07 16:38:00 <@Zlopez:matrix.org> I have one thing for open floor, I did a meta ticket for all the RHEL 7 EOL work, so if anybody wants to help with anything look here https://pagure.io/fedora-infrastructure/issue/11815
2024-03-07 16:38:09 <@nirik:matrix.scrye.com> Oh, I forgot to mention, I will be out tomorrow morning too... I have a dentist appointment. ;(
2024-03-07 16:38:30 <@leo:fedora.im> still searching for the thing that differs on fedora infrastructure instead of my local one… hm.
2024-03-07 16:38:34 <@dtometzki:fedora.im> ouch
2024-03-07 16:39:10 <@Zlopez:matrix.org> I read that you tried to fix the haproxy issue
2024-03-07 16:40:57 <@nirik:matrix.scrye.com> I'm not sure what could be different/why apache isn't working there. :(
2024-03-07 16:41:12 <@leo:fedora.im> ah, me? yeah.
2024-03-07 16:41:38 <@leo:fedora.im> is it really the worst if we do the punch through NAT with a static port though? was the biggest issue that it isn’t static? or was it a general security issue
2024-03-07 16:42:31 <@nirik:matrix.scrye.com> well, it's just a pain, because we have to get them to do it (which could be a while till they get to it) and then if the port ever changes we would have to do it all over again...
2024-03-07 16:42:59 <@nirik:matrix.scrye.com> if we have to I suppose we could... but this should work via apache. ;(
2024-03-07 16:43:02 <@leo:fedora.im> yeah… that’s what i thought. we could make the port static but it’s probably better to figure out apache
2024-03-07 16:43:36 <@nirik:matrix.scrye.com> all I can now think of is that it's a rhel9/fedora39 difference in apache? perhaps try a f39 test vm?
2024-03-07 16:43:53 <@leo:fedora.im> will try that, yeah.
2024-03-07 16:44:14 <@leo:fedora.im> there's somehow a 302 happening somewhere between me, apache and OCP.
2024-03-07 16:44:21 <@leo:fedora.im> there's somehow a 302 happening somewhere between me, apache and OCP. i have no idea how, or where, or why.
2024-03-07 16:44:27 <@leo:fedora.im> there's somehow a 302 happening somewhere between me, apache and OCP. i have no idea how, or where, or why. more investigation needed :(
2024-03-07 16:44:40 <@nirik:matrix.scrye.com> thanks for coming everyone!
2024-03-07 16:44:43 <@nirik:matrix.scrye.com> !endmeeting