16:04:59 <nirik> #startmeeting Infrastructure (2023-11-09)
16:04:59 <zodbot> Meeting started Thu Nov  9 16:04:59 2023 UTC.
16:04:59 <zodbot> This meeting is logged and archived in a public location.
16:04:59 <zodbot> The chair is nirik. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:04:59 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:04:59 <zodbot> The meeting name has been set to 'infrastructure_(2023-11-09)'
16:05:00 <nirik> #meetingname infrastructure
16:05:00 <zodbot> The meeting name has been set to 'infrastructure'
16:05:00 <nirik> #chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp
16:05:00 <zodbot> Current chairs: bodanel dtometzki jnsamyak lenkaseg nb nirik patrikp zlopez
16:05:00 <nirik> #info Agenda is at: https://board.net/p/fedora-infra
16:05:00 <nirik> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
16:05:18 <nirik> #chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp
16:05:18 <zodbot> Current chairs: bodanel dtometzki jnsamyak lenkaseg nb nirik patrikp zlopez
16:05:24 <nirik> #info Agenda is at: https://board.net/p/fedora-infra
16:05:28 <nirik> #info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
16:05:37 <nirik> #topic namaste
16:05:43 <nirik> Who all is around for a infra meeting?
16:05:46 <zlopez> .hi
16:05:47 <zodbot> zlopez: zlopez 'Michal Konecny' <michal.konecny@pacse.eu>
16:06:31 <zlopez> I wonder if we shouldn't move to matrix, there is a nice progress on meetbot now :-)
16:06:58 <patrikp> .hi
16:06:59 <zodbot> patrikp: patrikp 'Patrik Polakovič' <ppolakov@redhat.com>
16:07:16 <nirik> we could... there's still a few small things, but we could try. perhaps next week?
16:07:28 <nirik> hi patrikp!
16:07:46 <zlopez> nirik: +1 to move it
16:08:16 <zlopez> We just need to update agenda for the matrix commands
16:08:21 <nirik> I also wonder if this time is still good with the DST stuff. It's a bit anoying for me as cats expect to be fed right in the middle of it. ;)
16:09:57 <nirik> but we can decide that sometime later.
16:09:59 <zlopez> It's using UTC
16:10:11 <nirik> yes.
16:10:36 <zlopez> Let's just wait six months and it will be back :-D
16:11:10 <zlopez> Still waiting till this DST nonsense will go away in EU
16:11:35 <pcreech> Crap....
16:11:48 <nirik> heh. well, yeah... also I have fesco meeting right after this which is also UTC, so we can't just move it an hour later for me anyhow.
16:11:48 <pcreech> My mind was still on pre-time-change meeting time
16:12:00 <nirik> pcreech: yeah, DST is the gift that keeps on giving.
16:12:15 <nirik> #chair pcreech
16:12:15 <zodbot> Current chairs: bodanel dtometzki jnsamyak lenkaseg nb nirik patrikp pcreech zlopez
16:12:26 <zlopez> It takes half a year to get used to it and then you can start again :-D
16:12:41 <pcreech> hehe
16:12:59 <zlopez> I still have one clock that is always showing UTC time :-)
16:13:24 <nirik> It was hard enough to move the cats to the new feeding time... but somehow they know...
16:13:35 <nirik> anyhow, pcreech, you want to take over?
16:13:54 <zlopez> You need to introduce the DST to cats, so they can adjust when that happens :-D
16:13:57 <pcreech> Sure! Was just getting oriented to where we were at :D
16:14:11 <pcreech> looks like _just started_
16:14:16 <nirik> yep.
16:14:29 <pcreech> ### Let new people say hello
16:14:30 <pcreech> ```
16:14:30 <pcreech> #topic New folks introductions
16:14:30 <pcreech> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
16:14:30 <pcreech> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
16:14:44 <pcreech> Any new people?
16:14:57 <nirik> FYI, I keep getting swamped and not progressing with onboarding docs... but hopefully I will get back to it more soon...
16:15:08 <nirik> and replacing that wiki page is high on the list. ;)
16:15:35 <pcreech> :D
16:16:07 <pcreech> assuming no new people
16:16:10 <pcreech> ### Determine who the next chair is
16:16:10 <pcreech> #topic Next chair
16:16:10 <pcreech> #info magic eight ball says:
16:16:10 <pcreech> #info chair 2023-11-09 pcreech
16:16:10 <pcreech> #info chair 2023-11-16 nirik
16:16:11 <pcreech> #info chair 2023-11-23 zlopez
16:16:22 <pcreech> Looks like we have quite the list already :D
16:16:28 * nirik nods
16:16:43 <pcreech> moving on then :D
16:16:47 <pcreech> #topic announcements and information
16:16:47 <pcreech> #info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0830 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
16:16:47 <pcreech> #info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
16:16:47 <pcreech> #info Fedora 39 released on 7th November
16:16:58 <zlopez> I can do the first matrix meeting, so I can adjust agenda as needed
16:17:29 <pcreech> ack
16:17:57 <pcreech> #topic Oncall
16:17:57 <pcreech> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
16:17:57 <pcreech> #info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
16:17:57 <pcreech> ## .oncalltakeeu .oncalltakeus
16:17:57 <pcreech> #info pcreech is on call from 2023-11-10 to 2023-11-16
16:17:57 <pcreech> #info ??? is on call from 2023-11-16 to 2023-11-23
16:18:01 <nirik> or I can next week? either way
16:18:18 <zlopez> Or we were talking about canceling that one as it's Thanksgiving in US
16:18:32 <nirik> yeah.
16:18:43 <nirik> you want to just take the one next week? ;) I don't mind at all.
16:19:01 <zlopez> I can
16:19:28 <zlopez> I can also take the oncall
16:20:11 <pcreech> Sold!
16:20:29 <nirik> +1
16:20:48 <zlopez> Ok, so I'm also chair for next week
16:21:00 <pcreech> #info zlopez is on call from 2023-11-16 to 2023-11-23
16:21:13 <pcreech> #info chair 2023-11-16 zlopez
16:22:24 <pcreech> And i'm thinking, since it's just us here, we could maybe try to do more oncalls next week, in case more people are around
16:22:53 <nirik> sure.
16:23:16 <pcreech> #info Summary of last week: (from current oncall )
16:24:14 <pcreech> zednorezec is current on call, don't see him in here
16:24:44 <zlopez> He said he will be unavailable and maybe get back till learning topic
16:25:30 <pcreech> .oncalltakeus
16:25:30 <zodbot> pcreech: Kneel before zod!
16:25:32 <pcreech> cool
16:26:04 <pcreech> O.o thought I put that in another window... :D
16:26:16 <pcreech> #topic Monitoring discussion [nirik]
16:26:16 <pcreech> #info https://nagios.fedoraproject.org/nagios
16:26:16 <pcreech> #info Go over existing out items and fix
16:26:29 <pcreech> Monitoring time!  nirik
16:27:06 <nirik> lets see
16:27:36 <nirik> mostly the same things as before...
16:27:47 <nirik> I might be able to do some cleanup now that we are out of freeze.
16:28:12 <nirik> Also, koji db has been alerting... it's really getting hammered. ;( Need to look at tunring it more or blocking abusing clients somehow
16:29:55 <nirik> nothing else really to report...
16:30:11 <pcreech> Sounds good :D
16:30:15 <pcreech> On to tickets?
16:30:39 <pcreech> #topic Revisit blocked tickets
16:30:40 <pcreech> #info Check if any blocked ticket is unblocked
16:30:40 <pcreech> #link https://pagure.io/fedora-infrastructure/issues?status=Open&tags=blocked&priority=0&close_status=
16:30:44 <nirik> sure (I may need to step away for a few in a few... cats are glaring at me)
16:30:54 <pcreech> Yay cats!
16:31:30 <pcreech> Any changes to the blocked tickets?
16:31:36 <nirik> So, on mailman3... everything is built now, but a few packages are in a copr pending landing in epel9... so we could start looking at making a mailman01.stg sometime...
16:32:17 <zlopez> The OIDC has also some progress with some services being decommissioned
16:32:21 <nirik> then ideally we import at least some of prod and test it a lot before looking at making a new one in prod and migrating the existing one to it.
16:33:14 <pcreech> verynice
16:33:49 <zlopez> We also now have IPA running on RHEL9 in staging :-)
16:34:08 <zlopez> At least one machine
16:34:14 <nirik> awesome! so, we still need to reinstall the ipa01.stg/ipa02.stg now right?
16:34:40 <zlopez> Yes and I noticed that the DNS service is missing on ipa03
16:35:16 <zlopez> I updated the ticket after running the playbook today
16:35:57 <zlopez> .ticket 10358
16:35:59 <zodbot> zlopez: Issue #10358: IPA backup fails on ipa03 and ipa02.stg - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10358
16:36:21 <nirik> yes, we don't have dns in ipa.
16:36:24 <nirik> we do our own dns
16:36:50 <nirik> so not sure why it should show in the others... we shouldn't have it enabled
16:37:07 <zlopez> Strange the ipa01 has DNS enabled
16:37:12 <zlopez> ipa01.stg
16:37:46 <zlopez> Also I'm not sure if we are using AD or NTP service on IPA
16:38:15 <nirik> nope, neither one.
16:39:07 <zlopez> Ok, then the deployment looks correct :-)
16:39:34 <nirik> cool. and it's replicating ok?
16:40:20 <zlopez> It was created by replicating the ipa01.stg, but I didn't tried to switch ipa01.stg or ipa02.stg to RHEL9
16:41:13 <zlopez> To switch ipa01 and ipa02 on staging needs to be tested first
16:42:08 <nirik> yeah. it does seem to be replicating fine.
16:42:12 <nirik> I used the nagios check:
16:42:55 <zlopez> There was only some errors when installing zabbix package, but that was temporary RHEL9 mirror issue
16:42:56 <nirik> oh wait
16:43:03 <nirik> no, it's not...
16:43:20 <nirik> [root@ipa01 ~][STG]# /usr/lib64/nagios/plugins/check_ipa_replication -u ldaps://localhost
16:43:20 <nirik> Warning - Warning: No replicas found
16:43:46 <zlopez> That is strange
16:43:59 <nirik> yeah...
16:44:42 <zlopez> Maybe we are missing some parameters in ipa-replica-install :/
16:45:27 <nirik> huh, the web ui does show it tho
16:46:01 <nirik> along with proxy.ipa.example
16:46:16 <smooge> that doesn't sound like a real host
16:47:13 <zlopez> Hm, you are right
16:47:27 <zlopez> I'm getting this error when checking the replica `cannot connect to 'ldap://rhel9.example.com:389': Bad file descriptor`
16:47:45 <zlopez> Oh sorry, forgot to change the hostname
16:48:02 <nirik> yeah, something still isn't right... ;)
16:48:43 <zlopez> Hm, this looks right
16:48:44 <zlopez> ipa01.stg.iad2.fedoraproject.org
16:48:44 <zlopez> last update status: Error (0) Replica acquired successfully: Incremental update succeeded
16:48:44 <zlopez> last update ended: 2023-11-09 16:39:52+00:00
16:49:24 <nirik> yeah.
16:49:33 <nirik> I've got to go feed cats, back in a few.
16:50:06 <zlopez> Do we want to move, I don't think we can solve the IPA issue here
16:50:14 <zlopez> At least it can be deployed now :-)
16:50:26 <pcreech> Yeah, I was letting the conversation go while it had some meat to it
16:50:38 <pcreech> and since we have about 10 minutes left, figured it was good time for open floor
16:50:40 <pcreech> #topic Open Floor
16:51:27 <zlopez> #info Next meeting will be on matrix https://matrix.to/#/#meeting-3:fedoraproject.org
16:51:51 <pcreech> Excited about the matrix meeting personally :D
16:52:27 <zlopez> I will probably leave the agenda for IRC in the board.net document in case anything will go wrong with matrix
16:52:52 <pcreech> Probably a good idea
16:54:37 <zlopez> I don't think it will be hard to change the agenda, just using different commands :-)
16:54:51 <pcreech> :D
16:55:05 <pcreech> and if nothing else, gonna end the meeting  a little early
16:55:25 <pcreech> #endmeeting