2024-08-01 16:00:19 <@pcreech:matrix.org> !startmeeting Infrastructure (2024-08-01)
2024-08-01 16:00:21 <@meetbot:fedora.im> Meeting started at 2024-08-01 16:00:19 UTC
2024-08-01 16:00:21 <@meetbot:fedora.im> The Meeting name is 'Infrastructure (2024-08-01)'
2024-08-01 16:00:33 <@pcreech:matrix.org> !info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
2024-08-01 16:00:33 <@pcreech:matrix.org> !meetingname infrastructure
2024-08-01 16:00:33 <@pcreech:matrix.org> !topic ahoy 
2024-08-01 16:00:33 <@pcreech:matrix.org> !info Agenda is at: https://board.net/p/fedora-infra
2024-08-01 16:00:33 <@pcreech:matrix.org> !info About our team: https://docs.fedoraproject.org/en-US/cpe/
2024-08-01 16:00:33 <@pcreech:matrix.org> !chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp
2024-08-01 16:00:34 <@meetbot:fedora.im> The Meeting Name is now infrastructure
2024-08-01 16:00:46 <@pcreech:matrix.org> !hi
2024-08-01 16:00:47 <@zodbot:fedora.im> Patrick Creech (pcreech17)
2024-08-01 16:00:48 <@nirik:matrix.scrye.com> morning everyone
2024-08-01 16:00:55 <@nirik:matrix.scrye.com> !hi
2024-08-01 16:00:56 <@zodbot:fedora.im> Kevin Fenzi (kevin) - he / him / his
2024-08-01 16:01:08 <@pcreech:matrix.org> Mornin!
2024-08-01 16:01:38 <@smilner:fedora.im> !hi
2024-08-01 16:01:39 <@zodbot:fedora.im> None (smilner)
2024-08-01 16:01:57 <@jnsamyak:matrix.org> !hi
2024-08-01 16:01:59 <@zodbot:fedora.im> Samyak Jain (jnsamyak) - he / him / his
2024-08-01 16:02:34 <@Zlopez:matrix.org> !hi
2024-08-01 16:02:37 <@zodbot:fedora.im> Michal Konecny (zlopez)
2024-08-01 16:02:39 <@carlwgeorge:matrix.org> !hi
2024-08-01 16:02:40 <@zodbot:fedora.im> Carl George (carlwgeorge) - he / him / his
2024-08-01 16:02:43 <@james:fedora.im> !hi
2024-08-01 16:02:44 <@zodbot:fedora.im> James Antill (james)
2024-08-01 16:02:54 <@lenkaseg:matrix.org> !hello
2024-08-01 16:02:56 <@zodbot:fedora.im> No Fedora Accounts users have the @lenkaseg:matrix.org Matrix Account defined
2024-08-01 16:04:33 <@pcreech:matrix.org> !topic New folks introductions
2024-08-01 16:04:33 <@pcreech:matrix.org> !info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
2024-08-01 16:04:33 <@pcreech:matrix.org> !info Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/
2024-08-01 16:04:48 <@pcreech:matrix.org> Any new folks?  I've had some internet issues so haven't been able to check for any intro e-mails
2024-08-01 16:06:20 <@nirik:matrix.scrye.com> Every day is a new day. Does that count as new? ;)
2024-08-01 16:06:47 <@pcreech:matrix.org> I mean... as an adhd person who's brain re-bootstraps every morning... i'd say so :D
2024-08-01 16:06:56 <@Zlopez:matrix.org> I didn't saw any introduction e-mail in infra mailing list last week
2024-08-01 16:07:11 <@Zlopez:matrix.org> But that doesn't mean there isn't anybody new around
2024-08-01 16:07:22 <@seddik:fedora.im> !hi 
2024-08-01 16:07:23 <@zodbot:fedora.im> seddik alaouiismaili (seddik)
2024-08-01 16:08:19 <@carlwgeorge:matrix.org> i guess i'm new, i don't remember if i've attended this meeting before
2024-08-01 16:08:43 <@pcreech:matrix.org> Welcome in!
2024-08-01 16:09:05 <@pcreech:matrix.org> Care to tell us a little about yourself? 
2024-08-01 16:11:24 <@carlwgeorge:matrix.org> the cpe folks know me, i'm on their team.  but for anyone else, howdy, my name is carl.  i'm an rpm nerd, been doing it since 2014.  i work for red hat since 2019, primarily on epel (this team https://communityblog.fedoraproject.org/cpe-to-staff-epel-work/ )
2024-08-01 16:12:12 <@Zlopez:matrix.org> Welcome on this meeting :-)
2024-08-01 16:12:15 <@nirik:matrix.scrye.com> Hi Carl. 👋
2024-08-01 16:12:39 <@pcreech:matrix.org> Awesome!  Thank you!
2024-08-01 16:13:05 <@pcreech:matrix.org> Ok, next up, chair time!
2024-08-01 16:13:07 <@pcreech:matrix.org> !info magic eight ball says:
2024-08-01 16:13:07 <@pcreech:matrix.org> !info chair 2024-08-01 pcreech
2024-08-01 16:13:07 <@pcreech:matrix.org> !info chair 2024-08-08 - cancelled (Flock)
2024-08-01 16:13:07 <@pcreech:matrix.org> !topic Next chair
2024-08-01 16:13:07 <@pcreech:matrix.org> !info chair 2024-08-15 - ???
2024-08-01 16:13:35 <@pcreech:matrix.org> Who wants to be up for the 15th?
2024-08-01 16:13:58 <@nirik:matrix.scrye.com> I guess I can if no one else... happy to defer to anyone else tho. ;)
2024-08-01 16:16:52 <@pcreech:matrix.org> nirik it is!
2024-08-01 16:17:01 <@pcreech:matrix.org> !info chair 2024-08-15 - nirik
2024-08-01 16:18:29 <@pcreech:matrix.org> !info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
2024-08-01 16:18:29 <@pcreech:matrix.org> !info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
2024-08-01 16:18:29 <@pcreech:matrix.org> !info Remaining issues with new mailman3 deployment are being investigated
2024-08-01 16:18:29 <@pcreech:matrix.org> !info Fedora Infra weekly meeting is cancelled for 2024-08-08 because of Flock 2024
2024-08-01 16:18:29 <@pcreech:matrix.org> !info Flock 2024 will happen from 7-11 Aug 2024. See you there!
2024-08-01 16:18:29 <@pcreech:matrix.org> !info Fedora 41 Mass rebuild is done
2024-08-01 16:18:29 <@pcreech:matrix.org> !topic announcements and information
2024-08-01 16:19:29 <@Zlopez:matrix.org> !info PDC is now stopped in Fedora Infrastructure, we are working on remaining things tied to it
2024-08-01 16:21:17 <@nirik:matrix.scrye.com> Looks like just one thing left (I hope)
2024-08-01 16:21:45 <@pcreech:matrix.org> Any other announcements or information before moving on?
2024-08-01 16:22:30 <@pcreech:matrix.org> !info ??? is on call from 2024-08-02 to 2024-08-08
2024-08-01 16:22:30 <@pcreech:matrix.org> !info ??? is on call from 2024-08-09 to 2024-08-15
2024-08-01 16:22:30 <@pcreech:matrix.org> !info zlopez is on call from 2024-07-26 to 2024-08-01
2024-08-01 16:22:30 <@pcreech:matrix.org> 
2024-08-01 16:22:30 <@pcreech:matrix.org> ##!oncall add <fas_name> (do not paste now, doesn't work yet)
2024-08-01 16:22:30 <@pcreech:matrix.org> !info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
2024-08-01 16:22:30 <@pcreech:matrix.org> !info https://fedoraproject.org/wiki/Infrastructure/Oncall
2024-08-01 16:22:30 <@pcreech:matrix.org> !topic Oncall
2024-08-01 16:23:13 <@pcreech:matrix.org> Not sure how flock would affect the oncalls for those ??? entries
2024-08-01 16:23:41 <@nirik:matrix.scrye.com> well, we could just have no one and people can file tickets.
2024-08-01 16:23:58 <@nirik:matrix.scrye.com> or I can take it and... just be away so they file tickets. ;)
2024-08-01 16:24:23 <@Zlopez:matrix.org> Most of us will be on the Flock
2024-08-01 16:25:02 <@pcreech:matrix.org> back to back nobody then?  (i assume with travel to + flock + travel home would be a lot for most people those two weeks0
2024-08-01 16:26:31 <@nirik:matrix.scrye.com> yeah, lets just set it to no one...
2024-08-01 16:26:38 <@nirik:matrix.scrye.com> I think it says to file a ticket then
2024-08-01 16:26:54 <@pcreech:matrix.org> 
2024-08-01 16:26:54 <@pcreech:matrix.org> !info no one is on call from 2024-08-09 to 2024-08-15
2024-08-01 16:26:54 <@pcreech:matrix.org> !info no one is on call from 2024-08-02 to 2024-08-08
2024-08-01 16:27:05 <@pcreech:matrix.org> done :D 
2024-08-01 16:27:15 <@pcreech:matrix.org> and that way, maybe nirik won't have so many noisy pings
2024-08-01 16:27:37 <@pcreech:matrix.org> !info Summary of last week: (from current oncall)
2024-08-01 16:27:41 <@pcreech:matrix.org> you're up zlopez!
2024-08-01 16:28:09 <@Zlopez:matrix.org> I didn't saw any ping whole week
2024-08-01 16:28:21 <@Zlopez:matrix.org> It was quiet on that regard
2024-08-01 16:28:42 <@Zlopez:matrix.org> And I already set oncall to no one, so let's see how it works
2024-08-01 16:28:46 <@Zlopez:matrix.org> !oncall
2024-08-01 16:28:48 <@zodbot:fedora.im> No one from Fedora Infrastructure is currently on call
2024-08-01 16:29:00 <@pcreech:matrix.org> nice
2024-08-01 16:29:18 <@pcreech:matrix.org> !info https://nagios.fedoraproject.org/nagios 
2024-08-01 16:29:18 <@pcreech:matrix.org> !topic Monitoring discussion [nirik]
2024-08-01 16:29:18 <@pcreech:matrix.org> !info Go over existing out items and fix
2024-08-01 16:29:31 <@nirik:matrix.scrye.com> so lets see...
2024-08-01 16:29:52 <@nirik:matrix.scrye.com> since last week I managed to get 2 down boxes to boot again...
2024-08-01 16:30:27 <@nirik:matrix.scrye.com> we are down to 4 down hosts. 2 of them are aarch64 boxes that need a onsite, 1 has memory problems, need to decide if we just retire it, or can replace memory, and the last one has a bad power supply
2024-08-01 16:30:45 <@nirik:matrix.scrye.com> we also have a staging virthost that lost a disk
2024-08-01 16:31:19 <@nirik:matrix.scrye.com> There's an outstanding fedora planet one because it's not sending messages.
2024-08-01 16:31:36 <@nirik:matrix.scrye.com> There's a ticket on that one. phsmoura has been on PTO, but if anyone else wants to work on that, please do.
2024-08-01 16:31:57 <@nirik:matrix.scrye.com> and ipa01.stg has a disk alert... haven't looked as to why yet. ;)
2024-08-01 16:32:24 <@Zlopez:matrix.org> I'm on the machine and it has really small `/` partition
2024-08-01 16:32:28 <@nirik:matrix.scrye.com> And thats all the current alerts.
2024-08-01 16:32:45 <@nirik:matrix.scrye.com> ah, well, we could grow it?
2024-08-01 16:32:58 <@nirik:matrix.scrye.com> or reinstall (as long as other servers can replicate back to it)
2024-08-01 16:33:34 <@Zlopez:matrix.org> Newermind, I'm on the wrong machine :-D
2024-08-01 16:33:47 <@nirik:matrix.scrye.com> ha. oops.
2024-08-01 16:34:02 <@nirik:matrix.scrye.com> anyhow, thats the alerts... hopefully it will be quiet this next week.
2024-08-01 16:34:09 <@pcreech:matrix.org> if it's ipa01.stg.iad2.fedoraproject.org, looks like it's more an inode issue than free space
2024-08-01 16:34:22 <@Zlopez:matrix.org> Plenty of tmux windows open :-D
2024-08-01 16:35:02 <@nirik:matrix.scrye.com> pcreech: note that on nagios 99% inodes means... 99% _free_ inodes. ;)
2024-08-01 16:35:14 <@pcreech:matrix.org> ah.  welp. :D
2024-08-01 16:35:19 <@nirik:matrix.scrye.com> it's that it's over 85% full on disk
2024-08-01 16:35:21 <@pcreech:matrix.org> it's been a while since i've nagios'ed
2024-08-01 16:35:35 <@pcreech:matrix.org> nevermind me :D
2024-08-01 16:35:49 <@nirik:matrix.scrye.com> it confuses basically everyone.
2024-08-01 16:36:14 <@Zlopez:matrix.org> I'm on correct machine now and only thing that stands out is `httpd` log around 600 MB
2024-08-01 16:37:13 <@Zlopez:matrix.org> The access log has 60 MB each day 
2024-08-01 16:37:26 <@Zlopez:matrix.org> Same for ssl_request_log
2024-08-01 16:37:39 <@Zlopez:matrix.org> I'm not sure if those are compressed, let me check
2024-08-01 16:38:00 <@nirik:matrix.scrye.com> anyhow, we don't need to fix it in meeting. ;) Nothing else nagios wise for me.
2024-08-01 16:38:04 <@Zlopez:matrix.org> They are not compressed
2024-08-01 16:38:33 <@Zlopez:matrix.org> I can update the logrotate to compress them as well
2024-08-01 16:38:46 <@pcreech:matrix.org> next up, let's talk about RHEL :D 
2024-08-01 16:38:49 <@pcreech:matrix.org> !ticket https://pagure.io/fedora-infrastructure/issue/11815
2024-08-01 16:38:49 <@pcreech:matrix.org> !info Check the progress on tickets related to RHEL7 EOL (end of June)
2024-08-01 16:38:49 <@pcreech:matrix.org> !topic RHEL7 EOL
2024-08-01 16:39:17 <@Zlopez:matrix.org> There are still few things missing here
2024-08-01 16:39:37 <@Zlopez:matrix.org> * github2fedmsg
2024-08-01 16:39:37 <@Zlopez:matrix.org> * PDC
2024-08-01 16:39:37 <@Zlopez:matrix.org> * fedimg
2024-08-01 16:40:50 <@Zlopez:matrix.org> PDC is currently stopped and will be decomissioned soon
2024-08-01 16:40:55 <@pcreech:matrix.org> Looks like last comment on the ticket was about a month ago... any value in adding another comment with an updated state?  or is it fairly similar anyways?
2024-08-01 16:41:21 <@Zlopez:matrix.org> It's still the same
2024-08-01 16:42:17 <@Zlopez:matrix.org> github2fedmsg is being replaced and hopefully we will have the first version of webhook2fedmsg running soon
2024-08-01 16:42:21 <@nirik:matrix.scrye.com> which tickets?
2024-08-01 16:42:26 <@nirik:matrix.scrye.com> oh, the overall one?
2024-08-01 16:43:13 <@nirik:matrix.scrye.com> On fedimg, I hope to get with David Duncan and sort out the perms.... will try and sync with him.
2024-08-01 16:43:32 <@Zlopez:matrix.org> That is the only thing missing I think
2024-08-01 16:44:37 <@nirik:matrix.scrye.com> well, with that we can test in stg and fix things up and then roll to prod
2024-08-01 16:44:59 <@nirik:matrix.scrye.com> There's still 2 virthosts... I just pinged again both sites about the onsite stuff I need.
2024-08-01 16:47:19 <@nirik:matrix.scrye.com> getting close, these things are just pesky
2024-08-01 16:47:42 <@Zlopez:matrix.org> Nice :-)
2024-08-01 16:48:32 <@pcreech:matrix.org> Cool, sounds like that's about it for the rhel stuff then :D
2024-08-01 16:48:34 <@pcreech:matrix.org> !link https://pagure.io/fedora-infrastructure/issues?status=Open&tags=blocked&priority=0&close_status=
2024-08-01 16:48:34 <@pcreech:matrix.org> !info Check if any blocked ticket is unblocked
2024-08-01 16:48:34 <@pcreech:matrix.org> !topic Revisit blocked tickets
2024-08-01 16:49:01 <@pcreech:matrix.org> Any blocked things with updates?
2024-08-01 16:49:15 <@nirik:matrix.scrye.com> nothing unblocked that I can see...
2024-08-01 16:49:28 <@nirik:matrix.scrye.com> possibly we could do the dmarc setting for all lists now
2024-08-01 16:50:55 <@pcreech:matrix.org> Cool.  Seeing as we have 10 minutes left...  skipping to open floor then unless someone has an opinion otherwise
2024-08-01 16:51:34 <@pcreech:matrix.org> !topic Open Floor
2024-08-01 16:51:52 <@pcreech:matrix.org> Anyone have anything to chat about during open floor?
2024-08-01 16:52:27 <@Zlopez:matrix.org> I fixed the ipa01.stg, but I don't understand why there is compressdelay in logrotate, nothing was compressed for httpd
2024-08-01 16:53:19 <@nirik:matrix.scrye.com> from the man page:
2024-08-01 16:53:24 <@nirik:matrix.scrye.com>               log file for some time."
2024-08-01 16:53:24 <@nirik:matrix.scrye.com> " It can be used when some program
2024-08-01 16:53:24 <@nirik:matrix.scrye.com>               cannot be told to close its logfile and thus might continue writing to the  previous
2024-08-01 16:54:01 <@Zlopez:matrix.org> I compressed that manually
2024-08-01 16:54:01 <@nirik:matrix.scrye.com> so perhaps ipa does this somehow?
2024-08-01 16:54:26 <@Zlopez:matrix.org> Could be
2024-08-01 16:56:08 <@Zlopez:matrix.org> But the next big thing is /var/log/journal, it has around 1 GB
2024-08-01 16:57:04 <@naraiank:fedora.im> !hi
2024-08-01 16:57:05 <@zodbot:fedora.im> Naraian K (naraiank) - he / him / his
2024-08-01 16:58:09 <@nirik:matrix.scrye.com> hey The Exorcist
2024-08-01 16:58:23 <@nirik:matrix.scrye.com> anyhow, I don't think I have anything for open floor...
2024-08-01 16:58:24 <@naraiank:fedora.im> Can't memory be added on hot server? I don't think we can but curious to know if it is possible..
2024-08-01 16:58:30 <@nirik:matrix.scrye.com> safe travels to everyone going to flock
2024-08-01 16:59:03 <@naraiank:fedora.im> Can't memory be added on hot server? I don't think we can, but curious to know if it is possible..
2024-08-01 16:59:15 <@nirik:matrix.scrye.com> you mean disk space for ipa01.stg? yeah, we could grow the disk... just a bit more complicated than fixing whats using it. ;)
2024-08-01 17:00:06 <@pcreech:matrix.org> !endmeeting