<@pcreech:matrix.org>
16:00:19
!startmeeting Infrastructure (2024-08-01)
<@meetbot:fedora.im>
16:00:21
Meeting started at 2024-08-01 16:00:19 UTC
<@meetbot:fedora.im>
16:00:21
The Meeting name is 'Infrastructure (2024-08-01)'
<@pcreech:matrix.org>
16:00:33
!info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
<@pcreech:matrix.org>
16:00:33
!meetingname infrastructure
<@pcreech:matrix.org>
16:00:33
!topic ahoy
<@pcreech:matrix.org>
16:00:33
!info Agenda is at: https://board.net/p/fedora-infra
<@pcreech:matrix.org>
16:00:33
!info About our team: https://docs.fedoraproject.org/en-US/cpe/
<@pcreech:matrix.org>
16:00:33
!chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp
<@meetbot:fedora.im>
16:00:34
The Meeting Name is now infrastructure
<@pcreech:matrix.org>
16:00:46
!hi
<@zodbot:fedora.im>
16:00:47
Patrick Creech (pcreech17)
<@nirik:matrix.scrye.com>
16:00:48
morning everyone
<@nirik:matrix.scrye.com>
16:00:55
!hi
<@zodbot:fedora.im>
16:00:56
Kevin Fenzi (kevin) - he / him / his
<@pcreech:matrix.org>
16:01:08
Mornin!
<@smilner:fedora.im>
16:01:38
!hi
<@zodbot:fedora.im>
16:01:39
None (smilner)
<@jnsamyak:matrix.org>
16:01:57
!hi
<@zodbot:fedora.im>
16:01:59
Samyak Jain (jnsamyak) - he / him / his
<@Zlopez:matrix.org>
16:02:34
!hi
<@zodbot:fedora.im>
16:02:37
Michal Konecny (zlopez)
<@carlwgeorge:matrix.org>
16:02:39
!hi
<@zodbot:fedora.im>
16:02:40
Carl George (carlwgeorge) - he / him / his
<@james:fedora.im>
16:02:43
!hi
<@zodbot:fedora.im>
16:02:44
James Antill (james)
<@lenkaseg:matrix.org>
16:02:54
!hello
<@zodbot:fedora.im>
16:02:56
No Fedora Accounts users have the @lenkaseg:matrix.org Matrix Account defined
<@pcreech:matrix.org>
16:04:33
!topic New folks introductions
<@pcreech:matrix.org>
16:04:33
!info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
<@pcreech:matrix.org>
16:04:33
!info Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/
<@pcreech:matrix.org>
16:04:48
Any new folks? I've had some internet issues so haven't been able to check for any intro e-mails
<@nirik:matrix.scrye.com>
16:06:20
Every day is a new day. Does that count as new? ;)
<@pcreech:matrix.org>
16:06:47
I mean... as an adhd person who's brain re-bootstraps every morning... i'd say so :D
<@Zlopez:matrix.org>
16:06:56
I didn't saw any introduction e-mail in infra mailing list last week
<@Zlopez:matrix.org>
16:07:11
But that doesn't mean there isn't anybody new around
<@seddik:fedora.im>
16:07:22
!hi
<@zodbot:fedora.im>
16:07:23
seddik alaouiismaili (seddik)
<@carlwgeorge:matrix.org>
16:08:19
i guess i'm new, i don't remember if i've attended this meeting before
<@pcreech:matrix.org>
16:08:43
Welcome in!
<@pcreech:matrix.org>
16:09:05
Care to tell us a little about yourself?
<@carlwgeorge:matrix.org>
16:11:24
the cpe folks know me, i'm on their team. but for anyone else, howdy, my name is carl. i'm an rpm nerd, been doing it since 2014. i work for red hat since 2019, primarily on epel (this team https://communityblog.fedoraproject.org/cpe-to-staff-epel-work/ )
<@Zlopez:matrix.org>
16:12:12
Welcome on this meeting :-)
<@nirik:matrix.scrye.com>
16:12:15
Hi Carl. 👋
<@pcreech:matrix.org>
16:12:39
Awesome! Thank you!
<@pcreech:matrix.org>
16:13:05
Ok, next up, chair time!
<@pcreech:matrix.org>
16:13:07
!info magic eight ball says:
<@pcreech:matrix.org>
16:13:07
!info chair 2024-08-01 pcreech
<@pcreech:matrix.org>
16:13:07
!info chair 2024-08-08 - cancelled (Flock)
<@pcreech:matrix.org>
16:13:07
!topic Next chair
<@pcreech:matrix.org>
16:13:07
!info chair 2024-08-15 - ???
<@pcreech:matrix.org>
16:13:35
Who wants to be up for the 15th?
<@nirik:matrix.scrye.com>
16:13:58
I guess I can if no one else... happy to defer to anyone else tho. ;)
<@pcreech:matrix.org>
16:16:52
nirik it is!
<@pcreech:matrix.org>
16:17:01
!info chair 2024-08-15 - nirik
<@pcreech:matrix.org>
16:18:29
!info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
<@pcreech:matrix.org>
16:18:29
!info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
<@pcreech:matrix.org>
16:18:29
!info Remaining issues with new mailman3 deployment are being investigated
<@pcreech:matrix.org>
16:18:29
!info Fedora Infra weekly meeting is cancelled for 2024-08-08 because of Flock 2024
<@pcreech:matrix.org>
16:18:29
!info Flock 2024 will happen from 7-11 Aug 2024. See you there!
<@pcreech:matrix.org>
16:18:29
!info Fedora 41 Mass rebuild is done
<@pcreech:matrix.org>
16:18:29
!topic announcements and information
<@Zlopez:matrix.org>
16:19:29
!info PDC is now stopped in Fedora Infrastructure, we are working on remaining things tied to it
<@nirik:matrix.scrye.com>
16:21:17
Looks like just one thing left (I hope)
<@pcreech:matrix.org>
16:21:45
Any other announcements or information before moving on?
<@pcreech:matrix.org>
16:22:30
!info ??? is on call from 2024-08-02 to 2024-08-08
<@pcreech:matrix.org>
16:22:30
!info ??? is on call from 2024-08-09 to 2024-08-15
<@pcreech:matrix.org>
16:22:30
!info zlopez is on call from 2024-07-26 to 2024-08-01
<@pcreech:matrix.org>
16:22:30
<@pcreech:matrix.org>
16:22:30
##!oncall add <fas_name> (do not paste now, doesn't work yet)
<@pcreech:matrix.org>
16:22:30
!info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
<@pcreech:matrix.org>
16:22:30
!info https://fedoraproject.org/wiki/Infrastructure/Oncall
<@pcreech:matrix.org>
16:22:30
!topic Oncall
<@pcreech:matrix.org>
16:23:13
Not sure how flock would affect the oncalls for those ??? entries
<@nirik:matrix.scrye.com>
16:23:41
well, we could just have no one and people can file tickets.
<@nirik:matrix.scrye.com>
16:23:58
or I can take it and... just be away so they file tickets. ;)
<@Zlopez:matrix.org>
16:24:23
Most of us will be on the Flock
<@pcreech:matrix.org>
16:25:02
back to back nobody then? (i assume with travel to + flock + travel home would be a lot for most people those two weeks0
<@nirik:matrix.scrye.com>
16:26:31
yeah, lets just set it to no one...
<@nirik:matrix.scrye.com>
16:26:38
I think it says to file a ticket then
<@pcreech:matrix.org>
16:26:54
<@pcreech:matrix.org>
16:26:54
!info no one is on call from 2024-08-09 to 2024-08-15
<@pcreech:matrix.org>
16:26:54
!info no one is on call from 2024-08-02 to 2024-08-08
<@pcreech:matrix.org>
16:27:05
done :D
<@pcreech:matrix.org>
16:27:15
and that way, maybe nirik won't have so many noisy pings
<@pcreech:matrix.org>
16:27:37
!info Summary of last week: (from current oncall)
<@pcreech:matrix.org>
16:27:41
you're up zlopez!
<@Zlopez:matrix.org>
16:28:09
I didn't saw any ping whole week
<@Zlopez:matrix.org>
16:28:21
It was quiet on that regard
<@Zlopez:matrix.org>
16:28:42
And I already set oncall to no one, so let's see how it works
<@Zlopez:matrix.org>
16:28:46
!oncall
<@zodbot:fedora.im>
16:28:48
No one from Fedora Infrastructure is currently on call
<@pcreech:matrix.org>
16:29:00
nice
<@pcreech:matrix.org>
16:29:18
!info https://nagios.fedoraproject.org/nagios
<@pcreech:matrix.org>
16:29:18
!topic Monitoring discussion [nirik]
<@pcreech:matrix.org>
16:29:18
!info Go over existing out items and fix
<@nirik:matrix.scrye.com>
16:29:31
so lets see...
<@nirik:matrix.scrye.com>
16:29:52
since last week I managed to get 2 down boxes to boot again...
<@nirik:matrix.scrye.com>
16:30:27
we are down to 4 down hosts. 2 of them are aarch64 boxes that need a onsite, 1 has memory problems, need to decide if we just retire it, or can replace memory, and the last one has a bad power supply
<@nirik:matrix.scrye.com>
16:30:45
we also have a staging virthost that lost a disk
<@nirik:matrix.scrye.com>
16:31:19
There's an outstanding fedora planet one because it's not sending messages.
<@nirik:matrix.scrye.com>
16:31:36
There's a ticket on that one. phsmoura has been on PTO, but if anyone else wants to work on that, please do.
<@nirik:matrix.scrye.com>
16:31:57
and ipa01.stg has a disk alert... haven't looked as to why yet. ;)
<@Zlopez:matrix.org>
16:32:24
I'm on the machine and it has really small `/` partition
<@nirik:matrix.scrye.com>
16:32:28
And thats all the current alerts.
<@nirik:matrix.scrye.com>
16:32:45
ah, well, we could grow it?
<@nirik:matrix.scrye.com>
16:32:58
or reinstall (as long as other servers can replicate back to it)
<@Zlopez:matrix.org>
16:33:34
Newermind, I'm on the wrong machine :-D
<@nirik:matrix.scrye.com>
16:33:47
ha. oops.
<@nirik:matrix.scrye.com>
16:34:02
anyhow, thats the alerts... hopefully it will be quiet this next week.
<@pcreech:matrix.org>
16:34:09
if it's ipa01.stg.iad2.fedoraproject.org, looks like it's more an inode issue than free space
<@Zlopez:matrix.org>
16:34:22
Plenty of tmux windows open :-D
<@nirik:matrix.scrye.com>
16:35:02
pcreech: note that on nagios 99% inodes means... 99% _free_ inodes. ;)
<@pcreech:matrix.org>
16:35:14
ah. welp. :D
<@nirik:matrix.scrye.com>
16:35:19
it's that it's over 85% full on disk
<@pcreech:matrix.org>
16:35:21
it's been a while since i've nagios'ed
<@pcreech:matrix.org>
16:35:35
nevermind me :D
<@nirik:matrix.scrye.com>
16:35:49
it confuses basically everyone.
<@Zlopez:matrix.org>
16:36:14
I'm on correct machine now and only thing that stands out is `httpd` log around 600 MB
<@Zlopez:matrix.org>
16:37:13
The access log has 60 MB each day
<@Zlopez:matrix.org>
16:37:26
Same for ssl_request_log
<@Zlopez:matrix.org>
16:37:39
I'm not sure if those are compressed, let me check
<@nirik:matrix.scrye.com>
16:38:00
anyhow, we don't need to fix it in meeting. ;) Nothing else nagios wise for me.
<@Zlopez:matrix.org>
16:38:04
They are not compressed
<@Zlopez:matrix.org>
16:38:33
I can update the logrotate to compress them as well
<@pcreech:matrix.org>
16:38:46
next up, let's talk about RHEL :D
<@pcreech:matrix.org>
16:38:49
!ticket https://pagure.io/fedora-infrastructure/issue/11815
<@pcreech:matrix.org>
16:38:49
!info Check the progress on tickets related to RHEL7 EOL (end of June)
<@pcreech:matrix.org>
16:38:49
!topic RHEL7 EOL
<@Zlopez:matrix.org>
16:39:17
There are still few things missing here
<@Zlopez:matrix.org>
16:39:37
* github2fedmsg
<@Zlopez:matrix.org>
16:39:37
* PDC
<@Zlopez:matrix.org>
16:39:37
* fedimg
<@Zlopez:matrix.org>
16:40:50
PDC is currently stopped and will be decomissioned soon
<@pcreech:matrix.org>
16:40:55
Looks like last comment on the ticket was about a month ago... any value in adding another comment with an updated state? or is it fairly similar anyways?
<@Zlopez:matrix.org>
16:41:21
It's still the same
<@Zlopez:matrix.org>
16:42:17
github2fedmsg is being replaced and hopefully we will have the first version of webhook2fedmsg running soon
<@nirik:matrix.scrye.com>
16:42:21
which tickets?
<@nirik:matrix.scrye.com>
16:42:26
oh, the overall one?
<@nirik:matrix.scrye.com>
16:43:13
On fedimg, I hope to get with David Duncan and sort out the perms.... will try and sync with him.
<@Zlopez:matrix.org>
16:43:32
That is the only thing missing I think
<@nirik:matrix.scrye.com>
16:44:37
well, with that we can test in stg and fix things up and then roll to prod
<@nirik:matrix.scrye.com>
16:44:59
There's still 2 virthosts... I just pinged again both sites about the onsite stuff I need.
<@nirik:matrix.scrye.com>
16:47:19
getting close, these things are just pesky
<@Zlopez:matrix.org>
16:47:42
Nice :-)
<@pcreech:matrix.org>
16:48:32
Cool, sounds like that's about it for the rhel stuff then :D
<@pcreech:matrix.org>
16:48:34
!link https://pagure.io/fedora-infrastructure/issues?status=Open&tags=blocked&priority=0&close_status=
<@pcreech:matrix.org>
16:48:34
!info Check if any blocked ticket is unblocked
<@pcreech:matrix.org>
16:48:34
!topic Revisit blocked tickets
<@pcreech:matrix.org>
16:49:01
Any blocked things with updates?
<@nirik:matrix.scrye.com>
16:49:15
nothing unblocked that I can see...
<@nirik:matrix.scrye.com>
16:49:28
possibly we could do the dmarc setting for all lists now
<@pcreech:matrix.org>
16:50:55
Cool. Seeing as we have 10 minutes left... skipping to open floor then unless someone has an opinion otherwise
<@pcreech:matrix.org>
16:51:34
!topic Open Floor
<@pcreech:matrix.org>
16:51:52
Anyone have anything to chat about during open floor?
<@Zlopez:matrix.org>
16:52:27
I fixed the ipa01.stg, but I don't understand why there is compressdelay in logrotate, nothing was compressed for httpd
<@nirik:matrix.scrye.com>
16:53:19
from the man page:
<@nirik:matrix.scrye.com>
16:53:24
log file for some time."
<@nirik:matrix.scrye.com>
16:53:24
" It can be used when some program
<@nirik:matrix.scrye.com>
16:53:24
cannot be told to close its logfile and thus might continue writing to the previous
<@Zlopez:matrix.org>
16:54:01
I compressed that manually
<@nirik:matrix.scrye.com>
16:54:01
so perhaps ipa does this somehow?
<@Zlopez:matrix.org>
16:54:26
Could be
<@Zlopez:matrix.org>
16:56:08
But the next big thing is /var/log/journal, it has around 1 GB
<@naraiank:fedora.im>
16:57:04
!hi
<@zodbot:fedora.im>
16:57:05
Naraian K (naraiank) - he / him / his
<@nirik:matrix.scrye.com>
16:58:09
hey The Exorcist
<@nirik:matrix.scrye.com>
16:58:23
anyhow, I don't think I have anything for open floor...
<@naraiank:fedora.im>
16:58:24
Can't memory be added on hot server? I don't think we can but curious to know if it is possible..
<@nirik:matrix.scrye.com>
16:58:30
safe travels to everyone going to flock
<@naraiank:fedora.im>
16:59:03
Can't memory be added on hot server? I don't think we can, but curious to know if it is possible..
<@nirik:matrix.scrye.com>
16:59:15
you mean disk space for ipa01.stg? yeah, we could grow the disk... just a bit more complicated than fixing whats using it. ;)
<@pcreech:matrix.org>
17:00:06
!endmeeting