<@Zlopez:matrix.org>
16:00:43
!startmeeting Infrastructure (2024-10-31)
<@meetbot:fedora.im>
16:00:46
Meeting started at 2024-10-31 16:00:43 UTC
<@meetbot:fedora.im>
16:00:46
The Meeting name is 'Infrastructure (2024-10-31)'
<@Zlopez:matrix.org>
16:00:53
!info About our team: https://docs.fedoraproject.org/en-US/cpe/
<@Zlopez:matrix.org>
16:00:53
!info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
<@Zlopez:matrix.org>
16:00:53
!meetingname infrastructure
<@Zlopez:matrix.org>
16:00:53
!topic ahoy
<@Zlopez:matrix.org>
16:00:53
!chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp
<@Zlopez:matrix.org>
16:00:53
!info Agenda is at: https://board.net/p/fedora-infra
<@meetbot:fedora.im>
16:00:54
The Meeting Name is now infrastructure
<@nirik:matrix.scrye.com>
16:01:13
👻spooky morning everyone 👻
<@Zlopez:matrix.org>
16:01:49
It's dark outside here :-D
<@Zlopez:matrix.org>
16:02:11
But spooky halloween everyone
<@nirik:matrix.scrye.com>
16:02:21
🌁 and raining here, but it is morning.
<@Zlopez:matrix.org>
16:02:33
I will be your seriall killer for this meeting 🗡️
<@Zlopez:matrix.org>
16:02:48
I will be your serial killer for this meeting 🗡️
<@seddik:fedora.im>
16:03:24
.hi
<@seddik:fedora.im>
16:03:40
!hi
<@zodbot:fedora.im>
16:03:41
seddik alaouiismaili (seddik)
<@Zlopez:matrix.org>
16:03:44
Which service will be my first victim today :-D
<@nirik:matrix.scrye.com>
16:05:16
could be any one? do we have any walking alone in the dark?
<@james:fedora.im>
16:05:28
All of them
<@phsmoura:fedora.im>
16:05:32
hello
<@thisisyaash:fedora.im>
16:06:03
!hi
<@zodbot:fedora.im>
16:06:04
Yashwanth Rathakrishnan (thisisyaash)
<@Zlopez:matrix.org>
16:06:49
Hi @thisisyaash:fedora.im, thanks for the PR for statusfpo :-)
<@thisisyaash:fedora.im>
16:07:19
thanks for merging it :)
<@Zlopez:matrix.org>
16:07:34
So let's go to the first victim of this meeting
<@Zlopez:matrix.org>
16:07:35
!info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
<@Zlopez:matrix.org>
16:07:35
!topic New folks introductions
<@Zlopez:matrix.org>
16:07:35
!info Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/
<@Zlopez:matrix.org>
16:08:06
So who is here for first time?
<@nirik:matrix.scrye.com>
16:09:49
I was here last week, but it's the first time I am here this week. ;)
<@Zlopez:matrix.org>
16:10:44
Let's move on, my first victim is eluding me
<@Zlopez:matrix.org>
16:10:52
!topic Next chair
<@Zlopez:matrix.org>
16:10:52
!info magic eight ball says:
<@Zlopez:matrix.org>
16:10:52
!info chair 2024-10-31 - zlopez
<@Zlopez:matrix.org>
16:10:52
!info chair 2024-11-07 - ?
<@Zlopez:matrix.org>
16:10:52
!info chair 2024-11-14 - ?
<@Zlopez:matrix.org>
16:11:41
So who wants to be my next victim, I wanted to say chair the next meeting
<@Zlopez:matrix.org>
16:13:48
Any volunteer?
<@nirik:matrix.scrye.com>
16:14:48
I suppose I can if no one else is wanting it.
<@Zlopez:matrix.org>
16:16:06
We should probably try to clean the chairs and just rotate through them
<@Zlopez:matrix.org>
16:16:23
What do you think?
<@Zlopez:matrix.org>
16:17:15
But for now I will assign it to you
<@Zlopez:matrix.org>
16:17:17
!info chair 2024-11-07 - nirik
<@nirik:matrix.scrye.com>
16:17:44
It's a great way for new folks to get more involved too... but yeah, if it's always the same people we should figure out something else. ;)
<@seddik:fedora.im>
16:18:15
i can take 14-11
<@seddik:fedora.im>
16:18:21
I can take 14-11
<@Zlopez:matrix.org>
16:18:39
We can always assign the chair to volunteer but if nobody is volunteering just pick the next one on chair list
<@Zlopez:matrix.org>
16:18:51
@seddik:fedora.im It's yours :-)
<@Zlopez:matrix.org>
16:19:00
!info chair 2024-11-14 - saibug
<@Zlopez:matrix.org>
16:19:42
I think we are covered for now
<@Zlopez:matrix.org>
16:19:55
Let's go haunting somewhere else :-D
<@Zlopez:matrix.org>
16:20:13
!info CLE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0900 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
<@Zlopez:matrix.org>
16:20:13
!info CLE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
<@Zlopez:matrix.org>
16:20:13
!topic announcements and information
<@Zlopez:matrix.org>
16:20:27
!info F41 freeze is over
<@nirik:matrix.scrye.com>
16:20:30
!info we are out of f41 final freeze now
<@james:fedora.im>
16:21:11
Change all the things!
<@Zlopez:matrix.org>
16:21:19
!info dist-git has issues returning 502/503 - we are aware of that and working on it
<@nirik:matrix.scrye.com>
16:22:00
Yeah, thats a puzzling one. ;(
<@Zlopez:matrix.org>
16:22:36
I will continue investigating that tomorrow, spend some time on that today, but got distracted by other things
<@nirik:matrix.scrye.com>
16:23:15
!info pagure.io outage in a while (hopefully short)
<@Zlopez:matrix.org>
16:25:17
Let's continue to next one
<@Zlopez:matrix.org>
16:25:27
!info ? is on call from 2024-11-08 to 2024-11-14
<@Zlopez:matrix.org>
16:25:27
!info ? is on call from 2024-11-01 to 2024-11-07
<@Zlopez:matrix.org>
16:25:27
!topic Oncall
<@Zlopez:matrix.org>
16:25:27
!info https://fedoraproject.org/wiki/Infrastructure/Oncall
<@Zlopez:matrix.org>
16:25:27
!info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
<@Zlopez:matrix.org>
16:25:27
!info lenkaseg is on call from 2024-10-25 to 2024-10-31
<@Zlopez:matrix.org>
16:25:38
I will take the next one
<@Zlopez:matrix.org>
16:25:49
!info zlopez is on call from 2024-11-01 to 2024-11-07
<@Zlopez:matrix.org>
16:27:16
Hm, there was still zardian in oncall
<@Zlopez:matrix.org>
16:27:22
!infra oncall list
<@zodbot:fedora.im>
16:27:23
● @Zlopez:matrix.org (zlopez) Current Time for them: 17:27 (Europe/Prague)
<@zodbot:fedora.im>
16:27:23
The following people are oncall:
<@zodbot:fedora.im>
16:27:23
<@zodbot:fedora.im>
16:27:23
If they do not respond, please file a ticket (https://pagure.io/fedora-infrastructure/issues)
<@Zlopez:matrix.org>
16:27:31
I added myself now
<@Zlopez:matrix.org>
16:28:05
Anybody wants to take oncall for 2024-11-08 to 2024-11-14
<@Zlopez:matrix.org>
16:28:28
?
<@nirik:matrix.scrye.com>
16:28:37
I could
<@Zlopez:matrix.org>
16:29:41
!info nirik is on call from 2024-11-08 to 2024-11-14
<@Zlopez:matrix.org>
16:30:01
I think we can use the same process as for chairs in future
<@Zlopez:matrix.org>
16:30:11
I will look into that for next week
<@nirik:matrix.scrye.com>
16:30:37
feel free to propose a new process... ;)
<@nirik:matrix.scrye.com>
16:31:04
side note: I see status is failing to build now, not due to any of the recent changes, seems to be a github actions change?
<@Zlopez:matrix.org>
16:31:07
Will do and we can try it next week
<@Zlopez:matrix.org>
16:31:31
I assume they changed the container they are using for running them
<@nirik:matrix.scrye.com>
16:31:39
not sure.
<@Zlopez:matrix.org>
16:31:39
I was looking at that as well
<@nirik:matrix.scrye.com>
16:32:00
well, we should discuss the change before just doing it, having a concrete proposal would be good for that
<@Zlopez:matrix.org>
16:32:11
It should be just change of the pip command we are using as one of the arguments is not recognised anymore
<@Zlopez:matrix.org>
16:32:29
Where should I submit the proposal?
<@Zlopez:matrix.org>
16:32:39
Infra mailing list?
<@nirik:matrix.scrye.com>
16:32:47
right, but ryan had to add that option I think to make it work before... :)
<@nirik:matrix.scrye.com>
16:32:53
sure! or discussion?
<@Zlopez:matrix.org>
16:33:47
I don't want to automate that just make the decision process easier :-)
<@Zlopez:matrix.org>
16:33:57
Let me submit it to both
<@Zlopez:matrix.org>
16:35:39
Let us continue
<@Zlopez:matrix.org>
16:35:44
!info Summary of last week: (from current oncall)
<@Zlopez:matrix.org>
16:36:11
As @lenkaseg:matrix.org wasn't even assigned to oncall I don't know if she did notice any
<@Zlopez:matrix.org>
16:36:24
I didn't saw any oncall ping in last week
<@nirik:matrix.scrye.com>
16:37:00
yeah. I don't recall any
<@Zlopez:matrix.org>
16:37:45
Let's move on then
<@Zlopez:matrix.org>
16:37:53
!topic Monitoring discussion [nirik]
<@Zlopez:matrix.org>
16:37:53
!info Go over existing items and fix them
<@Zlopez:matrix.org>
16:37:53
!info https://nagios.fedoraproject.org/nagios
<@nirik:matrix.scrye.com>
16:38:24
lets see.
<@nirik:matrix.scrye.com>
16:39:00
theres a fmn alert, but thats Aurélien B working on the caching
<@nirik:matrix.scrye.com>
16:39:16
There's 2 down aarch64 machines, but I am going to shuffle things around and they should be gone soon
<@nirik:matrix.scrye.com>
16:39:35
pagure02 disk space (but that should be fixed in an hour or so)
<@Zlopez:matrix.org>
16:39:40
The FMN is now processing again, as the amount of messages in queue is getting lower
<@nirik:matrix.scrye.com>
16:39:54
disk space on ipa01.stg I haven't looked at... logs of logs?
<@nirik:matrix.scrye.com>
16:40:26
pagure is being hit hard again, I'm going to block some more ips
<@Zlopez:matrix.org>
16:40:27
I remember that there were some logs that weren't compressed, so I just did the compression
<@Zlopez:matrix.org>
16:40:54
It seems that logrotate doesn't work as it should
<@nirik:matrix.scrye.com>
16:41:05
:(
<@nirik:matrix.scrye.com>
16:41:22
anyhow, thats all the current alerts... hopefully we can have some quiet time soon...
<@Zlopez:matrix.org>
16:41:34
Or maybe there is just more traffic on staging as the dist-git test instances are using staging accounts
<@Zlopez:matrix.org>
16:42:25
One thing to add I restarted openqa01 today as it was stuck on some CPU stall and didn't wanted to even shutdown
<@Zlopez:matrix.org>
16:42:52
But it's a VM, so I just shut it down forcefully and started it again
<@nirik:matrix.scrye.com>
16:43:00
ok. Odd.
<@Zlopez:matrix.org>
16:43:14
It's running OK now
<@Zlopez:matrix.org>
16:43:39
What do we want to do now?
<@nirik:matrix.scrye.com>
16:43:49
I also had to muck with bvmhost-x86-03 yesterday... it went unresponsive and then had a disk drop out and had some raid issues... took a while to get it back. It's up and down a disk, but I need to look and see if that disk is really bad or not, because some things say it's fine.
<@nirik:matrix.scrye.com>
16:44:01
sorry, thats bvmhost-x86-03.stg
<@Zlopez:matrix.org>
16:44:52
I saw the messages in noc channel, it looked really strange
<@Zlopez:matrix.org>
16:45:17
Let us vote by giving thumbs up
<@Zlopez:matrix.org>
16:45:32
Revisit blocked tickets
<@Zlopez:matrix.org>
16:45:37
Fedora Infra backlog refinement
<@Zlopez:matrix.org>
16:45:47
Think of learning topic
<@Zlopez:matrix.org>
16:45:49
Open floor
<@Zlopez:matrix.org>
16:45:53
End early
<@nirik:matrix.scrye.com>
16:46:12
we don't have too much time left...
<@nirik:matrix.scrye.com>
16:46:31
we could hit on blocked really quick then open floor?
<@Zlopez:matrix.org>
16:46:38
So let's skip to open floor
<@Zlopez:matrix.org>
16:46:44
!topic Open Floor
<@thisisyaash:fedora.im>
16:47:41
actually i noticed that the recently merged PR in github is failing the checks
<@nirik:matrix.scrye.com>
16:48:12
for status? yeah, it seems to have changed and no longer accepts a flag we were passing to pip
<@thisisyaash:fedora.im>
16:48:14
actually i noticed that the recently merged PR in github is failing the checks (the check is failed)
<@Zlopez:matrix.org>
16:48:20
Today on Fedora prioritisation meeting we talked about zabbix and what is the situation around it
<@Zlopez:matrix.org>
16:48:37
It's not related to your PR
<@nirik:matrix.scrye.com>
16:48:42
I was gonna try and get to that, but it was too early for me. ;)
<@thisisyaash:fedora.im>
16:49:32
i see!
<@thisisyaash:fedora.im>
16:49:32
i thought i messed up bcz after my PR getting merged
<@thisisyaash:fedora.im>
16:49:32
<@thisisyaash:fedora.im>
16:49:32
and the check is failed
<@thisisyaash:fedora.im>
16:49:32
so that's why!
<@Zlopez:matrix.org>
16:50:21
From what I understand the zabbix work is now more or less waiting for us to start using it and start converting the checks we have for nagios
<@nirik:matrix.scrye.com>
16:51:03
well, it's still really noisy, we need to make it not alert every 2 minutes. ;)
<@nirik:matrix.scrye.com>
16:51:28
althought I guess james did some of that. I should look again.
<@nirik:matrix.scrye.com>
16:51:50
There's also a 7.0 version out we should look at to see if we can move to
<@Zlopez:matrix.org>
16:51:55
It is really noisy that's true
<@nirik:matrix.scrye.com>
16:52:52
but then yeah, we should look at whats in nagios and make sure it's in zabbix.
<@nirik:matrix.scrye.com>
16:53:18
I'm not sure if zabbix is monitoring all the hosts nagios is? but it might be.
<@Zlopez:matrix.org>
16:54:03
It would be nice to check
<@nirik:matrix.scrye.com>
16:54:07
perhaps we could break up things into milestones or something and track them?
<@nirik:matrix.scrye.com>
16:54:27
I don't know if thats more tickets, or just some planning in the existing one(s)...
<@Zlopez:matrix.org>
16:54:49
I would like as to move this forward, but there are so much things happening it's hard to move forward everything
<@nirik:matrix.scrye.com>
16:55:15
very much so yeah.
<@nirik:matrix.scrye.com>
16:55:36
lots of big big things on the horizon (mostly next year, but need some planning this year)
<@Zlopez:matrix.org>
16:55:55
There are so many interesting things happening in Fedora and not only in infrastructure, it's hard to keep track of everything :-)
<@nirik:matrix.scrye.com>
16:56:06
it's a firehose for sure.
<@Zlopez:matrix.org>
16:56:25
We should at least finish the RHEL 7 EOL till end of year
<@nirik:matrix.scrye.com>
16:56:46
yes, I would very much like to finish that off.
<@Zlopez:matrix.org>
16:56:58
I need to poke the work on webhook2fedmsg and announce sunset of github2fedmsg
<@nirik:matrix.scrye.com>
16:57:14
I think there was a article / announcement?
<@Zlopez:matrix.org>
16:57:38
Yes, I scheduled that one
<@nirik:matrix.scrye.com>
16:57:54
https://communityblog.fedoraproject.org/announcing-webhook-to-fedora-messaging/
<@Zlopez:matrix.org>
16:58:31
I see only one request for migration
<@Zlopez:matrix.org>
16:58:52
https://pagure.io/fedora-infra/w2fm-migration/issues
<@nirik:matrix.scrye.com>
16:59:01
yeah, thats not great. ;( Probibly should reach out to each person that has github2fedmsg enabled.
<@nirik:matrix.scrye.com>
16:59:25
also, coming up: we need to move some f39 things to f40 before f39 goes eol
<@Zlopez:matrix.org>
16:59:39
+1 on that
<@nirik:matrix.scrye.com>
16:59:45
and also we need to setup process for communishift cleanup/announce/etc
<@Zlopez:matrix.org>
17:00:05
@dkirwan:fedora.im is working on that
<@nirik:matrix.scrye.com>
17:00:24
yep.
<@nirik:matrix.scrye.com>
17:00:49
we were gonna do that at f39 eol tho, so 1 month left and need to give people time to react
<@Zlopez:matrix.org>
17:00:50
I will check with him
<@Zlopez:matrix.org>
17:02:14
Could we get the list of users of github2fedmsg from db?
<@Zlopez:matrix.org>
17:02:23
!endmeeting