<@patrikp:matrix.org>
16:00:10
!startmeeting Infrastructure (2024-09-26)
<@meetbot:fedora.im>
16:00:11
Meeting started at 2024-09-26 16:00:10 UTC
<@meetbot:fedora.im>
16:00:12
The Meeting name is 'Infrastructure (2024-09-26)'
<@patrikp:matrix.org>
16:00:17
!meetingname infrastructure
<@meetbot:fedora.im>
16:00:18
The Meeting Name is now infrastructure
<@patrikp:matrix.org>
16:00:23
!chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp
<@patrikp:matrix.org>
16:00:29
!info Agenda is at: https://board.net/p/fedora-infra
<@patrikp:matrix.org>
16:00:36
!info About our team: https://docs.fedoraproject.org/en-US/cpe/
<@patrikp:matrix.org>
16:00:42
!info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
<@nirik:matrix.scrye.com>
16:00:47
morning
<@patrikp:matrix.org>
16:00:49
!topic ahoy
<@james:fedora.im>
16:00:50
!hi
<@zodbot:fedora.im>
16:00:51
James Antill (james)
<@patrikp:matrix.org>
16:00:52
Hello!
<@Zlopez:matrix.org>
16:00:55
!hi
<@zodbot:fedora.im>
16:00:57
Michal Konecny (zlopez)
<@patrikp:matrix.org>
16:01:25
!topic New folks introductions
<@patrikp:matrix.org>
16:01:30
!info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
<@patrikp:matrix.org>
16:01:35
!info Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/
<@patrikp:matrix.org>
16:01:45
Do we have anyone new joining us today? Come and say hi. 👋
<@Zlopez:matrix.org>
16:03:04
@patrikp:matrix.org Feel free to copy over multiple lines, only line that needs to be pasted separately is !startmeeting
<@patrikp:matrix.org>
16:03:36
Does it? Alright, let me try. Looks like nobody new is here, we can move on...
<@patrikp:matrix.org>
16:03:49
!topic Next chair
<@patrikp:matrix.org>
16:03:49
!info chair 2024-10-03 - lenkaseg
<@patrikp:matrix.org>
16:03:49
!info chair 2024-09-26 - patrikp
<@patrikp:matrix.org>
16:03:49
!info magic eight ball says:
<@patrikp:matrix.org>
16:03:49
!info chair 2024-10-10 - ???
<@patrikp:matrix.org>
16:04:04
Any takers for October 10?
<@Zlopez:matrix.org>
16:04:20
I can take it
<@patrikp:matrix.org>
16:04:37
!info chair 2024-10-10 - zlopez
<@patrikp:matrix.org>
16:05:00
!info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
<@patrikp:matrix.org>
16:05:00
!info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
<@patrikp:matrix.org>
16:05:00
!topic announcements and information
<@patrikp:matrix.org>
16:05:46
There are announcements about F41 Beta but I suppose they were already covered last week.
<@Zlopez:matrix.org>
16:05:58
!Info CPE Update for Q2 published https://communityblog.fedoraproject.org/cpe-update-q2-2024/
<@patrikp:matrix.org>
16:05:58
Any other announcements?
<@nirik:matrix.scrye.com>
16:06:25
!info nirik out on pto tomorrow
<@Zlopez:matrix.org>
16:06:50
It seems meetbot skipped my message
<@patrikp:matrix.org>
16:07:05
Is it because of the capital I?
<@nirik:matrix.scrye.com>
16:07:35
could be
<@patrikp:matrix.org>
16:07:38
!info CPE Update for Q2 published https://communityblog.fedoraproject.org/cpe-update-q2-2024/
<@Zlopez:matrix.org>
16:07:49
Didn't realized that
<@patrikp:matrix.org>
16:08:22
Any other announcements?
<@patrikp:matrix.org>
16:08:37
Moving on...
<@patrikp:matrix.org>
16:08:44
!info https://fedoraproject.org/wiki/Infrastructure/Oncall
<@patrikp:matrix.org>
16:08:44
!info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
<@patrikp:matrix.org>
16:08:44
!topic Oncall
<@patrikp:matrix.org>
16:08:53
!info lenkaseg is on call from 2024-09-20 to 2024-09-27
<@patrikp:matrix.org>
16:08:53
!info ? is on call from 2024-09-27 to 2024-10-03
<@patrikp:matrix.org>
16:09:13
Any takers for the next week in regards to oncall?
<@nirik:matrix.scrye.com>
16:10:02
I think I forgot to move it to lenkaseg last week, so I have had it two weeks. ;)
<@patrikp:matrix.org>
16:10:30
Ah, alright, so Lenka has Oct 3 - 10?
<@nirik:matrix.scrye.com>
16:11:27
not sure, wouldn't want to sign her up for something without checking.
<@Zlopez:matrix.org>
16:11:43
I don't see her on the meeting, so I'm not sure if it's fine to assign the oncall to her when she doesn't know about it
<@patrikp:matrix.org>
16:11:53
Good point.
<@patrikp:matrix.org>
16:12:14
Any takers for Sep 27 to Oct 3?
<@Zlopez:matrix.org>
16:12:57
I can take it
<@Zlopez:matrix.org>
16:13:17
Hopefully it will work better for me this time
<@patrikp:matrix.org>
16:13:40
!info zlopez is on call from 2024-09-27 to 2024-10-03
<@patrikp:matrix.org>
16:14:01
!info Summary of last week: (from current oncall)
<@nirik:matrix.scrye.com>
16:14:37
lets see... several oncall calls.
<@nirik:matrix.scrye.com>
16:14:44
some for read only koji builders.
<@nirik:matrix.scrye.com>
16:15:11
one from adamw about proxy01.stg not responding...
<@leo:fedora.im>
16:15:16
hey!
<@lenkaseg:matrix.org>
16:16:23
Hi all!
<@nirik:matrix.scrye.com>
16:16:24
one from when bvmhost-x86-03 was unresponsive... taking down koji and registry (those vm's are on it)
<@Zlopez:matrix.org>
16:16:47
Hi @leo:fedora.im @lenkaseg:matrix.org
<@lenkaseg:matrix.org>
16:17:15
Assign the oncall to me please!
<@nirik:matrix.scrye.com>
16:17:34
actually it was two from that bvmhost thing. ;)
<@nirik:matrix.scrye.com>
16:17:38
thats all of them that I saw.
<@Zlopez:matrix.org>
16:18:03
@lenkaseg:fedora.im I already assigned it to me, but let's add you for next week
<@lenkaseg:matrix.org>
16:18:14
(Sorry, the new school year started and I'm a bit busy around this time moving kids to trainings)
<@patrikp:matrix.org>
16:18:19
Would you like Oct 3 - 10 Lenka?
<@lenkaseg:matrix.org>
16:18:26
Yup!
<@patrikp:matrix.org>
16:18:40
!info lenkaseg is on call from 2024-10-03 to 2024-10-10
<@Zlopez:matrix.org>
16:19:11
I forgot that in Spain it's starting at end of September, here the kids are already going to school for three weeks
<@patrikp:matrix.org>
16:19:48
Moving on then.
<@patrikp:matrix.org>
16:19:56
!info Go over existing items and fix them
<@patrikp:matrix.org>
16:19:56
!info https://nagios.fedoraproject.org/nagios
<@patrikp:matrix.org>
16:19:56
!topic Monitoring discussion [nirik]
<@nirik:matrix.scrye.com>
16:21:40
lets see.
<@nirik:matrix.scrye.com>
16:21:59
2 aarch64 machines down with bad memory.
<@Zlopez:matrix.org>
16:22:26
I see only one in nagios
<@nirik:matrix.scrye.com>
16:22:35
pagure02 close to disk alerting... usually this can be fixed up by nuking the autogenerated releases made for forks...
<@nirik:matrix.scrye.com>
16:22:53
buildhw-a64-23.iad2.fedoraproject.org and openqa-a64-worker01.iad2.fedoraproject.org
<@nirik:matrix.scrye.com>
16:23:28
openqa-p09-worker02.iad2.fedoraproject.org is alerting on /boot space... this machine just needs reinstalled to make /boot bigger sometime.
<@Zlopez:matrix.org>
16:23:30
Oh, I was looking at the services
<@nirik:matrix.scrye.com>
16:25:07
thats all I see.. several things got cleared this last week.
<@nirik:matrix.scrye.com>
16:25:17
the planet messages are working now. kudos phsmoura
<@nirik:matrix.scrye.com>
16:25:31
and a drive was replaced that was bad. kudos David Kirwan
<@nirik:matrix.scrye.com>
16:26:22
Oh, one more monitoring note:
<@nirik:matrix.scrye.com>
16:26:48
We now have staging zabiix sending to a staging room and production to another room...
<@nirik:matrix.scrye.com>
16:27:02
it's very noisy and we need to fix that, but if you want to see what zabbix is alerting on you can join those.
<@nirik:matrix.scrye.com>
16:27:12
#noc-zabbix-testing:fedora.im
<@Zlopez:matrix.org>
16:27:29
Already joined
<@nirik:matrix.scrye.com>
16:27:51
The staging one I think we should move. it's in the staging bots room right now.
<@nirik:matrix.scrye.com>
16:28:49
thats all on monitoring.
<@patrikp:matrix.org>
16:29:42
Alright, next topic. Now I'm a bit confused as to what should be next. Looks like we don't have a learning topic for this week.
<@patrikp:matrix.org>
16:30:02
RHEL7 EOL? Blocked tickets?
<@Zlopez:matrix.org>
16:30:02
I don't think we have
<@nirik:matrix.scrye.com>
16:30:03
yeah, we usually decide. Or a while back we used to alternate.
<@Zlopez:matrix.org>
16:30:32
We didn't do a Backlog refinement for some time
<@nirik:matrix.scrye.com>
16:30:51
sure, I'm game to do whichever...
<@patrikp:matrix.org>
16:31:02
!info Refine oldest tickets on Fedora Infra tracker
<@patrikp:matrix.org>
16:31:02
!topic Fedora Infra backlog refinement
<@patrikp:matrix.org>
16:31:02
!link https://pagure.io/fedora-infrastructure/issues?status=Open&order_key=last_updated&order=asc
<@nirik:matrix.scrye.com>
16:32:08
Zlopez: you want to run things? or patrikp? we usually go one by one and discuss.
<@Zlopez:matrix.org>
16:32:23
I can run it
<@Zlopez:matrix.org>
16:32:29
!ticket 10458
<@zodbot:fedora.im>
16:32:31
● **Last Updated:** 9 months ago
<@zodbot:fedora.im>
16:32:31
**fedora-infrastructure #10458** (https://pagure.io/fedora-infrastructure/issue/10458):**Help me move my discourse bots to production?**
<@zodbot:fedora.im>
16:32:31
<@zodbot:fedora.im>
16:32:31
● **Opened:** 2 years ago by mattdm
<@zodbot:fedora.im>
16:32:31
● **Assignee:** Not Assigned
<@Zlopez:matrix.org>
16:32:49
This one is waiting for mattdm
<@nirik:matrix.scrye.com>
16:32:51
So, this is on my list of 'projects looking for people' that I intend to send today
<@nirik:matrix.scrye.com>
16:33:07
I'd like to get someone on our side that can 'own' it and nag him about it and get the info, etc.
<@Zlopez:matrix.org>
16:33:36
That sounds good, I think we have a few tickets like that
<@Zlopez:matrix.org>
16:34:20
!ticket 12204
<@nirik:matrix.scrye.com>
16:34:21
yep.
<@zodbot:fedora.im>
16:34:22
**fedora-infrastructure #12204** (https://pagure.io/fedora-infrastructure/issue/12204):**buildvm-x86-05.iad2.fedoraproject.org: Read-only file system: '/var/tmp/koji/tasks/2642/123992642'**
<@zodbot:fedora.im>
16:34:22
<@zodbot:fedora.im>
16:34:22
● **Opened:** 6 hours ago by ppisar
<@zodbot:fedora.im>
16:34:22
● **Last Updated:** 6 hours ago
<@zodbot:fedora.im>
16:34:22
● **Assignee:** Not Assigned
<@Zlopez:matrix.org>
16:34:29
Wrong one
<@Zlopez:matrix.org>
16:34:46
!ticket 11013
<@zodbot:fedora.im>
16:34:47
<@zodbot:fedora.im>
16:34:47
● **Opened:** a year ago by adamwill
<@zodbot:fedora.im>
16:34:47
● **Last Updated:** 7 months ago
<@zodbot:fedora.im>
16:34:47
**fedora-infrastructure #11013** (https://pagure.io/fedora-infrastructure/issue/11013):**Redeploy openQA workers with consistent storage configuration**
<@zodbot:fedora.im>
16:34:47
● **Assignee:** Not Assigned
<@nirik:matrix.scrye.com>
16:34:47
so many tickets. ;)
<@nirik:matrix.scrye.com>
16:35:19
oh yeah, we should ask adamwill about this and when we can do it...
<@nirik:matrix.scrye.com>
16:35:28
I also want to upgrade the qa vmhosts.
<@nirik:matrix.scrye.com>
16:35:45
I am not sure we have the desired configuration decided yet tho.
<@Zlopez:matrix.org>
16:35:51
It's marked as low-trouble and sitting there for more than 2 years :/
<@nirik:matrix.scrye.com>
16:36:24
yeah. people or machines are always busy. ;(
<@Zlopez:matrix.org>
16:37:22
It seems that adamwill did some investigation around that
<@nirik:matrix.scrye.com>
16:37:34
how about we try and schedule it for the 8th/9th... then we have a goal. ;)
<@nirik:matrix.scrye.com>
16:37:41
(the week before final freeze starts)
<@nirik:matrix.scrye.com>
16:38:26
or at least propose that in the ticket. ;)
<@nirik:matrix.scrye.com>
16:38:47
it will probibly need to be announced since updates won't flow without gating tests.
<@Zlopez:matrix.org>
16:39:12
That sounds like good idea
<@nirik:matrix.scrye.com>
16:39:44
shall I update the ticket? or you want to?
<@Zlopez:matrix.org>
16:40:54
Could you update it, it seems you know what you want to do
<@Zlopez:matrix.org>
16:41:02
Could you update it? It seems you know what you want to do
<@nirik:matrix.scrye.com>
16:41:21
yep. I can.
<@Zlopez:matrix.org>
16:41:54
Next ticket
<@Zlopez:matrix.org>
16:42:01
!ticket 11512
<@zodbot:fedora.im>
16:42:03
● **Assignee:** Not Assigned
<@zodbot:fedora.im>
16:42:03
● **Last Updated:** 7 months ago
<@zodbot:fedora.im>
16:42:03
● **Opened:** a year ago by kevin
<@zodbot:fedora.im>
16:42:03
<@zodbot:fedora.im>
16:42:03
**fedora-infrastructure #11512** (https://pagure.io/fedora-infrastructure/issue/11512):**container registry pruning**
<@Zlopez:matrix.org>
16:42:50
Should we still consider this being an issue when most of our container workflow is ending up in quay.io?
<@nirik:matrix.scrye.com>
16:43:21
well... it's a short term problem still for us...
<@nirik:matrix.scrye.com>
16:43:30
let me get the graph
<@nirik:matrix.scrye.com>
16:44:18
I mean, it could be just from more use and perhaps we should just not worry about it too much
<@nirik:matrix.scrye.com>
16:44:47
There was someone who said they wanted to look at it, but I didn't see any updates from them
<@nirik:matrix.scrye.com>
16:45:30
so, I guess I'm ok just closing it... but we should try and push forward on the quay.io move.
<@Zlopez:matrix.org>
16:45:46
Probably didn't have spare time to look into it
<@james:fedora.im>
16:45:56
What is the rough netapp limit to expand to?
<@nirik:matrix.scrye.com>
16:47:41
looks like there's 20T more on that aggregate, but of course we couldn't allocate it all.
<@nirik:matrix.scrye.com>
16:47:51
it gets mad if you go over 90% usage
<@nirik:matrix.scrye.com>
16:48:05
in any case, there's still room.
<@Zlopez:matrix.org>
16:48:35
Let's wait with closing that till flatpaks are moved to quay.io
<@james:fedora.im>
16:48:43
yeh, I'd just worry about it more if it was 10T and a lot less at ~2 years of expansion.
<@nirik:matrix.scrye.com>
16:49:35
I think now we should just close it...
<@nirik:matrix.scrye.com>
16:50:00
FYI, there's 459 fedora flatpaks now
<@Zlopez:matrix.org>
16:50:27
Much more than I thought 👍️
<@Zlopez:matrix.org>
16:50:34
Let me close it than
<@nirik:matrix.scrye.com>
16:50:52
yselkowitz is a flatpak building superstar. ;)
<@Zlopez:matrix.org>
16:51:42
I tried to build one in past, but I hit some roadblock
<@Zlopez:matrix.org>
16:51:55
I still have 2 flatpaks on flathub
<@patrikp:matrix.org>
16:52:37
Alright, less than 8 minutes to go. You want to do one more or shall we go to open floor?
<@Zlopez:matrix.org>
16:52:48
Issue closed as Will not/Can not fix
<@Zlopez:matrix.org>
16:53:11
@nirik:matrix.scrye.com One more?
<@nirik:matrix.scrye.com>
16:53:28
sure
<@Zlopez:matrix.org>
16:53:52
!ticket 11807
<@zodbot:fedora.im>
16:53:53
● **Opened:** 7 months ago by kevin
<@zodbot:fedora.im>
16:53:53
● **Assignee:** Not Assigned
<@zodbot:fedora.im>
16:53:53
● **Last Updated:** 7 months ago
<@zodbot:fedora.im>
16:53:53
**fedora-infrastructure #11807** (https://pagure.io/fedora-infrastructure/issue/11807):**move resultsdb-ci-listener deployment pipeline**
<@zodbot:fedora.im>
16:53:53
<@Zlopez:matrix.org>
16:54:18
I think this one just fall through the cracks
<@Zlopez:matrix.org>
16:54:30
As it seems easy to do
<@nirik:matrix.scrye.com>
16:55:22
yeah, someone just needs to do it.
<@Zlopez:matrix.org>
16:56:04
I will share it on monday standup as it's a good way to learn how the s2i in openshift works and it's easy to do
<@nirik:matrix.scrye.com>
16:56:32
👍
<@Zlopez:matrix.org>
16:57:06
Or if anybody here want's to take it feel free I can guide you through it
<@Zlopez:matrix.org>
16:57:16
Or if anybody here wants to take it feel free I can guide you through it
<@patrikp:matrix.org>
16:57:32
I will assign it to myself.
<@patrikp:matrix.org>
16:58:09
And with that, we are pretty much at time. Any closing thoughts?
<@Zlopez:matrix.org>
16:58:29
Nice work with the tickets today :-)
<@patrikp:matrix.org>
16:59:16
I suppose that's it then. Have a good rest of the day everyone!
<@patrikp:matrix.org>
16:59:21
!endmeeting