<@lenkaseg:fedora.im>
16:00:01
!startmeeting Infrastructure (2024-10-03)
<@meetbot:fedora.im>
16:00:02
Meeting started at 2024-10-03 16:00:01 UTC
<@meetbot:fedora.im>
16:00:03
The Meeting name is 'Infrastructure (2024-10-03)'
<@lenkaseg:fedora.im>
16:00:12
!topic ahoy
<@lenkaseg:fedora.im>
16:00:12
!meetingname infrastructure
<@lenkaseg:fedora.im>
16:00:12
!info About our team: https://docs.fedoraproject.org/en-US/cpe/
<@lenkaseg:fedora.im>
16:00:12
!info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
<@lenkaseg:fedora.im>
16:00:12
!info Agenda is at: https://board.net/p/fedora-infra
<@lenkaseg:fedora.im>
16:00:12
!chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp
<@meetbot:fedora.im>
16:00:12
The Meeting Name is now infrastructure
<@lenkaseg:fedora.im>
16:00:23
Hi all!
<@james:fedora.im>
16:00:57
!hi
<@zodbot:fedora.im>
16:01:02
James Antill (james)
<@zardian:matrix.org>
16:01:26
Hi everyone!!
<@Zlopez:matrix.org>
16:01:45
!hi
<@zodbot:fedora.im>
16:01:56
Michal Konecny (zlopez)
<@lenkaseg:fedora.im>
16:02:52
Nice to see you all here!
<@zardian:matrix.org>
16:02:57
!hi
<@lenkaseg:fedora.im>
16:02:58
!info Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/
<@lenkaseg:fedora.im>
16:02:58
!topic New folks introductions
<@lenkaseg:fedora.im>
16:02:58
!info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
<@zodbot:fedora.im>
16:03:03
No Fedora Accounts users have the @zardian:matrix.org Matrix Account defined
<@lenkaseg:fedora.im>
16:03:18
Do we have here some new folks today?
<@lenkaseg:fedora.im>
16:03:58
If you want to say hi and share some background
<@zardian:matrix.org>
16:04:24
1st time attending this meeting. Do i need to connect my matrix account with my fedora account?
<@nirik:matrix.scrye.com>
16:04:30
morning.
<@Zlopez:matrix.org>
16:04:47
@zardian:matrix.org Welcome!
<@nirik:matrix.scrye.com>
16:04:58
zardian: nope... but you can if you want bots to award you cookies or the like.
<@zardian:matrix.org>
16:05:24
Thanks!!
<@zardian:matrix.org>
16:05:24
oh ok
<@lenkaseg:fedora.im>
16:05:47
!next
<@lenkaseg:fedora.im>
16:06:55
ok, nevermind :)
<@lenkaseg:fedora.im>
16:06:58
!topic Next chair
<@lenkaseg:fedora.im>
16:06:58
!info magic eight ball says:
<@lenkaseg:fedora.im>
16:06:58
!info chair 2024-10-10 - zlopez
<@abompard:matrix.org>
16:06:58
There is no next stage in this meeting template. We're done here.
<@lenkaseg:fedora.im>
16:06:59
Some volunteers for chairing this meeting next week (10.) or next next week (17.)?
<@lenkaseg:fedora.im>
16:07:00
I mean any, not some
<@zardian:matrix.org>
16:07:15
Ya sure
<@lenkaseg:fedora.im>
16:08:03
Hey, I didn't see all those messages from zardian and nirik
<@lenkaseg:fedora.im>
16:08:09
Got them all at once now
<@nirik:matrix.scrye.com>
16:08:15
The joys of federation?
<@lenkaseg:fedora.im>
16:08:21
Not sure what;s happening on my side
<@lenkaseg:fedora.im>
16:09:12
zardian: Welcome!
<@zardian:matrix.org>
16:10:09
Hey Everyone, I have been using Fedora for 2 years now. I'm from India and is currently in the last year of my college. I have been interested in the devops and infra admin tasks and tools so I'm also learning the basics of linux, networking, containers and kubernetes.
<@zardian:matrix.org>
16:11:27
Wish to experience and learn how these thing are implemented and are being used in the production env of such large projects.
<@nirik:matrix.scrye.com>
16:12:26
welcome! Look forward to talking more with you.
<@lenkaseg:fedora.im>
16:13:29
zardian: Nice! Feel free to look around, join meetings and ask questions! We can help you find something to work on :)
<@zardian:matrix.org>
16:13:31
Infrastructure/GettingStarted and it stated that I have to ask about being let in the "Infrastructure Apprentice" group here.
<@zardian:matrix.org>
16:13:31
Thanks, I did went through the documentation of
<@nirik:matrix.scrye.com>
16:14:34
yeah, we can get you setup... ask in #admin:fedoraproject.org after the meeting?
<@zardian:matrix.org>
16:14:47
Ok, Thanks!
<@lenkaseg:fedora.im>
16:14:52
Good start!
<@zardian:matrix.org>
16:15:04
Thats it from me then.
<@lenkaseg:fedora.im>
16:15:47
Ok, let's get to choosing next next and next next next chair!
<@lenkaseg:fedora.im>
16:16:09
going once
<@lenkaseg:fedora.im>
16:16:27
going twice
<@nirik:matrix.scrye.com>
16:16:30
this is for the 23rd?
<@nirik:matrix.scrye.com>
16:16:43
17th, sorry
<@lenkaseg:fedora.im>
16:16:49
ah, yes, 17 and 24
<@lenkaseg:fedora.im>
16:17:00
for 10th we have zlope
<@lenkaseg:fedora.im>
16:17:03
for 10th we have zlopez
<@nirik:matrix.scrye.com>
16:17:14
I can take the 17th
<@lenkaseg:fedora.im>
16:17:14
I can do 17th I guess
<@Zlopez:matrix.org>
16:17:33
I think only the one for 10th is actually occupied
<@lenkaseg:fedora.im>
16:17:51
I was first! And anyways, you chaired for me when I forgot some time ago
<@lenkaseg:fedora.im>
16:18:00
!info chair 2024-10-17 - lenkaseg
<@Zlopez:matrix.org>
16:18:06
At your service :-)
<@lenkaseg:fedora.im>
16:18:39
!topic announcements and information
<@lenkaseg:fedora.im>
16:18:39
!info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
<@lenkaseg:fedora.im>
16:18:39
!info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
<@lenkaseg:fedora.im>
16:19:01
(I hope I'm not missing any messages, apparently I se them with delay)
<@nirik:matrix.scrye.com>
16:19:21
!info f41 final freeze starts 2024-10-15
<@nirik:matrix.scrye.com>
16:19:36
!info several planned outages next week...
<@nirik:matrix.scrye.com>
16:20:10
we have a networking thing that might affect pagure.io, a reinstall/upgrade on openqa stuff, and a koji upgrade
<@nirik:matrix.scrye.com>
16:22:03
Thats all I had. (should have added to the doc)
<@Zlopez:matrix.org>
16:22:28
!info Because of internal changes in CPE Team the I&R weekly reports will be limited in upcoming weeks
<@lenkaseg:fedora.im>
16:23:08
Yep, we have to figure out new format
<@lenkaseg:fedora.im>
16:23:16
let's move on ..
<@lenkaseg:fedora.im>
16:23:18
!info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
<@lenkaseg:fedora.im>
16:23:18
<@lenkaseg:fedora.im>
16:23:18
!topic Oncall
<@lenkaseg:fedora.im>
16:23:18
!info https://fedoraproject.org/wiki/Infrastructure/Oncall
<@lenkaseg:fedora.im>
16:23:27
!info zlopez is on call from 2024-09-27 to 2024-10-03
<@lenkaseg:fedora.im>
16:23:27
!info lenkaseg is on call from 2024-10-03 to 2024-10-10
<@lenkaseg:fedora.im>
16:23:27
!info ? is on call from 2024-10-10 to 2024-10-17
<@lenkaseg:fedora.im>
16:24:00
Anybody interested in oncall duty for the week 10-17?
<@zardian:matrix.org>
16:25:09
can I shadow someone to see what and how they do stuff when oncall? But first, what does this mean?
<@lenkaseg:fedora.im>
16:25:54
zardian: Definitely or you can take oncall yourself, there's nothing to worry about.
<@zardian:matrix.org>
16:26:31
can I shadow someone to see what and how they do stuff when oncall? But first, what does being oncall mean?
<@lenkaseg:fedora.im>
16:26:32
When someone encounters a problem and runs oncall command in some linked matrix channels, it should ping you
<@nirik:matrix.scrye.com>
16:26:50
There's a doc on it. ;)
<@nirik:matrix.scrye.com>
16:27:03
https://docs.fedoraproject.org/en-US/infra/day_to_day_fedora/#_the_oncall_role_in_our_team
<@lenkaseg:fedora.im>
16:27:06
And then you help that person either solve the problem or open an issue in infrastructure issue tracker
<@zardian:matrix.org>
16:29:07
I see, will have to read the doc then. Count me in for the last oncall that is vacant, till then I can get the hang of it.
<@lenkaseg:fedora.im>
16:29:28
zardian: Excellent!
<@lenkaseg:fedora.im>
16:29:40
!info zardian is on call from 2024-10-10 to 2024-10-17
<@nirik:matrix.scrye.com>
16:29:53
It's usually not too much... but you can watch our channels and see over the coming weeks too.
<@lenkaseg:fedora.im>
16:30:13
not sure if you don't need to be linked with FAS for that tbh
<@Zlopez:matrix.org>
16:31:02
I think that you need to, otherwise it will not ping you
<@zardian:matrix.org>
16:31:10
Ya, Thats what I was thinking, will be following on with you all in the main chat group this week to see how somethings are done.
<@lenkaseg:fedora.im>
16:32:03
zardian: Do you have an account here? https://accounts.fedoraproject.org/
<@zardian:matrix.org>
16:32:19
yes
<@zardian:matrix.org>
16:33:04
how do I link it with matrix account?
<@nirik:matrix.scrye.com>
16:33:51
There's a matrix id in the profile settings, just set it to your @user:server
<@Zlopez:matrix.org>
16:33:54
On https://accounts.fedoraproject.org/ go to settings and there is Matrix nickname you can fill
<@lenkaseg:fedora.im>
16:34:44
Should look like this
<@lenkaseg:fedora.im>
16:35:05
!info Summary of last week: (from current oncall)
<@zardian:matrix.org>
16:35:33
added
<@Zlopez:matrix.org>
16:35:34
I did receive only one ping regarding the builds being stuck on newRepo in koji
<@Zlopez:matrix.org>
16:35:56
After some digging I restarted kojira and it caught up again
<@zardian:matrix.org>
16:37:07
!hi
<@zodbot:fedora.im>
16:37:11
Aman Singh (zardian)
<@zardian:matrix.org>
16:37:21
yup working
<@lenkaseg:fedora.im>
16:37:22
let's move on to
<@lenkaseg:fedora.im>
16:37:25
!topic Monitoring discussion [nirik]
<@lenkaseg:fedora.im>
16:37:25
!info Go over existing items and fix them
<@lenkaseg:fedora.im>
16:37:25
!info https://nagios.fedoraproject.org/nagios
<@lenkaseg:fedora.im>
16:37:31
over to nirik!
<@zardian:matrix.org>
16:37:42
Thanks!!
<@zardian:matrix.org>
16:37:42
yup, working.
<@nirik:matrix.scrye.com>
16:37:53
lets see...
<@nirik:matrix.scrye.com>
16:38:16
2 down aarch64 machines with bad memory as before.
<@nirik:matrix.scrye.com>
16:38:42
mailman01.stg has low disk space. I think this is Zlopez's indexer?
<@nirik:matrix.scrye.com>
16:39:06
openqa-p09-worker01 has low space on /boot, but this should get fixed next week when we reinstall openqa stuff
<@nirik:matrix.scrye.com>
16:39:32
and there's a few cert expiring in 57 days things. Should get solved before that matters.
<@nirik:matrix.scrye.com>
16:40:05
I've noted a numed of rabbitmq alerts of late... not sure whats causing them... for the messaging bridges or the like
<@Zlopez:matrix.org>
16:40:24
@nirik:matrix.scrye.com Again? I need to increase the volume size again, but the fulltext search index is almost finished for all mailing lists
<@nirik:matrix.scrye.com>
16:40:57
<@nirik:matrix.scrye.com>
16:40:57
Disk_Space_/
<@nirik:matrix.scrye.com>
16:40:57
WARNING 10-03-2024 16:18:11 0d 20h 22m 36s 3/3 DISK WARNING - free space: / 37962 MiB (13.00% inode=100%):
<@nirik:matrix.scrye.com>
16:40:57
<@nirik:matrix.scrye.com>
16:40:57
<@nirik:matrix.scrye.com>
16:40:57
This service has 1 comment associated with it This service problem has been acknowledged
<@nirik:matrix.scrye.com>
16:40:57
mailman01.stg.iad2.fedoraproject.org
<@nirik:matrix.scrye.com>
16:41:00
WARNING 10-03-2024 16:18:11 0d 20h 22m 36s 3/3 DISK WARNING - free space: / 37962 MiB (13.00% inode=100%): ```
<@nirik:matrix.scrye.com>
16:41:00
mailman01.stg.iad2.fedoraproject.org
<@nirik:matrix.scrye.com>
16:41:00
<@nirik:matrix.scrye.com>
16:41:00
<@nirik:matrix.scrye.com>
16:41:00
Disk_Space_/
<@nirik:matrix.scrye.com>
16:41:00
<@nirik:matrix.scrye.com>
16:41:00
This service has 1 comment associated with it This service problem has been acknowledged
<@Zlopez:matrix.org>
16:41:22
I need to check if there isn't some temp files from the generation
<@nirik:matrix.scrye.com>
16:41:31
is the space likely to free up after with temp files.. yeah...
<@Zlopez:matrix.org>
16:42:12
I will wait till it's finished and then see if it still needs some space
<@nirik:matrix.scrye.com>
16:42:13
Last thing on monitoring, we do have zabbix going to two rooms (one stg/one prod) if anyone wants to watch. It's very noisy tho...
<@Zlopez:matrix.org>
16:42:43
It's hard to watch
<@nirik:matrix.scrye.com>
16:43:55
acutally seems prod one stopped... will let ryanlerch poke at it.
<@nirik:matrix.scrye.com>
16:44:12
anyhow thats it on monitoring I think.
<@Zlopez:matrix.org>
16:44:14
Let's poke it with stick :-)
<@lenkaseg:fedora.im>
16:45:48
next topic?
<@nirik:matrix.scrye.com>
16:46:11
sure.
<@lenkaseg:fedora.im>
16:46:13
blocked tickets or backlog refimenet?
<@lenkaseg:fedora.im>
16:46:26
blocked tickets or backlog refinement?
<@lenkaseg:fedora.im>
16:46:48
I think you looked at many tickets last time, do you remember where did you get?
<@lenkaseg:fedora.im>
16:46:52
(not to repeat)
<@Zlopez:matrix.org>
16:47:10
Or we can go directly to open floor as we have only 13 minutes left
<@lenkaseg:fedora.im>
16:47:26
!topic Open Floor
<@lenkaseg:fedora.im>
16:47:57
we have some brainstormed ideas
<@lenkaseg:fedora.im>
16:47:57
btw there was not any learning topic for very long time
<@lenkaseg:fedora.im>
16:48:15
koji thing, zabbix, ...
<@nirik:matrix.scrye.com>
16:49:07
yeah, I'd be happy to pontificate on things if people tell me what they want.
<@lenkaseg:fedora.im>
16:49:12
Or is there something you're currently working on that would be nice to tell others about?
<@lenkaseg:fedora.im>
16:50:50
I'd be interested in the koji stuff
<@Zlopez:matrix.org>
16:50:52
Maybe a debuging session of something in nagios would be nice learning topic
<@nirik:matrix.scrye.com>
16:52:51
I'm happy to pontificate on koji...
<@nirik:matrix.scrye.com>
16:54:38
perhaps a session about the block retired stuff? I know that was complex...
<@nirik:matrix.scrye.com>
16:54:47
or toddlers/poddlers in general?
<@lenkaseg:fedora.im>
16:55:23
I can talk about the block retired, but that's hardly interesting :)
<@lenkaseg:fedora.im>
16:55:33
toddlers/poddlers sounds good!
<@lenkaseg:fedora.im>
16:56:10
Btw the PR is open: https://pagure.io/fedora-infra/toddlers/pull-request/261
<@Zlopez:matrix.org>
16:56:18
I can talk about toddlers and probably about poddlers as well
<@lenkaseg:fedora.im>
16:56:22
I tested it to my best knowledge
<@lenkaseg:fedora.im>
16:56:52
now CI failing for 2 lines of uncovered code, but it should work
<@nirik:matrix.scrye.com>
16:57:01
cool!
<@Zlopez:matrix.org>
16:57:26
I will review it tomorrow
<@lenkaseg:fedora.im>
16:57:34
for the moment I use this script to ask someone to block the packages manually: https://lenkaseg.fedorapeople.org/check_retired_if_blocked.py
<@zodbot:fedora.im>
16:58:16
zlopez has already given cookies to lenkaseg during the F40 timeframe
<@lenkaseg:fedora.im>
16:58:27
for the moment I use this script to see unblocked and to ask someone to block the packages manually: https://lenkaseg.fedorapeople.org/check\_retired\_if\_blocked.py
<@zodbot:fedora.im>
16:58:36
kevin has already given cookies to lenkaseg during the F40 timeframe
<@lenkaseg:fedora.im>
16:59:13
Ok, we're out of time, nice to see you'all!
<@lenkaseg:fedora.im>
16:59:24
!endmeeting