<@Zlopez:matrix.org>
16:00:28
!startmeeting Infrastructure (2024-07-18)
<@meetbot:fedora.im>
16:00:30
Meeting started at 2024-07-18 16:00:28 UTC
<@meetbot:fedora.im>
16:00:30
The Meeting name is 'Infrastructure (2024-07-18)'
<@Zlopez:matrix.org>
16:00:37
!info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
<@Zlopez:matrix.org>
16:00:37
!chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp
<@Zlopez:matrix.org>
16:00:37
!meetingname infrastructure
<@Zlopez:matrix.org>
16:00:37
!topic namaste
<@Zlopez:matrix.org>
16:00:37
!info About our team: https://docs.fedoraproject.org/en-US/cpe/
<@Zlopez:matrix.org>
16:00:37
!info Agenda is at: https://board.net/p/fedora-infra
<@meetbot:fedora.im>
16:00:38
The Meeting Name is now infrastructure
<@Zlopez:matrix.org>
16:00:54
Ahoy everyone!
<@markrosenbaum:fedora.im>
16:01:01
Heyo!
<@Zlopez:matrix.org>
16:01:22
Welcome to infrastructure meeting for this week
<@nirik:matrix.scrye.com>
16:01:43
morning
<@Zlopez:matrix.org>
16:01:51
Hope you are ready for today show :-)
<@Zlopez:matrix.org>
16:03:14
Let's wait a little for more folks to join, but as it is summer I don't think there will be much people to show up
<@Zlopez:matrix.org>
16:04:13
How is everyone?
<@markrosenbaum:fedora.im>
16:04:33
Pretty good, not much going on around here at the moment
<@nirik:matrix.scrye.com>
16:04:58
ok here... cat immoblizing one arm, so its hard to type tho...
<@nirik:matrix.scrye.com>
16:05:14
🐈
<@Zlopez:matrix.org>
16:05:28
The summer is usually slow, plenty of folks going on vacation :-)
<@markrosenbaum:fedora.im>
16:05:43
Haha, your arm belongs to the cat now
<@Zlopez:matrix.org>
16:06:04
All your arms belong to cat!
<@nirik:matrix.scrye.com>
16:06:09
pretty much...
<@Zlopez:matrix.org>
16:06:28
I think we waited long enough, let us continue
<@Zlopez:matrix.org>
16:06:35
!info Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/
<@Zlopez:matrix.org>
16:06:35
!topic New folks introductions
<@Zlopez:matrix.org>
16:06:35
!info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
<@Zlopez:matrix.org>
16:06:43
Anybody new around?
<@markrosenbaum:fedora.im>
16:08:23
I'm new-ish as of maybe 1-2 weeks ago
<@Zlopez:matrix.org>
16:09:02
I already saw the documentation work you are doing, nice job :-)
<@Zlopez:matrix.org>
16:09:37
Welcome to Fedora
<@markrosenbaum:fedora.im>
16:09:45
Thanks, just helping out where I can
<@nirik:matrix.scrye.com>
16:09:46
Yes, Welcome!
<@Zlopez:matrix.org>
16:10:25
We all trying to do that, to help where we can
<@Zlopez:matrix.org>
16:11:09
Mark Rosenbaum: Do you need any help?
<@markrosenbaum:fedora.im>
16:11:55
Not so far, everything looks pretty good
<@markrosenbaum:fedora.im>
16:12:28
It was nice finding that we did indeed have an archive so it's asciidoc time right now
<@markrosenbaum:fedora.im>
16:12:54
Then off to update the actual documentation
<@markrosenbaum:fedora.im>
16:13:02
Then off to update the actual documentation information
<@Zlopez:matrix.org>
16:13:30
The asciidoc format is sometimes confusing
<@markrosenbaum:fedora.im>
16:14:21
Honestly it isn't that bad, spent some time studying it on a quest to fix a broken link so I have a basic understanding now
<@Zlopez:matrix.org>
16:15:06
In case of fedora infra documentation there is a script which will start local preview, so you can check how it looks before submitting changes
<@jeffiscow:fedora.im>
16:15:47
I'm new have not been to a infra meeting and just started looking into what its all about.
<@markrosenbaum:fedora.im>
16:15:58
I've honestly just been using https://asciidoclive.com/edit/scratch/1
<@nirik:matrix.scrye.com>
16:16:13
yeah, thats been very handy for me
<@markrosenbaum:fedora.im>
16:16:14
I've honestly just been using https://asciidoclive.com/
<@markrosenbaum:fedora.im>
16:16:28
Welcome!
<@Zlopez:matrix.org>
16:17:22
I think we spend plenty of time on this topic already, let's move on, we can discuss that more on open floor
<@Zlopez:matrix.org>
16:17:32
!info chair 2024-08-01 pcreech
<@Zlopez:matrix.org>
16:17:32
!info chair 2024-07-25 nirik
<@Zlopez:matrix.org>
16:17:32
!topic Next chair
<@Zlopez:matrix.org>
16:17:32
!info magic eight ball says:
<@Zlopez:matrix.org>
16:17:32
!info chair 2024-08-15 - ???
<@Zlopez:matrix.org>
16:17:32
!info chair 2024-08-08 - cancelled (Flock)
<@jeffiscow:fedora.im>
16:17:41
!hi
<@zodbot:fedora.im>
16:17:41
jeff bucher (jeffiscow) - he / him / his
<@Zlopez:matrix.org>
16:17:56
jeffiscow (he/him): Welcome to fedora infra
<@nirik:matrix.scrye.com>
16:18:06
welcome jeffiscow (he/him)
<@Zlopez:matrix.org>
16:18:29
Feel free to reach to us, if you need any guidance
<@jeffiscow:fedora.im>
16:18:41
Hello everyone I'm just trying to learn and soak in as much info as I can.
<@Zlopez:matrix.org>
16:19:06
So I already marked 2024-08-08 as cancelled, as it is at the same time as Flock 2024
<@Zlopez:matrix.org>
16:19:32
And we are looking for a volunteer to run after Flock infra meeting
<@Zlopez:matrix.org>
16:20:23
We don't need to fill it up today and leave it for next meeting
<@nirik:matrix.scrye.com>
16:20:33
yeah, lets leave it for when more folks are around
<@Zlopez:matrix.org>
16:21:39
!info Flock 2024 will happen from 7-11 Aug 2024. See you there!
<@Zlopez:matrix.org>
16:21:39
!info Fedora Infra weekly meeting is cancelled for 2024-08-08 because of Flock 2024
<@Zlopez:matrix.org>
16:21:39
!info Fedora 41 Mass rebuild in progress
<@Zlopez:matrix.org>
16:21:39
!info Remaining issues with new mailman3 deployment are being investigated
<@Zlopez:matrix.org>
16:21:39
!info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
<@Zlopez:matrix.org>
16:21:39
!info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
<@Zlopez:matrix.org>
16:21:39
!topic announcements and information
<@Zlopez:matrix.org>
16:22:08
I added few announcements to the list, does anybody has anything else?
<@nirik:matrix.scrye.com>
16:22:29
I was going to mention the mass rebuild and flock, but now I don't have to. ;)
<@Zlopez:matrix.org>
16:23:09
Hm, what happened?
<@aggraxis:fedora.im>
16:23:33
nothing? you already mentioned it. lol
<@nirik:matrix.scrye.com>
16:23:33
hum?
<@Zlopez:matrix.org>
16:23:51
I see the announcements posted twice
<@Zlopez:matrix.org>
16:24:04
But the second seems to be visible only to me :-D
<@aggraxis:fedora.im>
16:24:09
I just see it once with the bot responding
<@Zlopez:matrix.org>
16:24:17
Let me refresh the browser
<@nirik:matrix.scrye.com>
16:24:44
I only see once.
<@Zlopez:matrix.org>
16:24:49
That's better
<@markrosenbaum:fedora.im>
16:24:53
Fire has spawned at work 😭, gtg for now yall. If I'm not back before it ends have a great meeting
<@nirik:matrix.scrye.com>
16:25:00
well, I mean meetbot repeats them as links...
<@nirik:matrix.scrye.com>
16:25:20
or infos... whatever
<@Zlopez:matrix.org>
16:25:20
That wasn't it it looked like I have it sent twice
<@nirik:matrix.scrye.com>
16:25:25
strange
<@Zlopez:matrix.org>
16:25:34
After refresh it's gone
<@Zlopez:matrix.org>
16:25:55
So anything else for announcements?
<@Zlopez:matrix.org>
16:27:25
It doesn't seem so, let's look what else awaits us
<@Zlopez:matrix.org>
16:27:33
!topic Oncall
<@Zlopez:matrix.org>
16:27:33
!info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
<@Zlopez:matrix.org>
16:27:33
!info https://fedoraproject.org/wiki/Infrastructure/Oncall
<@Zlopez:matrix.org>
16:27:40
!info zlopez is on call from 2024-07-12 to 2024-07-18
<@Zlopez:matrix.org>
16:27:40
!info ??? is on call from 2024-07-19 to 2024-07-25
<@Zlopez:matrix.org>
16:27:40
!info ??? is on call from 2024-07-26 to 2024-08-01
<@nirik:matrix.scrye.com>
16:27:52
I can take the next stint
<@Zlopez:matrix.org>
16:28:00
Sold!
<@Zlopez:matrix.org>
16:28:08
!info nirik is on call from 2024-07-19 to 2024-07-25
<@Zlopez:matrix.org>
16:28:40
Anybody want to go for week after that?
<@Zlopez:matrix.org>
16:29:11
!oncall
<@zodbot:fedora.im>
16:29:13
The following people are oncall:
<@zodbot:fedora.im>
16:29:13
If they do not respond, please file a ticket (https://pagure.io/fedora-infrastructure/issues)
<@zodbot:fedora.im>
16:29:13
● @nirik:matrix.scrye.com (kevin) Current Time for them: 09:29 (US/Pacific)
<@zodbot:fedora.im>
16:29:13
<@Zlopez:matrix.org>
16:29:27
Changed the oncall :-)
<@nirik:matrix.scrye.com>
16:29:33
thanks
<@Zlopez:matrix.org>
16:30:31
Let's keep the decision for next week as there isn't much folks around
<@Zlopez:matrix.org>
16:30:39
!info Summary of last week: (from current oncall)
<@Zlopez:matrix.org>
16:30:43
That would be me
<@Zlopez:matrix.org>
16:30:52
I didn't saw any ping for the whole week
<@nirik:matrix.scrye.com>
16:31:15
yeah...been quiet on that front
<@Zlopez:matrix.org>
16:32:01
But there was the issue with the mail spammer on issue tracker
<@Zlopez:matrix.org>
16:32:30
It seems that it's now gone I didn't saw any message yesterday or today from the account
<@nirik:matrix.scrye.com>
16:33:14
so, yeah, I need to reply to that ticket...
<@nirik:matrix.scrye.com>
16:33:27
I also removed their watch from the project, so perhaps thats what stopped it.
<@Zlopez:matrix.org>
16:33:40
That could be it
<@Zlopez:matrix.org>
16:33:47
How can somebody do that?
<@nirik:matrix.scrye.com>
16:34:03
because I thiink it was commit to issue -> email to user -> user sends reply back
<@nirik:matrix.scrye.com>
16:34:14
there's a pagure-admin command. I will add it to the ticket.
<@nirik:matrix.scrye.com>
16:34:58
pagure-admin update-watch fedora-infrastructure <username> -s 0
<@Zlopez:matrix.org>
16:35:17
It would be probably best to update it in docs as well
<@nirik:matrix.scrye.com>
16:35:31
yeah, I can.
<@Zlopez:matrix.org>
16:35:32
I already have PR open for pagure spam guide https://pagure.io/infra-docs-fpo/pull-request/308
<@nirik:matrix.scrye.com>
16:35:42
ok.
<@Zlopez:matrix.org>
16:35:52
Just to mention the sanitize-spam-user command
<@nirik:matrix.scrye.com>
16:36:24
On a somewhat related note... pagure.io has been under heavy load lately. It looks like tons of random .cn ip's doing a HEAD on forked project files then a GET...
<@nirik:matrix.scrye.com>
16:36:42
I have tried to block them, but don't know if I caught legit traffic in there. ;(
<@nirik:matrix.scrye.com>
16:37:17
and it's a lot of IP's, so it's in a ipset right now... not in ansible (was waiting to see what it broke before pushing any commits)
<@Zlopez:matrix.org>
16:38:06
I thought that we aren't adding those to ansible
<@nirik:matrix.scrye.com>
16:39:05
I did add some ip's we blocked... but adding these would be... tricky. I am not sure how to handle it. I was hoping perhaps by rejecting them they would stop and go away.
<@Zlopez:matrix.org>
16:39:36
So it will be in firewall rules for pagure?
<@nirik:matrix.scrye.com>
16:39:50
yes, it's a iptables rule to check the ipset...
<@nirik:matrix.scrye.com>
16:39:53
so far:
<@nirik:matrix.scrye.com>
16:39:57
pkts bytes target prot opt in out source destination
<@nirik:matrix.scrye.com>
16:39:57
1032414 61793439 REJECT tcp -- any any anywhere anywhere
<@nirik:matrix.scrye.com>
16:40:06
about a million packets blocked
<@Zlopez:matrix.org>
16:40:29
That looks like some botnet
<@nirik:matrix.scrye.com>
16:40:56
could be. it's really hard to isolate because it uses different user agents, it's hitting different projects, etc.
<@nirik:matrix.scrye.com>
16:41:05
or it could be some AI scraper.
<@nirik:matrix.scrye.com>
16:41:20
anyhow, will let it run and try and think what we could do longer term.
<@Zlopez:matrix.org>
16:41:34
Everything is AI these days :/
<@Zlopez:matrix.org>
16:41:46
Let me move to next topic
<@Zlopez:matrix.org>
16:41:51
!info Go over existing out items and fix
<@Zlopez:matrix.org>
16:41:51
!topic Monitoring discussion [nirik]
<@Zlopez:matrix.org>
16:41:51
!info https://nagios.fedoraproject.org/nagios
<@nirik:matrix.scrye.com>
16:42:02
so, looking better here this week...
<@nirik:matrix.scrye.com>
16:42:22
I managed to get firmware updated and reinstalled the old emag aarch64 boxes, so they are all up now and off nagios
<@nirik:matrix.scrye.com>
16:42:41
5 down hosts right now:
<@nirik:matrix.scrye.com>
16:42:52
one aarch64 box with memory errors. Not sure we can easily fix it.
<@nirik:matrix.scrye.com>
16:43:08
2 buildhw-x86's that wont power on (needs an onsite power cycle as the next step)
<@nirik:matrix.scrye.com>
16:43:21
2 buildhw-a64's that never came back from reboot and I can't get to mgmt (needs onsite)
<@Zlopez:matrix.org>
16:43:22
Sounds like a hardware issue
<@nirik:matrix.scrye.com>
16:43:43
otherwise we are looking pretty reasonable
<@nirik:matrix.scrye.com>
16:43:55
yeah, bad memory I think. I will look at it more later today.
<@Zlopez:matrix.org>
16:44:20
I can see copr-be.aws error
<@nirik:matrix.scrye.com>
16:44:50
yeah, we should ask them about that.
<@nirik:matrix.scrye.com>
16:44:55
I think their limit is too low...
<@nirik:matrix.scrye.com>
16:45:05
it's a check for a test build finishing in X time
<@Zlopez:matrix.org>
16:45:13
And fedbadges message queue alert, but I assume this is related to mass rebuild
<@nirik:matrix.scrye.com>
16:46:05
yeah, not sure on that one, but it did clear up
<@nirik:matrix.scrye.com>
16:46:11
or did it?
<@Zlopez:matrix.org>
16:46:24
Not yet, I still see it in nagios
<@nirik:matrix.scrye.com>
16:46:30
ugh, no...
<@nirik:matrix.scrye.com>
16:46:34
we need to look at that.
<@nirik:matrix.scrye.com>
16:46:44
600k messages is... bad
<@nirik:matrix.scrye.com>
16:47:10
if that gets too high it can affect other queues...
<@Zlopez:matrix.org>
16:47:55
Yeah, the number is very high
<@nirik:matrix.scrye.com>
16:48:20
I can look for anything obvious... if not we could perhaps ask ryanlerch to look? was he involved in the porting to openshift?
<@Zlopez:matrix.org>
16:48:38
Maybe the consumer is just stuck
<@nirik:matrix.scrye.com>
16:49:45
it's not got any errors...
<@Zlopez:matrix.org>
16:50:06
Let's move to open floor as we have only 10 minutes left
<@Zlopez:matrix.org>
16:50:19
!topic Open Floor
<@Zlopez:matrix.org>
16:50:48
Does the fedbadges running in Openshift now?
<@nirik:matrix.scrye.com>
16:51:28
Flock is coming up fast...
<@nirik:matrix.scrye.com>
16:51:52
I plan to gather the workshop ideas and post to the discussion thread. Perhaps we can decide some things we want to do in advance.
<@aggraxis:fedora.im>
16:52:12
I'm looking forward to seeing you guys
<@Zlopez:matrix.org>
16:52:23
I'm already working on the Fedora Infra overview presentation
<@nirik:matrix.scrye.com>
16:52:33
Paul Maconi: you gonna make it? cool... it's a fun conference (I think)
<@nirik:matrix.scrye.com>
16:52:41
I need to work on my other talk
<@Zlopez:matrix.org>
16:52:48
I should have first version ready tomorrow, I will share that with you nirik
<@aggraxis:fedora.im>
16:52:56
bringing my oldest with me, too. I think he'll get inspired by it
<@nirik:matrix.scrye.com>
16:53:13
sure, happy to review.
<@nirik:matrix.scrye.com>
16:53:20
nice. :)
<@Zlopez:matrix.org>
16:53:48
Should I try and restart the consumer pods in badges project?
<@Zlopez:matrix.org>
16:54:13
I don't see any message processed till yesterday
<@nirik:matrix.scrye.com>
16:54:25
I actually tried that the other day, but it can't hurt to try again. ;)
<@jeffiscow:fedora.im>
16:54:41
will there be a steam of flock?
<@aggraxis:fedora.im>
16:54:54
yes I think so
<@nirik:matrix.scrye.com>
16:55:01
jeffiscow (he/him): not sure, but I hope so.
<@Zlopez:matrix.org>
16:55:29
It should be, not sure if you could also ask questions through it
<@jeffiscow:fedora.im>
16:55:32
I'm sure well have recording of and talks and stuff
<@nirik:matrix.scrye.com>
16:55:43
yep. Thats standard.
<@nirik:matrix.scrye.com>
16:56:07
There's also a matrix channel where there is a lot of discussion before, during and after the event. ;)
<@Zlopez:matrix.org>
16:56:57
I'm probably not there
<@nirik:matrix.scrye.com>
16:57:17
#flock:fedoraproject.org
<@Zlopez:matrix.org>
16:58:20
Joined :-)
<@jeffiscow:fedora.im>
16:58:39
I really hope to go inperson one day seem like a good time.
<@Zlopez:matrix.org>
16:58:50
I'm doing the new build of fedbadges as the last one happened on 10th July, maybe there is something new
<@Zlopez:matrix.org>
16:59:31
We are at the end of this meeting
<@Zlopez:matrix.org>
16:59:43
It was a nice discussion with all of you :-)
<@Zlopez:matrix.org>
16:59:57
Thanks for joining
<@Zlopez:matrix.org>
17:00:03
!endmeeting
<@aggraxis:fedora.im>
17:00:04
thanks for running the meeting Zlopez !