2024-07-18 16:00:28 <@Zlopez:matrix.org> !startmeeting Infrastructure (2024-07-18)
2024-07-18 16:00:30 <@meetbot:fedora.im> Meeting started at 2024-07-18 16:00:28 UTC
2024-07-18 16:00:30 <@meetbot:fedora.im> The Meeting name is 'Infrastructure (2024-07-18)'
2024-07-18 16:00:37 <@Zlopez:matrix.org> !info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
2024-07-18 16:00:37 <@Zlopez:matrix.org> !chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp
2024-07-18 16:00:37 <@Zlopez:matrix.org> !meetingname infrastructure
2024-07-18 16:00:37 <@Zlopez:matrix.org> !topic namaste
2024-07-18 16:00:37 <@Zlopez:matrix.org> !info About our team: https://docs.fedoraproject.org/en-US/cpe/
2024-07-18 16:00:37 <@Zlopez:matrix.org> !info Agenda is at: https://board.net/p/fedora-infra
2024-07-18 16:00:38 <@meetbot:fedora.im> The Meeting Name is now infrastructure
2024-07-18 16:00:54 <@Zlopez:matrix.org> Ahoy everyone!
2024-07-18 16:01:01 <@markrosenbaum:fedora.im> Heyo!
2024-07-18 16:01:22 <@Zlopez:matrix.org> Welcome to infrastructure meeting for this week
2024-07-18 16:01:43 <@nirik:matrix.scrye.com> morning
2024-07-18 16:01:51 <@Zlopez:matrix.org> Hope you are ready for today show :-)
2024-07-18 16:03:14 <@Zlopez:matrix.org> Let's wait a little for more folks to join, but as it is summer I don't think there will be much people to show up
2024-07-18 16:04:13 <@Zlopez:matrix.org> How is everyone?
2024-07-18 16:04:33 <@markrosenbaum:fedora.im> Pretty good, not much going on around here at the moment
2024-07-18 16:04:58 <@nirik:matrix.scrye.com> ok here... cat immoblizing one arm, so its hard to type tho...
2024-07-18 16:05:14 <@nirik:matrix.scrye.com> 🐈
2024-07-18 16:05:28 <@Zlopez:matrix.org> The summer is usually slow, plenty of folks going on vacation :-)
2024-07-18 16:05:43 <@markrosenbaum:fedora.im> Haha, your arm belongs to the cat now
2024-07-18 16:06:04 <@Zlopez:matrix.org> All your arms belong to cat!
2024-07-18 16:06:09 <@nirik:matrix.scrye.com> pretty much...
2024-07-18 16:06:28 <@Zlopez:matrix.org> I think we waited long enough, let us continue
2024-07-18 16:06:35 <@Zlopez:matrix.org> !info Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/
2024-07-18 16:06:35 <@Zlopez:matrix.org> !topic New folks introductions
2024-07-18 16:06:35 <@Zlopez:matrix.org> !info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
2024-07-18 16:06:43 <@Zlopez:matrix.org> Anybody new around?
2024-07-18 16:08:23 <@markrosenbaum:fedora.im> I'm new-ish as of maybe 1-2 weeks ago
2024-07-18 16:09:02 <@Zlopez:matrix.org> I already saw the documentation work you are doing, nice job :-)
2024-07-18 16:09:37 <@Zlopez:matrix.org> Welcome to Fedora
2024-07-18 16:09:45 <@markrosenbaum:fedora.im> Thanks, just helping out where I can
2024-07-18 16:09:46 <@nirik:matrix.scrye.com> Yes, Welcome!
2024-07-18 16:10:25 <@Zlopez:matrix.org> We all trying to do that, to help where we can
2024-07-18 16:11:09 <@Zlopez:matrix.org> Mark Rosenbaum: Do you need any help?
2024-07-18 16:11:55 <@markrosenbaum:fedora.im> Not so far, everything looks pretty good
2024-07-18 16:12:28 <@markrosenbaum:fedora.im> It was nice finding that we did indeed have an archive so it's asciidoc time right now
2024-07-18 16:12:54 <@markrosenbaum:fedora.im> Then off to update the actual documentation
2024-07-18 16:13:02 <@markrosenbaum:fedora.im> Then off to update the actual documentation information
2024-07-18 16:13:30 <@Zlopez:matrix.org> The asciidoc format is sometimes confusing
2024-07-18 16:14:21 <@markrosenbaum:fedora.im> Honestly it isn't that bad, spent some time studying it on a quest to fix a broken link so I have a basic understanding now
2024-07-18 16:15:06 <@Zlopez:matrix.org> In case of fedora infra documentation there is a script which will start local preview, so you can check how it looks before submitting changes
2024-07-18 16:15:47 <@jeffiscow:fedora.im> I'm new have not been to a infra meeting and just started looking into what its all about.
2024-07-18 16:15:58 <@markrosenbaum:fedora.im> I've honestly just been using https://asciidoclive.com/edit/scratch/1
2024-07-18 16:16:13 <@nirik:matrix.scrye.com> yeah, thats been very handy for me
2024-07-18 16:16:14 <@markrosenbaum:fedora.im> I've honestly just been using https://asciidoclive.com/
2024-07-18 16:16:28 <@markrosenbaum:fedora.im> Welcome!
2024-07-18 16:17:22 <@Zlopez:matrix.org> I think we spend plenty of time on this topic already, let's move on, we can discuss that more on open floor
2024-07-18 16:17:32 <@Zlopez:matrix.org> !info chair 2024-08-01 pcreech
2024-07-18 16:17:32 <@Zlopez:matrix.org> !info chair 2024-07-25 nirik
2024-07-18 16:17:32 <@Zlopez:matrix.org> !topic Next chair
2024-07-18 16:17:32 <@Zlopez:matrix.org> !info magic eight ball says:
2024-07-18 16:17:32 <@Zlopez:matrix.org> !info chair 2024-08-15 - ???
2024-07-18 16:17:32 <@Zlopez:matrix.org> !info chair 2024-08-08 - cancelled (Flock)
2024-07-18 16:17:41 <@jeffiscow:fedora.im> !hi
2024-07-18 16:17:41 <@zodbot:fedora.im> jeff bucher (jeffiscow) - he / him / his
2024-07-18 16:17:56 <@Zlopez:matrix.org> jeffiscow (he/him): Welcome to fedora infra
2024-07-18 16:18:06 <@nirik:matrix.scrye.com> welcome jeffiscow (he/him)
2024-07-18 16:18:29 <@Zlopez:matrix.org> Feel free to reach to us, if you need any guidance
2024-07-18 16:18:41 <@jeffiscow:fedora.im> Hello everyone I'm just trying to learn and soak in as much info as I can.
2024-07-18 16:19:06 <@Zlopez:matrix.org> So I already marked 2024-08-08 as cancelled, as it is at the same time as Flock 2024
2024-07-18 16:19:32 <@Zlopez:matrix.org> And we are looking for a volunteer to run after Flock infra meeting
2024-07-18 16:20:23 <@Zlopez:matrix.org> We don't need to fill it up today and leave it for next meeting
2024-07-18 16:20:33 <@nirik:matrix.scrye.com> yeah, lets leave it for when more folks are around
2024-07-18 16:21:39 <@Zlopez:matrix.org> !info Flock 2024 will happen from 7-11 Aug 2024. See you there!
2024-07-18 16:21:39 <@Zlopez:matrix.org> !info Fedora Infra weekly meeting is cancelled for 2024-08-08 because of Flock 2024
2024-07-18 16:21:39 <@Zlopez:matrix.org> !info Fedora 41 Mass rebuild in progress
2024-07-18 16:21:39 <@Zlopez:matrix.org> !info Remaining issues with new mailman3 deployment are being investigated
2024-07-18 16:21:39 <@Zlopez:matrix.org> !info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
2024-07-18 16:21:39 <@Zlopez:matrix.org> !info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
2024-07-18 16:21:39 <@Zlopez:matrix.org> !topic announcements and information
2024-07-18 16:22:08 <@Zlopez:matrix.org> I added few announcements to the list, does anybody has anything else?
2024-07-18 16:22:29 <@nirik:matrix.scrye.com> I was going to mention the mass rebuild and flock, but now I don't have to. ;)
2024-07-18 16:23:09 <@Zlopez:matrix.org> Hm, what happened?
2024-07-18 16:23:33 <@aggraxis:fedora.im> nothing? you already mentioned it. lol
2024-07-18 16:23:33 <@nirik:matrix.scrye.com> hum?
2024-07-18 16:23:51 <@Zlopez:matrix.org> I see the announcements posted twice
2024-07-18 16:24:04 <@Zlopez:matrix.org> But the second seems to be visible only to me :-D
2024-07-18 16:24:09 <@aggraxis:fedora.im> I just see it once with the bot responding
2024-07-18 16:24:17 <@Zlopez:matrix.org> Let me refresh the browser
2024-07-18 16:24:44 <@nirik:matrix.scrye.com> I only see once.
2024-07-18 16:24:49 <@Zlopez:matrix.org> That's better
2024-07-18 16:24:53 <@markrosenbaum:fedora.im> Fire has spawned at work 😭, gtg for now yall. If I'm not back before it ends have a great meeting
2024-07-18 16:25:00 <@nirik:matrix.scrye.com> well, I mean meetbot repeats them as links...
2024-07-18 16:25:20 <@nirik:matrix.scrye.com> or infos... whatever
2024-07-18 16:25:20 <@Zlopez:matrix.org> That wasn't it it looked like I have it sent twice
2024-07-18 16:25:25 <@nirik:matrix.scrye.com> strange
2024-07-18 16:25:34 <@Zlopez:matrix.org> After refresh it's gone
2024-07-18 16:25:55 <@Zlopez:matrix.org> So anything else for announcements?
2024-07-18 16:27:25 <@Zlopez:matrix.org> It doesn't seem so, let's look what else awaits us
2024-07-18 16:27:33 <@Zlopez:matrix.org> !topic Oncall
2024-07-18 16:27:33 <@Zlopez:matrix.org> !info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
2024-07-18 16:27:33 <@Zlopez:matrix.org> !info https://fedoraproject.org/wiki/Infrastructure/Oncall
2024-07-18 16:27:40 <@Zlopez:matrix.org> !info zlopez is on call from 2024-07-12 to 2024-07-18
2024-07-18 16:27:40 <@Zlopez:matrix.org> !info ??? is on call from 2024-07-19 to 2024-07-25
2024-07-18 16:27:40 <@Zlopez:matrix.org> !info ??? is on call from 2024-07-26 to 2024-08-01
2024-07-18 16:27:52 <@nirik:matrix.scrye.com> I can take the next stint
2024-07-18 16:28:00 <@Zlopez:matrix.org> Sold!
2024-07-18 16:28:08 <@Zlopez:matrix.org> !info nirik is on call from 2024-07-19 to 2024-07-25
2024-07-18 16:28:40 <@Zlopez:matrix.org> Anybody want to go for week after that?
2024-07-18 16:29:11 <@Zlopez:matrix.org> !oncall
2024-07-18 16:29:13 <@zodbot:fedora.im> The following people are oncall:
2024-07-18 16:29:13 <@zodbot:fedora.im> If they do not respond, please file a ticket (https://pagure.io/fedora-infrastructure/issues)
2024-07-18 16:29:13 <@zodbot:fedora.im> ● @nirik:matrix.scrye.com (kevin) Current Time for them: 09:29 (US/Pacific)
2024-07-18 16:29:13 <@zodbot:fedora.im> 
2024-07-18 16:29:27 <@Zlopez:matrix.org> Changed the oncall :-)
2024-07-18 16:29:33 <@nirik:matrix.scrye.com> thanks
2024-07-18 16:30:31 <@Zlopez:matrix.org> Let's keep the decision for next week as there isn't much folks around
2024-07-18 16:30:39 <@Zlopez:matrix.org> !info Summary of last week: (from current oncall)
2024-07-18 16:30:43 <@Zlopez:matrix.org> That would be me
2024-07-18 16:30:52 <@Zlopez:matrix.org> I didn't saw any ping for the whole week
2024-07-18 16:31:15 <@nirik:matrix.scrye.com> yeah...been quiet on that front
2024-07-18 16:32:01 <@Zlopez:matrix.org> But there was the issue with the mail spammer on issue tracker
2024-07-18 16:32:30 <@Zlopez:matrix.org> It seems that it's now gone I didn't saw any message yesterday or today from the account
2024-07-18 16:33:14 <@nirik:matrix.scrye.com> so, yeah, I need to reply to that ticket...
2024-07-18 16:33:27 <@nirik:matrix.scrye.com> I also removed their watch from the project, so perhaps thats what stopped it.
2024-07-18 16:33:40 <@Zlopez:matrix.org> That could be it
2024-07-18 16:33:47 <@Zlopez:matrix.org> How can somebody do that?
2024-07-18 16:34:03 <@nirik:matrix.scrye.com> because I thiink it was commit to issue -> email to user -> user sends reply back
2024-07-18 16:34:14 <@nirik:matrix.scrye.com> there's a pagure-admin command. I will add it to the ticket.
2024-07-18 16:34:58 <@nirik:matrix.scrye.com> pagure-admin update-watch fedora-infrastructure  <username> -s 0
2024-07-18 16:35:17 <@Zlopez:matrix.org> It would be probably best to update it in docs as well
2024-07-18 16:35:31 <@nirik:matrix.scrye.com> yeah, I can.
2024-07-18 16:35:32 <@Zlopez:matrix.org> I already have PR open for pagure spam guide https://pagure.io/infra-docs-fpo/pull-request/308
2024-07-18 16:35:42 <@nirik:matrix.scrye.com> ok.
2024-07-18 16:35:52 <@Zlopez:matrix.org> Just to mention the sanitize-spam-user command
2024-07-18 16:36:24 <@nirik:matrix.scrye.com> On a somewhat related note... pagure.io has been under heavy load lately. It looks like tons of random .cn ip's doing a HEAD on forked project files then a GET...
2024-07-18 16:36:42 <@nirik:matrix.scrye.com> I have tried to block them, but don't know if I caught legit traffic in there. ;(
2024-07-18 16:37:17 <@nirik:matrix.scrye.com> and it's a lot of IP's, so it's in a ipset right now... not in ansible (was waiting to see what it broke before pushing any commits)
2024-07-18 16:38:06 <@Zlopez:matrix.org> I thought that we aren't adding those to ansible
2024-07-18 16:39:05 <@nirik:matrix.scrye.com> I did add some ip's we blocked... but adding these would be... tricky. I am not sure how to handle it. I was hoping perhaps by rejecting them they would stop and go away.
2024-07-18 16:39:36 <@Zlopez:matrix.org> So it will be in firewall rules for pagure?
2024-07-18 16:39:50 <@nirik:matrix.scrye.com> yes, it's a iptables rule to check the ipset...
2024-07-18 16:39:53 <@nirik:matrix.scrye.com> so far:
2024-07-18 16:39:57 <@nirik:matrix.scrye.com> pkts      bytes target     prot opt in     out     source               destination         
2024-07-18 16:39:57 <@nirik:matrix.scrye.com>  1032414 61793439 REJECT     tcp  --  any    any     anywhere             anywhere
2024-07-18 16:40:06 <@nirik:matrix.scrye.com> about a million packets blocked
2024-07-18 16:40:29 <@Zlopez:matrix.org> That looks like some botnet
2024-07-18 16:40:56 <@nirik:matrix.scrye.com> could be. it's really hard to isolate because it uses different user agents, it's hitting different projects, etc.
2024-07-18 16:41:05 <@nirik:matrix.scrye.com> or it could be some AI scraper.
2024-07-18 16:41:20 <@nirik:matrix.scrye.com> anyhow, will let it run and try and think what we could do longer term.
2024-07-18 16:41:34 <@Zlopez:matrix.org> Everything is AI these days :/
2024-07-18 16:41:46 <@Zlopez:matrix.org> Let me move to next topic
2024-07-18 16:41:51 <@Zlopez:matrix.org> !info Go over existing out items and fix
2024-07-18 16:41:51 <@Zlopez:matrix.org> !topic Monitoring discussion [nirik]
2024-07-18 16:41:51 <@Zlopez:matrix.org> !info https://nagios.fedoraproject.org/nagios 
2024-07-18 16:42:02 <@nirik:matrix.scrye.com> so, looking better here this week...
2024-07-18 16:42:22 <@nirik:matrix.scrye.com> I managed to get firmware updated and reinstalled the old emag aarch64 boxes, so they are all up now and off nagios
2024-07-18 16:42:41 <@nirik:matrix.scrye.com> 5 down hosts right now:
2024-07-18 16:42:52 <@nirik:matrix.scrye.com> one aarch64 box with memory errors. Not sure we can easily fix it.
2024-07-18 16:43:08 <@nirik:matrix.scrye.com> 2 buildhw-x86's that wont power on (needs an onsite power cycle as the next step)
2024-07-18 16:43:21 <@nirik:matrix.scrye.com> 2 buildhw-a64's that never came back from reboot and I can't get to mgmt (needs onsite)
2024-07-18 16:43:22 <@Zlopez:matrix.org> Sounds like a hardware issue
2024-07-18 16:43:43 <@nirik:matrix.scrye.com> otherwise we are looking pretty reasonable
2024-07-18 16:43:55 <@nirik:matrix.scrye.com> yeah, bad memory I think. I will look at it more later today.
2024-07-18 16:44:20 <@Zlopez:matrix.org> I can see copr-be.aws error
2024-07-18 16:44:50 <@nirik:matrix.scrye.com> yeah, we should ask them about that.
2024-07-18 16:44:55 <@nirik:matrix.scrye.com> I think their limit is too low...
2024-07-18 16:45:05 <@nirik:matrix.scrye.com> it's a check for a test build finishing in X time
2024-07-18 16:45:13 <@Zlopez:matrix.org> And fedbadges message queue alert, but I assume this is related to mass rebuild
2024-07-18 16:46:05 <@nirik:matrix.scrye.com> yeah, not sure on that one, but it did clear up
2024-07-18 16:46:11 <@nirik:matrix.scrye.com> or did it?
2024-07-18 16:46:24 <@Zlopez:matrix.org> Not yet, I still see it in nagios
2024-07-18 16:46:30 <@nirik:matrix.scrye.com> ugh, no...
2024-07-18 16:46:34 <@nirik:matrix.scrye.com> we need to look at that.
2024-07-18 16:46:44 <@nirik:matrix.scrye.com> 600k messages is... bad
2024-07-18 16:47:10 <@nirik:matrix.scrye.com> if that gets too high it can affect other queues...
2024-07-18 16:47:55 <@Zlopez:matrix.org> Yeah, the number is very high
2024-07-18 16:48:20 <@nirik:matrix.scrye.com> I can look for anything obvious... if not we could perhaps ask ryanlerch to look? was he involved in the porting to openshift?
2024-07-18 16:48:38 <@Zlopez:matrix.org> Maybe the consumer is just stuck
2024-07-18 16:49:45 <@nirik:matrix.scrye.com> it's not got any errors...
2024-07-18 16:50:06 <@Zlopez:matrix.org> Let's move to open floor as we have only 10 minutes left
2024-07-18 16:50:19 <@Zlopez:matrix.org> !topic Open Floor
2024-07-18 16:50:48 <@Zlopez:matrix.org> Does the fedbadges running in Openshift now?
2024-07-18 16:51:28 <@nirik:matrix.scrye.com> Flock is coming up fast...
2024-07-18 16:51:52 <@nirik:matrix.scrye.com> I plan to gather the workshop ideas and post to the discussion thread. Perhaps we can decide some things we want to do in advance.
2024-07-18 16:52:12 <@aggraxis:fedora.im> I'm looking forward to seeing you guys
2024-07-18 16:52:23 <@Zlopez:matrix.org> I'm already working on the Fedora Infra overview presentation
2024-07-18 16:52:33 <@nirik:matrix.scrye.com> Paul Maconi: you gonna make it? cool... it's a fun conference (I think)
2024-07-18 16:52:41 <@nirik:matrix.scrye.com> I need to work on my other talk
2024-07-18 16:52:48 <@Zlopez:matrix.org> I should have first version ready tomorrow, I will share that with you nirik 
2024-07-18 16:52:56 <@aggraxis:fedora.im> bringing my oldest with me, too. I think he'll get inspired by it
2024-07-18 16:53:13 <@nirik:matrix.scrye.com> sure, happy to review.
2024-07-18 16:53:20 <@nirik:matrix.scrye.com> nice. :)
2024-07-18 16:53:48 <@Zlopez:matrix.org> Should I try and restart the consumer pods in badges project?
2024-07-18 16:54:13 <@Zlopez:matrix.org> I don't see any message processed till yesterday
2024-07-18 16:54:25 <@nirik:matrix.scrye.com> I actually tried that the other day, but it can't hurt to try again. ;)
2024-07-18 16:54:41 <@jeffiscow:fedora.im> will there be a steam of flock?
2024-07-18 16:54:54 <@aggraxis:fedora.im> yes I think so
2024-07-18 16:55:01 <@nirik:matrix.scrye.com> jeffiscow (he/him): not sure, but I hope so.
2024-07-18 16:55:29 <@Zlopez:matrix.org> It should be, not sure if you could also ask questions through it
2024-07-18 16:55:32 <@jeffiscow:fedora.im> I'm sure well have recording of and talks and stuff
2024-07-18 16:55:43 <@nirik:matrix.scrye.com> yep. Thats standard.
2024-07-18 16:56:07 <@nirik:matrix.scrye.com> There's also a matrix channel where there is a lot of discussion before, during and after the event. ;)
2024-07-18 16:56:57 <@Zlopez:matrix.org> I'm probably not there
2024-07-18 16:57:17 <@nirik:matrix.scrye.com> #flock:fedoraproject.org
2024-07-18 16:58:20 <@Zlopez:matrix.org> Joined :-)
2024-07-18 16:58:39 <@jeffiscow:fedora.im> I really hope to go inperson one day seem like a good time.
2024-07-18 16:58:50 <@Zlopez:matrix.org> I'm doing the new build of fedbadges as the last one happened on 10th July, maybe there is something new
2024-07-18 16:59:31 <@Zlopez:matrix.org> We are at the end of this meeting
2024-07-18 16:59:43 <@Zlopez:matrix.org> It was a nice discussion with all of you :-)
2024-07-18 16:59:57 <@Zlopez:matrix.org> Thanks for joining
2024-07-18 17:00:03 <@Zlopez:matrix.org> !endmeeting
2024-07-18 17:00:04 <@aggraxis:fedora.im> thanks for running the meeting Zlopez !