2025-03-27 17:02:03 <@Zlopez:matrix.org> !startmeeting Infrastructure (2025-03-20)
2025-03-27 17:02:05 <@meetbot:fedora.im> Meeting started at 2025-03-27 17:02:03 UTC
2025-03-27 17:02:05 <@meetbot:fedora.im> The Meeting name is 'Infrastructure (2025-03-20)'
2025-03-27 17:02:13 <@Zlopez:matrix.org> !meetingname infrastructure
2025-03-27 17:02:13 <@Zlopez:matrix.org> !topic ahoy
2025-03-27 17:02:13 <@Zlopez:matrix.org> !info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
2025-03-27 17:02:13 <@Zlopez:matrix.org> !info About our team: https://docs.fedoraproject.org/en-US/cle/
2025-03-27 17:02:13 <@Zlopez:matrix.org> !info Agenda is at: https://board.net/p/fedora-infra
2025-03-27 17:02:13 <@Zlopez:matrix.org> !chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp
2025-03-27 17:02:14 <@meetbot:fedora.im> The Meeting Name is now infrastructure
2025-03-27 17:02:38 <@Zlopez:matrix.org> !hi
2025-03-27 17:02:40 <@zodbot:fedora.im> Michal Konecny (zlopez)
2025-03-27 17:03:03 <@Zlopez:matrix.org> I'm taking over today meeting instead of @pcreech:matrix.org
2025-03-27 17:03:11 <@nirik:matrix.scrye.com> morning
2025-03-27 17:03:59 <@Zlopez:matrix.org> Welcome everyone to today infra meeting 👋
2025-03-27 17:04:20 <@nirik:matrix.scrye.com> well, all two of us anyhow. ;)
2025-03-27 17:04:49 <@Zlopez:matrix.org> All two of us welcome here :-D
2025-03-27 17:07:20 <@Zlopez:matrix.org> !info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
2025-03-27 17:07:20 <@Zlopez:matrix.org> !topic New folks introductions
2025-03-27 17:07:20 <@Zlopez:matrix.org> !info Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/
2025-03-27 17:07:26 <@Zlopez:matrix.org> Anyone new around?
2025-03-27 17:09:26 <@Zlopez:matrix.org> It doesn't seem so
2025-03-27 17:09:27 <@nirik:matrix.scrye.com> must be everyone is having a party and didn't invite us. ;)
2025-03-27 17:10:02 <@Zlopez:matrix.org> Something like that, maybe it's Friday already and everybody already left :-D
2025-03-27 17:10:04 <@Zlopez:matrix.org> !topic Next chair
2025-03-27 17:10:04 <@Zlopez:matrix.org> !info magic eight ball says:
2025-03-27 17:10:04 <@Zlopez:matrix.org> !info chair 2025-04-24 - ???
2025-03-27 17:10:04 <@Zlopez:matrix.org> !info chair 2025-03-27 - @pcreech
2025-03-27 17:10:04 <@Zlopez:matrix.org> !info chair 2025-04-03 - @Zlopez
2025-03-27 17:10:04 <@Zlopez:matrix.org> !info chair 2025-04-10 - ???
2025-03-27 17:10:04 <@Zlopez:matrix.org> !info chair 2025-04-17 - ???
2025-03-27 17:10:33 <@Zlopez:matrix.org> @nirik:matrix.scrye.com Do you want to take one of those dates?
2025-03-27 17:11:34 <@nirik:matrix.scrye.com> could do the 17th?
2025-03-27 17:11:48 <@nirik:matrix.scrye.com> or I could take next week since you took over this week?
2025-03-27 17:12:10 <@Zlopez:matrix.org> I can take the 10th instead, that sounds good
2025-03-27 17:12:30 <@Zlopez:matrix.org> !info chair 2025-04-03 - @nirik
2025-03-27 17:12:41 <@Zlopez:matrix.org> !info chair 2025-04-10 - @Zlopez
2025-03-27 17:13:13 <@nirik:matrix.scrye.com> Sounds good
2025-03-27 17:13:34 <@Zlopez:matrix.org> As we are only 2 here, let's skip to next topic
2025-03-27 17:13:35 <@Zlopez:matrix.org> !info CLE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
2025-03-27 17:13:35 <@Zlopez:matrix.org> !topic announcements and information
2025-03-27 17:13:35 <@Zlopez:matrix.org> !info CLE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
2025-03-27 17:13:54 <@Zlopez:matrix.org> Is the time still correct?
2025-03-27 17:14:41 <@nirik:matrix.scrye.com> no, it's 19UTC now
2025-03-27 17:14:50 <@nirik:matrix.scrye.com> (For the second one)
2025-03-27 17:14:52 <@Zlopez:matrix.org> We will probably adjust it as well after DST in EU
2025-03-27 17:15:01 <@nirik:matrix.scrye.com> !info final freeze starts next tuesday
2025-03-27 17:15:13 <@Zlopez:matrix.org> @info The DST in EU is happening this weekend
2025-03-27 17:15:19 <@gwmngilfen:fedora.im> finally 😉
2025-03-27 17:15:40 <@Zlopez:matrix.org> !info The DST in EU is happening this weekend
2025-03-27 17:15:58 <@nirik:matrix.scrye.com> sadly, it doesn't really save any daylight at all. ;)
2025-03-27 17:16:01 <@Zlopez:matrix.org> I fixed the time of the meeting in agenda
2025-03-27 17:17:41 <@Zlopez:matrix.org> Not anymore, I hope they just cancel it one day
2025-03-27 17:17:56 <@nirik:matrix.scrye.com> so say we all.
2025-03-27 17:18:05 <@markrosenbaum:fedora.im> We can hope
2025-03-27 17:18:55 <@Zlopez:matrix.org> !info OpenID EOL in Fedora Infra is set to 20-05-2025
2025-03-27 17:19:07 <@Zlopez:matrix.org> Anything else to announce?
2025-03-27 17:21:19 <@nirik:matrix.scrye.com> not off hand
2025-03-27 17:22:14 <@Zlopez:matrix.org> OK, let's continue
2025-03-27 17:22:25 <@Zlopez:matrix.org> !info ??? is on call from 2025-04-11 to 2025-04-17
2025-03-27 17:22:25 <@Zlopez:matrix.org> !info ??? is on call from 2025-04-18 to 2025-04-24
2025-03-27 17:22:25 <@Zlopez:matrix.org> !info ??? is on call from 2025-04-04 to 2025-04-10
2025-03-27 17:22:25 <@Zlopez:matrix.org> !info @markrosenbaum is on call from 2025-03-28 to 2025-04-03
2025-03-27 17:22:25 <@Zlopez:matrix.org> !info @nirik is on call from 2025-03-21 to 2025-03-27
2025-03-27 17:22:25 <@Zlopez:matrix.org> !info https://docs.fedoraproject.org/en-US/infra/day_to_day_fedora/#_the_oncall_role_in_our_team
2025-03-27 17:22:25 <@Zlopez:matrix.org> !topic Oncall
2025-03-27 17:22:45 <@nirik:matrix.scrye.com> I noticed no oncall calls.
2025-03-27 17:22:48 <@smooge:fedora.im> next week is freeze again for final?
2025-03-27 17:22:53 <@smooge:fedora.im> oh crap sorry
2025-03-27 17:22:56 <@smooge:fedora.im> wrong channel
2025-03-27 17:22:56 <@nirik:matrix.scrye.com> yep
2025-03-27 17:23:06 <@Zlopez:matrix.org> I will take the one from 2025-04-04 to 2025-04-10
2025-03-27 17:23:20 <@Zlopez:matrix.org> !info @Zlopez is on call from 2025-04-04 to 2025-04-10
2025-03-27 17:23:42 <@Zlopez:matrix.org> Anybody wants the other weeks?
2025-03-27 17:24:15 <@markrosenbaum:fedora.im> I’d personally have to see as they get closer 
2025-03-27 17:26:02 <@Zlopez:matrix.org> !info Summary of last week: (from current oncall)
2025-03-27 17:26:17 <@Zlopez:matrix.org> As @nirik:matrix.scrye.com already mentioned there were no oncall pings
2025-03-27 17:26:23 <@Zlopez:matrix.org> !topic Monitoring discussion [nirik]
2025-03-27 17:26:23 <@Zlopez:matrix.org> !info Go over existing items and fix them
2025-03-27 17:26:23 <@Zlopez:matrix.org> !info https://nagios.fedoraproject.org/nagios 
2025-03-27 17:26:29 <@nirik:matrix.scrye.com> lets see...
2025-03-27 17:26:45 <@nirik:matrix.scrye.com> just the same ones as last week.
2025-03-27 17:27:06 <@nirik:matrix.scrye.com> 1 host down, 1 host logdetective02, and a bunch of datagrepper checks
2025-03-27 17:27:21 <@nirik:matrix.scrye.com> we have been getting a lot of mailman alerts... but hopefully better now
2025-03-27 17:27:35 <@Zlopez:matrix.org> Working on that, but the reviews are taking a plenty of time
2025-03-27 17:27:53 <@Zlopez:matrix.org> Hopefully the improvement will work
2025-03-27 17:28:05 <@nirik:matrix.scrye.com> yeah... is someone reviewing now? or just waiting?
2025-03-27 17:28:58 <@Zlopez:matrix.org> I got few comments few days ago, addressed them and now waiting for approval
2025-03-27 17:29:18 <@Zlopez:matrix.org> Then the second package will have to go as well
2025-03-27 17:29:33 <@Zlopez:matrix.org> Fixed the oncall in the meantime
2025-03-27 17:29:34 <@nirik:matrix.scrye.com> ok. Let me know if you need reviewers... I can try and do it sometime
2025-03-27 17:29:35 <@Zlopez:matrix.org> !oncall
2025-03-27 17:29:37 <@zodbot:fedora.im> ● @markrosenbaum:fedora.im (markrosenbaum) Current Time for them: 13:29 (US/Eastern)
2025-03-27 17:29:37 <@zodbot:fedora.im> The following people are oncall:
2025-03-27 17:29:37 <@zodbot:fedora.im> If they do not respond, please file a ticket (https://pagure.io/fedora-infrastructure/issues)
2025-03-27 17:29:37 <@zodbot:fedora.im> 
2025-03-27 17:30:07 <@Zlopez:matrix.org> Thanks
2025-03-27 17:30:21 <@Zlopez:matrix.org> Now I have a topic for discussion
2025-03-27 17:30:27 <@Zlopez:matrix.org> #topic Blocking IPs process - zlopez
2025-03-27 17:30:30 <@nirik:matrix.scrye.com> I have one too. Might be the same one?
2025-03-27 17:30:42 <@Zlopez:matrix.org> !topic Blocking IPs process - zlopez
2025-03-27 17:31:04 <@nirik:matrix.scrye.com> yeah, AI scrapers. :)
2025-03-27 17:31:42 <@Zlopez:matrix.org> We had a lot of IP added as blocked lately, so it would be nice to have some kind of unified process, so we don't need to look for them all over the place, when somebody can't connect
2025-03-27 17:32:03 <@nirik:matrix.scrye.com> Yep. So... do we care if they are public or not?
2025-03-27 17:32:15 <@nirik:matrix.scrye.com> I go back and forth on it.
2025-03-27 17:32:37 <@nirik:matrix.scrye.com> If we don't care we should put them in ansible. If we do, I guess we could use ansible-private.
2025-03-27 17:32:53 <@Zlopez:matrix.org> I don't think the IP address is actually something that we should keep private
2025-03-27 17:33:09 <@Zlopez:matrix.org> It's a public IP at the end
2025-03-27 17:33:49 <@Zlopez:matrix.org> Maybe somebody can use it as a blocklist for their server
2025-03-27 17:34:00 <@nirik:matrix.scrye.com> also, our process for finding IP's to block is not very set. At least for me, I tend to try a bunch of places to look for them, it's more feeling that a specific set process.
2025-03-27 17:34:04 <@Zlopez:matrix.org> We also should have an option to add comment to each entry
2025-03-27 17:34:17 <@Zlopez:matrix.org> With reason for blocking
2025-03-27 17:34:22 <@nirik:matrix.scrye.com> yeah, I think we should use a ipset for this (they do support comments)
2025-03-27 17:34:53 <@nirik:matrix.scrye.com> but the reason is only not too clear too tho right... "was hitting src.fp.o a lot from this network" ?
2025-03-27 17:34:56 <@Zlopez:matrix.org> Same I usually just got by feeling when blocking them, looking for patterns that look suspicious to me
2025-03-27 17:35:03 <@Zlopez:matrix.org> Same I usually just go by feeling when blocking them, looking for patterns that look suspicious to me
2025-03-27 17:35:32 <@nirik:matrix.scrye.com> so, we might consider the proof of work thing I guess instead of trying to manually block things.
2025-03-27 17:35:42 <@markrosenbaum:fedora.im> Or yk, check if they are blocked
2025-03-27 17:36:13 <@nirik:matrix.scrye.com> (side note: ipsets also easily allow you to say 'is this specific ip matching any of the nets in the set')
2025-03-27 17:36:27 <@Zlopez:matrix.org> I'm for using something that would help us with this
2025-03-27 17:36:36 <@markrosenbaum:fedora.im> I wonder if there’s a way to have the list of blocked IPs not affected by the IP block
2025-03-27 17:37:39 <@markrosenbaum:fedora.im> If we do make them public
2025-03-27 17:37:44 <@nirik:matrix.scrye.com> My two concerns about the proof of work thing (anubis?) are: 1) if a bunch of people hit a specific file/thing it could be very slow and 2) if the scrapers decide to drop "Mozilla" from their agent to avoid it, it will become an arms race to figure out what they are using.
2025-03-27 17:37:54 <@Zlopez:matrix.org> What do you mean exactly?
2025-03-27 17:38:38 <@nirik:matrix.scrye.com> depending on where its blocked I guess... if we are talking about blocking ips globally then no, but if we are only blocking say proxies then pagure.io would still be unblocked.
2025-03-27 17:38:43 <@Zlopez:matrix.org> I understand the concerns
2025-03-27 17:39:31 <@Zlopez:matrix.org> Is there anything else that is not behind proxies that the AI scrapers are targeting?
2025-03-27 17:39:38 <@markrosenbaum:fedora.im> Could we host like a json of blocked IPs on some other site?
2025-03-27 17:39:41 <@Zlopez:matrix.org> Or just pagure?
2025-03-27 17:39:42 <@nirik:matrix.scrye.com> pagure.io
2025-03-27 17:40:09 <@nirik:matrix.scrye.com> just pagure that I know of off hand, everything else is behind proxies.
2025-03-27 17:40:18 <@nirik:matrix.scrye.com> we could... just more work
2025-03-27 17:41:33 <@nirik:matrix.scrye.com> So, right now we have some blocks on pagure.io (which I put back after reboot yesterday) and no blocks on proxies.
2025-03-27 17:41:59 <@nirik:matrix.scrye.com> and things seem fine. I am not sure if thats because they stopped, or because they are just busy elsewhere.
2025-03-27 17:42:17 <@Zlopez:matrix.org> I assume the later
2025-03-27 17:42:26 <@markrosenbaum:fedora.im> I definitely wouldn't assume they're gone
2025-03-27 17:42:49 <@nirik:matrix.scrye.com> on pagure, I also increased resources a lot... many more cpus, etc...
2025-03-27 17:43:44 <@Zlopez:matrix.org> We tweaked the httpd resources as well to get better performance
2025-03-27 17:43:48 <@nirik:matrix.scrye.com> I guess my thought right now is to not be hasty. ;) But explore what it would take to setup a ipset thing in ansible and what it would take to deploy anubis if we had to.
2025-03-27 17:44:19 <@Zlopez:matrix.org> I'm OK with opening spike tickets for that, so we don't forget about it
2025-03-27 17:44:51 <@nirik:matrix.scrye.com> sure.
2025-03-27 17:45:19 <@Zlopez:matrix.org> Let me add that to my TODO, I will do that tomorrow
2025-03-27 17:45:44 <@nirik:matrix.scrye.com> also I think James was gonna mention nftables handling here... I am not sure we could do much directly there tho (or differently from iptables)
2025-03-27 17:46:49 <@nirik:matrix.scrye.com> Oh, one other thing:
2025-03-27 17:47:05 <@Zlopez:matrix.org> We will see, I'm not against using nftables if that will make it easier
2025-03-27 17:47:41 <@nirik:matrix.scrye.com> well, it could rate limit... but I don't think that would help this particular thing
2025-03-27 17:48:03 <@Zlopez:matrix.org> I meant with the ipset setup in ansible
2025-03-27 17:48:32 <@nirik:matrix.scrye.com> There is also mod_qos in apache. That could at least allow us to prioritize our ip's over others...
2025-03-27 17:49:12 <@Zlopez:matrix.org> All of that could be mentioned in the spike tickets :-)
2025-03-27 17:49:22 <@nirik:matrix.scrye.com> So, perhaps we could just make a ticket with all these ideas, explore them as time permits and rediscuss once we have more data?
2025-03-27 17:49:27 <@nirik:matrix.scrye.com> yeah
2025-03-27 17:50:26 <@nirik:matrix.scrye.com> Finally, should we leave the things blocked that are blocked right now on pagure.io? or clear it out? I had left it clear, but re-added them when some folks were noticing high load.
2025-03-27 17:51:02 <@Zlopez:matrix.org> I think we can safely remove them now and see if the issue returns
2025-03-27 17:51:09 <@nirik:matrix.scrye.com> ok
2025-03-27 17:51:15 <@nirik:matrix.scrye.com> we can always add them back
2025-03-27 17:51:40 <@Zlopez:matrix.org> We are almost at end of the meeting, so let me switch to open floor
2025-03-27 17:51:59 <@Zlopez:matrix.org> !topic Open Floor
2025-03-27 17:52:08 <@Zlopez:matrix.org> Anything to discuss on open floor?
2025-03-27 17:52:34 <@nirik:matrix.scrye.com> Thanks to James, Gwmngilfen and phsmoura for helping out with updates/reboots this week. I appreciate it!
2025-03-27 17:53:02 <@jcline:fedora.im> I did have one thing, which is really a question about a question...
2025-03-27 17:53:18 <@nirik:matrix.scrye.com> meta!
2025-03-27 17:53:29 <@jcline:fedora.im> 
2025-03-27 17:53:29 <@jcline:fedora.im> I've been doing some work on the client side for Sigul, and before I invested more time into I wanted to know where the infra folks would like signing to go longer term. I inquired with the konflux folks about their signing story and they said some people internally are looking at Vault, but that there's a way to hook in via messages and use something like Sigul.
2025-03-27 17:53:29 <@jcline:fedora.im> Konflux seems a bit handy-wavy to me still, but in any case, I wanted to bring up the question of what folks _want_ to run. Continue with Sigul in some form, or switch to something off-the-shelf like Vault?
2025-03-27 17:53:29 <@jcline:fedora.im> 
2025-03-27 17:53:29 <@jcline:fedora.im> I think this is probably best discussed async, so my real question is where is the best place to start this kind of conversation? Discourse?
2025-03-27 17:53:55 <@nirik:matrix.scrye.com> discourse is probibly fine.
2025-03-27 17:54:19 <@nirik:matrix.scrye.com> IMHO, konflux is a ways off, so I think we are on sigul for a while...
2025-03-27 17:54:47 <@Zlopez:matrix.org> Konflux will take some time
2025-03-27 17:55:04 <@nirik:matrix.scrye.com> and I am not sure about vault. I don't like depending on external non free stuff if we can avoid it... but perhaps thats just me
2025-03-27 17:55:54 <@jcline:fedora.im> I had to admin a vault at a prior job and personally, I wouldn't want to run it. But I'm absolutely biased, as well, since the Sigul stuff has been interesting to work on.
2025-03-27 17:56:00 <@Zlopez:matrix.org> It's not only you, we should avoid things we don't have control on, especially not open ones
2025-03-27 17:56:23 <@nirik:matrix.scrye.com> I had some hope we could leverage sigstore, but it doesn't seem like it's really built for our use cases.
2025-03-27 17:56:35 <@nirik:matrix.scrye.com> https://www.sigstore.dev/
2025-03-27 17:57:55 <@jcline:fedora.im> I can certainly look at it to double check
2025-03-27 17:58:31 <@jcline:fedora.im> Okay, well I will start a discourse discussion just to give folks more opportunity to think about it all and weigh in
2025-03-27 17:58:42 <@nirik:matrix.scrye.com> Jeremy Cline: so if sigul was something we would at least be still using for a year or two... what was the next thing you were thinking of there? replace the bridge/vault part?
2025-03-27 17:59:00 <@Zlopez:matrix.org> We have only two minutes left, so discourse discussion will be definitely better
2025-03-27 17:59:30 <@jcline:fedora.im> Yes. Well, I have a client that does a limited set of commands, so I would add all the existing commands to that and stabilize the interface a bit, and then move to the bridge/vault part
2025-03-27 17:59:49 <@zodbot:fedora.im> zlopez has already given cookies to jcline during the F41 timeframe
2025-03-27 18:00:36 <@Zlopez:matrix.org> We are at the end of our time
2025-03-27 18:00:37 <@jcline:fedora.im> The bridge I don't think will be too much additional work, the server bit is a bit more. It might be possible to do the server bit-by-bit so it's not an all-or-nothing situation, but I'd have to think carefully about that
2025-03-27 18:00:51 <@zodbot:fedora.im> kevin gave a cookie to jcline. They now have 89 cookies, 6 of which were obtained in the Fedora 41 release cycle
2025-03-27 18:01:19 <@Zlopez:matrix.org> Thanks everybody for coming today :-)
2025-03-27 18:01:32 <@Zlopez:matrix.org> Let's see you next week at the same time
2025-03-27 18:01:33 <@Zlopez:matrix.org> !endmeeting