<@Zlopez:matrix.org>
17:02:03
!startmeeting Infrastructure (2025-03-20)
<@meetbot:fedora.im>
17:02:05
Meeting started at 2025-03-27 17:02:03 UTC
<@meetbot:fedora.im>
17:02:05
The Meeting name is 'Infrastructure (2025-03-20)'
<@Zlopez:matrix.org>
17:02:13
!meetingname infrastructure
<@Zlopez:matrix.org>
17:02:13
!topic ahoy
<@Zlopez:matrix.org>
17:02:13
!info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
<@Zlopez:matrix.org>
17:02:13
!info About our team: https://docs.fedoraproject.org/en-US/cle/
<@Zlopez:matrix.org>
17:02:13
!info Agenda is at: https://board.net/p/fedora-infra
<@Zlopez:matrix.org>
17:02:13
!chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp
<@meetbot:fedora.im>
17:02:14
The Meeting Name is now infrastructure
<@Zlopez:matrix.org>
17:02:38
!hi
<@zodbot:fedora.im>
17:02:40
Michal Konecny (zlopez)
<@Zlopez:matrix.org>
17:03:03
I'm taking over today meeting instead of @pcreech:matrix.org
<@nirik:matrix.scrye.com>
17:03:11
morning
<@Zlopez:matrix.org>
17:03:59
Welcome everyone to today infra meeting 👋
<@nirik:matrix.scrye.com>
17:04:20
well, all two of us anyhow. ;)
<@Zlopez:matrix.org>
17:04:49
All two of us welcome here :-D
<@Zlopez:matrix.org>
17:07:20
!info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
<@Zlopez:matrix.org>
17:07:20
!topic New folks introductions
<@Zlopez:matrix.org>
17:07:20
!info Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/
<@Zlopez:matrix.org>
17:07:26
Anyone new around?
<@Zlopez:matrix.org>
17:09:26
It doesn't seem so
<@nirik:matrix.scrye.com>
17:09:27
must be everyone is having a party and didn't invite us. ;)
<@Zlopez:matrix.org>
17:10:02
Something like that, maybe it's Friday already and everybody already left :-D
<@Zlopez:matrix.org>
17:10:04
!topic Next chair
<@Zlopez:matrix.org>
17:10:04
!info magic eight ball says:
<@Zlopez:matrix.org>
17:10:04
!info chair 2025-04-24 - ???
<@Zlopez:matrix.org>
17:10:04
!info chair 2025-03-27 - @pcreech
<@Zlopez:matrix.org>
17:10:04
!info chair 2025-04-03 - @Zlopez
<@Zlopez:matrix.org>
17:10:04
!info chair 2025-04-10 - ???
<@Zlopez:matrix.org>
17:10:04
!info chair 2025-04-17 - ???
<@Zlopez:matrix.org>
17:10:33
@nirik:matrix.scrye.com Do you want to take one of those dates?
<@nirik:matrix.scrye.com>
17:11:34
could do the 17th?
<@nirik:matrix.scrye.com>
17:11:48
or I could take next week since you took over this week?
<@Zlopez:matrix.org>
17:12:10
I can take the 10th instead, that sounds good
<@Zlopez:matrix.org>
17:12:30
!info chair 2025-04-03 - @nirik
<@Zlopez:matrix.org>
17:12:41
!info chair 2025-04-10 - @Zlopez
<@nirik:matrix.scrye.com>
17:13:13
Sounds good
<@Zlopez:matrix.org>
17:13:34
As we are only 2 here, let's skip to next topic
<@Zlopez:matrix.org>
17:13:35
!info CLE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
<@Zlopez:matrix.org>
17:13:35
!topic announcements and information
<@Zlopez:matrix.org>
17:13:35
!info CLE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0800 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
<@Zlopez:matrix.org>
17:13:54
Is the time still correct?
<@nirik:matrix.scrye.com>
17:14:41
no, it's 19UTC now
<@nirik:matrix.scrye.com>
17:14:50
(For the second one)
<@Zlopez:matrix.org>
17:14:52
We will probably adjust it as well after DST in EU
<@nirik:matrix.scrye.com>
17:15:01
!info final freeze starts next tuesday
<@Zlopez:matrix.org>
17:15:13
@info The DST in EU is happening this weekend
<@gwmngilfen:fedora.im>
17:15:19
finally 😉
<@Zlopez:matrix.org>
17:15:40
!info The DST in EU is happening this weekend
<@nirik:matrix.scrye.com>
17:15:58
sadly, it doesn't really save any daylight at all. ;)
<@Zlopez:matrix.org>
17:16:01
I fixed the time of the meeting in agenda
<@Zlopez:matrix.org>
17:17:41
Not anymore, I hope they just cancel it one day
<@nirik:matrix.scrye.com>
17:17:56
so say we all.
<@markrosenbaum:fedora.im>
17:18:05
We can hope
<@Zlopez:matrix.org>
17:18:55
!info OpenID EOL in Fedora Infra is set to 20-05-2025
<@Zlopez:matrix.org>
17:19:07
Anything else to announce?
<@nirik:matrix.scrye.com>
17:21:19
not off hand
<@Zlopez:matrix.org>
17:22:14
OK, let's continue
<@Zlopez:matrix.org>
17:22:25
!info ??? is on call from 2025-04-11 to 2025-04-17
<@Zlopez:matrix.org>
17:22:25
!info ??? is on call from 2025-04-18 to 2025-04-24
<@Zlopez:matrix.org>
17:22:25
!info ??? is on call from 2025-04-04 to 2025-04-10
<@Zlopez:matrix.org>
17:22:25
!info @markrosenbaum is on call from 2025-03-28 to 2025-04-03
<@Zlopez:matrix.org>
17:22:25
!info @nirik is on call from 2025-03-21 to 2025-03-27
<@Zlopez:matrix.org>
17:22:25
!info https://docs.fedoraproject.org/en-US/infra/day_to_day_fedora/#_the_oncall_role_in_our_team
<@Zlopez:matrix.org>
17:22:25
!topic Oncall
<@nirik:matrix.scrye.com>
17:22:45
I noticed no oncall calls.
<@smooge:fedora.im>
17:22:48
next week is freeze again for final?
<@smooge:fedora.im>
17:22:53
oh crap sorry
<@smooge:fedora.im>
17:22:56
wrong channel
<@nirik:matrix.scrye.com>
17:22:56
yep
<@Zlopez:matrix.org>
17:23:06
I will take the one from 2025-04-04 to 2025-04-10
<@Zlopez:matrix.org>
17:23:20
!info @Zlopez is on call from 2025-04-04 to 2025-04-10
<@Zlopez:matrix.org>
17:23:42
Anybody wants the other weeks?
<@markrosenbaum:fedora.im>
17:24:15
I’d personally have to see as they get closer
<@Zlopez:matrix.org>
17:26:02
!info Summary of last week: (from current oncall)
<@Zlopez:matrix.org>
17:26:17
As @nirik:matrix.scrye.com already mentioned there were no oncall pings
<@Zlopez:matrix.org>
17:26:23
!topic Monitoring discussion [nirik]
<@Zlopez:matrix.org>
17:26:23
!info Go over existing items and fix them
<@Zlopez:matrix.org>
17:26:23
!info https://nagios.fedoraproject.org/nagios
<@nirik:matrix.scrye.com>
17:26:29
lets see...
<@nirik:matrix.scrye.com>
17:26:45
just the same ones as last week.
<@nirik:matrix.scrye.com>
17:27:06
1 host down, 1 host logdetective02, and a bunch of datagrepper checks
<@nirik:matrix.scrye.com>
17:27:21
we have been getting a lot of mailman alerts... but hopefully better now
<@Zlopez:matrix.org>
17:27:35
Working on that, but the reviews are taking a plenty of time
<@Zlopez:matrix.org>
17:27:53
Hopefully the improvement will work
<@nirik:matrix.scrye.com>
17:28:05
yeah... is someone reviewing now? or just waiting?
<@Zlopez:matrix.org>
17:28:58
I got few comments few days ago, addressed them and now waiting for approval
<@Zlopez:matrix.org>
17:29:18
Then the second package will have to go as well
<@Zlopez:matrix.org>
17:29:33
Fixed the oncall in the meantime
<@nirik:matrix.scrye.com>
17:29:34
ok. Let me know if you need reviewers... I can try and do it sometime
<@Zlopez:matrix.org>
17:29:35
!oncall
<@zodbot:fedora.im>
17:29:37
● @markrosenbaum:fedora.im (markrosenbaum) Current Time for them: 13:29 (US/Eastern)
<@zodbot:fedora.im>
17:29:37
The following people are oncall:
<@zodbot:fedora.im>
17:29:37
If they do not respond, please file a ticket (https://pagure.io/fedora-infrastructure/issues)
<@zodbot:fedora.im>
17:29:37
<@Zlopez:matrix.org>
17:30:07
Thanks
<@Zlopez:matrix.org>
17:30:21
Now I have a topic for discussion
<@Zlopez:matrix.org>
17:30:27
#topic Blocking IPs process - zlopez
<@nirik:matrix.scrye.com>
17:30:30
I have one too. Might be the same one?
<@Zlopez:matrix.org>
17:30:42
!topic Blocking IPs process - zlopez
<@nirik:matrix.scrye.com>
17:31:04
yeah, AI scrapers. :)
<@Zlopez:matrix.org>
17:31:42
We had a lot of IP added as blocked lately, so it would be nice to have some kind of unified process, so we don't need to look for them all over the place, when somebody can't connect
<@nirik:matrix.scrye.com>
17:32:03
Yep. So... do we care if they are public or not?
<@nirik:matrix.scrye.com>
17:32:15
I go back and forth on it.
<@nirik:matrix.scrye.com>
17:32:37
If we don't care we should put them in ansible. If we do, I guess we could use ansible-private.
<@Zlopez:matrix.org>
17:32:53
I don't think the IP address is actually something that we should keep private
<@Zlopez:matrix.org>
17:33:09
It's a public IP at the end
<@Zlopez:matrix.org>
17:33:49
Maybe somebody can use it as a blocklist for their server
<@nirik:matrix.scrye.com>
17:34:00
also, our process for finding IP's to block is not very set. At least for me, I tend to try a bunch of places to look for them, it's more feeling that a specific set process.
<@Zlopez:matrix.org>
17:34:04
We also should have an option to add comment to each entry
<@Zlopez:matrix.org>
17:34:17
With reason for blocking
<@nirik:matrix.scrye.com>
17:34:22
yeah, I think we should use a ipset for this (they do support comments)
<@nirik:matrix.scrye.com>
17:34:53
but the reason is only not too clear too tho right... "was hitting src.fp.o a lot from this network" ?
<@Zlopez:matrix.org>
17:34:56
Same I usually just got by feeling when blocking them, looking for patterns that look suspicious to me
<@Zlopez:matrix.org>
17:35:03
Same I usually just go by feeling when blocking them, looking for patterns that look suspicious to me
<@nirik:matrix.scrye.com>
17:35:32
so, we might consider the proof of work thing I guess instead of trying to manually block things.
<@markrosenbaum:fedora.im>
17:35:42
Or yk, check if they are blocked
<@nirik:matrix.scrye.com>
17:36:13
(side note: ipsets also easily allow you to say 'is this specific ip matching any of the nets in the set')
<@Zlopez:matrix.org>
17:36:27
I'm for using something that would help us with this
<@markrosenbaum:fedora.im>
17:36:36
I wonder if there’s a way to have the list of blocked IPs not affected by the IP block
<@markrosenbaum:fedora.im>
17:37:39
If we do make them public
<@nirik:matrix.scrye.com>
17:37:44
My two concerns about the proof of work thing (anubis?) are: 1) if a bunch of people hit a specific file/thing it could be very slow and 2) if the scrapers decide to drop "Mozilla" from their agent to avoid it, it will become an arms race to figure out what they are using.
<@Zlopez:matrix.org>
17:37:54
What do you mean exactly?
<@nirik:matrix.scrye.com>
17:38:38
depending on where its blocked I guess... if we are talking about blocking ips globally then no, but if we are only blocking say proxies then pagure.io would still be unblocked.
<@Zlopez:matrix.org>
17:38:43
I understand the concerns
<@Zlopez:matrix.org>
17:39:31
Is there anything else that is not behind proxies that the AI scrapers are targeting?
<@markrosenbaum:fedora.im>
17:39:38
Could we host like a json of blocked IPs on some other site?
<@Zlopez:matrix.org>
17:39:41
Or just pagure?
<@nirik:matrix.scrye.com>
17:39:42
pagure.io
<@nirik:matrix.scrye.com>
17:40:09
just pagure that I know of off hand, everything else is behind proxies.
<@nirik:matrix.scrye.com>
17:40:18
we could... just more work
<@nirik:matrix.scrye.com>
17:41:33
So, right now we have some blocks on pagure.io (which I put back after reboot yesterday) and no blocks on proxies.
<@nirik:matrix.scrye.com>
17:41:59
and things seem fine. I am not sure if thats because they stopped, or because they are just busy elsewhere.
<@Zlopez:matrix.org>
17:42:17
I assume the later
<@markrosenbaum:fedora.im>
17:42:26
I definitely wouldn't assume they're gone
<@nirik:matrix.scrye.com>
17:42:49
on pagure, I also increased resources a lot... many more cpus, etc...
<@Zlopez:matrix.org>
17:43:44
We tweaked the httpd resources as well to get better performance
<@nirik:matrix.scrye.com>
17:43:48
I guess my thought right now is to not be hasty. ;) But explore what it would take to setup a ipset thing in ansible and what it would take to deploy anubis if we had to.
<@Zlopez:matrix.org>
17:44:19
I'm OK with opening spike tickets for that, so we don't forget about it
<@nirik:matrix.scrye.com>
17:44:51
sure.
<@Zlopez:matrix.org>
17:45:19
Let me add that to my TODO, I will do that tomorrow
<@nirik:matrix.scrye.com>
17:45:44
also I think James was gonna mention nftables handling here... I am not sure we could do much directly there tho (or differently from iptables)
<@nirik:matrix.scrye.com>
17:46:49
Oh, one other thing:
<@Zlopez:matrix.org>
17:47:05
We will see, I'm not against using nftables if that will make it easier
<@nirik:matrix.scrye.com>
17:47:41
well, it could rate limit... but I don't think that would help this particular thing
<@Zlopez:matrix.org>
17:48:03
I meant with the ipset setup in ansible
<@nirik:matrix.scrye.com>
17:48:32
There is also mod_qos in apache. That could at least allow us to prioritize our ip's over others...
<@Zlopez:matrix.org>
17:49:12
All of that could be mentioned in the spike tickets :-)
<@nirik:matrix.scrye.com>
17:49:22
So, perhaps we could just make a ticket with all these ideas, explore them as time permits and rediscuss once we have more data?
<@nirik:matrix.scrye.com>
17:49:27
yeah
<@nirik:matrix.scrye.com>
17:50:26
Finally, should we leave the things blocked that are blocked right now on pagure.io? or clear it out? I had left it clear, but re-added them when some folks were noticing high load.
<@Zlopez:matrix.org>
17:51:02
I think we can safely remove them now and see if the issue returns
<@nirik:matrix.scrye.com>
17:51:09
ok
<@nirik:matrix.scrye.com>
17:51:15
we can always add them back
<@Zlopez:matrix.org>
17:51:40
We are almost at end of the meeting, so let me switch to open floor
<@Zlopez:matrix.org>
17:51:59
!topic Open Floor
<@Zlopez:matrix.org>
17:52:08
Anything to discuss on open floor?
<@nirik:matrix.scrye.com>
17:52:34
Thanks to James, Gwmngilfen and phsmoura for helping out with updates/reboots this week. I appreciate it!
<@jcline:fedora.im>
17:53:02
I did have one thing, which is really a question about a question...
<@nirik:matrix.scrye.com>
17:53:18
meta!
<@jcline:fedora.im>
17:53:29
<@jcline:fedora.im>
17:53:29
I've been doing some work on the client side for Sigul, and before I invested more time into I wanted to know where the infra folks would like signing to go longer term. I inquired with the konflux folks about their signing story and they said some people internally are looking at Vault, but that there's a way to hook in via messages and use something like Sigul.
<@jcline:fedora.im>
17:53:29
Konflux seems a bit handy-wavy to me still, but in any case, I wanted to bring up the question of what folks _want_ to run. Continue with Sigul in some form, or switch to something off-the-shelf like Vault?
<@jcline:fedora.im>
17:53:29
<@jcline:fedora.im>
17:53:29
I think this is probably best discussed async, so my real question is where is the best place to start this kind of conversation? Discourse?
<@nirik:matrix.scrye.com>
17:53:55
discourse is probibly fine.
<@nirik:matrix.scrye.com>
17:54:19
IMHO, konflux is a ways off, so I think we are on sigul for a while...
<@Zlopez:matrix.org>
17:54:47
Konflux will take some time
<@nirik:matrix.scrye.com>
17:55:04
and I am not sure about vault. I don't like depending on external non free stuff if we can avoid it... but perhaps thats just me
<@jcline:fedora.im>
17:55:54
I had to admin a vault at a prior job and personally, I wouldn't want to run it. But I'm absolutely biased, as well, since the Sigul stuff has been interesting to work on.
<@Zlopez:matrix.org>
17:56:00
It's not only you, we should avoid things we don't have control on, especially not open ones
<@nirik:matrix.scrye.com>
17:56:23
I had some hope we could leverage sigstore, but it doesn't seem like it's really built for our use cases.
<@nirik:matrix.scrye.com>
17:56:35
https://www.sigstore.dev/
<@jcline:fedora.im>
17:57:55
I can certainly look at it to double check
<@jcline:fedora.im>
17:58:31
Okay, well I will start a discourse discussion just to give folks more opportunity to think about it all and weigh in
<@nirik:matrix.scrye.com>
17:58:42
Jeremy Cline: so if sigul was something we would at least be still using for a year or two... what was the next thing you were thinking of there? replace the bridge/vault part?
<@Zlopez:matrix.org>
17:59:00
We have only two minutes left, so discourse discussion will be definitely better
<@jcline:fedora.im>
17:59:30
Yes. Well, I have a client that does a limited set of commands, so I would add all the existing commands to that and stabilize the interface a bit, and then move to the bridge/vault part
<@zodbot:fedora.im>
17:59:49
zlopez has already given cookies to jcline during the F41 timeframe
<@Zlopez:matrix.org>
18:00:36
We are at the end of our time
<@jcline:fedora.im>
18:00:37
The bridge I don't think will be too much additional work, the server bit is a bit more. It might be possible to do the server bit-by-bit so it's not an all-or-nothing situation, but I'd have to think carefully about that
<@zodbot:fedora.im>
18:00:51
kevin gave a cookie to jcline. They now have 89 cookies, 6 of which were obtained in the Fedora 41 release cycle
<@Zlopez:matrix.org>
18:01:19
Thanks everybody for coming today :-)
<@Zlopez:matrix.org>
18:01:32
Let's see you next week at the same time
<@Zlopez:matrix.org>
18:01:33
!endmeeting