<@nirik:matrix.scrye.com>
16:02:06
!startmeeting Infrastructure (2025-09-18)
<@meetbot:fedora.im>
16:02:07
Meeting started at 2025-09-18 16:02:06 UTC
<@meetbot:fedora.im>
16:02:08
The Meeting name is 'Infrastructure (2025-09-18)'
<@nirik:matrix.scrye.com>
16:02:17
!info Agenda is at: https://board.net/p/fedora-infra
<@nirik:matrix.scrye.com>
16:02:17
!chair @nirik:matrix.scrye.com @zlopez:matrix.org @nb:fedora.im bodanel @dtometzki:fedora.im @jnsamyak:matrix.org @lenkaseg:fedora.im @patrikp:matrix.org @james:fedora.im
<@nirik:matrix.scrye.com>
16:02:17
!meetingname infrastructure
<@nirik:matrix.scrye.com>
16:02:17
!info About our team: https://docs.fedoraproject.org/en-US/cle/
<@nirik:matrix.scrye.com>
16:02:17
!topic Hola y bienvenido
<@nirik:matrix.scrye.com>
16:02:17
!info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
<@meetbot:fedora.im>
16:02:18
The Meeting Name is now infrastructure
<@nirik:matrix.scrye.com>
16:03:16
anyone around today besides me? :)
<@phsmoura:fedora.im>
16:03:29
sorry
<@phsmoura:fedora.im>
16:03:34
got distracted
<@phsmoura:fedora.im>
16:03:35
!startmeeting Infrastructure (2025-09-18)
<@meetbot:fedora.im>
16:03:36
Meeting already in progress
<@phsmoura:fedora.im>
16:03:52
!info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
<@phsmoura:fedora.im>
16:03:52
!topic Hola y bienvenido
<@phsmoura:fedora.im>
16:03:52
!info About our team: https://docs.fedoraproject.org/en-US/cle/
<@phsmoura:fedora.im>
16:03:52
!info Agenda is at: https://board.net/p/fedora-infra
<@phsmoura:fedora.im>
16:03:52
!chair @nirik:matrix.scrye.com @zlopez:matrix.org @nb:fedora.im bodanel @dtometzki:fedora.im @jnsamyak:matrix.org @lenkaseg:fedora.im @patrikp:matrix.org @james:fedora.im
<@phsmoura:fedora.im>
16:03:52
!meetingname infrastructure
<@meetbot:fedora.im>
16:03:53
The Meeting Name is now infrastructure
<@phsmoura:fedora.im>
16:04:00
hello
<@phsmoura:fedora.im>
16:04:16
!info Getting Started Guide: https://docs.fedoraproject.org/en-US/infra/gettingstarted/
<@phsmoura:fedora.im>
16:04:16
!info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
<@phsmoura:fedora.im>
16:04:16
!topic New folks introductions
<@phsmoura:fedora.im>
16:04:28
lets see if there are new people around
<@zlopez:fedora.im>
16:05:44
!hi
<@zodbot:fedora.im>
16:05:45
Michal Konecny (zlopez)
<@phsmoura:fedora.im>
16:06:22
!info chair 2025-10-09 - ???
<@phsmoura:fedora.im>
16:06:22
!topic Next chair
<@phsmoura:fedora.im>
16:06:22
!info chair 2025-10-02 - ???
<@phsmoura:fedora.im>
16:06:22
!info chair 2025-09-25 - ???
<@phsmoura:fedora.im>
16:06:22
!info magic eight ball says:
<@phsmoura:fedora.im>
16:06:56
anyone would like to chair this meeting in next weeks?
<@zlopez:fedora.im>
16:07:05
I can take next week
<@nirik:matrix.scrye.com>
16:08:01
I could do the 2nd I think...
<@phsmoura:fedora.im>
16:08:09
anyone would like to take 10-02 or 10-09?
<@phsmoura:fedora.im>
16:09:56
!info chair 2025-10-02 - nirik
<@phsmoura:fedora.im>
16:09:56
!info chair 2025-09-25 - Michal Konečný
<@phsmoura:fedora.im>
16:10:16
!info CLE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1900 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
<@phsmoura:fedora.im>
16:10:16
!topic announcements and information
<@phsmoura:fedora.im>
16:10:16
!info CLE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0815 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
<@phsmoura:fedora.im>
16:10:16
!info F43 beta infra. freeze in effect
<@phsmoura:fedora.im>
16:10:36
is there any other announcement?
<@nirik:matrix.scrye.com>
16:10:53
!infra f43 beta freeze is over. ;)
<@zodbot:fedora.im>
16:10:53
● oncall <subcommand> [...] - oncall
<@zodbot:fedora.im>
16:10:53
**Usage:** !infra <subcommand> [...]
<@zodbot:fedora.im>
16:10:53
<@zodbot:fedora.im>
16:10:53
● status - get a list of the ongoing and planned outages
<@nirik:matrix.scrye.com>
16:11:00
!info f43 beta freeze is over. ;)
<@zlopez:fedora.im>
16:11:05
!info F43 Beta is out
<@james:fedora.im>
16:11:23
wooo
<@phsmoura:fedora.im>
16:13:49
!topic Oncall
<@phsmoura:fedora.im>
16:13:49
!info on call from 2025-09-12 to 2025-09-18 - Gwmngilfen
<@phsmoura:fedora.im>
16:13:49
!info on call from 2025-10-10 to 2025-10-16 - ???
<@phsmoura:fedora.im>
16:13:49
!info on call from 2025-10-03 to 2025-10-09 - ???
<@phsmoura:fedora.im>
16:13:49
!info on call from 2025-09-26 to 2025-10-02 - ???
<@phsmoura:fedora.im>
16:13:49
!info on call from 2025-09-19 to 2025-09-25 - nirik
<@phsmoura:fedora.im>
16:13:49
!info https://docs.fedoraproject.org/en-US/infra/day_to_day_fedora/#_the_oncall_role_in_our_team
<@phsmoura:fedora.im>
16:14:24
anyone would like to be oncall during the available weeks?
<@zlopez:fedora.im>
16:15:26
I can the next one that is free
<@zlopez:fedora.im>
16:15:34
2025-09-26 to 2025-10-02
<@phsmoura:fedora.im>
16:18:59
ok so thats what we have for the next weeks
<@phsmoura:fedora.im>
16:18:59
!info on call from 2025-10-10 to 2025-10-16 - ???
<@phsmoura:fedora.im>
16:18:59
!info on call from 2025-10-03 to 2025-10-09 - ???
<@phsmoura:fedora.im>
16:18:59
!info on call from 2025-09-26 to 2025-10-02 - Michal Konečný
<@phsmoura:fedora.im>
16:18:59
!info on call from 2025-09-19 to 2025-09-25 - nirik
<@phsmoura:fedora.im>
16:19:48
!info Summary of last week: (Gwmngilfen)
<@phsmoura:fedora.im>
16:20:11
is he around? for some reason I cant tag people here...
<@zlopez:fedora.im>
16:20:36
Gwmngilfen: Are you around?
<@phsmoura:fedora.im>
16:20:37
Gwmngilfen: here :)
<@zlopez:fedora.im>
16:21:00
I think this time doesn't really work for him
<@phsmoura:fedora.im>
16:21:32
ok, so we should move on?
<@nirik:matrix.scrye.com>
16:21:37
yep
<@zlopez:fedora.im>
16:21:44
I didn't saw any oncall ping this week
<@zlopez:fedora.im>
16:22:00
So probably not much was broken
<@phsmoura:fedora.im>
16:22:11
!info Go over existing items and fix them
<@phsmoura:fedora.im>
16:22:11
!info https://nagios.fedoraproject.org/nagios
<@phsmoura:fedora.im>
16:22:11
!topic Monitoring discussion [nirik]
<@nirik:matrix.scrye.com>
16:22:34
so, I cleaned up a few of the outstanding ones after the freeze...
<@nirik:matrix.scrye.com>
16:22:43
just back to the ones we have had for a while.
<@nirik:matrix.scrye.com>
16:23:01
Oh and I need to get a new wildcard cert for fedoraproject.org...
<@nirik:matrix.scrye.com>
16:23:12
(in the next 52 days)
<@zlopez:fedora.im>
16:23:26
That is close
<@nirik:matrix.scrye.com>
16:24:12
nothing really much more on this this week unless someone has questions?
<@phsmoura:fedora.im>
16:25:33
ok, moving on then
<@phsmoura:fedora.im>
16:25:37
Backlog refinement ?
<@nirik:matrix.scrye.com>
16:25:53
no, I was going to talk about anubis today. ;)
<@zlopez:fedora.im>
16:25:56
Looking at the F43 release schedule we should probably renew the cert before final freeze
<@nirik:matrix.scrye.com>
16:26:05
at least thats what we decided last week...
<@phsmoura:fedora.im>
16:26:11
ah sorry
<@phsmoura:fedora.im>
16:26:31
!topic Learning topic
<@phsmoura:fedora.im>
16:26:31
!info Anubis in Fedora [nirik] on 2025-09-18
<@nirik:matrix.scrye.com>
16:27:12
So, for those of you who don't know it, anubis is a application that proxies connections and acts on them based on a set of rules.
<@nirik:matrix.scrye.com>
16:27:53
The default ruleset first just denys a bunch of ai crawler network ips
<@nirik:matrix.scrye.com>
16:28:53
Then, things that have "Mozilla" in their user agent (basically anything that wants to use javascript) are issued a challenge... their browser completes that and sets a cookie with the signed result... anubis then can look for that cookie and pass that particular client until the cookie expires.
<@nirik:matrix.scrye.com>
16:29:11
There are also various other rules and you can set your own.
<@nirik:matrix.scrye.com>
16:29:51
so, anubis sits as a proxy... outside client request -> apache (for ssl termination) -> anubis -> back to apache on localhost to serve the request
<@nirik:matrix.scrye.com>
16:30:40
https://github.com/TecharoHQ/anubis/blob/00679aed66f4eabb66847d47a0d016d2bb09a04d/data/botPolicies.yaml is the current default policy you can include...
<@nirik:matrix.scrye.com>
16:31:02
it also allows 'good' crawlers like the internet archive and such
<@nirik:matrix.scrye.com>
16:31:41
There's a pretty large number of sites using it: https://anubis.techaro.lol/docs/user/known-instances
<@nirik:matrix.scrye.com>
16:32:16
It's written in go with some npm / js...
<@nirik:matrix.scrye.com>
16:32:49
It's available as a native fedora package or a container from upstream
<@nirik:matrix.scrye.com>
16:32:58
(or of course you can build it yourself)
<@zlopez:fedora.im>
16:34:05
It seems to be the best protection about AI scrapers right now
<@nirik:matrix.scrye.com>
16:34:13
I'm trying to think of what else to go over. :) Any questions?
<@zlopez:fedora.im>
16:34:31
It seems to be the best protection against AI scrapers right now
<@nirik:matrix.scrye.com>
16:34:43
yeah, it does a pretty nice job. I have enabled it for a number of staging sites (see call for testing on the mailing list) and the amount of BW / CPU saved is... amazing
<@zlopez:fedora.im>
16:34:48
Where it's running right now in Fedora?
<@nirik:matrix.scrye.com>
16:34:53
I posted a few graphs in noc.
<@nirik:matrix.scrye.com>
16:35:16
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/thread/7MICBGOI3NEBEE47XIWR5FAL3WFDBRAE/
<@zlopez:fedora.im>
16:35:20
When do we plan to get it to production, pagure.io really needs that
<@nirik:matrix.scrye.com>
16:35:28
after that post I also enabled src.stg.fedoraproject.org
<@zlopez:fedora.im>
16:35:53
When do we plan to get it to production? pagure.io really needs that
<@nirik:matrix.scrye.com>
16:36:00
My plan is to enable pagure.io today. I think all thats in place... since those machines are rhel8 and not behind a proxy they are using the container.
<@gwmngilfen:fedora.im>
16:36:27
!hi
<@zodbot:fedora.im>
16:36:28
Greg Sutcliffe (gwmngilfen) - he / him / his
<@nirik:matrix.scrye.com>
16:36:30
For the other apps, I need to rework the ansible role some, but I mostly have that done.
<@nirik:matrix.scrye.com>
16:36:42
The testing showed a important issue:
<@gwmngilfen:fedora.im>
16:37:03
sorry, cooking 🙂 - I got zero pings 🙂
<@nirik:matrix.scrye.com>
16:37:28
by default if you don't specify, anubis will generate a new random key when it starts. It uses this to sign the challenge responses it gives users in cookies.
<@nirik:matrix.scrye.com>
16:37:54
but with out proxies if each of them has a different key, users will get a new challenge everytime they hit a different proxy
<@nirik:matrix.scrye.com>
16:38:07
which happens all the time.
<@nirik:matrix.scrye.com>
16:38:39
So, to avoid that I am going to set a key for all the proxies, so they will validate each others challenge cookies... so you should only get one and then it should work on the rest.
<@nirik:matrix.scrye.com>
16:39:09
I tested this in staging and it seemed to work fine
<@zlopez:fedora.im>
16:39:24
That sounds good. Are those ssl keys, or are we talking about some other keys?
<@nirik:matrix.scrye.com>
16:40:32
they are ED25519 keys... but yes, just a random string...
<@nirik:matrix.scrye.com>
16:40:38
https://anubis.techaro.lol/docs/admin/installation#key-generation
<@nirik:matrix.scrye.com>
16:41:14
so, my plan is pagure.io today, work on the proxy role... and as soon as that looks good in staging roll out to the sites I tested in staging to prod
<@nirik:matrix.scrye.com>
16:41:28
oh yeah, let me mention that...
<@zlopez:fedora.im>
16:41:36
Can we see how much was blocked by Anubis?
<@nirik:matrix.scrye.com>
16:41:55
for proxies websites there's a new 'anubis' variable... if it's set to true on a site it just sets everything for that site.
<@nirik:matrix.scrye.com>
16:42:13
it logs to journal on the proxies... so yeah, it logs DENYs I think.
<@zlopez:fedora.im>
16:42:49
Just interested if we can see some statistics
<@zlopez:fedora.im>
16:43:17
But you already mentioned graphs about saved CPU
<@nirik:matrix.scrye.com>
16:44:31
yeah... here's koji.stg interface over the last month:
<@zlopez:fedora.im>
16:45:02
The drop is after enabling Anubis?
<@nirik:matrix.scrye.com>
16:45:20
The last one... the middle part I think is us blocking some endpoints.
<@nirik:matrix.scrye.com>
16:45:59
anyhow, I think it will help us out a lot.
<@nirik:matrix.scrye.com>
16:46:14
I picked the sites to enable based on how much traffic they were getting, we may need to enable more later.
<@nirik:matrix.scrye.com>
16:47:00
Any other questions on this?
<@zlopez:fedora.im>
16:47:11
It definitely looks like it's helping
<@nirik:matrix.scrye.com>
16:47:18
Oh, I also have a SOP with a lot of this, but there's not much tuning unless you want to tweak the policy a bunch
<@zlopez:fedora.im>
16:47:23
Did anybody complained about the javascript part?
<@nirik:matrix.scrye.com>
16:48:09
well, not that I know of, but I am sure we will get some complaints once it's in prod.
<@nirik:matrix.scrye.com>
16:49:13
but with kernel.org and the un and lots of places using it... not sure what alternative we have
<@nirik:matrix.scrye.com>
16:49:23
and also most of our stuff doesn't work without js anyhow
<@zlopez:fedora.im>
16:49:52
I assume this is the lesser evil than being flooded by AI scrapers
<@james:fedora.im>
16:50:02
Yeh, I've seen a bunch of random websites enable anubis or similar this year ... so I'd guess users are getting used to it anyway.
<@nirik:matrix.scrye.com>
16:51:03
Oh, one note about the cute little mascot/icon... the author asks people to not replace it (but doesn't require this it's open source). They are selling a commercially supported version that allows businesses to do that
<@nirik:matrix.scrye.com>
16:51:30
it's really so fast here I can't even really see it
<@nirik:matrix.scrye.com>
16:52:45
and for some fun: https://fosstodon.org/@nirik/115216714667752660 (what is the plural of a collection of anubis servers)
<@phsmoura:fedora.im>
16:54:36
cool, lets move to open floor if there is anything else...
<@phsmoura:fedora.im>
16:54:38
!topic Open Floor
<@zlopez:fedora.im>
16:54:54
I would say Anubi 🙂
<@nirik:matrix.scrye.com>
16:55:13
thats what the author said too. :) Sorry I missed that as an option in the poll
<@james:fedora.im>
16:55:17
haha: Manyubis
<@zlopez:fedora.im>
16:56:02
I think it's from latin
<@nirik:matrix.scrye.com>
16:56:38
anubis is the greekized version of the egyptian. ;)
<@zlopez:fedora.im>
16:58:27
It's ancient Greek, I always think that as latin
<@zlopez:fedora.im>
16:58:38
It's ancient Greek, I always think that is latin
<@phsmoura:fedora.im>
17:00:14
thanks everyone :)
<@phsmoura:fedora.im>
17:00:14
ok, lets go back then
<@phsmoura:fedora.im>
17:00:18
!endmeeting