#meeting-1:fedoraproject.org log

<@tdawson:fedora.im>

18:00:08

!startmeeting EPEL (2024-05-22)

<@meetbot:fedora.im>

18:00:10

Meeting started at 2024-05-22 18:00:08 UTC

<@meetbot:fedora.im>

18:00:11

The Meeting name is 'EPEL (2024-05-22)'

<@tdawson:fedora.im>

18:00:16

!meetingname epel

<@tdawson:fedora.im>

18:00:21

!topic aloha

<@nirik:matrix.scrye.com>

18:00:53

morning. 🌄

<@tdawson:fedora.im>

18:01:09

Morning nirik

<@carlwgeorge:matrix.org>

18:01:17

!hi

<@zodbot:fedora.im>

18:01:22

Carl George (carlwgeorge) - he / him / his

<@tdawson:fedora.im>

18:01:36

!hi

<@zodbot:fedora.im>

18:01:38

Troy Dawson (tdawson)

<@tdawson:fedora.im>

18:01:49

Aha ... it finally shows my name.

<@davide:cavalca.name>

18:02:27

!hi

<@tdawson:fedora.im>

18:02:27

Turns out I somehow had my Fedora info set to private. I didn't know until it broke the elections.

<@zodbot:fedora.im>

18:02:31

Davide Cavalca (dcavalca) - he / him / his

<@nhanlon:beeper.com>

18:02:42

!hi

<@zodbot:fedora.im>

18:02:45

Neil Hanlon (neil) - he / him / his

<@tdawson:fedora.im>

18:02:46

Hi Carl George and Davide Cavalca

<@tdawson:fedora.im>

18:02:59

Hi Neil Hanlon

<@nhanlon:beeper.com>

18:03:55

but will try

<@dherrera:fedora.im>

18:05:00

!hi

<@zodbot:fedora.im>

18:05:02

Diego Herrera (dherrera) - he / him / his

<@tdawson:fedora.im>

18:05:07

We're a little lite today

<@tdawson:fedora.im>

18:05:12

Hi Diego Herrera

<@tdawson:fedora.im>

18:05:18

!topic End Of Life (EOL)

<@tdawson:fedora.im>

18:05:26

RHEL 7 / epel-7 will go EOL on 2024-06-30 https://endoflife.date/rhel CentOS Stream 8 / epel-8-next goes EOL in 2024-05-31 CentOS Stream 9 / epel-9-next goes EOL in 2027-05-31 https://endoflife.date/centos-stream

<@tdawson:fedora.im>

18:05:47

Two more weeks, and that list get's shorter.

<@tdawson:fedora.im>

18:06:11

!topic EPEL Issues https://pagure.io/epel/issues https://pagure.io/epel/issues?tags=meeting&status=Open

<@conan_kudo:matrix.org>

18:06:25

!hi

<@zodbot:fedora.im>

18:06:28

Neal Gompa (ngompa) - he / him / his

<@tdawson:fedora.im>

18:06:36

Hi Conan Kudo

<@nirik:matrix.scrye.com>

18:06:46

I can hardly wait.

<@tdawson:fedora.im>

18:07:03

There aren't any issues marked with "meeting" ... so moving along

<@tdawson:fedora.im>

18:07:11

!topic Old Business

<@tdawson:fedora.im>

18:07:30

Does anyone have any old business they want to bring up?

<@carlwgeorge:matrix.org>

18:07:40

i noticed something today in bodhi regarding updates policy

<@carlwgeorge:matrix.org>

18:07:49

looks like there is a related issue

<@carlwgeorge:matrix.org>

18:07:55

!link https://pagure.io/epel/issue/272

<@carlwgeorge:matrix.org>

18:08:48

i'll comment there as well for posterity, but in summary i realized that bodhi lets you create epel updates that only require +1 karma, but our docs say minimum is +3

<@carlwgeorge:matrix.org>

18:09:16

example: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2024-38d250bafc

<@tdawson:fedora.im>

18:09:39

That is true, and sadly, something I abuse until we brought it up a couple weeks ago.

<@carlwgeorge:matrix.org>

18:09:47

so we should either update to docs to match what bodhi does, or fix bodhi to match our docs (i don't have strong feelings either way tbh)

<@tdawson:fedora.im>

18:09:49

That is true, and sadly, something I abused until we brought it up a couple weeks ago.

<@nirik:matrix.scrye.com>

18:10:14

I think possibly thats fixed in adamw's big bodhi rework... but perhaps he didn't touch epel there.

<@adamwill:fedora.im>

18:10:37

no, it'll fix that

<@carlwgeorge:matrix.org>

18:10:49

fedora is also out of sync, by policy updates after beta freeze are minimum +2, but bodhi will also let you set a fedora update to just +1

<@adamwill:fedora.im>

18:10:59

well, assuming the appropriate config is set in ansible plays

<@adamwill:fedora.im>

18:11:05

yes, it fixes that too

<@carlwgeorge:matrix.org>

18:11:08

!link https://docs.fedoraproject.org/en-US/epel/epel-policy-updates/#stable_releases

<@carlwgeorge:matrix.org>

18:11:18

!link https://docs.fedoraproject.org/en-US/fesco/Updates_Policy/#karma-requirements

<@tdawson:fedora.im>

18:12:07

So I guess that is the question, do we want to enforce that? (change bodhi) or change our policy to allow +1 ?

<@carlwgeorge:matrix.org>

18:13:15

i'm actually fine with the +1 minimum, +3 default way that bodhi does it now. in the example i gave it came in handy (cve fixes for chromium).

<@adamwill:fedora.im>

18:13:37

you should understand what those numbers *mean*

<@adamwill:fedora.im>

18:13:47

the number you set at update creation time is the *automatic push threshold*

<@adamwill:fedora.im>

18:14:06

it's the amount of positive karma at which bodhi will automatically push the update stable (assuming it meets gating requirements)

<@carlwgeorge:matrix.org>

18:14:20

oh good point

<@tdawson:fedora.im>

18:14:25

Oh ... so if someone doesn't setup any karma, by default they would get +1 ?

<@adamwill:fedora.im>

18:14:47

the other threshold concept we have is the 'minimum push threshold'

<@carlwgeorge:matrix.org>

18:15:04

the new update page defaults the drop down to +3/-3, i'm not sure what the cli does

<@adamwill:fedora.im>

18:15:19

which is a karma number below which the update *cannot* be pushed stable manually

<@adamwill:fedora.im>

18:15:43

whatever you set the autopush threshold to when creating the update, if the update's karma exceeds the minimum push threshold it can be pushed *manually*

<@adamwill:fedora.im>

18:15:52

whatever you set the autopush threshold to when creating the update, if the update's karma exceeds the policy minimum push threshold it can be pushed _manually_

<@carlwgeorge:matrix.org>

18:16:00

ideally the auto-push threshold won't be able to be set lower than the minimum push threshold

<@adamwill:fedora.im>

18:16:52

the minimum push threshold can be configured per distribution (fedora, epel etc) and/or per release number and release phase

<@adamwill:fedora.im>

18:17:10

Carl George: yes, that is one of the things my changes fix (although there are some interesting corner cases)

<@carlwgeorge:matrix.org>

18:17:29

awesome, glad to hear help in on the way

<@carlwgeorge:matrix.org>

18:17:33

thanks for your work on that

<@zodbot:fedora.im>

18:17:51

carlwgeorge gave a cookie to adamwill. They now have 246 cookies, 4 of which were obtained in the Fedora 40 release cycle

<@zodbot:fedora.im>

18:17:56

dherrera gave a cookie to adamwill. They now have 247 cookies, 5 of which were obtained in the Fedora 40 release cycle

<@zodbot:fedora.im>

18:18:07

kevin gave a cookie to adamwill. They now have 248 cookies, 6 of which were obtained in the Fedora 40 release cycle

<@adamwill:fedora.im>

18:18:10

as i interpret the policy right now, the config we should set is to make the minimum push threshold for all epel releases at all times 3

<@carlwgeorge:matrix.org>

18:18:33

yup, that's what our policy is now

<@adamwill:fedora.im>

18:18:35

with my changes to bodhi, that would mean you could not set the autopush number below 3 when creating an update

<@carlwgeorge:matrix.org>

18:18:49

i didn't see anyone comment that they feel strongly about lowering it

<@tdawson:fedora.im>

18:19:20

Not strongly. And if someone really does have a problem, we can always re-visit it.

<@adamwill:fedora.im>

18:19:46

as things stand, i believe the current behaviour is "all epel updates can be manually pushed at +2, and epel updates with the autopush threshold set to +1 will be pushed at +1"

<@nirik:matrix.scrye.com>

18:20:05

I suspect (but don't know) that epel updates get a lot less karma than fedora ones... but I could be wrong.

<@tdawson:fedora.im>

18:20:43

I depends on the update, but in general, epel updates usually timeout into stable version karma into stable.

<@carlwgeorge:matrix.org>

18:20:51

if i'm reading things correctly, that matches the fedora updates policy, right?

<@adamwill:fedora.im>

18:21:29

yes, because afaics we don't actually specify a different policy for epel karma rn

<@adamwill:fedora.im>

18:21:36

we specify *days* to 7 for epel

<@adamwill:fedora.im>

18:21:54

(which means epel updates can be pushed stable after 7 days regardless of karma)

<@nirik:matrix.scrye.com>

18:21:55

I think also now we have a 'it must have been pushed to testing before it can go stable' check, which didn't use to be the case.

<@carlwgeorge:matrix.org>

18:22:06

we did semi-recently change the time threshold for epel to match fedora stable branches

<@adamwill:fedora.im>

18:22:21

nirik: I don't think that's actually right, but would have to dive back in.

<@carlwgeorge:matrix.org>

18:22:48

i think also matching the fedora karma requirements would be nice

<@adamwill:fedora.im>

18:22:49

Carl George: afaics it's just blanket 7 days for epel, which matches the written policy

<@nirik:matrix.scrye.com>

18:23:14

I am pretty sure I have hit that, but thats a sidebar....

<@carlwgeorge:matrix.org>

18:24:32

that's probably enough time on this in this meeting, we can probably table this until the bodhi changes land and then examine if things are working the way we want them to

<@smooge:fedora.im>

18:24:52

...

<@adamwill:fedora.im>

18:25:51

as things stand, you'll get the same change fedora does: the +1 loophole will be gone and all updates will need +2, or 7 days in testing

<@tdawson:fedora.im>

18:26:07

Sounds good. Are there any other Old Business?

<@smooge:fedora.im>

18:26:24

I have sort of Old sort of New sort of Open

<@tdawson:fedora.im>

18:26:28

Sounds good. Is there any other Old Business?

<@tdawson:fedora.im>

18:26:48

Stephen J Smoogen: Sounds like a wedding ... go for it.

<@smooge:fedora.im>

18:27:29

Its data on this graph https://data-analysis.fedoraproject.org/csv-reports/images/epel-stacked.png and the data I am going to refer to is blue

<@smooge:fedora.im>

18:27:44

I think I have most of the stuff with a wedding :)

<@tdawson:fedora.im>

18:28:14

Ha ... cuz it's blue :)

<@smooge:fedora.im>

18:28:19

So starting a couple of months ago, the EPEL-7 numbers went through the roof and have stayed that way

<@nirik:matrix.scrye.com>

18:28:51

yes, they are hamming the mirrorlists pretty hard.

<@smooge:fedora.im>

18:28:58

mattdm asked if I could find anything and I did a rough guess of systems per various days on that time.

<@nirik:matrix.scrye.com>

18:29:01

yes, they are hammering the mirrorlists pretty hard.

<@smooge:fedora.im>

18:29:08

It turns out to be Amazon

<@carlwgeorge:matrix.org>

18:29:41

amazon linux, or just whatever distro running on aws

<@smooge:fedora.im>

18:29:57

where before hand we might see a couple dozen outbound ipv4 ips with EPEL-7 data, we are now seeing entire subnets of ipv4 from .1 to .254

<@nirik:matrix.scrye.com>

18:30:18

someone avoiding managed nat gateway charges? ;)

<@smooge:fedora.im>

18:30:24

I do not know if it is CentOS-7, RHEL-7, or Amazon Linux-2

<@smooge:fedora.im>

18:31:04

It was a couple thousand subnets so I think it might have been some sort of config change

<@tdawson:fedora.im>

18:32:21

What is our biggest concern here? The fact that they are running out of time? Or that when things move to archive, they will overwhelm out archive servers?

<@nirik:matrix.scrye.com>

18:32:26

or dropping some caching layer

<@smooge:fedora.im>

18:32:34

because EL-7/Amazon Linux 2 use yum all it says is `urlgrabber/X.Y yum/X.Y` in the logs

<@smooge:fedora.im>

18:32:58

they will definitely overwhelm the archive servers

<@smooge:fedora.im>

18:33:08

they are already hammering the mirrors a lot

<@carlwgeorge:matrix.org>

18:33:22

do the archive servers have any rate limiting in place?

<@smooge:fedora.im>

18:33:42

the archive servers are pretty much dl.fedoraproject.org and one or two other sites

<@smooge:fedora.im>

18:33:55

so the rate limiting is 'ooops no more bandwidth'

<@nirik:matrix.scrye.com>

18:33:56

uh... yeah, it's not just that, it's more...

<@smooge:fedora.im>

18:34:38

When I looked at a couple they only archived Fedora that I could find

<@nirik:matrix.scrye.com>

18:34:40

theres 10ish or so in there.

<@nirik:matrix.scrye.com>

18:35:06

but really my concern is the mirrorlists.

<@smooge:fedora.im>

18:35:06

but that was a while ago.

<@carlwgeorge:matrix.org>

18:35:26

is it possible for a mirror to sync just the epel archive, not the entire archive (fedora+epel)?

<@nirik:matrix.scrye.com>

18:35:45

sure.

<@smooge:fedora.im>

18:35:45

anyway.. I just wanted to say 'hey I figured out why!' and then we need to figure out if anyone at amazon can figure it out

<@carlwgeorge:matrix.org>

18:36:04

i only see fedora-archive in the wiki page, with a comment that it's both fedora and epel

<@carlwgeorge:matrix.org>

18:36:13

!link https://fedoraproject.org/wiki/Infrastructure/Mirroring#Suggested_rsync_modules

<@smooge:fedora.im>

18:36:34

rsync --exclude will exclude whatever you want

<@tdawson:fedora.im>

18:36:48

Does anyone have any Amazon contacts that can look into this?

<@nirik:matrix.scrye.com>

18:37:19

davdunc is the only one I know off hand.

<@tdawson:fedora.im>

18:37:39

That's who I was thinking of too.

<@carlwgeorge:matrix.org>

18:38:33

help us davdunc kenobi, you're our only hope

<@tdawson:fedora.im>

18:39:22

Does anyone want to contact him?

<@carlwgeorge:matrix.org>

18:39:55

i can shoot him a text and ask him to check his matrix notifications

<@tdawson:fedora.im>

18:40:08

Thank you Carl George

<@smooge:fedora.im>

18:40:16

"who dis? new phone"

<@tdawson:fedora.im>

18:40:33

Let's move on ...

<@tdawson:fedora.im>

18:40:56

I'm going to move to Open Topic ... it doesn't seem to get enough time in past meetings ...

<@tdawson:fedora.im>

18:41:12

!topic General Issues / Open Floor

<@carlwgeorge:matrix.org>

18:41:13

i've got a quick thing

<@nirik:matrix.scrye.com>

18:41:20

I had a small note... can go after Carl

<@davide:cavalca.name>

18:41:25

I also have a thing for open floor

<@tdawson:fedora.im>

18:41:33

Carl George: go for it.

<@carlwgeorge:matrix.org>

18:42:32

it came to my attention that there is an rce exploit for cacti. cacti is orphaned, but still in f39 and epel7/8/9. per policy it is the "collective responsibility" of packagers to take care of packages hanging out in such a state, so i did the updates on those branches. if you care about cacti, please test and karma.

<@carlwgeorge:matrix.org>

18:42:36

!link https://bodhi.fedoraproject.org/updates/?search=cacti-1.2.27-1

<@smooge:fedora.im>

18:43:10

thank you

<@carlwgeorge:matrix.org>

18:43:10

!link https://github.com/Cacti/cacti/security/advisories/GHSA-pfh9-gwm6-86vp

<@tdawson:fedora.im>

18:43:20

Thank you for doing that.

<@carlwgeorge:matrix.org>

18:43:44

oh, wrong link, one sec

<@carlwgeorge:matrix.org>

18:44:08

apparently there are two rce's, one high, one critical. here's the critical one.

<@carlwgeorge:matrix.org>

18:44:14

!link https://github.com/Cacti/cacti/security/advisories/GHSA-7cmj-g5qc-pj88

<@nirik:matrix.scrye.com>

18:44:48

thats both good and bad. ;) Now people will think it's maintained and you touched it last. ;)

<@carlwgeorge:matrix.org>

18:45:04

yeah hence that last email (which caused me to look at it again)

<@carlwgeorge:matrix.org>

18:45:24

anyways, that's all i had on that one. there are also chromium updates in testing with high cve fixes, but those aren't mine.

<@carlwgeorge:matrix.org>

18:45:47

!link https://bodhi.fedoraproject.org/updates/?search=chromium-125.0.6422.60

<@tdawson:fedora.im>

18:45:58

Thanks Carl George ... ok nirik go for it.

<@nirik:matrix.scrye.com>

18:46:46

The epel 8 buildroot in koji is updated now... for all your epel8 building needs.

<@carlwgeorge:matrix.org>

18:47:42

on a related note, i remembered to file the issue for the epel 8.9 snapshot before that, and it was completed

<@nirik:matrix.scrye.com>

18:47:57

yes, thanks for that!

<@carlwgeorge:matrix.org>

18:48:03

!link https://pagure.io/releng/issue/12121

<@carlwgeorge:matrix.org>

18:48:10

calendar reminders work, who knew

<@zodbot:fedora.im>

18:48:10

tdawson gave a cookie to kevin. They now have 669 cookies, 10 of which were obtained in the Fedora 40 release cycle

<@zodbot:fedora.im>

18:48:13

tdawson gave a cookie to carlwgeorge. They now have 53 cookies, 6 of which were obtained in the Fedora 40 release cycle

<@zodbot:fedora.im>

18:48:37

smooge gave a cookie to carlwgeorge. They now have 54 cookies, 7 of which were obtained in the Fedora 40 release cycle

<@zodbot:fedora.im>

18:48:40

yselkowitz gave a cookie to kevin. They now have 670 cookies, 11 of which were obtained in the Fedora 40 release cycle

<@zodbot:fedora.im>

18:48:47

yselkowitz has already given cookies to carlwgeorge during the F40 timeframe

<@zodbot:fedora.im>

18:48:49

smooge gave a cookie to kevin. They now have 671 cookies, 12 of which were obtained in the Fedora 40 release cycle

<@zodbot:fedora.im>

18:48:54

ngompa gave a cookie to carlwgeorge. They now have 55 cookies, 8 of which were obtained in the Fedora 40 release cycle

<@nirik:matrix.scrye.com>

18:48:58

cookie party!

<@tdawson:fedora.im>

18:49:03

I'm glad that cookies are easier to give now.

<@carlwgeorge:matrix.org>

18:49:14

nom nom 🍪

<@zodbot:fedora.im>

18:49:22

ngompa gave a cookie to kevin. They now have 672 cookies, 13 of which were obtained in the Fedora 40 release cycle

<@zodbot:fedora.im>

18:49:25

dcavalca gave a cookie to tdawson. They now have 69 cookies, 5 of which were obtained in the Fedora 40 release cycle

<@zodbot:fedora.im>

18:49:25

kevin gave a cookie to carlwgeorge. They now have 56 cookies, 9 of which were obtained in the Fedora 40 release cycle

<@smooge:fedora.im>

18:49:42

ok so with 8.10 out that means its finally stable

<@tdawson:fedora.im>

18:49:58

Correct

<@carlwgeorge:matrix.org>

18:50:04

no it's not really stable until it hits eol

<@zodbot:fedora.im>

18:50:10

kevin gave a cookie to tdawson. They now have 70 cookies, 6 of which were obtained in the Fedora 40 release cycle

<@nirik:matrix.scrye.com>

18:50:28

nothing is ever stable. :)

<@carlwgeorge:matrix.org>

18:50:44

at least that's what jim perrin always taught me

<@smooge:fedora.im>

18:51:01

Carl George: which is why I am finally updating my RHEL5 systems to RHEL6

<@tdawson:fedora.im>

18:51:06

Thank nirik and Carl George ... ok Davide Cavalca go for it

<@davide:cavalca.name>

18:51:26

drgn got moved to RHEL, so it's been removed from EPEL

<@davide:cavalca.name>

18:51:34

that's fine, but we noticed it broke our packit CI: https://github.com/osandov/drgn/commit/bee9d25a442efa642c01e2ec04330f24553e8762

<@davide:cavalca.name>

18:52:04

one reason is it's missing a BR (and I'll file a jira for that as I think that'd be useful to have in CRB)

<@davide:cavalca.name>

18:52:22

but the actual question I wanted to ask is: do we have a suggested way to do packit CI for packages that are now in RHEL?

<@smooge:fedora.im>

18:53:29

do them from centos stream?

<@carlwgeorge:matrix.org>

18:54:05

how is packit ci doing "epel" builds now?

<@davide:cavalca.name>

18:54:14

packit does builds in copr

<@davide:cavalca.name>

18:54:22

it can build against any copr chroot

<@carlwgeorge:matrix.org>

18:54:49

i'm not sure i understand the question/problem then

<@davide:cavalca.name>

18:55:54

I'm fine with building against CentOS Stream here, and it seems the reasonable thing to do for me as well (as I don't think we can build against RHEL in copr)

<@carlwgeorge:matrix.org>

18:56:13

copr has rhel and rhel+epel chroots

<@davide:cavalca.name>

18:56:15

I was asking because this caught us by surprise and we may wanna call it out in the "package is moving to rhel" docs

<@carlwgeorge:matrix.org>

18:57:26

like the bz template?

<@carlwgeorge:matrix.org>

18:58:16

ah i think is see the problem better, drgn and libkdumpfile were moved from epel to rhel, but libkdumpfile-devel (a build dep) is being filtered out

<@carlwgeorge:matrix.org>

18:58:20

https://bugzilla.redhat.com/show_bug.cgi?id=2242594

<@davide:cavalca.name>

18:58:59

yeah, "your package moved to rhel but a subpackage will be filtered out" would also be good to call out

<@smooge:fedora.im>

18:59:37

ugh..

<@nirik:matrix.scrye.com>

18:59:44

random idea: a 'epel release notes' for rhel minor releases...

<@carlwgeorge:matrix.org>

19:00:06

a great idea for someone else to write

<@tdawson:fedora.im>

19:02:06

At the time the EPEL2RHEL bug's are created, i'm not sure if the developers even know which packages are being dropped, though I could be wrong there. But from looking at the code, I think they only know the source package name.

<@carlwgeorge:matrix.org>

19:02:38

best course of action for this one is to file an issue to request libkdumpfile-devel in crb

<@tdawson:fedora.im>

19:02:42

"They" being the team/scripts that create the bug.

<@carlwgeorge:matrix.org>

19:03:38

epel10 will help with this, as we'll be able to retire in the leading branch early and notice this stuff sooner

<@smooge:fedora.im>

19:03:47

second best course.. use `koji -p stream` to download the centos stream one and install those in your system

<@carlwgeorge:matrix.org>

19:03:49

possibly in time to request the crb inclusion before the minor version release

<@tdawson:fedora.im>

19:04:44

Oh ... I just saw the time. Sorry, but I'm going to have to cut this last discussion short.

<@smooge:fedora.im>

19:05:12

oops

<@tdawson:fedora.im>

19:05:21

Thank you all for coming, and thank you all for the good discussions.

<@tdawson:fedora.im>

19:05:34

I'll talk to you all next week, if not sooner.

<@davide:cavalca.name>

19:05:35

thanks!

<@tdawson:fedora.im>

19:05:48

!endmeeting