20:00:32 #startmeeting EPEL (2022-03-22) 20:00:32 Meeting started Wed Mar 23 20:00:32 2022 UTC. 20:00:32 This meeting is logged and archived in a public location. 20:00:32 The chair is tdawson. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions. 20:00:32 Useful Commands: #action #agreed #halp #info #idea #link #topic. 20:00:32 The meeting name has been set to 'epel_(2022-03-22)' 20:00:32 #meetingname epel 20:00:32 The meeting name has been set to 'epel' 20:00:32 #chair nirik tdawson pgreco carlwgeorge salimma dcavalca 20:00:32 Current chairs: carlwgeorge dcavalca nirik pgreco salimma tdawson 20:00:32 #topic aloha 20:00:41 morning 20:00:42 .hi 20:00:45 pgreco: pgreco 'Pablo Sebastian Greco' 20:01:03 Hi Ebeneezer_Smooge nirik and pgreco 20:01:24 thought I was late 20:01:30 .hi 20:01:31 carlwgeorge: carlwgeorge 'Carl George' 20:01:34 Hi carlwgeorge 20:01:34 .hi 20:01:35 rcallicotte: rcallicotte 'Robby Callicotte' 20:01:37 .hi 20:01:38 salimma: salimma 'Michel Alexandre Salim' 20:01:39 .hi 20:01:41 dherrera: dherrera 'None' 20:01:49 Hi rcallicotte and salimma and dherrera 20:02:06 Ebeneezer_Smooge: Nope, right on time. 20:04:27 Just like a wizard! 20:05:03 Ah, we've finally figured out what Ebeneezer_Smooge is. :) Ebeneerzer the grey 20:05:15 #topic EPEL Issues https://pagure.io/epel/issues 20:05:16 https://pagure.io/epel/issues?tags=meeting&status=Open 20:05:24 I am a Wizard harry 20:05:26 :) 20:05:39 only one issue tagged, really? 20:05:58 Yep, but it's a doozy, and it's the week we said we'd look at it. 20:06:12 So, salimma, looks like you are up. 20:06:13 yeah. I posted an update on the issue itself 20:06:51 yup. ok, so last month we started looking at EPEL CVEs - so far limited to those with severity high or more 20:07:22 er, correction, priority high or more, but those correspond to severity anyway 20:08:03 I've just added some more functionality so we now see p50/p90/p99 of the bugs, and I've started grouping the report by status as well (though this is not available for the reports for previous weeks) 20:08:33 What does p50/p90/p99 mean? Is that refering to age? 20:08:41 it's... a bit concerning. for NEW bugs, p50 is 480 days. for ASSIGNED bugs, p50 is ... 948 days 20:08:47 percentiles 20:09:07 percentiles of what? 20:09:12 yup, so the median NEW bug has been open for 480 days, the 90th pctile is 1437 days 20:09:21 percentile of all bugs in the same category 20:09:30 I think this is probibly largely due to people tossing things into epel and ignoring them... :( 20:09:31 Oh ... ok 20:10:03 nirik: definitely. though the ASSIGNED ones are even more worrying 20:10:32 nirik, well they also don't get that large in Fedora because they autoclose usually every ~365 days 20:10:38 well, that might well be due to "Oh no, I need to update this to fix the CVE, but it's an incompatible upgrade and epel doesn't do that, so I just won't do anything" 20:10:47 the ones left in NEW, clearly people don't actually care about the package, but the ones in ASSIGNED sound like "oh, I should work on that" and then people don't have time 20:10:53 or yeah, that 20:11:01 yeah, but in fedora you can just upgrade to upstream latest much more easily. 20:11:35 so.. should we consider having a similar policy to FTBFS/FTI ? if it's more than a month, "orphan" the package, except IDK how we do that for a branch 20:11:51 I think it's also a matter of the team creating the CVE's also just tossing them over the wall to EPEL. There really isn't anything in the bugs that really say what the CVE's are for, or what version(s) are affected. It's basically ... here's a CVE, you do the work. 20:12:16 yeah, there's some of that too I am sure... 20:12:20 well first we need to clarify clearly what can be done. I have had to 'correct' people 4 times this month that yes we do allow updates and you don't have to backport for 10 years 20:12:20 maybe the follow up is either we decide to retire that branch or we request epel-packagers-sig to be added 20:12:47 yeah, the CVE report itself certainly could be improved. but.. it's really a matter of just checking on cve.mitre.org in most cases 20:12:50 I think we should only orphan as a last resort.. 20:13:01 I closed most of the nodejs CVE's in epel ... for all of them I had to go at least two layers deep to figure out if it applied or not. 20:13:07 hum... only 2 assigned? 20:13:10 "ASSIGNED": 2 20:13:23 speaking of CVE tickets, here's my example: https://bugzilla.redhat.com/show_bug.cgi?id=2044550 20:13:25 ah, so the stats are really bad just because of outliers then 20:13:35 (the ASSIGNED stats I mean) 20:14:04 it has 6 CVEs in it, 2 of which are fixed already, 2 are closed by upstream as WONTFIX and 2 remain open - what do I do with this? 20:14:14 yeah, pam-kwallet in EPEL 7 and containerd in EPEL 7 20:14:46 Rathann|Mobile: if you can push an update with the 2 that are fixed I'd say do that... unless upstream is going to fix the other 2 soon... 20:14:57 nirik: already done 20:15:27 so in this case the bug stays open but it only blocks the remaining two CVE trackers, I guess? 20:15:44 oh 20:16:00 yeah. 20:16:05 but ugh, looks like the security team only filed one tracking bug for all the CVEs 20:16:05 so I'm supposed to remove the fixed CVEs from blocks list, got it 20:16:21 yeah, annoying 20:16:33 I have a number of CVE's against ansible... where there's no clear indication when or if they will ever be fixed, so they sit there. ;( 20:17:12 and... I can't actually remove the CVE tickets from the Blocks: list 20:17:15 sounds like we should have a discussion with the security team about what to do about unactionable CVEs? 20:17:30 Rathann|Mobile: huh, they get locked in? I wonder how 20:17:39 and another where it was fixed, but then the fix was reverted because it caused lots of problems. ;) 20:17:54 oh, makes sense - you don't have permission to edit that bug, and editing the blocks will modify the blocking on that other bug 20:18:02 salimma: honestly all we can do is find a way to filter out the ones that are waiting on upstream 20:18:30 perhaps a whiteboard thing? 'notactionablerightnow' 20:18:31 yeah. though... on our side we should probably decide on what to do if the status is stuck on NEW, right? 20:18:44 .hi dcavalca 20:18:45 davide: Sorry, but user 'davide' does not exist 20:18:51 yeah, having a flag or whiteboard would help 20:18:57 maybe status CLOSED-DEFERRED 20:18:57 .hello dcavalca 20:18:57 Sorry I'm late 20:18:58 salimma: dcavalca 'Davide Cavalca' 20:19:17 Hi davide ... better late than never, and I think you exist. :) 20:19:23 ideally bugzilla has a "BLOCKED" status 20:19:52 this sort of thing is also good for a activity day/workshop... ie, look at CVE's and update them or offer to help push fixes, etc. 20:21:20 yeah.. so what can we decide on? I can try going through some of them, see what can actually be fixed and do PRs for them, and use the stalled request policy in case the maintainer doesn't react 20:21:43 but as smooge was saying perhaps we also need to clarify the EPEL upgrade / support policy 20:23:16 Clarify it in what way? 20:23:29 would a email to epel-devel/epel-announce/devel-announce explaining the policy be helpfull? or should we perhaps add comments to cve bugs? ;) 20:23:50 Yes to both 20:24:25 I'm thinking a page describing our CVE effort that has links to any relevant policy would be nice, then we can email that to the lists nirik mentioned 20:24:33 to my understanding the policy for incompatible upgrades is https://docs.fedoraproject.org/en-US/epel/epel-policy-incompatible-upgrades/ 20:24:47 salimma: +1 20:24:49 it would be. I thought I put out a couple a long time ago but not sure 20:24:57 But, the update doesn't HAVE to be incompatible. 20:25:00 nice. do we have one for lifecycle? let me see... 20:25:02 it would be good to put to the lists again. I thought I put out a couple a long time ago but not sure 20:25:26 because I've heard Fedora maintainers saying "I don't want to maintain this for 10 years" a few times 20:25:37 tdawson: it doesn't have to be, no. but it could be... 20:25:42 Correct 20:26:05 I guess we can point to https://docs.fedoraproject.org/en-US/epel/epel-about-history-philosophy/#_philosophy 20:26:08 which says SHOULD not MUST 20:26:57 Also correct. 20:27:03 * nirik nods 20:27:20 Who wants to write the email? 20:27:37 annd https://docs.fedoraproject.org/en-US/epel/epel-policy/#policy_for_orphan_and_retired_packages so you can obviously retire packages 20:27:53 should we wait for the CVE page ? 20:28:11 I can write the email. but yeah should I write the CVE page first? 20:28:15 I'm ok waiting 20:28:25 yeah, it's been a longstanding issue anyway 20:28:31 alright, I'll draft something and put up a PR 20:29:35 Do we want to visit this next week? Or wait another month? 20:30:02 * nirik would defer to salimma's availability 20:30:04 let's do a week this time since we might have a documentation and announcement coming up, but maybe a month after that 20:30:15 OK, can do. 20:30:47 Anything else before we move on? 20:31:05 not from me 20:31:24 move along, nothing to see here 20:31:29 #topic Old Business 20:32:01 We have the "Depending on HA" discussion for last week. 20:32:12 I see that carlwgeorge wrote a very good policy in the email. 20:32:22 policy proposal I should say 20:32:44 if people feel good about it, i'm happy to pr the docs with it 20:32:45 Did everyone get a chance to read it ? 20:32:55 * nirik looks it up again 20:33:05 tldr, requires only allowed on baseos/appstream/crb, recommends allowed on ha/rs 20:33:08 link anyone? 20:33:18 or the equivalent repos in 7 20:33:27 https://lists.fedoraproject.org/archives/list/epel-devel@lists.fedoraproject.org/thread/VAYJH5EFAIWHKUKA7LIMASBHYVSW75IW/ ? 20:33:46 as is fashionable, i sent it shortly before the meeting to make sure no one could read it ahead of time :P 20:33:50 yeah, weak depenndencies sound fine 20:34:14 should we limit it to additional channels that are available at no additional cost, or do we not care? 20:34:37 carlwgeorge: but that's how you make sure people *do* read it :) 20:34:38 I don't like it. ;) 20:34:45 i don't care about cost or not, as stream and rebuilds probably offer them free anyways 20:35:00 Ya, I don't care about cost either. 20:35:10 * salimma trying to remember which 'how to run meetings' book recommends this. basically have a reading period in the meeting rather than assuming people pre-read 20:35:10 there's 10gigillion rhel channels with all sorts of stuff in them. I don't think we should allow deps on them. 20:35:20 This would actually be great for the mongodb support packages in epel. 20:35:29 fyi, ha is available in the free devsub, but Eighth_Doctor tells me the entry level paid rhel sub doesn't include it, so it's a mixed bag 20:35:33 I'd prefer to bring HA and RS into the repos we build against 20:35:38 maybe limit to "channels that are available in rebuilds"? 20:35:48 nirik: recommends only, so the don't fail if not available 20:35:50 we should absolutely not do that since the paid subs don't have it 20:36:16 the devsub is a bad match for folks who actually have rhel for real 20:36:37 again, recommends only 20:36:37 we don't have a rhel rebuild for rhel 9 yet to qualify which channels will be available 20:36:38 I think it's a can o worms... 20:37:39 salimma: fesco had a rule that anything posted sooner than a day before a meeting shouldn't be discussed in that meeting, it should wait until next one... but thats been bent a bit 20:37:49 :) 20:37:53 i haven't been able to gather the stats yet, but i do recall that there are some partially shipped packages in ha/rs, and adding those to the "target base" would mean getting into that problem with -epel variants of the packages to ship the missing stuff 20:38:03 nirik: bent how, using timezone shenanigans? :) 20:38:08 I've lost track of who's for the proposal, and who is against it or has concerns. 20:38:40 I'm tentative +1 for allowing weak deps, but prefer we have stronger criteria for which additional channels qualify 20:38:49 I guess my main desire to add HA is so no one has to maintain that stuff in epel, but if we would have to anyhow, I guess thats moot. 20:38:50 I'm currently +1 too. 20:38:59 -1 from me. :) 20:39:00 #info proposed policy "All EPEL package dependencies (build-time or runtime) MUST ALWAYS be satisfiable within the Target Base or EPEL itself. Weak package dependencies are allowed on packages from additional RHEL channels that are not part of the Target Base, such as the HighAvailability channel." 20:39:41 i am currently against the proposal. weak deps in rpm/dnf in EL8 early on was 'broken' in some ways to the maintainers. I don't know if htey have been fixed 20:39:56 i'll gather the stats this week on which epel packages would become in violation of policy if we add ha to the base 20:40:27 the way it works now, if you have a recommends that can't be satisfied, it's simply ignored 20:40:32 * pgreco returns from a "quick work call" :( and reads back 20:40:38 which is why i think it's a good compromise 20:40:41 i am also currently against it for the reason that I agree with the FESCO rule of needing some time before making a decision 20:40:51 I don't like the idea of adding HA to the base. 20:41:03 i accept full blame for waiting till the last minute to get my reply to that email out 20:41:17 Ebeneezer_Smooge: Good point. I'm ok waiting another week on this . 20:41:18 i've been promising dherrera a reply since before the last meeting, i'm just a procrastinator 20:41:29 I would prefer we defer a week, gather some stats and then re-vote 20:41:34 Shame on you carlwgeorge :) 20:41:35 agreed, we don't have to force a decision today or anything 20:41:43 happens 20:42:13 I don't like the idea of adding HA to the build, but I am curious how many of the packages would conflict. 20:42:39 Anyway, sounds like this will be deferred until next week. 20:42:48 also how does one test that weak-deps will 'work' / 'not-work' to make sure the proposal is ok 20:42:56 +1 for deferring, tbh i don't feel like i understand the tradeoffs well enough to decide now 20:43:23 #info proposal voting is deferred until next week. 20:43:25 Ebeneezer_Smooge: find a weak dep, exclude it, and install the package with the dep 20:44:19 Are we ok moving on at this point? 20:44:22 I am 20:44:29 * carlwgeorge nods 20:44:29 yeah 20:44:32 #topic EPEL-7 20:44:32 CentOS 7 will go EOL on 30 June, 2024 20:44:44 Other than the CVE's, anything for EPEL7? 20:45:02 not from me 20:45:18 #topic EPEL-8 20:45:46 just a FYI... 20:46:08 I have a scratch build for ansible 5 building against epel8-next... (and python38) 20:46:19 Cool 20:47:03 Oh, and last week I said I was going to update the KDE Plasma Desktop ... well, that isn't happening. There was a wayland dependency that stopped it. So I just patched the packages that needed patching. 20:47:34 That's all I have for EPEL8. 20:47:49 #topic EPEL-9 20:49:03 Although I did most of my work last week in epel9 ... none of it really needs to be discussed here ... anyone else have any epel9 stuff ? 20:49:17 I'm looking at bringing neovim to EPEL9 now that I've sorted out the upgrade for EPEL8, this is the first batch of dependencies - https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-61461ae87a 20:49:19 oh one last EL8 item.. CentOS Stream 8 goes EOL in 2024-05-31 20:49:39 so do we have an idea about epel8-next then? 20:49:50 epel8-next eol matches c8s 20:50:12 * nirik nods 20:50:19 Yep 20:50:30 no problem. I just missed that point 20:50:37 salimma: nice ... you managed to get them all in one update. 20:50:57 https://pagure.io/fedscm-admin/blob/main/f/fedscm_admin/__init__.py#_63 20:50:58 tdawson: well, another batch coming since I forgot to branch some and I'm escalating one package 20:51:25 #topic EPEL-Packaging-SIG 20:52:03 I updated the epel-packaging-sig document with a "get to work" section, that I think listed everything we talked about last week. 20:52:15 carlwgeorge, should the epel8-playground get changed in that link you gave? 20:52:27 https://docs.fedoraproject.org/en-US/epel/epel-packagers-sig/#_get_to_work 20:53:10 probably, yeah 20:53:27 Feel free to update it if you think of other things for the sig. 20:53:32 tdawson: nice. I'll link the CVE page to this when it's up 20:53:45 i'm not sure of the impact of it being wrong as i'm not sure what all reads that file, but worth correcting regardless 20:53:58 tdawson: what would you say is our official eol date of epel8-playground? 20:54:06 s/is/was/ 20:54:26 carlwgeorge: January 31 2022 ... or possibly Feb 1 20:54:37 2022-02-01 .. what he said 20:55:00 Or ... you can make it Feb 2 ... just for a funner number 2022-02-02 20:55:24 #topic General Issues / Open Floor 20:56:03 oh, this just came to mind 20:56:17 Ebeneezer_Smooge: got an example for you, in c8s chrony recommends timedatex, and `dnf --exclude timedatex install chrony` works 20:56:21 salimma: oh, I noticed postorius got orphaned... 20:56:26 Davide and I have been slowly adding packit configs to the in-house projects we maintain in Fedora 20:56:52 might be worth looking into for packages that are lightly maintained in EPEL so we know if the Fedora spec rebuilds fine 20:56:59 (kind of like ELN, but for stable EPEL) 20:57:11 carlwgeorge, thanks for finding that 20:57:17 nirik: oh woops, I need to claim that back again. yeah, I pushed that too soon but need to chase some deps 20:57:45 I picked it back up and gave it back to you for you 20:58:07 yeah, it's hard to keep track of that stack... ;( 20:58:18 Conan Kudo: ah thanks 20:58:26 was wondering why it's not marked orphaned 20:58:30 let me claim the bugs real quick 20:59:07 it's just a case of juggling too many things in this case - and not checking for FTI :( 20:59:08 salimma: I'm not sure why we want to make sure the Fedora packages build in EPEL ... what does that give us? 20:59:45 I'm always happy when the ones I maintain build in both Fedora and EPEL, but there are many packages that keep them seperate. 20:59:55 tdawson: yeah... probably not necessary for most packages 21:00:13 I guess it matters for some where we know we want to keep them up to date 21:00:23 e.g. the neovim dependencies - or say KDE 21:00:29 Packit is useful when upstream is friendly and they accidentally break epel by adding some new dependency 21:00:31 we're out of time but i had one more quick open floor thing i wanted to get people thinking about 21:00:42 True, and if you know them, I see no problem in putting them in. 21:00:46 go for it carl 21:00:48 carlwgeorge: ok 21:00:48 if people can stick around for another minute or 3 21:00:58 Yeah sure 21:01:05 Unless someone else needs the channel 21:01:27 IIRC there's normally nobody after us 21:01:28 I can stick around 21:01:32 i'm observing rumblings about the stalled epel process leaving some fedora maintainers feeling rushed or gone around 21:01:56 hmm.. is 1 week + 1 week too rushed? 21:02:01 yeah, it's a balancing act... 21:02:10 not placing any blame, but there was an example where the policy was followed, and the maintainer didn't reply at all until after the releng ticket for perms was filed, and then it proceeded 21:02:36 it was more miscommunication than anything, and timing, but i would like us to consider tapping the brakes on the process ever so slightly 21:02:50 from my experience it's normally maintainer saying "I don't have time but feel free, I gave you ACL" and total silence 21:03:02 That happened to me (and I might be the example) ... someone didn't reply at all, I got the permissions, and found they had built the package the day before. 21:03:04 but haven't heard of the maintainer then complaining (yet) 21:03:21 my ideas were to modify to 3x checks 1 wk apiece (add one more check), or to keep the 2x checks, but extend to 2wk intervals 21:03:40 yeah... +1 on waiting a bit. FWIW I normally wait > 1 week before needinfo-ing, just because I forgot to do so faster 21:03:54 the former would be a 3 wk total process, the later would be 4 wk 21:03:57 how about a balance, first check after a week, then wait two weeks for second check 21:04:03 otherwise we get accused of nagging 21:04:12 Using needinfo more aggressively is also an option 21:04:17 but after the second check, one week to filing rel-eng, as now 21:04:24 We could make it in the policy if we wanna be sure they maintainer sees it 21:04:26 the thing is we don't currently do needinfo 21:04:31 +1 to formalizing needsinfo into the process 21:04:31 oh, we should clarify that the first ping involves a needinfo - I normally always do that 21:04:33 But i expect some will find it annoying 21:04:53 needinfo is a anoying hammer for sure. ;) 21:04:53 Yeah i don't usually needinfo because it feels aggressive, but if we make it the policy I'm cool with it 21:04:54 needs info on the first ping would be annoying imo 21:05:00 I already wait a fairly long time because I don't sync back with them exactly on each week 21:05:03 so two pings 21:05:05 I'm ok revisiting this. Let's give it a week of emails and thought, and I'll put it in the Old Business next week. 21:05:19 first ping after a week, second ping after an additional two weeks with needinfo, then after a week, releng? 21:05:20 yup, like i said, just give it some thought 21:05:25 perhaps a list thread to discuss over the next week? 21:05:31 i can put it on list too 21:05:43 I'd probably do needinfo after the first ping 21:05:47 so second ping needinfo 21:05:51 carlwgeorge: Just not 30 minutes before next weeks meeting ;) 21:05:54 hehe 21:05:59 Yeah that seems reasonable 21:06:02 no promises 21:06:06 but i'll try 21:06:24 Thank you everyone for the good discussions today. 21:06:34 We're over time, but I think it was worth it. 21:06:34 I guess my dislike of needinfo is that it's often not clear what info is needed... but in this case it's mostly clear 21:06:37 thank you tdawson and others 21:06:42 thanks everyone 21:06:43 thannks all 21:06:49 Thanks folks 21:06:50 I'll talk to you next week if not sooner. 21:06:59 #endmeeting