20:00:34 <tdawson> #startmeeting EPEL (2023-05-03)
20:00:34 <zodbot> Meeting started Wed May  3 20:00:34 2023 UTC.
20:00:34 <zodbot> This meeting is logged and archived in a public location.
20:00:34 <zodbot> The chair is tdawson. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
20:00:34 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
20:00:34 <zodbot> The meeting name has been set to 'epel_(2023-05-03)'
20:00:35 <tdawson> #meetingname epel
20:00:35 <zodbot> The meeting name has been set to 'epel'
20:00:37 <tdawson> #chair nirik tdawson pgreco carlwgeorge salimma dcavalca dherrera gotmax23 smooge
20:00:37 <zodbot> Current chairs: carlwgeorge dcavalca dherrera gotmax23 nirik pgreco salimma smooge tdawson
20:00:38 <tdawson> #topic aloha
20:01:10 <carlwgeorge> .hi
20:01:11 <zodbot> carlwgeorge: carlwgeorge 'Carl George' <carl@redhat.com>
20:01:13 <nirik> morning
20:01:18 <tdawson> Hi carlwgeorge
20:01:22 <tdawson> Morning nirik
20:01:24 <Eighth_Doctor> .hello ngompa
20:01:25 <zodbot> Eighth_Doctor: ngompa 'Neal Gompa' <ngompa13@gmail.com>
20:01:28 <jonathanspw> .hi
20:01:31 <zodbot> jonathanspw: jonathanspw 'Jonathan Wright' <jonathan@almalinux.org>
20:01:48 <tdawson> Hi jonathanspw
20:01:52 <tdawson> Hello Eighth_Doctor
20:03:01 <neil> .hi
20:03:02 <zodbot> neil: neil 'Neil Hanlon' <neil@shrug.pw>
20:03:07 <neil> heya folks
20:03:26 <tdawson> Hi neil
20:03:43 <yselkowitz[m]> .hello yselkowitz
20:03:44 <zodbot> yselkowitz[m]: yselkowitz 'Yaakov Selkowitz' <yselkowi@redhat.com>
20:04:29 <tdawson> Hello yselkowitz[m]
20:05:08 <tdawson> #topic End Of Life (EOL)
20:05:09 <tdawson> RHEL 7 / epel-7 will go EOL on 2024-06-30
20:05:11 <tdawson> CentOS Stream 8 / epel-8-next goes EOL in 2024-05-31
20:05:12 <tdawson> CentOS Stream 9 / epel-9-next goes EOL in 2027-05-31
20:05:23 <tdawson> #topic EPEL Issues https://pagure.io/epel/issues
20:05:25 <tdawson> https://pagure.io/epel/issues?tags=meeting&status=Open
20:05:48 <tdawson> We only have one issue marked meeting that is open
20:05:52 <tdawson> .epel 226
20:05:53 <zodbot> tdawson: Issue #226: Incompatible update in apptainer-suid-1.1.8 - epel - Pagure.io - https://pagure.io/epel/issue/226
20:06:28 <tdawson> This issue has also been discussed on the mailling list.
20:06:40 <tdawson> Does anyone have the link to that handy?
20:06:55 <nirik> I fear i failed to re-read it closely. ;(
20:07:02 <carlwgeorge> https://lists.fedoraproject.org/archives/list/epel-devel@lists.fedoraproject.org/thread/LHQDDPAAVGSQBIELADJXL2YQNDK2YCMY/
20:07:16 <tdawson> Thank you
20:07:31 <neil> me too nirik
20:07:58 <nirik> I see carlwgeorge just posted some in that thread.
20:08:19 <carlwgeorge> yeah some last minute follow ups to some comments, but doesn't change my preferences that i posted last week
20:09:20 <tdawson> Summary is that apptainer wants to push a incompatible change for epel 7 8 and 9
20:09:20 <carlwgeorge> tldr; as rhel 8/9 are not affected by the cve, i don't feel an incompat upgrade is justified in epel 8/9.  for epel7, i would prefer to wait for the nvd rating and then decide.
20:09:39 <DrDaveD> How long does nvd take to rate?
20:10:01 <tdawson> Hi DrDaveD
20:11:06 <nirik> yeah, 8 and 9 seem like they shouldn't be incompatibly upgraded... 7... I guess I'd say ok to... if the maintainer is raring to fix it.
20:11:49 <DrDaveD> In Carl's last comment, he seems to have misunderstood the point.  I am not saying that the original CVE CVE-2022-1184 was too low.  I am saying that setuid-root apptainer (prior to 1.1.8) and setuid-root singularity raises the priority of that type of vulnerability because they expose the raw filesystem data to writing by users.
20:11:58 <yselkowitz[m]> I would think we'd be stricter with 7 than 8/9??
20:12:39 <carlwgeorge> yselkowitz[m]: in general yes, to be "maintained similar to rhel" like the policy says, the bar should be higher and higher to do an incompat upgrade as we get later in the lifecycle
20:12:39 <Eighth_Doctor> we should be, yes
20:12:48 <nirik> well, in some senses yeah... it is the oldest stable release
20:13:07 <DrDaveD> But when there are high priority vulnerabilities, they need to be addressed.
20:13:21 <DrDaveD> High severity, I should say.
20:13:23 <carlwgeorge> DrDaveD: but red hat rated it with low privileges factored in
20:13:58 <DrDaveD> Yes CVE-2022-1184 was medium severity.  The problem is apptainer
20:14:25 <tdawson> What is the difference between "exposing the raw filesystem" and ... not, I'm not understanding that part.
20:14:43 <carlwgeorge> is it not the same basic vulnerability, and the reason that CVE-2023-30549 doesn't apply to epel8/9?
20:15:23 <DrDaveD> Normally only privileged users can mount filesystems, so only root has write access to the raw data.  Setuid-root singularity & apptainer use kernel filesystem drivers to mount user-owned filesystems.
20:16:07 <carlwgeorge> sounds like low privs to me, which is how red hat and nvd rated CVE-2022-1184
20:16:08 <tdawson> And something like fuse, doesn't use the kernel drivers to mount things?
20:17:16 <DrDaveD> tdawson: correct, fuse doesn't use kernel filesystem drivers.  The fuse implementation is in the kernel, but the kernel developers have cleared that by exposing it to mounts in unprivileged user namespaces.
20:18:58 <neil> if I'm understanding it correctly, it's a mechanism of how apptainer (and singularity-ce) mount the user filesystems which exposes it. i.e., a non-root user is allowed to mount an unprivileged filesystem in privileged space. It sounds to me the privileges are different between the two (2022-1184 vs current discussion)
20:19:43 <DrDaveD> carlwgeorge: CVE-2022-1184 is not exploitable with low privs except perhaps from a USB-mounted filesystem.  Since that requires physical access which can more easily be exploited in other ways to get root privileges, they wouldn't consider that a potential privilege escalation.
20:20:08 <carlwgeorge> CVE-2022-1184 was literally scored as low privs and only made it to 5.5
20:20:35 <Eighth_Doctor> note that unpriv'd fuse isn't a thing in RHEL 7
20:20:42 <Eighth_Doctor> it was introduced in RHEL 8
20:20:56 <carlwgeorge> the priv levels for the score are just none/low/high, there is no further granularity than that
20:21:17 <DrDaveD> carlwgeorge: That's because they didn't factor in possible privilege escalation.  However, what setuid-root apptainer & singularity do allow exploitation by people logged in from anywhere.
20:21:44 <DrDaveD> Eight_Doctor: not true, unpriv'ed fuse mounts does work in RHEL 7.  Only unprivileged overlayfs doesn't work there.
20:22:41 <carlwgeorge> if the privs are rated as low then how could priv escalation not have been factored in?
20:24:14 <carlwgeorge> this discussion is solidifying my opinion that the best course of action is to wait on the final nvd score to determine how to handle apptainer in epel7, since they likely understand this better than all of us
20:24:21 <DrDaveD> They only treated it as denial of service, I assume because they believed that was the only additional capability it could provide to a user who plugged in a USB filesystem.  Such users can already escalate their privilege by booting a live linux or something.
20:24:26 * nirik stays out of it as I didn't really read up on the issue in detail.
20:25:27 <neil> i was going to say that it seems we're in some manner debating the finer points of CVSS v3 scoring and I'm not an expert by any means, so.. also thinking waiting seems reasonable... depending on the time it takes to come back
20:25:28 <tdawson> I'm leaning towards delaying the vote also, but while DrDaveD is here, I do have a couple more questions.
20:25:48 <DrDaveD> I have been in touch with nvd, and they have at least acknowledged that they will fix the CVSS scores to go with the published scores instead of the one that github gave them, old scores.
20:26:33 <yselkowitz[m]> shouldn't it be RH security's score that takes priority?
20:26:41 <DrDaveD> But I have no idea how long it will take them to get around to it.  Does anybody know that?  Btw I will probably not be able to join the meeting next week.
20:27:00 <tdawson> My first question.  Everything I see shows that the user has to have access to the machine already.  Is this correct?
20:27:05 <carlwgeorge> yselkowitz[m]: red hat didn't score CVE-2023-30549, just CVE-2022-1184
20:27:31 <yselkowitz[m]> will they be scoring it?
20:27:36 <carlwgeorge> no idea
20:27:46 <DrDaveD> tdawson: they need to have a local account for CVE-2023-30549, yes
20:27:55 <tdawson> The CVE's are listing "denial of service" but I'm seeing you list "corruption of data" , which is obviously worse.  But is there any way for them to jailbreak out?
20:28:12 <carlwgeorge> but the former is arguably a duplicate of the latter (at the very least related), which is why the conversation about severity has centered around it
20:28:14 <DrDaveD> yselkowitz: CVE-2023-30549 is not Red Hat's problem, it's Apptainer's problem, so I don't expect Red Hat to score it.
20:29:36 <DrDaveD> tdawson: there is not yet a known "jailbreak" (privilege escalation) but I was informed by the ext4 filesystem owner that he believed any data corruption could be turned into a privilege escalation by a kernel security expert.  He pointed me to an article as an example.
20:29:51 <tdawson> That is what I ws thinking.
20:30:36 <carlwgeorge> "could be turned into a privilege escalation" sounds way too arbitrary and subjective to be part of the actual scoring
20:31:16 <neil> my "gut" tells me that the situation with how red hat evaluated CVE-2022-1184 is different than CVE-2023-30549. as we just discussed, apptainer isn't part of RHEL so they would not be considering a user that has such a binary installed. that said, CVSS does not provide enough granularity, I feel, on the privilege level required due to this
20:31:36 <tdawson> carlwgeorge: It actually sounds reasonable to me.  Many of the other gain root exploits were just "they can just write random stuff" and they managed to get it to be an exploit.
20:32:01 <DrDaveD> carlwgeorge: but there's a big difference between data corruption and a normal denial of service, which can be caused by a panic or something like that.  That is a "potential privilege escalation" which gets a higher rating.
20:32:03 <neil> Agreed there. The risk has to take into account potential other exploits being combined
20:33:00 <tdawson> DrDaveD: This is hinted to, and maybe you said it, but I'm going to ask it plainly.  In your and your ext4 expert opinion, can the data be corrupted on RHEL 8 and 9 machines?
20:33:09 <tdawson> Currently.
20:33:14 <carlwgeorge> but again, none of us are experts in this, so lets wait for the real rating and then decide
20:33:38 <neil> carlwgeorge: but i like to cosplay as an infosec expert...
20:33:44 <tdawson> *laughs*
20:34:46 <DrDaveD> tdawson: not with a currently public CVE, but because any such vulnerability is given a medium or low severity, they can exist for quite some time before they get patched, even on RHEL 8 & 9.
20:34:50 <nirik> does that mean you have a dark hoodie? ;)
20:35:12 <neil> nirik: don't leave home without one
20:35:12 <tdawson> DrDaveD: Thank you for the honest answer.
20:35:37 * carlwgeorge smells an embargo
20:35:37 <DrDaveD> That general attack approach was explained by the ext4 kernel filesystem owner, and he gave me CVE-2022-1184 just as an example.
20:35:57 <tdawson> I think I'm going to lean with carlwgeorge in wanting to delay, but I do feel more informed.
20:36:09 <DrDaveD> Also keep in mind that RHEL8/9 have unprivileged user  namespaces enabled by default, so it is not a very incompatible change there.
20:36:33 <DrDaveD> Apptainer has an unprivileged alternative that is nearly identical behavior, based on fuse2fs.
20:36:57 <carlwgeorge> "not very incompatible" can you elaborate?
20:37:02 <DrDaveD> It also works on RHEL7, but the system administrator has to enable unprivileged user namespaces there.
20:37:39 <DrDaveD> There are some slight differences in unprivileged user namespaces which almost all users don't notice.  In particular supplemental groups aren't imported, only the currently selected group.
20:38:19 <carlwgeorge> my understanding was that even on rhel8/9, it's an incompatible change.  there is behavior that works now that won't work without user config changes if the update is shipped.
20:38:45 <carlwgeorge> by policy that's incompatible
20:38:51 <DrDaveD> Also, if the apptainer-suid rpm is installed, with apptainer-1.1.8 if somebody tries to use an ext3 overlay filesystem (which only a small subset of users even do) by default they will get an error that tells them to add the --userns option.  If apptainer-suid is not installed it just works.
20:39:10 <carlwgeorge> > All updates should strive to avoid situations that require manual intervention to keep the package functioning after update.
20:39:43 <DrDaveD> Yes but the policy allows for changes that are done because for security reasons.
20:39:43 <tdawson> So ... if a user is using apptainer-suid to mount an ext4 file system, they wouldn't see any difference?
20:40:16 <DrDaveD> Actually apptainer only supports mounting ext3 filesystems, but they use the ext4 driver ...
20:40:46 <carlwgeorge> DrDaveD: no.  the policy allows requesting exceptions (usually around security impact), which is the process we are in now.  it doesn't blanket allow them.
20:41:05 <DrDaveD> And as I said, they will need to add a --userns option, and some very small number might notice the change in supplemental groups.
20:41:17 <DrDaveD> carlwgeorge: understood
20:42:59 * tdawson puts his "keep the meeting moving" hat on.
20:43:01 <DrDaveD> I am not opposed to waiting another week for NVD to respond, but if they haven't answered by then, we need to also consider the impact on all the machines that are exposed to the vulnerability and not delay it indefinitely.
20:43:10 <DrDaveD> The policy requires a 2 week wait anyway.
20:43:57 <tdawson> It's sounding like an unanimous "put it off at least one week"
20:44:15 <tdawson> Thank you DrDaveD for comming and answering our questions.
20:44:24 <DrDaveD> You're welcome
20:45:05 <tdawson> I'm going to timebox this, and move on.
20:45:14 <tdawson> #topic Old Business
20:45:40 <tdawson> Did anyone have any old business that they wanted to bring up?
20:45:57 <carlwgeorge> epel 10 thing, regarding dist tags i would like to hold a vote in two weeks to pick an option
20:46:19 <carlwgeorge> mainly because that will let me talk about it clearly at red hat summit as "this is the dist tag we are planning on using"
20:46:25 <tdawson> So, that is May 17, correct?
20:46:30 <carlwgeorge> yessir
20:46:48 * nirik nods. sounds reasonable
20:47:14 <tdawson> Sounds good.  It will be good to get that part off the list of things to decide.
20:47:32 <carlwgeorge> of course that isn't binding if something unforeseen comes up and we're forced to change course
20:47:46 <carlwgeorge> plans are just that, a plan
20:47:56 <tdawson> Correct.  But we'll have a way decided to try to go.
20:48:05 <carlwgeorge> https://discussion.fedoraproject.org/t/epel-10-bikeshedding-dist-tag/80510
20:48:16 <carlwgeorge> ^ discussion thread for those that still haven't weighed in
20:48:48 <carlwgeorge> i think at the may 17 meeting we can do a ranked choice vote since we have three options with mixed support
20:49:41 <tdawson> Yep.  neil pointed out a web page we can use, though I prefer keeing everything here in the meeting so it's recorded ... maybe some combination of both.
20:50:33 <carlwgeorge> yeah doing the vote on the site and then reporting the full result in chat is fine by me
20:50:48 <tdawson> That could work.
20:51:31 <neil> the site also produces a report with a breakdown of how the election was done, too
20:51:33 <neil> so that can be saved
20:51:51 <tdawson> Since we're getting low on time, I'm moving to open floor.   If there are other old business, feel free to open floor them.
20:51:59 * carlwgeorge wishes he had ranked choice for real life voting
20:52:00 <tdawson> #topic General Issues / Open Floor
20:52:11 <tdawson> *laughs*
20:52:20 <carlwgeorge> i think we missed one issue that didn't get tagged for the meeting
20:52:21 <carlwgeorge> https://pagure.io/epel/issue/227
20:52:41 <carlwgeorge> considering the time we should probably bump it to next week's meeting, as it doesn't look like a simple topic
20:53:13 <tdawson> Huh ... I didn't even see that one, which is why I didn't tag it.
20:53:20 <yselkowitz[m]> next week, but I'm sceptical given the context
20:54:40 <tdawson> Yep, that one seems like a multi-week one.  Mainly because someone needs to write the new policy.
20:54:56 <tdawson> policy isn't right ... new wording.
20:55:07 <neil> LawyerGPT
20:55:12 <tdawson> :)
20:55:25 <carlwgeorge> yes a concrete wording of the changes being asked for is needed, preferably in pull request form
20:56:14 <carlwgeorge> at the very least we should probably remove the note about "need some way..." from the page until we actually decide what that way is
20:56:58 <tdawson> One thing I wanted to bring up is that May is "RHEL Update Month", RHEL 8.8 and 9.2 will be released at some point this month.
20:57:12 <neil> how much we gotta pay to know the exact dates? ;)
20:57:32 <tdawson> If you have changes you want, start getting them ready.
20:57:33 <carlwgeorge> no red hat pays you to know the dates, it's called getting hired lol
20:57:39 <tdawson> :)
20:57:40 <neil> lol
20:58:11 <neil> I'm tempted to say "everyone has a price" but I don't _really_ mean it outside of the joke, lol
20:58:31 <tdawson> One thing we can say, is that they won't get released on the same week.  There is to much ... bandwidth, same teams doing the release, other reasons ...
20:59:14 <neil> yeah, i mean it's fairly easy to work out +/- a week given prior release dates
20:59:58 <tdawson> Anyway, so get your changes ready.  Especially if you know you'll need a rebuild.
21:00:02 <neil> i know we're late, but if anyone has recommended reading on EPEL after/during a RHEL release, I wouldn't mind a link
21:00:32 <tdawson> I'm not sure I know what you are asking.
21:00:49 <carlwgeorge> i don't think we have a guide to that, but tldr would be make sure your packages install on the new rhel minor version, if not fix them
21:01:10 <tdawson> And with that, looks like our time is up.
21:01:17 <neil> makes sense. didn't know if there was some check to see if RHEL added something currently in epel, or what not
21:01:19 <carlwgeorge> ideally you would already know about it because of centos stream and epel-next, and it's just a matter of repeating the build in the epel branch
21:01:43 <tdawson> Thank you all for the good discussions.  And thank you for all your work for EPEL and it's community.
21:01:49 <carlwgeorge> there is, and if it's the case you should have a bugzilla on it
21:01:58 <carlwgeorge> several months ahead of time i believe
21:02:02 <neil> thanks tdawson!
21:02:04 <tdawson> #endmeeting