2026-05-27 15:30:23 <@marmijo:fedora.im> !startmeeting fedora_coreos_meeting
2026-05-27 15:30:24 <@meetbot:fedora.im> Meeting started at 2026-05-27 15:30:23 UTC
2026-05-27 15:30:24 <@meetbot:fedora.im> The Meeting name is 'fedora_coreos_meeting'
2026-05-27 15:30:28 <@marmijo:fedora.im> !topic roll call
2026-05-27 15:31:11 <@hricky:fedora.im> !hi
2026-05-27 15:31:13 <@zodbot:fedora.im> Hristo Marinov: Hristo Marinov (hricky) - he / him / his
2026-05-27 15:32:21 <@cadejacobson:matrix.org> !hi
2026-05-27 15:32:23 <@zodbot:fedora.im> Cade: Cade Jacobson (cadejacobson)
2026-05-27 15:32:35 <@peytonrobertson:matrix.org> !hi
2026-05-27 15:32:37 <@zodbot:fedora.im> No Fedora Accounts users have the @peytonrobertson:matrix.org Matrix Account defined
2026-05-27 15:34:17 <@marmijo:fedora.im> Good morning/afternoon/evening. Let's wait a little bit longer for more folks to attend
2026-05-27 15:34:17 <@dustymabe:matrix.org> !hi
2026-05-27 15:34:22 <@zodbot:fedora.im> dustymabe: Dusty Mabe (dustymabe) - he / him / his
2026-05-27 15:34:52 <@jbtrystram:matrix.org> !hi
2026-05-27 15:34:53 <@zodbot:fedora.im> jbtrystram: Jean-Baptiste Trystram (jbtrystram) - he / him / his
2026-05-27 15:35:49 <@rapneset:matrix.org> !hi
2026-05-27 15:35:51 <@zodbot:fedora.im> rapneset: Rolv Apneseth (rapneset)
2026-05-27 15:35:56 <@angelcr:matrix.org> !hi
2026-05-27 15:35:58 <@zodbot:fedora.im> Angel Cervera Roldan: Angel Cervera Roldan (acervera)
2026-05-27 15:36:09 <@marmijo:fedora.im> !topic Action items from last meeting
2026-05-27 15:36:14 <@siosm:fedora.im> !hi
2026-05-27 15:36:16 <@zodbot:fedora.im> Timothée Ravier: Timothée Ravier (siosm) - he / him / his
2026-05-27 15:36:19 <@marmijo:fedora.im> !info there are no action items from the last meeting
2026-05-27 15:37:08 <@marmijo:fedora.im> There's not much to discuss with the F45 release schedule at the moment. We can move on to the main topics unless there's something specific to bring up.
2026-05-27 15:38:08 <@marmijo:fedora.im> Alright, let's move into the main topics. We have 4, so I'll go in the order the meeting label was added. Please stop me if something is more pressing!
2026-05-27 15:38:24 <@marmijo:fedora.im> !topic Compress container images using zstd
2026-05-27 15:38:30 <@marmijo:fedora.im> !link https://github.com/coreos/fedora-coreos-tracker/issues/2150
2026-05-27 15:39:57 <@marmijo:fedora.im> Timothée Ravier: would you like to introduce this one?
2026-05-27 15:40:24 <@siosm:fedora.im> sure
2026-05-27 15:40:43 <@siosm:fedora.im> The idea is to switch to zstd by default for layer compression for the container images
2026-05-27 15:41:23 <@siosm:fedora.im> That should reduce network bandwith required for update and potentially make them faster but I have not done benchmarks to confirm that.
2026-05-27 15:41:54 <@siosm:fedora.im> Work would be to do those benchmarks and then do the switch
2026-05-27 15:41:57 <@dustymabe:matrix.org> current compression is gzip ?
2026-05-27 15:42:02 <@siosm:fedora.im> yes
2026-05-27 15:42:26 <@nemric:relativit.fr> !hi
2026-05-27 15:42:28 <@zodbot:fedora.im> Nemric: Emeric Chassagne (nemric)
2026-05-27 15:43:42 <@hricky:fedora.im> Is the testing similar to what we did for initramfs?
2026-05-27 15:44:30 <@siosm:fedora.im> Not exactly as this is for the container image not the initrd but it's a similar idea
2026-05-27 15:46:23 <@marmijo:fedora.im> It looks like the first step would be to run a benchmark to understand the speed and bandwidth improvements
2026-05-27 15:47:18 <@siosm:fedora.im> yes
2026-05-27 15:48:02 <@siosm:fedora.im> I don't think there is much more to talk about. We can do a vote that it's what we want to do and we'll get to it at some point?
2026-05-27 15:48:46 <@spresti:fedora.im> !hi 
2026-05-27 15:48:48 <@zodbot:fedora.im> spresti: Steven Presti (spresti)
2026-05-27 15:48:50 <@dustymabe:matrix.org> +1 from my side. we can bundle that into `next` for some time because we are wanting to experiment with a few things there anyway
2026-05-27 15:49:29 <@jbtrystram:matrix.org> +1 from my side
2026-05-27 15:49:31 <@marmijo:fedora.im> !info PROPOSED: Investigate switching Fedora CoreOS container image layer compression from gzip to zstd, including running benchmarks to confirm improvements in network bandwidth and update speed, with the goal of enabling zstd by default if results are positive.
2026-05-27 15:49:50 <@marmijo:fedora.im> +1
2026-05-27 15:49:59 <@hricky:fedora.im> +1
2026-05-27 15:49:59 <@hricky:fedora.im> If the same tools can be used as for benchmarking initramfs compression, I can try.
2026-05-27 15:50:26 <@siosm:fedora.im> +1
2026-05-27 15:51:03 <@marmijo:fedora.im> !agreed Investigate switching Fedora CoreOS container image layer compression from gzip to zstd, including running benchmarks to confirm improvements in network bandwidth and update speed, with the goal of enabling zstd by default if results are positive.
2026-05-27 15:51:17 <@marmijo:fedora.im> !topic Denylist/blacklist a set of "unlikely to be used" kernel modules by default
2026-05-27 15:51:21 <@marmijo:fedora.im> https://github.com/coreos/fedora-coreos-tracker/issues/2152
2026-05-27 15:51:26 <@marmijo:fedora.im> !link https://github.com/coreos/fedora-coreos-tracker/issues/2152
2026-05-27 15:52:37 <@dustymabe:matrix.org> expressed my concerns in the ticket there
2026-05-27 15:53:47 <@siosm:fedora.im> Summary for this one is that if we want to keep up with vulnerabilities in the kernel, we'll likely have to start denylisting/blacklisting kernel modules for features likely 99% of Fedora CoreOS users never use
2026-05-27 15:53:54 <@cverna_:matrix.org> !hi
2026-05-27 15:54:16 <@zodbot:fedora.im> Clément Verna: Clement Verna (cverna) - he / him / his
2026-05-27 15:55:13 <@siosm:fedora.im> One example from the recent Dirty Frag vulnerabilities is rxrpc sockets
2026-05-27 15:55:49 <@siosm:fedora.im> This is needed for AFS support (Andrew File System) and this is mostly a datacenter filesystem
2026-05-27 15:56:12 <@siosm:fedora.im> so you would definitely know if you needed this module
2026-05-27 15:56:40 <@siosm:fedora.im> denylisting modules like that also does not prevent them from being loaded, it only makes it explicit
2026-05-27 15:57:12 <@spresti:fedora.im> Hmm, so are we are looking at it from the stance reducing security boundary footprint?
2026-05-27 15:58:52 <@jbtrystram:matrix.org> It's quite an opinionated move to make, but I see where you're coming from. I don't really have strong opinions. 
2026-05-27 15:58:52 <@jbtrystram:matrix.org> I would vote to stick with whatever is fedora default. 
2026-05-27 15:58:55 <@siosm:fedora.im> yes, the less modules can be loaded by unprivileged users, the smaller the attack surface and thus the less work we have to do to push updates urgently
2026-05-27 15:59:01 <@spresti:fedora.im> Naive question, could we run into the circumstance, where we ship a kernal that does not have modules enabled but a user enables said module? resulting in our priority not being to patch said issue and user fall into not knowing about that issue? 
2026-05-27 15:59:56 <@siosm:fedora.im> I'm not sure I understand your question. We would only deny modules available in the Fedora kernel
2026-05-27 16:00:17 <@siosm:fedora.im> (and that would only apply to feature modules, not hardware support ones)
2026-05-27 16:00:49 <@marmijo:fedora.im> Could we run into a situation where a user manually enables a denylisted module leaving them exposed to vulnerabilities without realizing it (since we wouldn't try to patch it).
2026-05-27 16:01:12 <@dustymabe:matrix.org> I think there is a lot of value we derive from just accepting whatever the fedora kernel maintainer's assessment of the security implications of a CVE is. 
2026-05-27 16:01:12 <@dustymabe:matrix.org> If we start being much different, that value starts to go away and we need our own dedicated team for that stuff 
2026-05-27 16:01:12 <@dustymabe:matrix.org> 
2026-05-27 16:01:38 <@spresti:fedora.im> Thank you marmijo for re-wording that
2026-05-27 16:01:57 <@siosm:fedora.im> yes, users that would enable a module that we deny would accept the risk associate with it.
2026-05-27 16:02:20 <@siosm:fedora.im> Note that this is not replacing the assesment from the Fedora's kernel maintainer
2026-05-27 16:02:32 <@siosm:fedora.im> they can only patch bugs so fast
2026-05-27 16:02:57 <@siosm:fedora.im> and we can not realistically realease updates daily
2026-05-27 16:05:16 <@spresti:fedora.im> ok, thank you. I agree it makes sense to reduce the attack surface where possible
2026-05-27 16:05:29 <@cverna_:matrix.org> I would be curious to explore the release updates daily, maybe rethinking the stream we have.
2026-05-27 16:05:36 <@dustymabe:matrix.org> 
2026-05-27 16:05:36 <@dustymabe:matrix.org> 1. extra maintenance over time
2026-05-27 16:05:36 <@dustymabe:matrix.org> 2. different kernel configuration than fedora ships
2026-05-27 16:05:36 <@dustymabe:matrix.org> I just think anything that adds to our load here has risks:
2026-05-27 16:05:36 <@dustymabe:matrix.org> 3. are we going to do the same thing downstream? If so then now we are differing from RHEL. If not then our upstream and downstream act different too 
2026-05-27 16:05:55 <@cverna_:matrix.org> I feel this is the general model towards patching CVEs
2026-05-27 16:06:18 <@siosm:fedora.im> Releasing daily is not realistic with auto updates enabled for every one
2026-05-27 16:06:25 <@angelcr:matrix.org> I also think it makes sense - specially given that it is not hard for a user to enable them in ignition.
2026-05-27 16:06:47 <@siosm:fedora.im> I don't uderstand why you're saying "different kernel configuration" Dusty
2026-05-27 16:07:06 <@siosm:fedora.im> And for 3 I would say that yes we should push that as well
2026-05-27 16:07:14 <@cverna_:matrix.org> I don't think everyone has auto updates enabled, at least from the countme stats we have
2026-05-27 16:07:32 <@dustymabe:matrix.org> "different kernel configuration" == "Fedora CoreOS isn't affected by CVE XYZ, but Fedora Cloud is" 
2026-05-27 16:07:35 <@cverna_:matrix.org> But I am side tracking the conversation :) 
2026-05-27 16:08:01 <@dustymabe:matrix.org> basically it requires more scrutiny each time a security issue happens
2026-05-27 16:08:01 <@siosm:fedora.im> If we move to daily updates then this pratically guarentees that people will disable auto updates
2026-05-27 16:08:25 <@siosm:fedora.im> We already need to do that?
2026-05-27 16:08:39 <@siosm:fedora.im> do decide if we need to fasttrack something or do a release early
2026-05-27 16:08:58 <@dustymabe:matrix.org> and while Timothée Ravier and rapneset are doing a good job of analyzing security issues now (@bgilbert used to help with this in the past). I'm not entirely confident we'll always have someone super well versed in security in this team and able to maintain the "list of modules that aren't supporting Fedora CoreOS use cases" 
2026-05-27 16:09:21 <@jcline:fedora.im> FWIW it'd be interesting to collaborate on a list with the Cloud variants, or as a general opt-in "I want to harden my install" package?
2026-05-27 16:09:48 <@siosm:fedora.im> Jeremy Cline happy to start something that works accross server variants at least
2026-05-27 16:09:52 <@dustymabe:matrix.org> yeah. basically I want this list to be determined by someone who is an expert in "insert use case here" 
2026-05-27 16:10:17 <@dustymabe:matrix.org> I don't think this group (coreos) is well enough versed to maintain it 
2026-05-27 16:10:37 <@dustymabe:matrix.org> the kernel surface area is vast
2026-05-27 16:10:57 <@dustymabe:matrix.org> and trying to play that game is not something we should enter into lightly
2026-05-27 16:11:07 <@siosm:fedora.im> note that the kernel surface is mostly drivers, and we would not look at that here, only features
2026-05-27 16:11:37 <@siosm:fedora.im> I understand you concerns Dusty but I don't think they are warranted
2026-05-27 16:11:40 <@jcline:fedora.im> Disabling a kernel module being built is a big hammer and will forever break someone's spacebar-heating workflow or whatever, but there's probably a good selection of "you likely don't need this for most use cases" that dropin configs to denylist would be nice for
2026-05-27 16:11:47 <@dustymabe:matrix.org> I don't know
2026-05-27 16:11:47 <@dustymabe:matrix.org> for example we still have the [kernel mitigations things](https://github.com/coreos/fedora-coreos-config/blob/0c62bd9d22a1f469729a9514d5064781db5f1a07/image-base.yaml#L12).. do we even still need them?
2026-05-27 16:11:47 <@dustymabe:matrix.org> 
2026-05-27 16:12:38 <@siosm:fedora.im> from memory yes
2026-05-27 16:12:41 <@dustymabe:matrix.org> Jeremy Cline: why couldn't we just make them modules that aren't loaded by default in the main kernel rpm (not something specific to a variant)? 
2026-05-27 16:13:48 <@siosm:fedora.im> the kernel package is common to all Fedora variants so you would have to create a list that works for all potential use cases of Fedora so essentially the list is empty as otherwise we would not have the module in the kernel in the first place
2026-05-27 16:14:23 <@jcline:fedora.im> Sure, if you want a more centralized list I don't think that's a problem either, and I have no strong opinion on where the list(s) are maintained. Obviously different variants have different targets and it's really a matter of who you want to wrap up in the specifics.
2026-05-27 16:15:14 <@jcline:fedora.im> IMO Fedora kernel team likely doesn't want to be interested in all the specifics so keeping a list there is harder than elsewhere
2026-05-27 16:15:57 <@jlebon:fedora.im> !hi
2026-05-27 16:15:58 <@zodbot:fedora.im> Jonathan Lebon: None (jlebon)
2026-05-27 16:16:04 <@siosm:fedora.im> We don't need to decide this today. I'll work with Jeremy to get this started and give it a try on the Atomic Desktops and we can revisit that later. This will likely be a change request for the Atomic Desktops.
2026-05-27 16:16:57 <@jlebon:fedora.im> reading the scrollback; this has some similarities to systemd presets where there's global lists and per-edition additions. one way to structure this is similarly a global list and a shared "workstation" and "server" sublist
2026-05-27 16:17:48 <@dustymabe:matrix.org> 
2026-05-27 16:17:48 <@dustymabe:matrix.org> basically I think having a "desktop use case" and "server use case" profile should suffice. and would allow sharing amongst the working groups  
2026-05-27 16:17:48 <@dustymabe:matrix.org> agree.
2026-05-27 16:18:02 <@jlebon:fedora.im> right
2026-05-27 16:18:22 <@jlebon:fedora.im> i will say though i think part of the motivation here is actually rooted in the fact that our release process is too manual
2026-05-27 16:19:01 <@jlebon:fedora.im> and so that's another place where pain could be alleviated
2026-05-27 16:19:43 <@siosm:fedora.im> While I agree that we can always improve the release process to be less manual, I disagree that it's the motivation for this. Releasing faster would not help.
2026-05-27 16:20:10 <@siosm:fedora.im> On the contrary, I want us to be able to release less often
2026-05-27 16:20:17 <@jlebon:fedora.im> I didn't say faster
2026-05-27 16:21:05 <@jlebon:fedora.im> (in the sense of releasing at a higher cadence)
2026-05-27 16:21:58 <@marmijo:fedora.im> Looks like we're wrapped up here pretty well. Should we do a proposed on this for now, or want me to just add a discussion summary to the issue?
2026-05-27 16:22:27 <@siosm:fedora.im> a summary is fine for me
2026-05-27 16:22:46 <@siosm:fedora.im> unless folks want to vote
2026-05-27 16:23:10 <@marmijo:fedora.im> I'm not sure we'll have enough time for our other 2 topics. jbtrystram are these topics pressing for today?
2026-05-27 16:23:34 <@jbtrystram:matrix.org> Not really :) 
2026-05-27 16:24:04 <@marmijo:fedora.im> okay, they'll be first on the list for next week!
2026-05-27 16:24:12 <@marmijo:fedora.im> !topic Open Floor
2026-05-27 16:28:18 <@peytonrobertson:matrix.org> https://github.com/coreos/afterburn/pull/1264
2026-05-27 16:28:18 <@peytonrobertson:matrix.org> Just plugging this PR again briefly! I've gone back and forth on some of the dependencies, but I think I've finally been able to keep the crates FIPS/OpenSSL compliant while completing the logic for password hashing:
2026-05-27 16:30:29 <@spresti:fedora.im> Awesome, thank you for working on that, I will try and take a look today or tomorrow
2026-05-27 16:31:29 <@marmijo:fedora.im> Thanks, all, for joining today and thanks for the lively discussion on the topics!
2026-05-27 16:31:35 <@marmijo:fedora.im> !endmeeting