2024-04-24 16:30:59 <@siosm:matrix.org> !startmeeting fedora_coreos_meeting
2024-04-24 16:31:03 <@meetbot:fedora.im> Meeting started at 2024-04-24 16:30:59 UTC
2024-04-24 16:31:03 <@meetbot:fedora.im> The Meeting name is 'fedora_coreos_meeting'
2024-04-24 16:31:19 <@siosm:matrix.org> !topic roll call
2024-04-24 16:31:21 <@siosm:matrix.org> !hi
2024-04-24 16:31:23 <@zodbot:fedora.im> Timothée Ravier (siosm) - he / him / his
2024-04-24 16:31:39 <@dustymabe:matrix.org> !hi
2024-04-24 16:31:41 <@zodbot:fedora.im> Dusty Mabe (dustymabe) - he / him / his
2024-04-24 16:32:29 <@jbtrystram:matrix.org> !hi
2024-04-24 16:32:36 <@jbrooks:matrix.org> !hi jasonbrooks
2024-04-24 16:32:45 <@zodbot:fedora.im> Jason Brooks (jasonbrooks) - he / him / his
2024-04-24 16:32:45 <@zodbot:fedora.im> Jean-Baptiste Trystram (jbtrystram) - he / him / his
2024-04-24 16:33:44 <@jlebon:fedora.im> !hi
2024-04-24 16:33:45 <@zodbot:fedora.im> None (jlebon)
2024-04-24 16:33:50 <@marmijo:fedora.im> !hi marmijo
2024-04-24 16:33:52 <@zodbot:fedora.im> Michael Armijo (marmijo)
2024-04-24 16:35:43 <@siosm:matrix.org> Lets' start!
2024-04-24 16:35:49 <@siosm:matrix.org> Let's start!
2024-04-24 16:35:53 <@siosm:matrix.org> !topic Action items from last meeting
2024-04-24 16:36:00 <@siosm:matrix.org> !link https://discussion.fedoraproject.org/t/fedora-coreos-community-meeting-minutes-2024-04-17/113434
2024-04-24 16:36:14 <@siosm:matrix.org> No action item from last meeting.
2024-04-24 16:36:39 <@siosm:matrix.org> :topic Remove i686 builds for all our packages
2024-04-24 16:36:44 <@siosm:matrix.org> !topic Remove i686 builds for all our packages
2024-04-24 16:36:55 <@siosm:matrix.org> !topic Remove i686 builds for all our packages
2024-04-24 16:37:07 <@siosm:matrix.org> !link https://github.com/coreos/fedora-coreos-tracker/issues/1716
2024-04-24 16:37:23 <@aaradhak:matrix.org> !hi aaradhak
2024-04-24 16:37:25 <@zodbot:fedora.im> Aashish Radhakrishnan (aaradhak)
2024-04-24 16:37:41 <@siosm:matrix.org> This is an small task if someone is interested in getting started with Fedora packaging.
2024-04-24 16:38:10 <@dustymabe:matrix.org> Thanks for opening that issue travier 
2024-04-24 16:38:31 <@siosm:matrix.org> We can mentor folks that want to learn how to do it
2024-04-24 16:38:49 <@ravanelli:matrix.org> !hi ravanelli
2024-04-24 16:38:58 <@aaradhak:matrix.org> I would like to look into it
2024-04-24 16:39:13 <@aaradhak:matrix.org> I am willing to look into it
2024-04-24 16:39:33 <@fifofonix:matrix.org> !hi
2024-04-24 16:40:16 <@siosm:matrix.org> Having https://github.com/coreos/fedora-coreos-tracker/issues/1631 would make it easier
2024-04-24 16:40:30 <@zodbot:fedora.im> Fifo Phonics (fifofonix)
2024-04-24 16:40:30 <@zodbot:fedora.im> Renata Ravanelli (ravanelli)
2024-04-24 16:40:33 <@siosm:matrix.org> Did we ever get to do it? I can file the issue after this meeting otherwise
2024-04-24 16:40:51 <@jlebon:fedora.im> I don't think so
2024-04-24 16:41:25 <@siosm:matrix.org> OK, I'll file the issue
2024-04-24 16:42:07 <@siosm:matrix.org> That's it for today in terms of tagged issues. Anyone wants to raise something else before we go to open floor?
2024-04-24 16:42:43 <@jlebon:fedora.im> should we discuss https://github.com/coreos/fedora-coreos-tracker/issues/1719 a bit?
2024-04-24 16:43:13 <@siosm:matrix.org> sure
2024-04-24 16:43:19 <@siosm:matrix.org> !topic Investigate UKI support in Fedora CoreOS
2024-04-24 16:43:25 <@siosm:matrix.org> !link https://github.com/coreos/fedora-coreos-tracker/issues/1719
2024-04-24 16:44:23 <@siosm:matrix.org> There is general push to get away from the split kernel / initramfs & kernel command line and move all thre into a single EFI binary that can be Secure Boot signed
2024-04-24 16:44:30 <@siosm:matrix.org> a Unified Kernel Image
2024-04-24 16:45:23 <@siosm:matrix.org> This is mostly interesting for security as it makes it significantly easier to implement a measured boot chain and then confidential computing (in the cloud or on capable hardware)
2024-04-24 16:46:15 <@siosm:matrix.org> This however clashes with some assumptions that we have today: the kernel command line being easily editable, the initramfs being easily re-generated, only the kernel being Secure Boot signed, etc.
2024-04-24 16:46:17 <@jlebon:fedora.im> IIUC a lot of things will not work in a UKI model, and so even on platforms where we already publish images, this would have to be a separate artifact, right? which is unfortunate
2024-04-24 16:46:36 <@siosm:matrix.org> not in all cases
2024-04-24 16:47:17 <@dustymabe:matrix.org> on platforms where we already publish images I think it would be fine (i.e. on AWS someone typically doesn't modify kargs)
2024-04-24 16:47:26 <@dustymabe:matrix.org> bare metal is the problem child here IIUC
2024-04-24 16:47:27 <@siosm:matrix.org> Probably at the cost of some disk space, we can extract the kernel initramfs & command line from a UKI to the system for backward compatibilitry
2024-04-24 16:47:33 <@spresti:fedora.im> !hi
2024-04-24 16:47:34 <@zodbot:fedora.im> Steven Presti (spresti)
2024-04-24 16:47:51 <@siosm:matrix.org> and get back the current behavior
2024-04-24 16:47:58 <@jlebon:fedora.im> travier: that could be interesting
2024-04-24 16:48:45 <@jlebon:fedora.im> dustymabe: kargs are relevant to the workloads too, not just the platform
2024-04-24 16:48:50 <@siosm:matrix.org> The biggest question is around signing, as we produce our own initramfs, so we would have to figure out a way to get them signed by Fedora
2024-04-24 16:49:38 <@siosm:matrix.org> There is always the option of "degrading" a UKI into a current setup so users that want to modify kargs can drop the UKI and use the current behavior (they won't get the benefits obviously)
2024-04-24 16:49:40 <@jlebon:fedora.im> i think we could get that going if the UKI is another 'image' in the list of artifacts
2024-04-24 16:49:54 <@jlebon:fedora.im> i think we could get that going if the UKI is another 'image' in the list of artifacts that we tell robosignatory to sign
2024-04-24 16:50:08 <@jlebon:fedora.im> but i guess that's another kind of signing
2024-04-24 16:50:26 <@siosm:matrix.org> Note that it's not the classic RPM / GPG signature here, it's the Secure Boot one, and I don't know how this works in Fedora
2024-04-24 16:50:34 <@siosm:matrix.org> it's only used for the kernel package right now
2024-04-24 16:50:40 <@jlebon:fedora.im> right, indeed
2024-04-24 16:51:11 <@siosm:matrix.org> Users can also enroll their own keys in some platforms, so they could sign their own UKIs and use that
2024-04-24 16:51:13 <@jlebon:fedora.im> even in traditional, there's tooling to go from UKI to non-UKI ?
2024-04-24 16:51:36 <@siosm:matrix.org> This is particularly useful to be able to change kernel arguments or make derived images for example
2024-04-24 16:51:57 <@siosm:matrix.org> I don't have a link at hand but I think there is
2024-04-24 16:52:23 <@siosm:matrix.org> The biggest item is thus: the ostree command line argument
2024-04-24 16:52:37 <@siosm:matrix.org> and this is where composefs "mostly" comes in
2024-04-24 16:52:58 <@siosm:matrix.org> https://github.com/coreos/fedora-coreos-tracker/issues/1718
2024-04-24 16:53:27 <@siosm:matrix.org> (but it's not strictly linked)
2024-04-24 16:53:45 <@jlebon:fedora.im> you probably meant a different link there :)
2024-04-24 16:54:15 <@siosm:matrix.org> whoops indeed: https://github.com/coreos/fedora-coreos-tracker/issues/1719
2024-04-24 16:54:27 <@siosm:matrix.org> the idea is to rely on a signature to select the ostree commit to boot instead of a hash
2024-04-24 16:54:48 <@siosm:matrix.org> ostree commit or composefs entry
2024-04-24 16:55:47 <@jbtrystram:matrix.org> travier: the UKI would contain the pubkey to validate the ostree commit signature ?
2024-04-24 16:55:49 <@siosm:matrix.org> we include the public key in the initramfs, and then sign the ostree/composefs hash with the private key and on boot you go through all ostree/composefs commits available and find the one matching the public key in the initramfs
2024-04-24 16:55:54 <@siosm:matrix.org> yes
2024-04-24 16:56:50 <@siosm:matrix.org> this however means that we can not share UKIs between boots anymore (like ostree does right now) but it's probably not that much of an issue (excepted that we have size issues in /boot right now)
2024-04-24 16:57:18 <@dustymabe:matrix.org> i feel like UKI is very much a lot of tradeoffs
2024-04-24 16:57:45 <@dustymabe:matrix.org> like sure, confidential computing is great, but I feel like most people won't bother or need it
2024-04-24 16:57:50 <@siosm:matrix.org> I had suggested other options (use a single key and include the hash of the deployment in the UKI name instead)
2024-04-24 16:58:08 <@jlebon:fedora.im> re. the general kargs problem, i wonder if there are proposals on ways to address that in upstream systemd
2024-04-24 16:58:29 <@siosm:matrix.org> It's need to get to a "Android" level of security for Linux in general
2024-04-24 16:58:36 <@siosm:matrix.org> It's needed to get to a "Android" level of security for Linux in general
2024-04-24 16:58:59 <@dustymabe:matrix.org> right, but android and linux are different
2024-04-24 16:59:05 <@dustymabe:matrix.org> android == appliance
2024-04-24 16:59:18 <@dustymabe:matrix.org> linux is many different things to many different people
2024-04-24 16:59:46 <@siosm:matrix.org> Yes, which is definitly it's harder for us as we have to keep the user freedom at the same time
2024-04-24 16:59:56 <@siosm:matrix.org> Yes, which is definitely why it's harder for us as we have to keep the user freedom at the same time
2024-04-24 17:00:25 <@siosm:matrix.org> https://github.com/systemd/systemd/issues/24539
2024-04-24 17:00:32 <@siosm:matrix.org> systemd issue around kernel arguments in UKIs
2024-04-24 17:02:09 <@siosm:matrix.org> So overall, this needs tooling, testing and integration in a variety of places in our build process.
2024-04-24 17:02:13 <@siosm:matrix.org> and time
2024-04-24 17:02:27 <@siosm:matrix.org> and time to work on it :/
2024-04-24 17:02:34 <@jlebon:fedora.im> we can't break existing flows, so this is about provide a new feature and letting users decide for themselves given the (mostly well-known) restrictions
2024-04-24 17:03:15 <@siosm:matrix.org> Agree
2024-04-24 17:03:31 <@jlebon:fedora.im> i think we could extend the osmet bits to cover the "metal-uki" image too
2024-04-24 17:03:36 <@siosm:matrix.org> It's probably easier to start with a new image first and investigate the integration after
2024-04-24 17:07:06 <@siosm:matrix.org> maybe we can talk about composefs a bit as well
2024-04-24 17:07:15 <@siosm:matrix.org> !topic Complete composefs integration in Fedora CoreOS
2024-04-24 17:07:23 <@siosm:matrix.org> !link https://github.com/coreos/fedora-coreos-tracker/issues/1718
2024-04-24 17:08:32 <@siosm:matrix.org> Composefs is a combination of filesystems (overlayfs, erofs) that creates an ostree like system without some of the downsides of ostree
2024-04-24 17:09:05 <@siosm:matrix.org> It also provides runtime integrity checks where ostree "only" provides offline integrity checks
2024-04-24 17:10:06 <@siosm:matrix.org> It also has potential performance improvements for deployments/updates as you don't need to create an entire filesystem tree for /usr anymore for each version as those are fully stored as EROFS images
2024-04-24 17:11:05 <@siosm:matrix.org> Combine with podman / container storage support, you can even gain disk space and memory saving if your containers use the same binaries as your system.
2024-04-24 17:11:20 <@siosm:matrix.org> So there are a lot of benefits to moving to composefs
2024-04-24 17:11:42 <@siosm:matrix.org> Combine with podman / container storage support, you can even gain disk space and memory <del>saving</del> de-duplication if your containers use the same binaries as your system.
2024-04-24 17:12:06 <@dustymabe:matrix.org> I think I need to understand this one more. I thought in OSTree it's just a hardlink farm basically
2024-04-24 17:12:34 <@siosm:matrix.org> yes, but you still need to create this hardlink farm for each deployment
2024-04-24 17:12:44 <@siosm:matrix.org> notably you can not hardlink symlinks and folders
2024-04-24 17:12:59 <@siosm:matrix.org> so ostree re-creates an entire filesystem tree for each deployment
2024-04-24 17:13:11 <@siosm:matrix.org> it's not that long, but still
2024-04-24 17:13:50 <@jlebon:fedora.im> i think the "online integrity check" + dedupe are the more interesting features
2024-04-24 17:13:55 <@siosm:matrix.org> with composefs, the filesystem hierarchy is stored in the EROFS filesystem, which is a read only FS
2024-04-24 17:14:30 <@jlebon:fedora.im> in the first phase AIUI, ostree will still create deployments anyway, but we would dynamically create the composefs image at boot
2024-04-24 17:15:02 <@jbtrystram:matrix.org> Which is populated with the hardlinks?
2024-04-24 17:15:15 <@siosm:matrix.org> yes
2024-04-24 17:15:22 <@siosm:matrix.org> yes, for the files
2024-04-24 17:18:13 <@jbtrystram:matrix.org> So once the ostree deployment is done, create a composefs image from it, if i understand you. What's the benefit? wouldn't we have two versions of the same
2024-04-24 17:18:44 <@siosm:matrix.org> the composefs EROFS file is very small as it's just metadata
2024-04-24 17:18:46 <@jbtrystram:matrix.org> So once the ostree deployment is done, create a composefs image from it, if i understand you. What's the benefit? wouldn't we have two versions of the  data
2024-04-24 17:18:59 <@siosm:matrix.org> it uses overlayfs to reference the files stored in the ostree repo
2024-04-24 17:19:43 <@jbtrystram:matrix.org> Gotcha. We add that result in podman storage and boom dedup?
2024-04-24 17:19:57 <@jbtrystram:matrix.org> (Yes i want to work on this)
2024-04-24 17:20:23 <@siosm:matrix.org> yes, if we share the "ostree"/composefs repo with the podman system container store then we can de-dup the files with the same hash
2024-04-24 17:20:58 <@siosm:matrix.org> alright, we're almost at time and I've talked a lot :)
2024-04-24 17:21:02 <@siosm:matrix.org> let's go to open floor?
2024-04-24 17:21:54 <@siosm:matrix.org> !topic Open Floor
2024-04-24 17:22:30 <@dustymabe:matrix.org> !info our `testing` stream has rebased to Fedora Linux 40 content! https://discussion.fedoraproject.org/t/fedora-coreos-rebasing-to-fedora-linux-40/114166
2024-04-24 17:22:35 <@dustymabe:matrix.org> !link https://discussion.fedoraproject.org/t/fedora-coreos-rebasing-to-fedora-linux-40/114166
2024-04-24 17:22:55 <@dustymabe:matrix.org> !info this will get promoted to `stable` in a few weeks time unless major issues are found
2024-04-24 17:28:59 <@jlebon:fedora.im> sounds like we can end this?
2024-04-24 17:29:08 <@siosm:matrix.org> yes
2024-04-24 17:29:14 <@siosm:matrix.org> !endmeeting