<@siosm:matrix.org>
16:30:59
!startmeeting fedora_coreos_meeting
<@meetbot:fedora.im>
16:31:03
Meeting started at 2024-04-24 16:30:59 UTC
<@meetbot:fedora.im>
16:31:03
The Meeting name is 'fedora_coreos_meeting'
<@siosm:matrix.org>
16:31:19
!topic roll call
<@siosm:matrix.org>
16:31:21
!hi
<@zodbot:fedora.im>
16:31:23
Timothée Ravier (siosm) - he / him / his
<@dustymabe:matrix.org>
16:31:39
!hi
<@zodbot:fedora.im>
16:31:41
Dusty Mabe (dustymabe) - he / him / his
<@jbtrystram:matrix.org>
16:32:29
!hi
<@jbrooks:matrix.org>
16:32:36
!hi jasonbrooks
<@zodbot:fedora.im>
16:32:45
Jason Brooks (jasonbrooks) - he / him / his
<@zodbot:fedora.im>
16:32:45
Jean-Baptiste Trystram (jbtrystram) - he / him / his
<@jlebon:fedora.im>
16:33:44
!hi
<@zodbot:fedora.im>
16:33:45
None (jlebon)
<@marmijo:fedora.im>
16:33:50
!hi marmijo
<@zodbot:fedora.im>
16:33:52
Michael Armijo (marmijo)
<@siosm:matrix.org>
16:35:43
Lets' start!
<@siosm:matrix.org>
16:35:49
Let's start!
<@siosm:matrix.org>
16:35:53
!topic Action items from last meeting
<@siosm:matrix.org>
16:36:00
!link https://discussion.fedoraproject.org/t/fedora-coreos-community-meeting-minutes-2024-04-17/113434
<@siosm:matrix.org>
16:36:14
No action item from last meeting.
<@siosm:matrix.org>
16:36:39
:topic Remove i686 builds for all our packages
<@siosm:matrix.org>
16:36:44
!topic Remove i686 builds for all our packages
<@siosm:matrix.org>
16:36:55
!topic Remove i686 builds for all our packages
<@siosm:matrix.org>
16:37:07
!link https://github.com/coreos/fedora-coreos-tracker/issues/1716
<@aaradhak:matrix.org>
16:37:23
!hi aaradhak
<@zodbot:fedora.im>
16:37:25
Aashish Radhakrishnan (aaradhak)
<@siosm:matrix.org>
16:37:41
This is an small task if someone is interested in getting started with Fedora packaging.
<@dustymabe:matrix.org>
16:38:10
Thanks for opening that issue travier
<@siosm:matrix.org>
16:38:31
We can mentor folks that want to learn how to do it
<@ravanelli:matrix.org>
16:38:49
!hi ravanelli
<@aaradhak:matrix.org>
16:38:58
I would like to look into it
<@aaradhak:matrix.org>
16:39:13
I am willing to look into it
<@fifofonix:matrix.org>
16:39:33
!hi
<@siosm:matrix.org>
16:40:16
Having https://github.com/coreos/fedora-coreos-tracker/issues/1631 would make it easier
<@zodbot:fedora.im>
16:40:30
Fifo Phonics (fifofonix)
<@zodbot:fedora.im>
16:40:30
Renata Ravanelli (ravanelli)
<@siosm:matrix.org>
16:40:33
Did we ever get to do it? I can file the issue after this meeting otherwise
<@jlebon:fedora.im>
16:40:51
I don't think so
<@siosm:matrix.org>
16:41:25
OK, I'll file the issue
<@siosm:matrix.org>
16:42:07
That's it for today in terms of tagged issues. Anyone wants to raise something else before we go to open floor?
<@jlebon:fedora.im>
16:42:43
should we discuss https://github.com/coreos/fedora-coreos-tracker/issues/1719 a bit?
<@siosm:matrix.org>
16:43:13
sure
<@siosm:matrix.org>
16:43:19
!topic Investigate UKI support in Fedora CoreOS
<@siosm:matrix.org>
16:43:25
!link https://github.com/coreos/fedora-coreos-tracker/issues/1719
<@siosm:matrix.org>
16:44:23
There is general push to get away from the split kernel / initramfs & kernel command line and move all thre into a single EFI binary that can be Secure Boot signed
<@siosm:matrix.org>
16:44:30
a Unified Kernel Image
<@siosm:matrix.org>
16:45:23
This is mostly interesting for security as it makes it significantly easier to implement a measured boot chain and then confidential computing (in the cloud or on capable hardware)
<@siosm:matrix.org>
16:46:15
This however clashes with some assumptions that we have today: the kernel command line being easily editable, the initramfs being easily re-generated, only the kernel being Secure Boot signed, etc.
<@jlebon:fedora.im>
16:46:17
IIUC a lot of things will not work in a UKI model, and so even on platforms where we already publish images, this would have to be a separate artifact, right? which is unfortunate
<@siosm:matrix.org>
16:46:36
not in all cases
<@dustymabe:matrix.org>
16:47:17
on platforms where we already publish images I think it would be fine (i.e. on AWS someone typically doesn't modify kargs)
<@dustymabe:matrix.org>
16:47:26
bare metal is the problem child here IIUC
<@siosm:matrix.org>
16:47:27
Probably at the cost of some disk space, we can extract the kernel initramfs & command line from a UKI to the system for backward compatibilitry
<@spresti:fedora.im>
16:47:33
!hi
<@zodbot:fedora.im>
16:47:34
Steven Presti (spresti)
<@siosm:matrix.org>
16:47:51
and get back the current behavior
<@jlebon:fedora.im>
16:47:58
travier: that could be interesting
<@jlebon:fedora.im>
16:48:45
dustymabe: kargs are relevant to the workloads too, not just the platform
<@siosm:matrix.org>
16:48:50
The biggest question is around signing, as we produce our own initramfs, so we would have to figure out a way to get them signed by Fedora
<@siosm:matrix.org>
16:49:38
There is always the option of "degrading" a UKI into a current setup so users that want to modify kargs can drop the UKI and use the current behavior (they won't get the benefits obviously)
<@jlebon:fedora.im>
16:49:40
i think we could get that going if the UKI is another 'image' in the list of artifacts
<@jlebon:fedora.im>
16:49:54
i think we could get that going if the UKI is another 'image' in the list of artifacts that we tell robosignatory to sign
<@jlebon:fedora.im>
16:50:08
but i guess that's another kind of signing
<@siosm:matrix.org>
16:50:26
Note that it's not the classic RPM / GPG signature here, it's the Secure Boot one, and I don't know how this works in Fedora
<@siosm:matrix.org>
16:50:34
it's only used for the kernel package right now
<@jlebon:fedora.im>
16:50:40
right, indeed
<@siosm:matrix.org>
16:51:11
Users can also enroll their own keys in some platforms, so they could sign their own UKIs and use that
<@jlebon:fedora.im>
16:51:13
even in traditional, there's tooling to go from UKI to non-UKI ?
<@siosm:matrix.org>
16:51:36
This is particularly useful to be able to change kernel arguments or make derived images for example
<@siosm:matrix.org>
16:51:57
I don't have a link at hand but I think there is
<@siosm:matrix.org>
16:52:23
The biggest item is thus: the ostree command line argument
<@siosm:matrix.org>
16:52:37
and this is where composefs "mostly" comes in
<@siosm:matrix.org>
16:52:58
https://github.com/coreos/fedora-coreos-tracker/issues/1718
<@siosm:matrix.org>
16:53:27
(but it's not strictly linked)
<@jlebon:fedora.im>
16:53:45
you probably meant a different link there :)
<@siosm:matrix.org>
16:54:15
whoops indeed: https://github.com/coreos/fedora-coreos-tracker/issues/1719
<@siosm:matrix.org>
16:54:27
the idea is to rely on a signature to select the ostree commit to boot instead of a hash
<@siosm:matrix.org>
16:54:48
ostree commit or composefs entry
<@jbtrystram:matrix.org>
16:55:47
travier: the UKI would contain the pubkey to validate the ostree commit signature ?
<@siosm:matrix.org>
16:55:49
we include the public key in the initramfs, and then sign the ostree/composefs hash with the private key and on boot you go through all ostree/composefs commits available and find the one matching the public key in the initramfs
<@siosm:matrix.org>
16:55:54
yes
<@siosm:matrix.org>
16:56:50
this however means that we can not share UKIs between boots anymore (like ostree does right now) but it's probably not that much of an issue (excepted that we have size issues in /boot right now)
<@dustymabe:matrix.org>
16:57:18
i feel like UKI is very much a lot of tradeoffs
<@dustymabe:matrix.org>
16:57:45
like sure, confidential computing is great, but I feel like most people won't bother or need it
<@siosm:matrix.org>
16:57:50
I had suggested other options (use a single key and include the hash of the deployment in the UKI name instead)
<@jlebon:fedora.im>
16:58:08
re. the general kargs problem, i wonder if there are proposals on ways to address that in upstream systemd
<@siosm:matrix.org>
16:58:29
It's need to get to a "Android" level of security for Linux in general
<@siosm:matrix.org>
16:58:36
It's needed to get to a "Android" level of security for Linux in general
<@dustymabe:matrix.org>
16:58:59
right, but android and linux are different
<@dustymabe:matrix.org>
16:59:05
android == appliance
<@dustymabe:matrix.org>
16:59:18
linux is many different things to many different people
<@siosm:matrix.org>
16:59:46
Yes, which is definitly it's harder for us as we have to keep the user freedom at the same time
<@siosm:matrix.org>
16:59:56
Yes, which is definitely why it's harder for us as we have to keep the user freedom at the same time
<@siosm:matrix.org>
17:00:25
https://github.com/systemd/systemd/issues/24539
<@siosm:matrix.org>
17:00:32
systemd issue around kernel arguments in UKIs
<@siosm:matrix.org>
17:02:09
So overall, this needs tooling, testing and integration in a variety of places in our build process.
<@siosm:matrix.org>
17:02:13
and time
<@siosm:matrix.org>
17:02:27
and time to work on it :/
<@jlebon:fedora.im>
17:02:34
we can't break existing flows, so this is about provide a new feature and letting users decide for themselves given the (mostly well-known) restrictions
<@siosm:matrix.org>
17:03:15
Agree
<@jlebon:fedora.im>
17:03:31
i think we could extend the osmet bits to cover the "metal-uki" image too
<@siosm:matrix.org>
17:03:36
It's probably easier to start with a new image first and investigate the integration after
<@siosm:matrix.org>
17:07:06
maybe we can talk about composefs a bit as well
<@siosm:matrix.org>
17:07:15
!topic Complete composefs integration in Fedora CoreOS
<@siosm:matrix.org>
17:07:23
!link https://github.com/coreos/fedora-coreos-tracker/issues/1718
<@siosm:matrix.org>
17:08:32
Composefs is a combination of filesystems (overlayfs, erofs) that creates an ostree like system without some of the downsides of ostree
<@siosm:matrix.org>
17:09:05
It also provides runtime integrity checks where ostree "only" provides offline integrity checks
<@siosm:matrix.org>
17:10:06
It also has potential performance improvements for deployments/updates as you don't need to create an entire filesystem tree for /usr anymore for each version as those are fully stored as EROFS images
<@siosm:matrix.org>
17:11:05
Combine with podman / container storage support, you can even gain disk space and memory saving if your containers use the same binaries as your system.
<@siosm:matrix.org>
17:11:20
So there are a lot of benefits to moving to composefs
<@siosm:matrix.org>
17:11:42
Combine with podman / container storage support, you can even gain disk space and memory <del>saving</del> de-duplication if your containers use the same binaries as your system.
<@dustymabe:matrix.org>
17:12:06
I think I need to understand this one more. I thought in OSTree it's just a hardlink farm basically
<@siosm:matrix.org>
17:12:34
yes, but you still need to create this hardlink farm for each deployment
<@siosm:matrix.org>
17:12:44
notably you can not hardlink symlinks and folders
<@siosm:matrix.org>
17:12:59
so ostree re-creates an entire filesystem tree for each deployment
<@siosm:matrix.org>
17:13:11
it's not that long, but still
<@jlebon:fedora.im>
17:13:50
i think the "online integrity check" + dedupe are the more interesting features
<@siosm:matrix.org>
17:13:55
with composefs, the filesystem hierarchy is stored in the EROFS filesystem, which is a read only FS
<@jlebon:fedora.im>
17:14:30
in the first phase AIUI, ostree will still create deployments anyway, but we would dynamically create the composefs image at boot
<@jbtrystram:matrix.org>
17:15:02
Which is populated with the hardlinks?
<@siosm:matrix.org>
17:15:15
yes
<@siosm:matrix.org>
17:15:22
yes, for the files
<@jbtrystram:matrix.org>
17:18:13
So once the ostree deployment is done, create a composefs image from it, if i understand you. What's the benefit? wouldn't we have two versions of the same
<@siosm:matrix.org>
17:18:44
the composefs EROFS file is very small as it's just metadata
<@jbtrystram:matrix.org>
17:18:46
So once the ostree deployment is done, create a composefs image from it, if i understand you. What's the benefit? wouldn't we have two versions of the data
<@siosm:matrix.org>
17:18:59
it uses overlayfs to reference the files stored in the ostree repo
<@jbtrystram:matrix.org>
17:19:43
Gotcha. We add that result in podman storage and boom dedup?
<@jbtrystram:matrix.org>
17:19:57
(Yes i want to work on this)
<@siosm:matrix.org>
17:20:23
yes, if we share the "ostree"/composefs repo with the podman system container store then we can de-dup the files with the same hash
<@siosm:matrix.org>
17:20:58
alright, we're almost at time and I've talked a lot :)
<@siosm:matrix.org>
17:21:02
let's go to open floor?
<@siosm:matrix.org>
17:21:54
!topic Open Floor
<@dustymabe:matrix.org>
17:22:30
!info our `testing` stream has rebased to Fedora Linux 40 content! https://discussion.fedoraproject.org/t/fedora-coreos-rebasing-to-fedora-linux-40/114166
<@dustymabe:matrix.org>
17:22:35
!link https://discussion.fedoraproject.org/t/fedora-coreos-rebasing-to-fedora-linux-40/114166
<@dustymabe:matrix.org>
17:22:55
!info this will get promoted to `stable` in a few weeks time unless major issues are found
<@jlebon:fedora.im>
17:28:59
sounds like we can end this?
<@siosm:matrix.org>
17:29:08
yes
<@siosm:matrix.org>
17:29:14
!endmeeting