<@dustymabe:matrix.org>
16:30:01
!startmeeting fedora_coreos_meeting
<@meetbot:fedora.im>
16:30:03
Meeting started at 2025-01-22 16:30:01 UTC
<@meetbot:fedora.im>
16:30:03
The Meeting name is 'fedora_coreos_meeting'
<@dustymabe:matrix.org>
16:30:24
!topic roll call
<@dustymabe:matrix.org>
16:30:26
!hi
<@zodbot:fedora.im>
16:30:28
Dusty Mabe (dustymabe) - he / him / his
<@jlebon:fedora.im>
16:30:52
!hi
<@zodbot:fedora.im>
16:30:53
None (jlebon)
<@siosm:matrix.org>
16:31:01
!hi
<@hricky:fedora.im>
16:31:02
!hi
<@apiaseck:matrix.org>
16:31:03
!hi
<@zodbot:fedora.im>
16:31:03
Hristo Marinov (hricky) - he / him / his
<@zodbot:fedora.im>
16:31:14
Timothée Ravier (siosm) - he / him / his
<@zodbot:fedora.im>
16:31:15
Adam Piasecki (c4rt0) - he / him / his
<@aaradhak:matrix.org>
16:31:29
!hi aaradhak
<@zodbot:fedora.im>
16:31:31
Aashish Radhakrishnan (aaradhak)
<@marmijo:fedora.im>
16:31:43
!hi
<@zodbot:fedora.im>
16:31:45
Michael Armijo (marmijo)
<@ravanelli:matrix.org>
16:32:02
!hi ravanelli
<@zodbot:fedora.im>
16:32:04
Renata Ravanelli (ravanelli)
<@dustymabe:matrix.org>
16:32:42
welcome all!
<@jbtrystram:matrix.org>
16:34:03
!hi
<@zodbot:fedora.im>
16:34:05
Jean-Baptiste Trystram (jbtrystram) - he / him / his
<@dustymabe:matrix.org>
16:34:59
!topic Action items from last meeting
<@gurssing:matrix.org>
16:35:01
!hi gursewak
<@zodbot:fedora.im>
16:35:03
Gursewak Singh (gursewak)
<@dustymabe:matrix.org>
16:35:52
We had one action item from last meeting:
<@dustymabe:matrix.org>
16:35:52
<@dustymabe:matrix.org>
16:35:52
> travier to open zincati issue to track migrating to sysusers.d in RPM
<@dustymabe:matrix.org>
16:37:04
anybody know if travier was able to do that?
<@marmijo:fedora.im>
16:37:18
yes, https://github.com/coreos/fedora-coreos-tracker/issues/1864 was opened
<@dustymabe:matrix.org>
16:37:37
marmijo: +1 - thanks
<@dustymabe:matrix.org>
16:38:30
!info travier opened https://github.com/coreos/fedora-coreos-tracker/issues/1864 for the zincati updates for sysuers
<@dustymabe:matrix.org>
16:38:55
!topic Move from OSTree to OCI for updates
<@dustymabe:matrix.org>
16:39:03
!link https://github.com/coreos/fedora-coreos-tracker/issues/1823
<@dustymabe:matrix.org>
16:39:30
jbtrystram: I see you added the meeting label to this one last week.
<@dustymabe:matrix.org>
16:40:57
want to introduce it?
<@jbtrystram:matrix.org>
16:41:52
YEah, I thought it would be good to discuss that, since we posted the change proposal
<@jbtrystram:matrix.org>
16:42:05
!link https://fedoraproject.org/wiki/Changes/CoreOSOstree2OCIUpdates
<@jbtrystram:matrix.org>
16:43:02
We want to switch from the ostree repository to quay (OCI registry) to distribute FCOS updates. This is a first step towards rebasing to bootable containers images, allowing users to derive their own images
<@jbtrystram:matrix.org>
16:43:56
We also plan to build tools to be able to derive images while keeping our update guidance (barrier releases) , but that's is still in the design phase
<@jbtrystram:matrix.org>
16:45:16
we will probably make the switch to the bootimages for the F42 rebase and migrate exisiting nodes between F42 and F43. (nothing written in stone yet)
<@dustymabe:matrix.org>
16:45:44
Notably.. this first phase (outlined in the change request) doesn't have much to do with "bootable containers". It's just changing from OSTree to OCI for pulling the update content.
<@mnguyen:fedora.im>
16:47:10
!hi mnguyen
<@zodbot:fedora.im>
16:47:12
Michael Nguyen (mnguyen)
<@jbtrystram:matrix.org>
16:47:32
they're real containers though (you can derive them)
<@dustymabe:matrix.org>
16:48:15
Interesting.. Reading the change request I would have thought we would be switching all nodes over to pull from OCI for F42
<@dustymabe:matrix.org>
16:48:15
> we will probably make the switch to the bootimages for the F42 rebase and migrate exisiting nodes between F42 and F43. (nothing written in stone yet)
<@dustymabe:matrix.org>
16:48:15
<@dustymabe:matrix.org>
16:48:45
<@dustymabe:matrix.org>
16:48:45
> they're real containers though (you can derive them)
<@dustymabe:matrix.org>
16:48:45
right, but that has existed for some time outside of this change
<@siosm:matrix.org>
16:48:59
While the resulting containers will have bootc and will be deriveable, it won't be the default supported path for now
<@siosm:matrix.org>
16:49:29
Which will most likely confuse a lot of people so we should make that really clear in the notes & docs
<@jbtrystram:matrix.org>
16:50:33
I think we agreed on switching bootimages first to see if there is any fallout
<@dustymabe:matrix.org>
16:51:20
right, but that can all happen on the switch to F42 (or during the beta <-> final window)
<@jlebon:fedora.im>
16:51:31
i was thinking something like: "switch new nodes to use OCI together with f42 rebase, switch upgrading nodes to use OCI some releases after"
<@dustymabe:matrix.org>
16:51:35
right, but that can all happen on the switch to F42 (or during the `beta` <-> `final` window)
<@dustymabe:matrix.org>
16:51:55
Jonathan Lebon: OK, that's in line with what jbtrystram said then.
<@dustymabe:matrix.org>
16:52:02
can we update the change request to make that clear?
<@jbtrystram:matrix.org>
16:53:17
!action @jbtrystram to update the change to make clear that bootimages will migrate first, then existing notes after. Also mention clearly that deriving the images won't be supported for now
<@dustymabe:matrix.org>
16:53:40
<@dustymabe:matrix.org>
16:53:40
> deriving the images won't be supported for now
<@dustymabe:matrix.org>
16:53:40
we need to be careful here.
<@dustymabe:matrix.org>
16:54:30
"supported" is a loaded term. basically we just need to make clear that this "change" changes nothing with respect to "loss of update functionality" that happens when delpoying/using derived containers for FCOS
<@dustymabe:matrix.org>
16:54:47
"supported" is a loaded term. basically we just need to make clear that this "change" changes nothing with respect to "loss of update functionality" that happens when deploying/using derived containers for FCOS
<@dustymabe:matrix.org>
16:54:56
It's simply no more or less supported than it was before
<@siosm:matrix.org>
16:56:00
It's difficult for people that don't have history / past experience to grasp that
<@siosm:matrix.org>
16:56:12
past experience with FCOS
<@dustymabe:matrix.org>
16:56:20
!info FCOS will move to default to pulling updates from OCI registry in the F42 time frame.
<@ravanelli:matrix.org>
16:56:23
Will we keep both ostree repository and quay (OCI registry) for a while, until we finally switch, or will we switch it already?
<@siosm:matrix.org>
16:56:37
I think we should still make this crystal clear. We are not saying it's forbidden, but that it's not ready
<@dustymabe:matrix.org>
16:56:39
> It's difficult for people that don't have history / past experience to grasp that
<@dustymabe:matrix.org>
16:56:39
given that, do you have suggestions?
<@dustymabe:matrix.org>
16:56:39
<@siosm:matrix.org>
16:57:17
For the Atomic Desktops for example, I had a section about bootc in the F41 release notes
<@jlebon:fedora.im>
16:57:53
I'd say "supported but with known issues"
<@dustymabe:matrix.org>
16:58:20
Renata Ravanelli: we will keep both ostree repo and registry for a while - a possible change request for f43 could be to decomission ostree repo based updates ?
<@siosm:matrix.org>
16:58:36
I don't think we should say that anything is supported if it does not follow our default update model
<@siosm:matrix.org>
16:59:18
Again, it's not about saying that it's forbidden, just setting expectations on what works compared to what we have right now
<@dustymabe:matrix.org>
16:59:21
ehh. :) again - "supported" is an overloaded term, so it's subject to interpretation on when it is appropriate to use that term or not
<@dustymabe:matrix.org>
16:59:39
maybe we collaborate on the update to the change request and then review it next week in the meeting here?
<@jlebon:fedora.im>
16:59:57
right, it has known issues. we just need to surface that. maybe "experimental support" is the right label
<@jlebon:fedora.im>
17:00:18
but hmm, i don't think this needs to go in the change proposal?
<@siosm:matrix.org>
17:00:38
It's more for the release notes from my perspective
<@siosm:matrix.org>
17:00:41
and annoucenements
<@siosm:matrix.org>
17:00:57
but adding clarifications to the release proposal does not hurt
<@dustymabe:matrix.org>
17:01:04
I would put it in the change proposal, but let's discuss with jbtrystram when he makes the updates and then we'll determine the appropriate place and wording for it
<@dustymabe:matrix.org>
17:01:16
can we move on to the next issue?
<@jlebon:fedora.im>
17:01:17
SGTM
<@dustymabe:matrix.org>
17:01:40
!topic Build FCOS using podman build
<@dustymabe:matrix.org>
17:01:52
!link https://github.com/coreos/fedora-coreos-tracker/issues/1861
<@dustymabe:matrix.org>
17:02:24
Jonathan Lebon: how about I intro this one and you tell me where I go wrong?
<@jlebon:fedora.im>
17:03:01
dustymabe: i believe in you :)
<@dustymabe:matrix.org>
17:04:08
OK. So in bootc upstream there are [conversations happening](https://gitlab.com/fedora/bootc/tracker/-/issues/32) about how building base images (the images we derive the rest of the world from) happens. Walters opened this issue to start socializing it with the FCOS community (us).
<@dustymabe:matrix.org>
17:05:40
I think the proposal (for us) boils down to us changing our build process to build FCOS using `podman` completely. Thus enabling anyone to easily `podman build` the FCOS base image. Eventually we can rebase on top of bootc/base-image for Fedora, but I don't think that is necessarily what this issue is about.
<@dustymabe:matrix.org>
17:06:35
the upstream bootc #32 issue has a lot of other listed goals but I'm not sure they are completely relevant for us to discuss here, but we can. Some of them I am still a bit confused on and will bring up in the next bootc community meeting
<@dustymabe:matrix.org>
17:07:39
<@dustymabe:matrix.org>
17:07:39
The enablement for this is mostly multi-stage container builds where you suck in the build tools that do the magic in earlier stages and then the final output is a container.
<@dustymabe:matrix.org>
17:07:39
> enabling anyone to easily podman build the FCOS base image
<@dustymabe:matrix.org>
17:08:21
jlebon-ai-bot: enhance!
<@dustymabe:matrix.org>
17:08:29
jlebon-ai-bot: enhance/correct!
<@jlebon:fedora.im>
17:08:54
:) it's quite good TBH. but i can add more colour
<@dustymabe:matrix.org>
17:09:40
IIUC this eventual change wouldn't really be user facing, but mostly an implementation detail of our build pipeline - let me know if that is incorrect
<@dustymabe:matrix.org>
17:11:55
while Jonathan Lebon is typing up more - any questions/concerns?
<@jlebon:fedora.im>
17:12:55
2. do a derivation, which is what you usually do when you e.g. add packages on top of `FROM fedora` for example. This works, but sometimes you really do want it to "look like a base image" at the end because it results in more efficient images and can fix things up. And that's the "build-chunked-oci" magic that happens in those Containerfiles.
<@jlebon:fedora.im>
17:12:55
The "move to `podman build`" at this point is uncontroversial (at least in those upstream discussions). The conversation at this point is mostly around "what interface/mechanism should we expose to users for them to customize the base images". and there are two broad approaches to that:
<@jlebon:fedora.im>
17:12:55
1. do another base image build, where you're actually composing from scratch. then the image gets chunked as usual, and it just looks like any base image.
<@jlebon:fedora.im>
17:13:46
what we do with the tier-x manifest inheritance via git submodules is closer to 1, just that we're not doing it via `podman build`
<@jlebon:fedora.im>
17:14:06
what I'd like to move FCOS to is 2, where we derive for real, but still "rechunk" for efficiency
<@jlebon:fedora.im>
17:15:27
to be clear though, the UX _for FCOS users_ is that they would normally not have to rechunk, just derive as you naturally would
<@dustymabe:matrix.org>
17:16:09
so `rpm-ostree experimental compose build-chunked-oci` is more closer to `2.` ?
<@jbtrystram:matrix.org>
17:16:24
is "rechunk" a synonym to "squash" here ?
<@jbtrystram:matrix.org>
17:16:50
or does rechunking preserves the smartness around how packages are distributed accross layers to reduce updates
<@jlebon:fedora.im>
17:16:58
> so rpm-ostree experimental compose build-chunked-oci is more closer to 2. ?
<@jlebon:fedora.im>
17:16:58
yes, it takes a the final derived rootfs and chunks it back up into layers
<@jlebon:fedora.im>
17:17:06
<@jlebon:fedora.im>
17:17:06
yes, it takes a the final derived rootfs and chunks it back up into layers
<@jlebon:fedora.im>
17:17:06
> so rpm-ostree experimental compose build-chunked-oci is more closer to 2. ?
<@jlebon:fedora.im>
17:17:17
this
<@jlebon:fedora.im>
17:17:17
> or does rechunking preserves the smartness around how packages are distributed accross layers to reduce updates
<@jlebon:fedora.im>
17:17:17
<@jbtrystram:matrix.org>
17:17:34
thanks
<@jlebon:fedora.im>
17:17:49
but you're right that it's like squashing in that all previous concepts of layers from regular `RUN dnf install ...` commands are lost
<@dustymabe:matrix.org>
17:19:01
but `build-chunked-oci` could and should be used if we were to go down the `1.` path too right?
<@dustymabe:matrix.org>
17:19:01
<@dustymabe:matrix.org>
17:19:01
basically chunking (or rechunking) would need to happen whether we were deriving or not?
<@dustymabe:matrix.org>
17:19:17
but `build-chunked-oci` could and should be used if we were to go down the `1.` path too right?
<@dustymabe:matrix.org>
17:19:17
basically chunking (or rechunking) would need to happen whether we were deriving or not in this new world where we use `podman build` for everything?
<@dustymabe:matrix.org>
17:19:17
<@jlebon:fedora.im>
17:20:32
yeah, i think we really want FCOS images we publish to be "from-scratch chunked" for optimal results
<@dustymabe:matrix.org>
17:20:51
Most of my understanding of this is derived from https://github.com/coreos/rpm-ostree/blob/main/docs/experimental-build-chunked-oci.md#example-as-part-of-a-containerfile
<@jlebon:fedora.im>
17:21:06
zstd:chunked will help a lot, but even then, chunking just makes zstd:chunked less wasteful because it doesn't have to inspect as many layers
<@jlebon:fedora.im>
17:21:30
zstd:chunked will help a lot, but even then, smart layering just makes zstd:chunked less wasteful because it doesn't have to inspect as many layers
<@dustymabe:matrix.org>
17:22:24
OK. are there any takeaways from this that we want to write down from the meeting? proposed/agreed ?
<@jbtrystram:matrix.org>
17:22:45
why do you qualify `zstd:chunked` as wasteful ?
<@jlebon:fedora.im>
17:22:59
the main argument for 2 in my mind is the much clearer CI story. you're enhancing a package set that has passed upstream CI.
<@jlebon:fedora.im>
17:22:59
whereas if you're recomposing a base image from scratch, you lose that. unless you introduce e.g. lockfiles. but that's not something I think we want to push to our users that'd want to also rechunk
<@jlebon:fedora.im>
17:24:33
jbtrystram: it's not, it just requires more processing client-side. my point was that maximizing layer sharing is still helpful there because it causes less processing
<@jlebon:fedora.im>
17:25:50
dustymabe: not sure. i guess agreement that this general approach is worth pursuing?
<@jbtrystram:matrix.org>
17:26:06
if both images end up properly rechunked, 2. is the most logical, except if we can pin packages (i.e something passed upstream bootc CI but not ours so we want to pin)
<@jbtrystram:matrix.org>
17:26:21
But i guess in that case we could pin on the previous bootc image
<@dustymabe:matrix.org>
17:26:47
!proposed FCOS is generally on board with changing FCOS builds to use `podman build` with multi-stage builds in the future.
<@jlebon:fedora.im>
17:27:37
+1
<@jlebon:fedora.im>
17:28:10
jbtrystram: rolling back packages has to be something that 100% works because that's an important use cases for users of layering as well
<@jlebon:fedora.im>
17:28:35
jbtrystram: rolling back packages has to be something that 100% works because that's an important use case for users of layering as well
<@dustymabe:matrix.org>
17:29:15
any other votes?
<@jlebon:fedora.im>
17:29:22
btw, if anyone isinterested in this stuff, you can join the bootc community meetings: https://docs.fedoraproject.org/en-US/bootc/community/#_meetings_and_calendar
<@jbtrystram:matrix.org>
17:29:22
+1
<@aaradhak:matrix.org>
17:29:26
+1
<@jlebon:fedora.im>
17:29:27
btw, if anyone is interested in this stuff, you can join the bootc community meetings: https://docs.fedoraproject.org/en-US/bootc/community/#\_meetings\_and\_calendar
<@apiaseck:matrix.org>
17:29:28
+1
<@marmijo:fedora.im>
17:30:04
+1
<@dustymabe:matrix.org>
17:30:05
!agreed FCOS is generally on board with changing FCOS builds to use `podman build` with multi-stage builds in the future.
<@dustymabe:matrix.org>
17:30:46
like I said.. there are some other parts of https://gitlab.com/fedora/bootc/tracker/-/issues/32 that are a bit more murky to me, but will try to hash those out in the bootc meeting
<@dustymabe:matrix.org>
17:30:50
!topic open floor
<@dustymabe:matrix.org>
17:30:58
anyone with anything for open floor today?
<@marmijo:fedora.im>
17:31:44
There were no new Fedora 42 change considerations for this week.
<@dustymabe:matrix.org>
17:32:03
!info we need more volunteers to participate in our [rotation](https://hackmd.io/@4rqq1dsYSVuBswOHTKSIBA/ByDq2EK5p) to run our weekly meetings (this meeting)
<@aaradhak:matrix.org>
17:32:49
I shall add my name for the upcoming meetings
<@jbtrystram:matrix.org>
17:33:15
I'll join the rotation back around the end of February :)
<@mnguyen:fedora.im>
17:34:15
I have school pickup that conflicts. When daylight savings comes around I may be able volunteer
<@dustymabe:matrix.org>
17:34:49
any other open floor topics? (sorry we ran over time today!)
<@dustymabe:matrix.org>
17:35:05
!endmeeting