2025-08-05 15:00:14 <@nimbinatus:matrix.org> !startmeeting Fedora bootc Initiative
2025-08-05 15:00:16 <@meetbot:fedora.im> Meeting started at 2025-08-05 15:00:14 UTC
2025-08-05 15:00:17 <@meetbot:fedora.im> The Meeting name is 'Fedora bootc Initiative'
2025-08-05 15:00:21 <@nimbinatus:matrix.org> !topic roll call
2025-08-05 15:00:27 <@nimbinatus:matrix.org> !hi
2025-08-05 15:00:29 <@zodbot:fedora.im> Laura Santamaria (nimbinatus) - she / her / hers
2025-08-05 15:00:29 <@walters:fedora.im> !hi
2025-08-05 15:00:31 <@zodbot:fedora.im> Colin Walters (walters)
2025-08-05 15:00:32 <@rsturla:fedora.im> !hi
2025-08-05 15:00:33 <@zodbot:fedora.im> None (rsturla)
2025-08-05 15:00:34 <@jeckersb:fedora.im> !hi
2025-08-05 15:00:35 <@zodbot:fedora.im> John Eckersberg (jeckersb)
2025-08-05 15:01:06 <@snthrailkill:matrix.org> !hi
2025-08-05 15:01:08 <@zodbot:fedora.im> Sean Thrailkill (snthrailkill)
2025-08-05 15:01:13 <@dustymabe:matrix.org> !hi
2025-08-05 15:01:16 <@zodbot:fedora.im> Dusty Mabe (dustymabe) - he / him / his
2025-08-05 15:01:24 <@hricky:fedora.im> !hi
2025-08-05 15:01:27 <@zodbot:fedora.im> Hristo Marinov (hricky) - he / him / his
2025-08-05 15:02:27 <@walters:fedora.im> not sure if this is a full agenda item but definitely something to track and have wider awareness of is https://gitlab.com/fedora/bootc/base-images/-/merge_requests/254#note_2669931106
2025-08-05 15:02:29 <@nimbinatus:matrix.org> !topic coreos convergence (standing topic)
2025-08-05 15:02:37 <@nimbinatus:matrix.org> (just to get the ball rolling)
2025-08-05 15:02:47 <@nimbinatus:matrix.org> or not! sorry Colin Walters . I can switch over to that
2025-08-05 15:03:18 <@nimbinatus:matrix.org> !topic https://gitlab.com/fedora/bootc/base-images/-/merge_requests/254#note_2669931106
2025-08-05 15:03:25 <@nimbinatus:matrix.org> !link https://gitlab.com/fedora/bootc/base-images/-/merge_requests/254#note_2669931106
2025-08-05 15:05:02 <@walters:fedora.im> We may need to update all of our examples and also describe this issue promimently because it disproportionally impacts bootc containers I think; app containers (w/out systemd especially) tend to be less sensitive to this
2025-08-05 15:05:26 <@dustymabe:matrix.org> mind a quick recap of the problem that this change solved? 
2025-08-05 15:05:32 <@dustymabe:matrix.org> and what the behavior change is
2025-08-05 15:05:52 <@walters:fedora.im> umask unpredictability in container builds leads to unpredictable permissions with a bare `COPY`
2025-08-05 15:06:00 <@dustymabe:matrix.org> or is this a specific example of a class of problem?
2025-08-05 15:06:51 <@walters:fedora.im> Previously e.g. https://github.com/coreos/coreos-assembler/pull/1683
2025-08-05 15:07:44 <@dustymabe:matrix.org> What's old is new again! 
2025-08-05 15:09:21 <@walters:fedora.im> I don't have more on this though, just raising awareness
2025-08-05 15:09:47 <@dustymabe:matrix.org> anything we can do in a `lint` ?
2025-08-05 15:10:29 <@walters:fedora.im> yes, we could probably easily replicate the systemd unit permissions test (or perhaps better argue for `systemctl list-units --static-checks` or something/)
2025-08-05 15:10:49 <@walters:fedora.im> That would catch common cases of umask skew either way
2025-08-05 15:11:03 <@dustymabe:matrix.org> awesome. Thanks. I'm still learning a lot in this space
2025-08-05 15:12:32 <@walters:fedora.im> !topic coreos convergence (standing topic)
2025-08-05 15:12:42 <@nimbinatus:matrix.org> !link https://github.com/coreos/fedora-coreos-tracker/issues/1726
2025-08-05 15:12:48 <@nimbinatus:matrix.org> :)
2025-08-05 15:13:29 <@dustymabe:matrix.org> Jonathan Lebon: is back today, but I think he is in another meeting currently.
2025-08-05 15:13:30 <@walters:fedora.im> We'll get a bootc release out soon which should unblock the osbuild bits
2025-08-05 15:13:54 <@dustymabe:matrix.org> Colin Walters: yeah. There are two different fronts we are working on. 
2025-08-05 15:14:27 <@dustymabe:matrix.org> 1. build CoreOS via container build
2025-08-05 15:14:27 <@dustymabe:matrix.org> 2. use install to-filesystem during our boot images creation
2025-08-05 15:14:44 <@dustymabe:matrix.org> `2.` is what you're referring to with the bootc release? 
2025-08-05 15:14:59 <@walters:fedora.im> yes
2025-08-05 15:15:40 <@dustymabe:matrix.org> I know JB is going to be away for a few weeks, so `2.` might wait until he gets back, so no rush on the release 
2025-08-05 15:16:45 <@walters:fedora.im> Yeah overall it's not a blocker for higher level efforts but would be a helpful cleanup
2025-08-05 15:17:33 <@dustymabe:matrix.org> Right now you can `cosa import` an externally built container (i.e. via `podman build`)
2025-08-05 15:17:33 <@dustymabe:matrix.org> 
2025-08-05 15:17:33 <@dustymabe:matrix.org> Yep.
2025-08-05 15:17:33 <@dustymabe:matrix.org> 
2025-08-05 15:17:33 <@dustymabe:matrix.org> As far as `1.` goes we are still working on it through various efforts.
2025-08-05 15:18:02 <@dustymabe:matrix.org> and soon you'll be able to `cosa build --with-buildah` to run a container build versus a `rpm-ostree compose tree` 
2025-08-05 15:18:30 <@dustymabe:matrix.org> and that is only a temporary thing until we can rely on konflux fully 
2025-08-05 15:19:14 <@dustymabe:matrix.org> but `cosa build --with-buildah` will still be useful for local dev use cases even when we use konflux in the future
2025-08-05 15:19:20 <@walters:fedora.im> The subthread here I think would be helpful to track and not sure if we have it written up is basically the day that the default e2e workflow of people working on (and automated CI for) many things in github.com/coreos  starts with just `podman build` and not `cosa build` will be a notable milestone
2025-08-05 15:20:20 <@walters:fedora.im> Of course it gets into https://gitlab.com/fedora/bootc/tracker/-/issues/2 fast
2025-08-05 15:21:12 <@dustymabe:matrix.org> Colin Walters: yeah. basically once we get to the point where we want it everywhere `cosa build --with-buildah` would just turn into the default, so you don't need `--with-buildah` any more
2025-08-05 15:21:59 <@walters:fedora.im> but won't that still output a `.ociarchive` into builds/ ?
2025-08-05 15:22:33 <@dustymabe:matrix.org> yes. for now
2025-08-05 15:23:41 <@walters:fedora.im> I totally understand why it does that, but my point is that I think we want to reach a spot where succeeding at working on say...console-login-helper-messages doesn't need to involve anything cosa and I can just `podman build` and run it using the same bootc-oriented tools used elsewhere
2025-08-05 15:24:31 <@walters:fedora.im> (while retaining backcompat with the whole coreos builds/stream stuff for good reasons - where necessary, but I'm just saying it doesn't need to always be the default for local dev or CI even)
2025-08-05 15:24:32 <@dustymabe:matrix.org> oh of course. and you can do that today
2025-08-05 15:24:57 <@dustymabe:matrix.org> you can already `git checkout fedora-coreos-config && podman build <options>` 
2025-08-05 15:25:13 <@dustymabe:matrix.org> you can already `git clone fedora-coreos-config && podman build <options>` and use that with bootc tools, correct?
2025-08-05 15:26:27 <@walters:fedora.im> Yeah though I'm more referring to the other projects, not f-c-c
2025-08-05 15:27:02 <@dustymabe:matrix.org> ahh - like console-login-helper messages? you want the CI to not be run on top of CoreOS but the bootc base image? 
2025-08-05 15:28:09 <@walters:fedora.im> We may need both sometimes, in some circumstances and I guess it's tricky as c-l-h-m does have some ignition integration
2025-08-05 15:29:13 <@dustymabe:matrix.org> Overall though. I'd say we are marching towards convergence as fast as we can :) 
2025-08-05 15:29:21 <@dustymabe:matrix.org> lots of progress over the past few months
2025-08-05 15:29:50 <@walters:fedora.im> OK I don't have too much more on this one at the moment
2025-08-05 15:30:02 <@dustymabe:matrix.org> F43 should be a big release for us on this front
2025-08-05 15:30:11 <@nimbinatus:matrix.org> There was another topic in the channel, I'll grab it here in a sec
2025-08-05 15:30:46 <@nimbinatus:matrix.org> !topic skip updates if booted container is a superset
2025-08-05 15:30:49 <@nimbinatus:matrix.org> !link https://github.com/bootc-dev/bootc/issues/1465
2025-08-05 15:31:04 <@dustymabe:matrix.org> ahh yes.
2025-08-05 15:31:53 <@dustymabe:matrix.org> TL;DR the idea here is that we can mark certain container builds as "containing" or "providing" other container images so the update logic knows there is no update necessary at this time.
2025-08-05 15:32:13 <@dustymabe:matrix.org> There are few places I think this would be useful:
2025-08-05 15:32:36 <@dustymabe:matrix.org> 1. building boot media that differs from update media (so the freshly installed/booted system won't immediately "upgrade")
2025-08-05 15:33:03 <@dustymabe:matrix.org> 2. when we implement localy layering plugins, the system will know not to upgrade because it's already booted into a container that is a superset of the target in the registry
2025-08-05 15:33:44 <@dustymabe:matrix.org> This idea came from a potential future state for CoreOS where we don't ship Ignition on every update, but just in the boot media (since it's only needed there) 
2025-08-05 15:34:14 <@dustymabe:matrix.org> but of course we do use zincati if CoreOS right now, so we can just implement the logic there
2025-08-05 15:34:36 <@dustymabe:matrix.org> but.. I think this would be useful in bootc longer term and coreos wants to integrate more with the upgrade logic here 
2025-08-05 15:34:46 <@dustymabe:matrix.org> so mostly this right now is an idea and I'm interested to know what others thing
2025-08-05 15:34:48 <@dustymabe:matrix.org> so mostly this right now is an idea and I'm interested to know what others think
2025-08-05 15:35:01 <@walters:fedora.im> can you give a response to my last comment here?
2025-08-05 15:35:16 <@walters:fedora.im> (I am also curious what others think about this whole topic as it's pretty wide ranging)
2025-08-05 15:35:36 <@walters:fedora.im> and if you aren't following some part of this please do just ask  for clarification!
2025-08-05 15:36:46 <@dustymabe:matrix.org> 
2025-08-05 15:36:46 <@dustymabe:matrix.org> Colin Walters: I was going to try to discuss your comment more with some of the CoreOS folks before responding.
2025-08-05 15:36:46 <@dustymabe:matrix.org> 
2025-08-05 15:36:46 <@dustymabe:matrix.org> TBH I'm not quite sure I understand completely what you are asking.
2025-08-05 15:36:46 <@dustymabe:matrix.org> Basically this would add another way to install and we already support too many different ways already? 
2025-08-05 15:37:46 <@walters:fedora.im> mmm...kind of but I guess more specifically can we dig into how it'd compare with say anaconda+kexec in the cloud? There's already folks doing that and I'd like to productize it more
2025-08-05 15:38:19 <@walters:fedora.im> (or anaconda via `systemctl switch-root` if as would normally be expected the kernels are compatible enough)
2025-08-05 15:39:30 <@walters:fedora.im> In that flow one can do fully arbitrary things to provision: the "provisioning image" is whatever you've chosen to inject via cloud-init but we'd make it pretty first-class to use Anaconda here
2025-08-05 15:39:54 <@walters:fedora.im> One could also pull down ignition at that time
2025-08-05 15:40:36 <@walters:fedora.im> So I guess to flesh it out a bit more, bootc wouldn't need any awareness of this, the firstboot flow could just `podman pull quay.io/fedora/fedora-coreos-ignition` e.g.
2025-08-05 15:40:55 <@dustymabe:matrix.org> Colin Walters: do those options you mention assume someone is going to provision + reboot in the cloud? 
2025-08-05 15:42:21 <@dustymabe:matrix.org> I guess what I'm trying to enable is someone starts with a bootimage (AMI) we (coreos) created and they don't want to reboot at all, they just pull their containers and run - and then don't immediately reboot because the update system recognizes they are already booted into something that provides what they would "upgrade" too already
2025-08-05 15:42:26 <@rsturla:fedora.im> 
2025-08-05 15:42:26 <@rsturla:fedora.im> I may be wayyy afield here, but I believe there's another potential solution, which is similar to the sysexts - the firstboot container has a single generic layer overlayed (term may not be correct) set of files on top of the base image.  
2025-08-05 15:42:26 <@rsturla:fedora.im> The disk image is essentially the output of `bootc install --overlay ghcr.io/coreos/firstboot:42` (proposed [here](https://github.com/bootc-dev/bootc/issues/7#issuecomment-2591390476)).  Since bootc would be aware of both the base and the overlay, it might be easy to just ignore that on future boots.
2025-08-05 15:42:29 <@walters:fedora.im> Well...yes the full anaconda flow would be a reboot.
2025-08-05 15:42:54 <@rsturla:fedora.im> I may be wayyy afield here, but I believe there's another potential solution, which is similar to the sysexts - the firstboot container has a single generic layer overlayed (term may not be correct) with a set of files on top of the base image required for only first boot.
2025-08-05 15:42:54 <@rsturla:fedora.im> 
2025-08-05 15:42:54 <@rsturla:fedora.im> The disk image is essentially the output of `bootc install --overlay ghcr.io/coreos/firstboot:42` (proposed [here](https://github.com/bootc-dev/bootc/issues/7#issuecomment-2591390476)).  Since bootc would be aware of both the base and the overlay, it might be easy to just ignore that on future boots.
2025-08-05 15:43:08 <@walters:fedora.im> dustymabe: Yeah, though the thing is of course by default there *is* still a reboot if the instance stays running at a later time
2025-08-05 15:43:36 <@dustymabe:matrix.org> but that only happens when there is a new update to upgrade to? 
2025-08-05 15:44:12 <@walters:fedora.im> Yes, but that will always happen eventually...I mean until such time as all the software is formally proven correct and is done 😄
2025-08-05 15:44:30 <@rsturla:fedora.im> The firstboot disk image is essentially the output of `bootc install --overlay ghcr.io/coreos/firstboot:42` (proposed [here](https://github.com/bootc-dev/bootc/issues/7#issuecomment-2591390476)).  Since bootc would be aware of both the base and the overlay, it might be easy to just ignore that on future boots with a `bootc switch --migrate-in-place` (without the overlay layer).
2025-08-05 15:44:30 <@rsturla:fedora.im> 
2025-08-05 15:44:30 <@rsturla:fedora.im> I may be wayyy afield here, but I believe there's another potential solution, which is similar to the sysexts - the firstboot container has a single generic layer overlayed (term may not be correct) with a set of files on top of the base image required for only first boot.
2025-08-05 15:44:35 <@walters:fedora.im> I think that's really close to what Dusty is talking about, though some detail as to whether the layers appear as an "extension" or not
2025-08-05 15:45:12 <@dustymabe:matrix.org> Robert Sturla: Thanks for the link. I'll have to look at it. 
2025-08-05 15:45:32 <@walters:fedora.im> Hmm...I guess if we want to try unifying things more it would be interesting to pursue a thread where we have optional ignition in the stock cloud images too, but man it'd be messy
2025-08-05 15:46:06 <@dustymabe:matrix.org> Yeah. One detail here is that we want the "firstboot" layer to drop out on upgrade
2025-08-05 15:46:42 <@dustymabe:matrix.org> but Colin Walters did bring up an interesting point that having that will kinda render "factory reset" hard to do
2025-08-05 15:46:43 <@rsturla:fedora.im> A `--mutate-in-place` may do that if you point to a firstboot-less image?
2025-08-05 15:47:00 <@walters:fedora.im> It doesn't seem worth the complexity to me still
2025-08-05 15:47:11 <@rsturla:fedora.im> A `--mutate-in-place` may do that if you point to a firstboot-less image?
2025-08-05 15:47:11 <@rsturla:fedora.im> Edit: actually, unlikely.  Ignore :) 
2025-08-05 15:47:36 <@jlebon:fedora.im> !hi
2025-08-05 15:47:37 <@zodbot:fedora.im> None (jlebon)
2025-08-05 15:47:40 <@walters:fedora.im> Like really concretely what are the user-visible high priority problems it would solve? The only one I can think of is the `/boot` space issue
2025-08-05 15:48:09 <@dustymabe:matrix.org> Colin Walters: the /boot space issue is one (that would help CoreOS)
2025-08-05 15:48:25 <@dustymabe:matrix.org> but also the local layering plugin use case (when we eventually implement thta) 
2025-08-05 15:48:36 <@jlebon:fedora.im> i think it's indeed worth investigating the sysext/generic overlay path to address this instead. the fact that it's also needed in the initrd makes this trickier though
2025-08-05 15:48:45 <@walters:fedora.im> !link https://github.com/coreos/fedora-coreos-tracker/issues/1465
2025-08-05 15:49:00 <@dustymabe:matrix.org> and also - what if a user creates a oneoff "fix" build for a problem they are having, but they want to continue to follow the upgrade ref (because they know the next one will fix their problem). This could be an answer for that too
2025-08-05 15:49:11 <@walters:fedora.im> OK but is the /boot space *the only* thing?
2025-08-05 15:49:51 <@jlebon:fedora.im> i guess it addresses any case that has a provisioning phase and you _don't_ want to pay for an immediate reboot
2025-08-05 15:50:07 <@dustymabe:matrix.org> 2. local layering could use it??
2025-08-05 15:50:07 <@dustymabe:matrix.org> 1. /boot space
2025-08-05 15:50:07 <@dustymabe:matrix.org> 3. "hotfix" build potentially could use it
2025-08-05 15:50:33 <@dustymabe:matrix.org> these are just ideas. I'm open to others 
2025-08-05 15:50:33 <@dustymabe:matrix.org> also open to re-discuss in the future
2025-08-05 15:50:33 <@dustymabe:matrix.org> 
2025-08-05 15:50:45 <@walters:fedora.im> Hmm I think Robert is right here this has super high overlap with the sysext-style model
2025-08-05 15:50:54 <@dustymabe:matrix.org> sysexts would be interested to try to explore here, but also none of our image building tools really support that right now 
2025-08-05 15:51:14 <@dustymabe:matrix.org> sysexts would be interested to try to explore here, but also none of our boot image building tools really support that right now 
2025-08-05 15:51:26 <@walters:fedora.im> well here's a sub-thread: I think bootc would gain support for this and we should stop trying to wrap everything bootc gains with osbuild schema and kickstarts
2025-08-05 15:51:40 <@walters:fedora.im> i.e. the osbuild json should allow passing arbitrary arguments to bootc, so we don't need to touch it
2025-08-05 15:52:36 <@dustymabe:matrix.org> seems reasonable, but I haven't thought about it deeply
2025-08-05 15:52:43 <@jlebon:fedora.im> so the sysext path would be: include an ignition sysext in the disk image, then in the initrd enable the sysext?
2025-08-05 15:53:20 <@dustymabe:matrix.org> Jonathan Lebon: there are also parts of Ignition that run in the real root
2025-08-05 15:53:36 <@dustymabe:matrix.org> and also what if we wanted to include cloud-init as a firstboot only thing in our disk images?
2025-08-05 15:53:41 <@dustymabe:matrix.org> that runs in the real root too 
2025-08-05 15:53:51 <@dustymabe:matrix.org> so we'd need a sysext for the initrd and the real root I think 
2025-08-05 15:54:09 <@jlebon:fedora.im> yeah, though i don't think anything in the real root needs the binary itself.
2025-08-05 15:54:29 <@dustymabe:matrix.org> `ignition-rmcfg` I think does
2025-08-05 15:54:33 <@jlebon:fedora.im> cloud-init is a bit different because normally AIUI you expect it to run on every boot
2025-08-05 15:54:40 <@dustymabe:matrix.org> `ignition-rmcfg` I think does - symlink to ignition 
2025-08-05 15:54:55 <@jlebon:fedora.im> i mean, it could still be a sysext too of course
2025-08-05 15:55:36 <@dustymabe:matrix.org> yeah. a sysext of that would be useful, because then if users wanted it on every boot they could opt in to keep it
2025-08-05 15:56:37 <@dustymabe:matrix.org> lukewarm on "superset/contains/proides" idea -> investigate sysexts more? 
2025-08-05 15:56:37 <@dustymabe:matrix.org> 
2025-08-05 15:56:37 <@dustymabe:matrix.org> ok. any summaries from this discussion?
2025-08-05 15:57:07 <@dustymabe:matrix.org> 
2025-08-05 15:57:07 <@dustymabe:matrix.org> lukewarm on "superset/contains/provides" idea -> investigate sysexts more? 
2025-08-05 15:57:07 <@dustymabe:matrix.org> ok. any summaries from this discussion?
2025-08-05 15:58:14 <@jlebon:fedora.im> another benefit of the sysext approach (apart from sidestepping the whole superset thing) is that it's easier to reuse by other users who want a similar experience
2025-08-05 15:58:20 <@walters:fedora.im> we're close to time and I think while this thread can continue it's probably going to need to do so in the background; maybe we should do a quick open floor?
2025-08-05 15:58:34 <@siosm:matrix.org> !hi
2025-08-05 15:58:38 <@nimbinatus:matrix.org> we can!
2025-08-05 15:58:39 <@zodbot:fedora.im> Timothée Ravier (siosm) - he / him / his
2025-08-05 15:58:43 <@nimbinatus:matrix.org> !topic open floor
2025-08-05 15:59:03 <@siosm:matrix.org> the second benefit of Dusty's idea would be that you are not including Ignition/cloud-init in all updates
2025-08-05 15:59:10 <@siosm:matrix.org> (sorry I'm late)
2025-08-05 15:59:20 <@nimbinatus:matrix.org> all good!
2025-08-05 15:59:27 <@dustymabe:matrix.org> hmm I don't know. I think the superset idea is very easily usable by other users
2025-08-05 15:59:46 <@walters:fedora.im> Sure but where is the issue someone filed where they were complaining about the update size that would be fixed by dropping ignition?
2025-08-05 16:00:26 <@walters:fedora.im> I dunno anyways I'm not opposed to this in the end to be clear but yeah my takeaway is we should do local layers/sysext style things first
2025-08-05 16:00:30 <@nimbinatus:matrix.org> (note that I'll be booting all of us to the general #bootc:fedoraproject.org channel in a minute so we can get out of the way for any meetings coming after us)
2025-08-05 16:00:48 <@siosm:matrix.org> there isn't but we know that /boot space is a premium and eventually fills up, we have had issues in the past
2025-08-05 16:00:59 <@walters:fedora.im> Right, we covered the `/boot` space earlier.
2025-08-05 16:01:22 <@walters:fedora.im> OK we should probably call this meeting done 😄 Thanks all for coming!
2025-08-05 16:01:22 <@dustymabe:matrix.org> ok. see you all in the bootc channel
2025-08-05 16:01:31 <@nimbinatus:matrix.org> !endmeeting