#meeting-1:fedoraproject.org log

<@nimbinatus:matrix.org>

15:00:14

!startmeeting Fedora bootc Initiative

<@meetbot:fedora.im>

15:00:16

Meeting started at 2025-08-05 15:00:14 UTC

<@meetbot:fedora.im>

15:00:17

The Meeting name is 'Fedora bootc Initiative'

<@nimbinatus:matrix.org>

15:00:21

!topic roll call

<@nimbinatus:matrix.org>

15:00:27

!hi

<@zodbot:fedora.im>

15:00:29

Laura Santamaria (nimbinatus) - she / her / hers

<@walters:fedora.im>

15:00:29

!hi

<@zodbot:fedora.im>

15:00:31

Colin Walters (walters)

<@rsturla:fedora.im>

15:00:32

!hi

<@zodbot:fedora.im>

15:00:33

None (rsturla)

<@jeckersb:fedora.im>

15:00:34

!hi

<@zodbot:fedora.im>

15:00:35

John Eckersberg (jeckersb)

<@snthrailkill:matrix.org>

15:01:06

!hi

<@zodbot:fedora.im>

15:01:08

Sean Thrailkill (snthrailkill)

<@dustymabe:matrix.org>

15:01:13

!hi

<@zodbot:fedora.im>

15:01:16

Dusty Mabe (dustymabe) - he / him / his

<@hricky:fedora.im>

15:01:24

!hi

<@zodbot:fedora.im>

15:01:27

Hristo Marinov (hricky) - he / him / his

<@walters:fedora.im>

15:02:27

not sure if this is a full agenda item but definitely something to track and have wider awareness of is https://gitlab.com/fedora/bootc/base-images/-/merge_requests/254#note_2669931106

<@nimbinatus:matrix.org>

15:02:29

!topic coreos convergence (standing topic)

<@nimbinatus:matrix.org>

15:02:37

(just to get the ball rolling)

<@nimbinatus:matrix.org>

15:02:47

or not! sorry Colin Walters . I can switch over to that

<@nimbinatus:matrix.org>

15:03:18

!topic https://gitlab.com/fedora/bootc/base-images/-/merge_requests/254#note_2669931106

<@nimbinatus:matrix.org>

15:03:25

!link https://gitlab.com/fedora/bootc/base-images/-/merge_requests/254#note_2669931106

<@walters:fedora.im>

15:05:02

We may need to update all of our examples and also describe this issue promimently because it disproportionally impacts bootc containers I think; app containers (w/out systemd especially) tend to be less sensitive to this

<@dustymabe:matrix.org>

15:05:26

mind a quick recap of the problem that this change solved?

<@dustymabe:matrix.org>

15:05:32

and what the behavior change is

<@walters:fedora.im>

15:05:52

umask unpredictability in container builds leads to unpredictable permissions with a bare `COPY`

<@dustymabe:matrix.org>

15:06:00

or is this a specific example of a class of problem?

<@walters:fedora.im>

15:06:51

Previously e.g. https://github.com/coreos/coreos-assembler/pull/1683

<@dustymabe:matrix.org>

15:07:44

What's old is new again!

<@walters:fedora.im>

15:09:21

I don't have more on this though, just raising awareness

<@dustymabe:matrix.org>

15:09:47

anything we can do in a `lint` ?

<@walters:fedora.im>

15:10:29

yes, we could probably easily replicate the systemd unit permissions test (or perhaps better argue for `systemctl list-units --static-checks` or something/)

<@walters:fedora.im>

15:10:49

That would catch common cases of umask skew either way

<@dustymabe:matrix.org>

15:11:03

awesome. Thanks. I'm still learning a lot in this space

<@walters:fedora.im>

15:12:32

!topic coreos convergence (standing topic)

<@nimbinatus:matrix.org>

15:12:42

!link https://github.com/coreos/fedora-coreos-tracker/issues/1726

<@nimbinatus:matrix.org>

15:12:48

<@dustymabe:matrix.org>

15:13:29

Jonathan Lebon: is back today, but I think he is in another meeting currently.

<@walters:fedora.im>

15:13:30

We'll get a bootc release out soon which should unblock the osbuild bits

<@dustymabe:matrix.org>

15:13:54

Colin Walters: yeah. There are two different fronts we are working on.

<@dustymabe:matrix.org>

15:14:27

1. build CoreOS via container build

<@dustymabe:matrix.org>

15:14:27

2. use install to-filesystem during our boot images creation

<@dustymabe:matrix.org>

15:14:44

`2.` is what you're referring to with the bootc release?

<@walters:fedora.im>

15:14:59

yes

<@dustymabe:matrix.org>

15:15:40

I know JB is going to be away for a few weeks, so `2.` might wait until he gets back, so no rush on the release

<@walters:fedora.im>

15:16:45

Yeah overall it's not a blocker for higher level efforts but would be a helpful cleanup

<@dustymabe:matrix.org>

15:17:33

Right now you can `cosa import` an externally built container (i.e. via `podman build`)

<@dustymabe:matrix.org>

15:17:33

<@dustymabe:matrix.org>

15:17:33

Yep.

<@dustymabe:matrix.org>

15:17:33

<@dustymabe:matrix.org>

15:17:33

As far as `1.` goes we are still working on it through various efforts.

<@dustymabe:matrix.org>

15:18:02

and soon you'll be able to `cosa build --with-buildah` to run a container build versus a `rpm-ostree compose tree`

<@dustymabe:matrix.org>

15:18:30

and that is only a temporary thing until we can rely on konflux fully

<@dustymabe:matrix.org>

15:19:14

but `cosa build --with-buildah` will still be useful for local dev use cases even when we use konflux in the future

<@walters:fedora.im>

15:19:20

The subthread here I think would be helpful to track and not sure if we have it written up is basically the day that the default e2e workflow of people working on (and automated CI for) many things in github.com/coreos starts with just `podman build` and not `cosa build` will be a notable milestone

<@walters:fedora.im>

15:20:20

Of course it gets into https://gitlab.com/fedora/bootc/tracker/-/issues/2 fast

<@dustymabe:matrix.org>

15:21:12

Colin Walters: yeah. basically once we get to the point where we want it everywhere `cosa build --with-buildah` would just turn into the default, so you don't need `--with-buildah` any more

<@walters:fedora.im>

15:21:59

but won't that still output a `.ociarchive` into builds/ ?

<@dustymabe:matrix.org>

15:22:33

yes. for now

<@walters:fedora.im>

15:23:41

I totally understand why it does that, but my point is that I think we want to reach a spot where succeeding at working on say...console-login-helper-messages doesn't need to involve anything cosa and I can just `podman build` and run it using the same bootc-oriented tools used elsewhere

<@walters:fedora.im>

15:24:31

(while retaining backcompat with the whole coreos builds/stream stuff for good reasons - where necessary, but I'm just saying it doesn't need to always be the default for local dev or CI even)

<@dustymabe:matrix.org>

15:24:32

oh of course. and you can do that today

<@dustymabe:matrix.org>

15:24:57

you can already `git checkout fedora-coreos-config && podman build <options>`

<@dustymabe:matrix.org>

15:25:13

you can already `git clone fedora-coreos-config && podman build <options>` and use that with bootc tools, correct?

<@walters:fedora.im>

15:26:27

Yeah though I'm more referring to the other projects, not f-c-c

<@dustymabe:matrix.org>

15:27:02

ahh - like console-login-helper messages? you want the CI to not be run on top of CoreOS but the bootc base image?

<@walters:fedora.im>

15:28:09

We may need both sometimes, in some circumstances and I guess it's tricky as c-l-h-m does have some ignition integration

<@dustymabe:matrix.org>

15:29:13

Overall though. I'd say we are marching towards convergence as fast as we can :)

<@dustymabe:matrix.org>

15:29:21

lots of progress over the past few months

<@walters:fedora.im>

15:29:50

OK I don't have too much more on this one at the moment

<@dustymabe:matrix.org>

15:30:02

F43 should be a big release for us on this front

<@nimbinatus:matrix.org>

15:30:11

There was another topic in the channel, I'll grab it here in a sec

<@nimbinatus:matrix.org>

15:30:46

!topic skip updates if booted container is a superset

<@nimbinatus:matrix.org>

15:30:49

!link https://github.com/bootc-dev/bootc/issues/1465

<@dustymabe:matrix.org>

15:31:04

ahh yes.

<@dustymabe:matrix.org>

15:31:53

TL;DR the idea here is that we can mark certain container builds as "containing" or "providing" other container images so the update logic knows there is no update necessary at this time.

<@dustymabe:matrix.org>

15:32:13

There are few places I think this would be useful:

<@dustymabe:matrix.org>

15:32:36

1. building boot media that differs from update media (so the freshly installed/booted system won't immediately "upgrade")

<@dustymabe:matrix.org>

15:33:03

2. when we implement localy layering plugins, the system will know not to upgrade because it's already booted into a container that is a superset of the target in the registry

<@dustymabe:matrix.org>

15:33:44

This idea came from a potential future state for CoreOS where we don't ship Ignition on every update, but just in the boot media (since it's only needed there)

<@dustymabe:matrix.org>

15:34:14

but of course we do use zincati if CoreOS right now, so we can just implement the logic there

<@dustymabe:matrix.org>

15:34:36

but.. I think this would be useful in bootc longer term and coreos wants to integrate more with the upgrade logic here

<@dustymabe:matrix.org>

15:34:46

so mostly this right now is an idea and I'm interested to know what others thing

<@dustymabe:matrix.org>

15:34:48

so mostly this right now is an idea and I'm interested to know what others think

<@walters:fedora.im>

15:35:01

can you give a response to my last comment here?

<@walters:fedora.im>

15:35:16

(I am also curious what others think about this whole topic as it's pretty wide ranging)

<@walters:fedora.im>

15:35:36

and if you aren't following some part of this please do just ask for clarification!

<@dustymabe:matrix.org>

15:36:46

<@dustymabe:matrix.org>

15:36:46

Colin Walters: I was going to try to discuss your comment more with some of the CoreOS folks before responding.

<@dustymabe:matrix.org>

15:36:46

<@dustymabe:matrix.org>

15:36:46

TBH I'm not quite sure I understand completely what you are asking.

<@dustymabe:matrix.org>

15:36:46

Basically this would add another way to install and we already support too many different ways already?

<@walters:fedora.im>

15:37:46

mmm...kind of but I guess more specifically can we dig into how it'd compare with say anaconda+kexec in the cloud? There's already folks doing that and I'd like to productize it more

<@walters:fedora.im>

15:38:19

(or anaconda via `systemctl switch-root` if as would normally be expected the kernels are compatible enough)

<@walters:fedora.im>

15:39:30

In that flow one can do fully arbitrary things to provision: the "provisioning image" is whatever you've chosen to inject via cloud-init but we'd make it pretty first-class to use Anaconda here

<@walters:fedora.im>

15:39:54

One could also pull down ignition at that time

<@walters:fedora.im>

15:40:36

So I guess to flesh it out a bit more, bootc wouldn't need any awareness of this, the firstboot flow could just `podman pull quay.io/fedora/fedora-coreos-ignition` e.g.

<@dustymabe:matrix.org>

15:40:55

Colin Walters: do those options you mention assume someone is going to provision + reboot in the cloud?

<@dustymabe:matrix.org>

15:42:21

I guess what I'm trying to enable is someone starts with a bootimage (AMI) we (coreos) created and they don't want to reboot at all, they just pull their containers and run - and then don't immediately reboot because the update system recognizes they are already booted into something that provides what they would "upgrade" too already

<@rsturla:fedora.im>

15:42:26

<@rsturla:fedora.im>

15:42:26

I may be wayyy afield here, but I believe there's another potential solution, which is similar to the sysexts - the firstboot container has a single generic layer overlayed (term may not be correct) set of files on top of the base image.

<@rsturla:fedora.im>

15:42:26

The disk image is essentially the output of `bootc install --overlay ghcr.io/coreos/firstboot:42` (proposed [here](https://github.com/bootc-dev/bootc/issues/7#issuecomment-2591390476)). Since bootc would be aware of both the base and the overlay, it might be easy to just ignore that on future boots.

<@walters:fedora.im>

15:42:29

Well...yes the full anaconda flow would be a reboot.

<@rsturla:fedora.im>

15:42:54

<@rsturla:fedora.im>

15:42:54

<@rsturla:fedora.im>

15:42:54

<@walters:fedora.im>

15:43:08

dustymabe: Yeah, though the thing is of course by default there *is* still a reboot if the instance stays running at a later time

<@dustymabe:matrix.org>

15:43:36

but that only happens when there is a new update to upgrade to?

<@walters:fedora.im>

15:44:12

Yes, but that will always happen eventually...I mean until such time as all the software is formally proven correct and is done 😄

<@rsturla:fedora.im>

15:44:30

The firstboot disk image is essentially the output of `bootc install --overlay ghcr.io/coreos/firstboot:42` (proposed [here](https://github.com/bootc-dev/bootc/issues/7#issuecomment-2591390476)). Since bootc would be aware of both the base and the overlay, it might be easy to just ignore that on future boots with a `bootc switch --migrate-in-place` (without the overlay layer).

<@rsturla:fedora.im>

15:44:30

<@rsturla:fedora.im>

15:44:30

<@walters:fedora.im>

15:44:35

I think that's really close to what Dusty is talking about, though some detail as to whether the layers appear as an "extension" or not

<@dustymabe:matrix.org>

15:45:12

Robert Sturla: Thanks for the link. I'll have to look at it.

<@walters:fedora.im>

15:45:32

Hmm...I guess if we want to try unifying things more it would be interesting to pursue a thread where we have optional ignition in the stock cloud images too, but man it'd be messy

<@dustymabe:matrix.org>

15:46:06

Yeah. One detail here is that we want the "firstboot" layer to drop out on upgrade

<@dustymabe:matrix.org>

15:46:42

but Colin Walters did bring up an interesting point that having that will kinda render "factory reset" hard to do

<@rsturla:fedora.im>

15:46:43

A `--mutate-in-place` may do that if you point to a firstboot-less image?

<@walters:fedora.im>

15:47:00

It doesn't seem worth the complexity to me still

<@rsturla:fedora.im>

15:47:11

A `--mutate-in-place` may do that if you point to a firstboot-less image?

<@rsturla:fedora.im>

15:47:11

Edit: actually, unlikely. Ignore :)

<@jlebon:fedora.im>

15:47:36

!hi

<@zodbot:fedora.im>

15:47:37

None (jlebon)

<@walters:fedora.im>

15:47:40

Like really concretely what are the user-visible high priority problems it would solve? The only one I can think of is the `/boot` space issue

<@dustymabe:matrix.org>

15:48:09

Colin Walters: the /boot space issue is one (that would help CoreOS)

<@dustymabe:matrix.org>

15:48:25

but also the local layering plugin use case (when we eventually implement thta)

<@jlebon:fedora.im>

15:48:36

i think it's indeed worth investigating the sysext/generic overlay path to address this instead. the fact that it's also needed in the initrd makes this trickier though

<@walters:fedora.im>

15:48:45

!link https://github.com/coreos/fedora-coreos-tracker/issues/1465

<@dustymabe:matrix.org>

15:49:00

and also - what if a user creates a oneoff "fix" build for a problem they are having, but they want to continue to follow the upgrade ref (because they know the next one will fix their problem). This could be an answer for that too

<@walters:fedora.im>

15:49:11

OK but is the /boot space *the only* thing?

<@jlebon:fedora.im>

15:49:51

i guess it addresses any case that has a provisioning phase and you _don't_ want to pay for an immediate reboot

<@dustymabe:matrix.org>

15:50:07

2. local layering could use it??

<@dustymabe:matrix.org>

15:50:07

1. /boot space

<@dustymabe:matrix.org>

15:50:07

3. "hotfix" build potentially could use it

<@dustymabe:matrix.org>

15:50:33

these are just ideas. I'm open to others

<@dustymabe:matrix.org>

15:50:33

also open to re-discuss in the future

<@dustymabe:matrix.org>

15:50:33

<@walters:fedora.im>

15:50:45

Hmm I think Robert is right here this has super high overlap with the sysext-style model

<@dustymabe:matrix.org>

15:50:54

sysexts would be interested to try to explore here, but also none of our image building tools really support that right now

<@dustymabe:matrix.org>

15:51:14

sysexts would be interested to try to explore here, but also none of our boot image building tools really support that right now

<@walters:fedora.im>

15:51:26

well here's a sub-thread: I think bootc would gain support for this and we should stop trying to wrap everything bootc gains with osbuild schema and kickstarts

<@walters:fedora.im>

15:51:40

i.e. the osbuild json should allow passing arbitrary arguments to bootc, so we don't need to touch it

<@dustymabe:matrix.org>

15:52:36

seems reasonable, but I haven't thought about it deeply

<@jlebon:fedora.im>

15:52:43

so the sysext path would be: include an ignition sysext in the disk image, then in the initrd enable the sysext?

<@dustymabe:matrix.org>

15:53:20

Jonathan Lebon: there are also parts of Ignition that run in the real root

<@dustymabe:matrix.org>

15:53:36

and also what if we wanted to include cloud-init as a firstboot only thing in our disk images?

<@dustymabe:matrix.org>

15:53:41

that runs in the real root too

<@dustymabe:matrix.org>

15:53:51

so we'd need a sysext for the initrd and the real root I think

<@jlebon:fedora.im>

15:54:09

yeah, though i don't think anything in the real root needs the binary itself.

<@dustymabe:matrix.org>

15:54:29

`ignition-rmcfg` I think does

<@jlebon:fedora.im>

15:54:33

cloud-init is a bit different because normally AIUI you expect it to run on every boot

<@dustymabe:matrix.org>

15:54:40

`ignition-rmcfg` I think does - symlink to ignition

<@jlebon:fedora.im>

15:54:55

i mean, it could still be a sysext too of course

<@dustymabe:matrix.org>

15:55:36

yeah. a sysext of that would be useful, because then if users wanted it on every boot they could opt in to keep it

<@dustymabe:matrix.org>

15:56:37

lukewarm on "superset/contains/proides" idea -> investigate sysexts more?

<@dustymabe:matrix.org>

15:56:37

<@dustymabe:matrix.org>

15:56:37

ok. any summaries from this discussion?

<@dustymabe:matrix.org>

15:57:07

<@dustymabe:matrix.org>

15:57:07

lukewarm on "superset/contains/provides" idea -> investigate sysexts more?

<@dustymabe:matrix.org>

15:57:07

ok. any summaries from this discussion?

<@jlebon:fedora.im>

15:58:14

another benefit of the sysext approach (apart from sidestepping the whole superset thing) is that it's easier to reuse by other users who want a similar experience

<@walters:fedora.im>

15:58:20

we're close to time and I think while this thread can continue it's probably going to need to do so in the background; maybe we should do a quick open floor?

<@siosm:matrix.org>

15:58:34

!hi

<@nimbinatus:matrix.org>

15:58:38

we can!

<@zodbot:fedora.im>

15:58:39

Timothée Ravier (siosm) - he / him / his

<@nimbinatus:matrix.org>

15:58:43

!topic open floor

<@siosm:matrix.org>

15:59:03

the second benefit of Dusty's idea would be that you are not including Ignition/cloud-init in all updates

<@siosm:matrix.org>

15:59:10

(sorry I'm late)

<@nimbinatus:matrix.org>

15:59:20

all good!

<@dustymabe:matrix.org>

15:59:27

hmm I don't know. I think the superset idea is very easily usable by other users

<@walters:fedora.im>

15:59:46

Sure but where is the issue someone filed where they were complaining about the update size that would be fixed by dropping ignition?

<@walters:fedora.im>

16:00:26

I dunno anyways I'm not opposed to this in the end to be clear but yeah my takeaway is we should do local layers/sysext style things first

<@nimbinatus:matrix.org>

16:00:30

(note that I'll be booting all of us to the general #bootc:fedoraproject.org channel in a minute so we can get out of the way for any meetings coming after us)

<@siosm:matrix.org>

16:00:48

there isn't but we know that /boot space is a premium and eventually fills up, we have had issues in the past

<@walters:fedora.im>

16:00:59

Right, we covered the `/boot` space earlier.

<@walters:fedora.im>

16:01:22

OK we should probably call this meeting done 😄 Thanks all for coming!

<@dustymabe:matrix.org>

16:01:22

ok. see you all in the bootc channel

<@nimbinatus:matrix.org>

16:01:31

!endmeeting