<@jlebon:fedora.im>
16:31:46
!startmeeting fedora_coreos_meeting
<@meetbot:fedora.im>
16:31:49
Meeting started at 2024-05-22 16:31:46 UTC
<@meetbot:fedora.im>
16:31:49
The Meeting name is 'fedora_coreos_meeting'
<@siosm:matrix.org>
16:31:52
!hi
<@jlebon:fedora.im>
16:31:54
!topic roll call
<@zodbot:fedora.im>
16:31:56
Timothée Ravier (siosm) - he / him / his
<@marmijo:fedora.im>
16:32:02
!hi
<@zodbot:fedora.im>
16:32:06
Michael Armijo (marmijo)
<@hricky:fedora.im>
16:32:22
!hi
<@zodbot:fedora.im>
16:32:24
Hristo Marinov (hricky) - he / him / his
<@ravanelli:matrix.org>
16:32:27
!hi ravanelli
<@zodbot:fedora.im>
16:32:31
Renata Ravanelli (ravanelli)
<@dustymabe:matrix.org>
16:32:37
!hi
<@zodbot:fedora.im>
16:32:40
Dusty Mabe (dustymabe) - he / him / his
<@jbrooks:matrix.org>
16:33:25
!hi jasonbrooks
<@zodbot:fedora.im>
16:33:29
Jason Brooks (jasonbrooks) - he / him / his
<@jbtrystram:matrix.org>
16:33:39
!hi
<@zodbot:fedora.im>
16:33:43
Jean-Baptiste Trystram (jbtrystram) - he / him / his
<@jlebon:fedora.im>
16:34:04
let's wait another minute
<@jlebon:fedora.im>
16:35:54
ok, let's get started!
<@jlebon:fedora.im>
16:35:55
!topic Action items from last meeting
<@jlebon:fedora.im>
16:36:25
ACTION: All to think about the python discussion
ACTION: ravanelli to contact the libteam maintainers to figure out if they intend to drop the package from fedora
<@aaradhak:matrix.org>
16:36:25
!hi aaradhak
<@blackwell:fedora.im>
16:36:56
!hi
<@jlebon:fedora.im>
16:37:02
i think a few of us have done the former. Renata Ravanelli did you do the second one yet?
<@zodbot:fedora.im>
16:37:16
Aashish Radhakrishnan (aaradhak)
<@zodbot:fedora.im>
16:37:17
Jason Blackwell (blackwell)
<@dogphilosopher:fedora.im>
16:37:41
!hi
<@zodbot:fedora.im>
16:37:45
Noel Miller (dogphilosopher)
<@jlebon:fedora.im>
16:37:57
dogphilosopher: 👋
<@ravanelli:matrix.org>
16:38:55
Jonathan Lebon: sorry, not yet. I will
<@jlebon:fedora.im>
16:39:15
Renata Ravanelli: ack, sounds good! i'll re-action it then
<@jlebon:fedora.im>
16:39:22
!action ravanelli to contact the libteam maintainers to figure out if they intend to drop the package from fedora
<@jlebon:fedora.im>
16:39:31
ok, let's move to topics
<@jlebon:fedora.im>
16:39:40
!topic Consider specializing mount units on first boot, after partitioning
<@jlebon:fedora.im>
16:39:50
!link https://github.com/coreos/fedora-coreos-tracker/issues/1732
<@jlebon:fedora.im>
16:40:00
travier: want to introduce this one?
<@siosm:matrix.org>
16:40:21
Shouldn't we do the Python discussion first?
<@siosm:matrix.org>
16:40:31
But I can speak about this one
<@siosm:matrix.org>
16:40:43
But I can also speak about this one
<@siosm:matrix.org>
16:42:03
The idea is that it's non-intuitive for users to use stable names for disks for partionning using Ignition and for a good reason: as soon as you do it, your config is not generic anymore
<@siosm:matrix.org>
16:42:35
moreover, you usually don't know that you need it until you ran into issues and it's usually too late then
<@jlebon:fedora.im>
16:43:03
i'm fine with swapping if you'd like. that's the order it was in in the checklist :)
<@siosm:matrix.org>
16:43:26
So instead of forcing everyone to have node specific configs, I think we should have something, maybe in Ignition, "specialize" the mount units to reference stable names
<@jlebon:fedora.im>
16:44:09
so i think there are two separate issues at play here
<@siosm:matrix.org>
16:44:47
we could run that logic on existing nodes as well as that would make the setup more robust to future name changes
<@jlebon:fedora.im>
16:45:06
the first is that it's hard to reference block devices in a reliable way in the storage.disks and storage.filesystems sections.
the second is that your mount unit can be written in a way that breaks between boots/in an upgrade.
<@jlebon:fedora.im>
16:46:11
the second issue I would argue is actually mostly a documentation problem
<@siosm:matrix.org>
16:46:38
to me it's all interconnected
<@jlebon:fedora.im>
16:46:48
not just of course. e.g. i think with_mount_units could be made smarter
<@siosm:matrix.org>
16:47:00
you are not going to write the mount units manually; they are written by ignition
<@jlebon:fedora.im>
16:47:40
right. so i think one improvement we could make here is to add a lint for this in butane
<@dustymabe:matrix.org>
16:48:42
today I think with_mount_units is butane sugar, right?
<@jlebon:fedora.im>
16:48:54
first, if you use `with_mount_unit: true`, the generated unit should use by-label/ instead, and warn if a label is not provided.
second, if you define a mount unit manually, we could check the What
<@jlebon:fedora.im>
16:49:03
the second could live in Ignition actually
<@siosm:matrix.org>
16:49:03
I think so, which is why this would have to live elsewhere
<@siosm:matrix.org>
16:50:07
The label helps, but it does not resolves things for existing nodes
<@siosm:matrix.org>
16:50:23
I'm +1 for adding docs about using labels
<@dustymabe:matrix.org>
16:50:24
what is the problem for existing nodes?
<@siosm:matrix.org>
16:50:55
Say you do a major unamed OS update from 8 to 9 and the disks are not discovered in the same order anymore
<@dustymabe:matrix.org>
16:51:57
and this is because people used unstable names in their units?
<@jlebon:fedora.im>
16:52:05
note this isn't just a CoreOS problem though. we should lean on leapp there where I could see this "specialization" live
<@dustymabe:matrix.org>
16:52:22
i.e. `/dev/sda6` versus `/dev/disk/by-label/my-data` ?
<@siosm:matrix.org>
16:52:33
unfortunately the names are stable until they aren't
<@jlebon:fedora.im>
16:52:37
dustymabe: yeah, exactly
<@siosm:matrix.org>
16:52:37
yes
<@siosm:matrix.org>
16:53:04
If you use `/dev/disk/by-label/my-data` then you immediately have a per-node config
<@siosm:matrix.org>
16:53:18
for the partitionning part
<@jlebon:fedora.im>
16:53:27
travier: how so? labels are generic. e.g. 'etcd-data'
<@jlebon:fedora.im>
16:53:36
the device backing them may not be
<@dustymabe:matrix.org>
16:53:36
hmm. part of me feels like we should fix it - hmm or rather, maybe we should do CLHM and tell the user how to fix and maybe even give them a migration script
<@dustymabe:matrix.org>
16:54:13
i'm not sure how I feel about automagically changing peoples mount units - when there might be some complicated units (i.e. we can't test everything and I don't think we should try)
<@siosm:matrix.org>
16:54:46
agree this is risky
<@dustymabe:matrix.org>
16:55:03
note I feel like this warning for the upcoming wifi removal has worked out well:
```
[dustymabe@media ~]$ ssh core@192.168.100.1
Fedora CoreOS 40.20240504.2.0
##########################################################################
WARNING: NetworkManager-wifi is a requested layered package on this
system, but no Wi-Fi drivers are requested. The Wi-Fi drivers will no
longer be included by default in the future.
More context and remediation steps are available in the following FAQ entry:
https://docs.fedoraproject.org/en-US/fedora-coreos/faq/#wifi-fw
To disable this warning, use:
sudo systemctl disable coreos-check-wireless-firmwares.service
##########################################################################
Tracker: https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/tag/coreos
```
<@siosm:matrix.org>
16:55:32
but the idea would be to provide an option for those that want to "specialize" existing installations and automatically specialize new installations
<@dustymabe:matrix.org>
16:55:37
so basically we can just let people know when they have units that might be unstable and then they can either take action or disable the warning
<@dustymabe:matrix.org>
16:55:54
right, we can give them instructions for how to improve their situation
<@siosm:matrix.org>
16:57:02
I also agree that working with leapp folks would be good, but this is RHEL only afaik
<@jlebon:fedora.im>
16:57:05
MOTD seems like it'd be a good fit here indeed
<@apiaseck:matrix.org>
16:57:11
Not sure if I understand this correctly but could we not mount the volumes in etc/fstab based on the uuid's?
<@siosm:matrix.org>
16:57:50
apiaseck: yes, that's the idea, but we need to make it easier for users to set this up
<@dustymabe:matrix.org>
16:57:57
travier: Jonathan Lebon, but I agree for newly provisioned systems it would be good to create mount units with more correct names if we can too
<@siosm:matrix.org>
16:59:26
(I don't think we should solve this / take a decision in this meeting but wanted to start the discussion)
<@jlebon:fedora.im>
17:00:02
what do folks think about an MOTD for this at least?
<@jlebon:fedora.im>
17:00:48
i can file an issue against butane also to see how we can improve things there
<@siosm:matrix.org>
17:01:17
I'm OK with that. We would also need actionable docs
<@jlebon:fedora.im>
17:01:57
yeah, we basically should audit all our docs for this IMO
<@jlebon:fedora.im>
17:02:28
and e.g. a section or at least a NOTE box on the storage page
<@jlebon:fedora.im>
17:03:02
travier: want to file a docs issue for this?
<@siosm:matrix.org>
17:03:16
Sure, will do
<@jlebon:fedora.im>
17:03:24
!action jlebon to file a butane issue to discuss improvements around device names
<@jlebon:fedora.im>
17:03:48
!action travier to file a docs issue to audit storage docs and add some details re. device naming
<@jlebon:fedora.im>
17:04:56
i've thought about something similar to your idea about selectors before, but using udev rules to create more symlinks
<@jlebon:fedora.im>
17:05:14
there might be something there worth exploring too
<@siosm:matrix.org>
17:05:30
👍
<@dustymabe:matrix.org>
17:05:40
i'm not sure how fancy we should get :)
<@jlebon:fedora.im>
17:05:48
ok, move on to the next topic?
<@jlebon:fedora.im>
17:06:15
!topic revisit python discussion
<@jlebon:fedora.im>
17:06:21
!link https://github.com/coreos/fedora-coreos-tracker/issues/1730
<@walters:fedora.im>
17:06:59
(tangentially related to this one) was the bootc one on the agenda for today?
<@jlebon:fedora.im>
17:07:27
Colin Walters: it's next in the list :)
<@jlebon:fedora.im>
17:07:33
(https://github.com/coreos/fcos-meeting-action/issues/88#issue-2310854304)
<@jlebon:fedora.im>
17:07:53
do folks want to discuss that one first instead?
<@siosm:matrix.org>
17:09:42
If folks are there for the bootc one we could move to it?
<@jlebon:fedora.im>
17:10:06
WFM
<@jlebon:fedora.im>
17:10:17
!topic Roadmap to Fedora Bootable Containers
<@jlebon:fedora.im>
17:10:21
!link https://github.com/coreos/fedora-coreos-tracker/issues/1726
<@walters:fedora.im>
17:10:48
I'm happy to talk about this one if it's ok, though travier wrote the issue
<@siosm:matrix.org>
17:11:01
go ahead!
<@walters:fedora.im>
17:11:38
OK so I think as most but not all know, I used to be on this team working on coreos and have for a while now been working on doing something different, but related that landed in RHEL in https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/using_image_mode_for_rhel_to_build_deploy_and_manage_operating_systems/index
<@walters:fedora.im>
17:12:23
There's a lot to say here but I think all the prior discussion around device naming etc. is a good example of a mature project discussion. However I also think CoreOS is well-defined and well maintained it is also a kind of "technological peak"
<@walters:fedora.im>
17:12:44
All the opinions the project has also constrain where it can go and what it can do
<@walters:fedora.im>
17:13:20
The fedora bootc project is all about basically saying "anywhere you can dnf, you can bootc" and I hope it feels literally "more fedora/rhel" in that aspect
<@walters:fedora.im>
17:13:39
And it's all about going all in on containers as the technical heart of things, the center of gravity
<@walters:fedora.im>
17:13:58
So this issue is about rebasing FCOS on bootc technologies; there's a lot of sub-details there
<@walters:fedora.im>
17:14:48
A lot of ways to slice this; one for example would be that we could cease maintaining FCOS rawhide separately (things like https://github.com/coreos/fedora-coreos-config/pull/3004 ) and always maintain bootc rawhide, and just base fcos on that
<@walters:fedora.im>
17:15:03
Getting to using container images on the wire for updates is a huge one
<@walters:fedora.im>
17:15:38
So I hope we can figure out how to make incremental progress on these things
<@walters:fedora.im>
17:16:45
It's important to note there are *very* fundamental differences in the fedora-bootc approach vs current FCOS - especially things like the "evergreen stream" model on top of fedora, that drives things like the python bits, etc. And the general assumption users start by deriving their own container images.
<@walters:fedora.im>
17:17:14
I stubbed out a table here https://docs.fedoraproject.org/en-US/bootc/fedora-coreos/ that could definitely use some expansion
<@jlebon:fedora.im>
17:18:40
right. i think another way to say that latter point is that FCOS is intended to be usable as is for the most part, while fedora-bootc very likely requires derivation. and that drives so much of the UX.
<@walters:fedora.im>
17:18:41
Overall I hope to finally achieve what I've been trying to do for years in having a shared "base image" between all the (current ostree, hopefully future container-based) fedora variants
<@dustymabe:matrix.org>
17:19:00
> And the general assumption users start by deriving their own container images.
Yeah. I think this is where we differ
<@walters:fedora.im>
17:20:29
btw one interesting view into this is that I started by forking the FCOS docs as the bootc docs, and that was *very* enlightening as to the differences. Compare https://docs.fedoraproject.org/en-US/bootc/sysconfig-network-configuration/ with https://docs.fedoraproject.org/en-US/fedora-coreos/sysconfig-network-configuration/
<@walters:fedora.im>
17:21:13
I mentioned this to dusty but I thought it was cool when I realized we can actually just document `podman run fedora-bootc nm-generator` as a way to run it (which we can with FCOS too! But the fcos docs don't think about fcos-as-container)
<@walters:fedora.im>
17:21:37
And another side thing here is a lot of the FCOS docs are "encode these files in butane" when it's WAY more ergonomic to just document a dockerfile `COPY` instruction
<@jbrooks:matrix.org>
17:21:46
I'm interested to know what benefits this group sees in the bootc approach, for instance, is the prospect of collaborating w/ other image-based Fedora variants attractive? Will it save FCOS time, etc
<@jlebon:fedora.im>
17:22:29
it'd be good to flesh out the CI story around the base images. i.e. how exactly do we share and contribute to versioned base images/image definitions. FCOS uses lockfiles right now which... definitely is a massive detail wrt the rest of Fedora both implementation-wise and conceptually
<@jlebon:fedora.im>
17:24:19
Jason Brooks: to answer your question more directly, i think i see a lot of potential benefits in sharing the workload to maintain stability
<@siosm:matrix.org>
17:24:24
Ideally we would integrate bootc into Fedora CI and release process to be able to gate package update on bootc image working.
<@dustymabe:matrix.org>
17:25:12
yes, travier that's the real value I see
<@walters:fedora.im>
17:25:12
one side note: a toplevel goal I have is that it's actually the fedora/centos bootc project, and we try aggressively to avoid letting docs, build system etc diverge between the two
<@siosm:matrix.org>
17:25:19
that would already be a massive benefit to FCOS that would help us move forward with getting closer to bootc
<@dustymabe:matrix.org>
17:25:46
can you elaborate on this?
<@jlebon:fedora.im>
17:26:22
travier: indeed. what i'd say is ideally we don't have lockfiles for the base images and sprint towards that gating CI instead
<@walters:fedora.im>
17:26:38
right now the docs describe *both* c9s and fedora:40, and we are sharing the image definitions pretty closely
<@dustymabe:matrix.org>
17:27:20
ok so you were just referring to c9s and fedora:40 bootc and not referring to FCOS there?
<@walters:fedora.im>
17:27:39
right
<@conan_kudo:matrix.org>
17:28:28
hey folks
<@conan_kudo:matrix.org>
17:28:29
!hi
<@zodbot:fedora.im>
17:28:35
Neal Gompa (ngompa) - he / him / his
<@dustymabe:matrix.org>
17:28:49
👋
<@conan_kudo:matrix.org>
17:28:56
if y'all don't mind, I've got a few questions here
<@siosm:matrix.org>
17:29:11
(Note we have 2 minutes left in this meeting)
<@siosm:matrix.org>
17:29:17
(Note: we have 2 minutes left in this meeting)
<@conan_kudo:matrix.org>
17:29:20
okay, then I'll make it quick
<@walters:fedora.im>
17:30:24
bigger picture we're obviously not going to change many things overnight; but we are spinning up more resources around this and so this will likely just become an ongoing merger/collaboration hopefully
<@conan_kudo:matrix.org>
17:30:53
1. with the new "image mode"/bootc/etc. stack, from the Kinoite perspective, does this mean we need a new backend to manage system updates?
2. how compatible with existing image build tools is this new stack?
3. does it _have_ to result in read-only environments?
<@jlebon:fedora.im>
17:31:03
also note for everyone that there are fedora-bootc community meetings on Tuesdays and Thursdays: https://gitlab.com/fedora/bootc/tracker#meetings
<@jlebon:fedora.im>
17:31:28
Conan Kudo: those sound like questions more for ^ :)
<@conan_kudo:matrix.org>
17:31:56
sure, I can take it there
<@siosm:matrix.org>
17:32:15
yeah, probably better for tomorrow's meeting
<@walters:fedora.im>
17:32:35
agreed, but briefly 1) no, rpm-ostree will continue to work. 2) compat, but note a big goal of bootc (vs ostree) is to own installation; there's `bootc install to-filesystem` and we are building things up there 3) yes but you can make a transient writable overlay if you want
<@siosm:matrix.org>
17:33:13
1) it will need one in the end
<@siosm:matrix.org>
17:33:26
1. it will need one in the end (once we remove rpm-ostree)
<@jlebon:fedora.im>
17:33:47
ok, we're over time now. let's do a quick open floor before closing
<@jlebon:fedora.im>
17:33:56
!topic Open Floor
<@jlebon:fedora.im>
17:34:05
anything anyone wants to bring up for open floor?
<@siosm:matrix.org>
17:34:33
We'll talk about Python next week 😅
<@dustymabe:matrix.org>
17:34:35
!info dusty is mostly AFK for the next few months while on leave (taking care of kids mostly)
<@jlebon:fedora.im>
17:35:36
alrighty, closing in 60s
<@jbtrystram:matrix.org>
17:36:14
Thanks Jonathan Lebon for running the meeting :)
<@jlebon:fedora.im>
17:36:37
!endmeeting