16:30:11 <dustymabe> #startmeeting fedora_coreos_meeting
16:30:11 <zodbot> Meeting started Wed Jun 15 16:30:11 2022 UTC.
16:30:11 <zodbot> This meeting is logged and archived in a public location.
16:30:11 <zodbot> The chair is dustymabe. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:30:11 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:30:11 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:30:18 <dustymabe> #topic roll call
16:30:21 <dustymabe> .hi
16:30:22 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:32:27 <lorbus> .hi
16:32:28 <zodbot> lorbus: lorbus 'Christian Glombek' <cglombek@redhat.com>
16:32:58 <dustymabe> #chair lorbus
16:32:58 <zodbot> Current chairs: dustymabe lorbus
16:33:04 <jmarrero> .hi
16:33:05 <zodbot> jmarrero: jmarrero 'Joseph Marrero' <jmarrero@redhat.com>
16:33:20 <jlebon> .hello2
16:33:21 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:33:26 <saqali> .hi
16:33:27 <zodbot> saqali: saqali 'Saqib Ali' <saqali@redhat.com>
16:34:06 <dustymabe> #chair jmarrero saqali
16:34:06 <zodbot> Current chairs: dustymabe jmarrero lorbus saqali
16:34:10 <dustymabe> #chair jlebon
16:34:10 <zodbot> Current chairs: dustymabe jlebon jmarrero lorbus saqali
16:34:33 <miabbott> .hello miabbott
16:34:34 <zodbot> miabbott: miabbott 'Micah Abbott' <miabbott@redhat.com>
16:35:27 <dustymabe> #chair miabbott
16:35:27 <zodbot> Current chairs: dustymabe jlebon jmarrero lorbus miabbott saqali
16:36:10 <dustymabe> ok let's get started
16:36:26 <dustymabe> #topic Action items from last meeting
16:36:33 <dustymabe> * cverna to open ticket for FCOS as an edition and update
16:36:34 <dustymabe> * dustymabe to schedule ad-hoc meeting to go over f37 changes ahead of next week's community meeting
16:36:36 <dustymabe> * jlebon to schedule meeting to discuss Fedora CoreOS layering use cases
16:36:51 <aaradhak[m]> .hi
16:36:52 <zodbot> aaradhak[m]: Sorry, but user 'aaradhak [m]' does not exist
16:37:02 <x3mboy> .hi
16:37:03 <zodbot> x3mboy: x3mboy 'Eduard Lucena' <eduardlucena@gmail.com>
16:37:13 <jlebon> #info jlebon has scheduled a meeting to discuss Fedora CoreOS layering use cases
16:37:29 <jlebon> i'm waiting to see if some primary stakeholders can make the suggested time before sending it out to the list
16:38:26 <dustymabe> #info dustymabe scheduled met with jlebon, lucab, saqali this morning to narrow down changes to discuss
16:38:38 <dustymabe> jlebon: +1
16:38:43 <dustymabe> cverna: around today?
16:38:48 <mnguyen> .hi
16:38:53 <dustymabe> #chair aaradhak[m] x3mboy mnguyen
16:38:53 <zodbot> Current chairs: aaradhak[m] dustymabe jlebon jmarrero lorbus miabbott mnguyen saqali x3mboy
16:39:24 <miabbott> cverna is probably still out on PTO/sick leave
16:39:24 <dustymabe> #action cverna to open ticket for FCOS as an edition and update
16:39:30 <dustymabe> +1
16:39:45 <dustymabe> ok let's jump in
16:39:52 <dustymabe> #topic New Package Request: qemu-user-static
16:40:03 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1088
16:40:52 <dustymabe> brought this up again since the packaging changes have landed (or are about to land)
16:41:02 <jdoss> .hi
16:41:03 <zodbot> jdoss: jdoss 'Joe Doss' <joe@solidadmin.com>
16:41:06 <ravanelli> .hi
16:41:07 <zodbot> ravanelli: ravanelli 'Renata Ravanelli' <renata.ravanelli@gmail.com>
16:41:08 <jdoss> I am alive.
16:41:11 <dustymabe> one example of the now representative changeset: https://github.com/coreos/fedora-coreos-tracker/issues/1088#issuecomment-1150393245
16:41:18 <dustymabe> #chair jdoss ravanelli
16:41:18 <zodbot> Current chairs: aaradhak[m] dustymabe jdoss jlebon jmarrero lorbus miabbott mnguyen ravanelli saqali x3mboy
16:41:30 <dustymabe> jdoss: lives!
16:41:33 <dustymabe> .chair jdoss
16:41:33 <zodbot> jdoss is seated in a chair with a nice view of a placid lake, unsuspecting that another chair is about to be slammed into them.
16:41:48 <jdoss> lol
16:42:35 <dustymabe> so it looks like we can essentially just install a single package that gives us the desired functionality on each architecture that we want to install them on
16:43:44 <dustymabe> so... should we move forward making this change? AND what platforms do we want to install what compat RPMS on?
16:43:51 <jdoss> I'd love qemu by default
16:44:15 <jlebon> i'm kinda meh on this change and feel like it falls better in the layering realm
16:45:03 <dustymabe> I might agree with you more if layering was a real solution right now, but it's not
16:45:07 <jdoss> yeah I am layering qemu and a few other things on server provision and I block other services from starting with systemd. I does add like 5min to my server coming on line to do work as they have to reboot.
16:45:15 <jlebon> but i understand also the podman team really want this baked in to make life easier
16:45:21 <dustymabe> jlebon: oh..
16:45:29 <dustymabe> you mean `rpm-ostree install` and not `coreos layering` ?
16:45:47 <jlebon> dustymabe: both :)
16:45:59 <jlebon> one available right now, one available Soon (tm)
16:46:06 <dustymabe> :)
16:46:35 <dustymabe> I think it could make sense to have a "batteries included" approach for non x86_64 architectures running x86_64 containers
16:46:59 <dustymabe> since x86_64 containers are the predominant offering right now
16:47:44 <dustymabe> jdoss: yeah I don't know if this change is going to give you what you are talking about
16:47:53 <dustymabe> this doesn't really give you all of qemu
16:48:02 <jdoss> yeah I just re-read the issue. bummer.
16:48:16 <jlebon> one thing that could be interesting is baking it in, but only for podman/moby-engine to use so we're not introducing non-container expectations
16:48:30 <jlebon> and then eventually pulling it out when layering becomes easier
16:49:18 <dustymabe> what would it be used for otherwise? running x86 binaries delivered by Ignition on aarch64 ?
16:49:51 <jlebon> right, yeah
16:50:17 <jlebon> but mainly so it's a clear contract with those apps so that we could revisit it more easily later
16:50:23 <dustymabe> ehh, I think that probably won't be super common (i.e. maybe not worth the effor the lock it down to just podman/moby)
16:51:10 <dustymabe> any other opinions on the current topic?
16:51:35 <dustymabe> miabbott: mnguyen bgilbert (in spirit) ravanelli etc..
16:52:03 <dustymabe> it was nice of the podman team to do the packaging work we requested.
16:52:52 <jlebon> how would we decide which arches to include?
16:53:12 <jlebon> definitely! it makes the layering UX much nicer too
16:53:14 <dustymabe> I think that is a followup
16:53:20 <dustymabe> first should we
16:53:24 <dustymabe> second how exactly
16:53:40 <jlebon> not necessarily.  if there's a risk of a slippery slope situation, then it might be better not to jump in at all
16:53:48 <dustymabe> ok let's flesh that out then
16:54:09 <dustymabe> proposal: just install `qemu-user-static-x86` on non x86_64 platforms
16:54:34 <dustymabe> primary FCOS image stays the same. aarch64 and s390x (and ppc64le in the future) now have an added rpm
16:55:15 <dustymabe> it's approximately 7-8M
16:55:58 <jlebon> if we can draw the line there and stick to it that sounds ok
16:56:30 <miabbott> i'm fine with including the x86 package temporarily to help the podman use case; revisiting it later when coreos layering is more mature seems appropriate
16:56:35 <dustymabe> I think the reasoning can be along the lines of "alot of container images exist for x86-64 but not the other architectures yet"
16:57:25 <jlebon> +1
16:57:36 <jmarrero> +1
16:57:59 <walters> I don't quite understand the "layering isn't ready so let's do something else" versus "let's push layering!!!1one!"  😃
16:58:22 <dustymabe> #proposed we will include the qemu-user-static-x86 package on non x86_64 FCOS images to allow access to the large inventory of containers only built for x86_64.
16:59:17 <dustymabe> +1 -1 ?
16:59:35 <miabbott> +1
16:59:36 <walters> Hmm so by not shipping on x86_64 it would mean using qemu for other purposes would be harder for other software or so?
16:59:36 <jlebon> weak +1 from me
17:00:08 <dustymabe> walters: I think the rationale of "allow access to the large inventory of containers only built for x86_64" doesn't really apply to x86_64
17:00:25 <ravanelli> +1
17:01:26 <dustymabe> i'll add "out of the box" to the agreed to make it more clear
17:01:26 <jlebon> i'd like to say we're not committing to this long-term, but may provide an alternate mechanism for the same end state
17:01:54 <dustymabe> jlebon: i.e. in the proposed text?
17:02:01 <jmarrero> Have we ever removed a package after providing it?
17:02:06 <miabbott> walters: i think the missing piece using layering in the podman use case is that we aren't able to generate qemu artifacts from the container image generated using layering. the podman use case wants to boot as quickly as possible, so having to rebase to a container image is probably a no-go for them
17:02:35 <miabbott> we can provide them a solution now and improve it via layering in the future
17:02:44 <jlebon> dustymabe: maybe just adding "for now, " at the start and "we may consider replacing this with layering in the future." at the end
17:02:59 <jlebon> with a layering mechanism*
17:03:03 <dustymabe> miabbott: ehh. that's part of it. Also, the "update" story for layering is missing. No zincati there at all.
17:03:29 <walters> miabbott: That's https://github.com/coreos/fedora-coreos-tracker/issues/1151
17:03:30 <jlebon> well, it's also a releng process change for them to maintain that
17:04:02 <miabbott> walters: yes, i know the tracker issue...but it's not reality today
17:04:28 <jlebon> firstboot layering might be more palatable for them
17:04:48 <dustymabe> #proposed For now we will include the qemu-user-static-x86 package on non x86_64 FCOS images to allow "out of the box" access to the large inventory of containers only built for x86_64. We may revisit this in the future once CoreOS layering is more mature or if there comes a time when containers for other architectures are as ubiquitous.
17:04:52 <jlebon> (e.g. the work in https://github.com/coreos/rpm-ostree/pull/3006)
17:04:52 <dustymabe> ok updated the proposed ^^
17:05:33 <dustymabe> look better?
17:05:34 <jlebon> ack +1
17:05:41 <ravanelli> +1
17:05:53 <lorbus> +1
17:06:04 <miabbott> +1
17:07:14 <jlebon> it's interesting if you look at e.g. https://koji.fedoraproject.org/koji/rpminfo?rpmID=30644202
17:07:24 <dustymabe> I assume jmarrero is still +1
17:07:28 <jlebon> there aren't that many files. just the binary and binfmt.d dropin
17:07:42 <jmarrero> +1
17:07:57 <dustymabe> jlebon: yeah
17:08:06 <dustymabe> minimal
17:08:07 <jlebon> would be much cheaper to layer it if it didn't involve rpm at all
17:08:22 <jlebon> anyway, don't want to sidetrack :)
17:08:29 <dustymabe> #agreed For now we will include the qemu-user-static-x86 package on non x86_64 FCOS images to allow "out of the box" access to the large inventory of containers only built for x86_64. We may revisit this in the future once CoreOS layering is more mature or if there comes a time when containers for other architectures are as ubiquitous.
17:08:44 <dustymabe> Thanks for the discussion there.
17:08:52 <jlebon> (like, they could literally just fetch a few files via Ignition into /usr/local and call it done -- obviously there's more to it, but...)
17:09:15 <dustymabe> #topic tracker: Fedora 37 changes considerations
17:09:19 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1222
17:09:30 <walters> I think what I'm looking for is to s/We may revisit this...once layering is more mature/We will continue to work on layering/ e.g.
17:09:59 <dustymabe> I think it's obvious that we are continuing to work on coreos layering
17:10:25 <dustymabe> we talked at the beginning of this meeting (action items) about forming a working group to flesh out use cases
17:11:19 <dustymabe> ok this one is about F37 changes
17:11:44 <dustymabe> a few of us met earlier today to discuss the existing changeset and weed out any trivial ones that we can ignore or have no impact on us
17:12:37 <dustymabe> all of the items in the description of https://github.com/coreos/fedora-coreos-tracker/issues/1222 that we can ignore or have no impact on us have a check mark ✔️
17:12:52 <dustymabe> all of the items that have a ⚠️  - we should discuss further
17:13:11 <dustymabe> each item in the list also has a NOTES: entry with some rationale
17:13:38 <dustymabe> we can start to go through each one of the ⚠️  items now to see what action we need to take
17:13:53 <dustymabe> in most cases we'll decide to open a ticket and further investigate in the separate ticket
17:14:11 <dustymabe> subitem 110. Signed RPM Contents
17:14:28 <dustymabe> This change addes a signed/verified hash of each file in each RPM
17:15:03 <dustymabe> There are no changes needed for us but we could choose to use this in the future if we wanted to leverage it for something
17:15:49 <dustymabe> I don't know if we should open an issue specifically for "investigate new features for X", probably not
17:16:02 <dustymabe> does everyone agree there shouldn't be any changes needed for FCOS for this?
17:16:24 <jlebon> yup definitely
17:16:41 <jlebon> i think of it more as the change encouraging a discussion about what our story is re. all the recent talks of IMA/fs-verity/composefs etc.. in FCOS
17:16:53 <dustymabe> indeed.
17:17:04 <dustymabe> Maybe a topic for a community video meeting?
17:17:22 <dustymabe> we could invite alexl/giuseppe
17:17:29 <jlebon> could be. interested in walters' thoughts
17:17:34 <dustymabe> +1
17:18:30 <walters> I think for now we just drop a note in the change that FCOS will not include the IMA digests...but that could change in the future
17:18:48 <dustymabe> WFM
17:18:58 <dustymabe> ok next subtopic
17:19:09 <dustymabe> subtopic 116. Strong crypto settings: phase 3, forewarning 1/2
17:19:33 <dustymabe> it looks like this is a pre-cursor to a F38/F39 change for stronger crypto policy defaults
17:20:14 <dustymabe> if nothing is going to change for F37 then obviously we don't need to do anything yet, but we could consider opting in our `next` stream early to the stricter defaults.. OR running automated tests against them
17:21:04 <jlebon> +1
17:21:10 <dustymabe> i.e. should we consider those options? We can open a ticket to track
17:21:50 <jlebon> i think we probably should yeah
17:22:20 <dustymabe> +1 will do
17:22:25 <dustymabe> subtopic 118. BIOS boot.iso with GRUB2
17:23:11 <dustymabe> My understanding is that the people maintaining syslinux eventually want to not maintain it. We use syslinux for ISO booting in BIOS mode on x86_64, but believe that we might be able to move to GRUB2 for this. Will take investigation, and we'll also need to consider RHCOS.
17:23:45 <dustymabe> *maintaining syslinux in Fedora*
17:24:29 <dustymabe> This shouldn't require any changes of us now, but if the syslinux stack ever does get fully dropped from Fedora we should be ready for it
17:24:43 <dustymabe> so adapting now would be ideal.
17:24:48 <dustymabe> thoughts?
17:24:49 <jlebon> maybe we should just match what the distro uses. e.g. if Fedora moves to GRUB2 for BIOS ISO, we move FCOS, but if RHEL doesn't, we don't move RHCOS
17:25:20 <jlebon> my understanding is that there might be some subtle compat differences between the two?
17:25:22 <dustymabe> jlebon: probably. unfortunately that means maintaining both stacks for longer
17:25:40 <jlebon> dustymabe: yeah, depends how hard that'd be in the cosa side
17:25:51 <jlebon> and also coreos-installer since it needs to know for ISO kargs
17:26:09 <dustymabe> indeed.
17:26:18 <dustymabe> I'll open a ticket for this investigation
17:26:31 <jlebon> but we don't want to be in a situation where e.g. the RHEL Server ISO boots on a machine, but not the RHCOS ISO
17:26:49 <jlebon> +1
17:27:01 <dustymabe> subtopic 207. Enable read only /sysroot for Fedora Silverblue & Kinoite
17:27:03 <miabbott> i would imagine RHEL dropping BIOS wouldn't happen until RHEL 10 at the earliest
17:27:26 <dustymabe> miabbott: well to be clear.. they aren't dropping BIOS
17:27:33 <dustymabe> they are switching the bootloader used on BIOS
17:27:39 <miabbott> _nods_
17:27:40 <dustymabe> for booting ISOs and such
17:28:08 <miabbott> understood; still the switch probably would happen as part of RHEL 10
17:28:14 <miabbott> if at all
17:28:20 <dustymabe> ok for this one we already default to read-only /sysroot but some of our very early systems they might not have that
17:28:27 <dustymabe> see https://github.com/coreos/fedora-coreos-tracker/issues/1222#issuecomment-1156676431
17:29:13 <dustymabe> I agree with jlebon that the reward/risk is low/high.
17:29:21 <dustymabe> a CLHM dropin would be a nice in between
17:29:45 <jlebon> i can split that out into a separate issue and we can decide on it next meeting
17:30:05 <dustymabe> jlebon: that works. I can create the issue when I create the others
17:30:12 <jlebon> dustymabe: ack thanks!
17:30:14 <dustymabe> next subtopic
17:30:32 <dustymabe> subtopic 209. Support FIDO Device Onboarding
17:31:11 <dustymabe> We weren't sure if there was anything we should do for this. This should *require* any changes, but we may want to enable new features by supporting it
17:31:25 <dustymabe> does anybody know any more about this change? maybe walters ?
17:32:35 <dustymabe> seems like a provisioning stack?
17:32:39 <jlebon> i've been meaning to dig into this to see how the trust system works because obviously credential provisioning is one big issue with Ignition and it sounds like this could help with that
17:32:50 <dustymabe> Package the rust implementation of the FIDO device onboarding stack including client, rendezvous service, owner onboarding service and prototype manufacturing service.
17:32:52 <dustymabe> Enable the client service by default for IoT Edition
17:32:54 <dustymabe> Add the client service to the IoT Edition deliverables
17:33:07 <jlebon> might not share any code at all and just ideas
17:33:23 <dustymabe> I mean I wonder if this is somehow an alternative provisioning vector to Ignition
17:33:33 <dustymabe> which wouldn't be great
17:34:13 <jlebon> yeah agreed. depends how the stack is split up
17:34:23 <dustymabe> I guess we can open an issue and ask for more information from peter/patrick
17:34:54 <dustymabe> ok moving on to open floor
17:34:59 <dustymabe> #topic open floor
17:35:01 <jlebon> anyway, i'd file this like the IMA thing: nothing really, just might make it a good time to spur some discussions
17:35:09 <jlebon> nothing really to do*
17:35:34 <dustymabe> jlebon: i.e. maybe instead of opening a ticket we see if someone wants to bring it as a topic to the community meeting?
17:35:38 <dustymabe> video*
17:36:06 <dustymabe> anyone with topics for open floor?
17:36:23 <dustymabe> (thanks for listening to me ramble in the meeting today)
17:36:40 <jlebon> dustymabe: might be worth filing a ticket still. would be an interesting research project/spike for someone
17:36:56 <jlebon> but not necessarily link it to the f37 rebase ticket
17:37:14 <dustymabe> jlebon: we've got a few skeletons of those in our issue tracker already :)
17:37:34 <jlebon> :)
17:37:48 <dustymabe> jlebon: if we don't linke back to the f37 rebase ticket (like the others) then maybe you can open a ticket for the IMA and FIDO ones and I'll do the others
17:38:06 <jlebon> dustymabe: ack sure
17:38:27 <dustymabe> #action jlebon to open investigation tickets for the IMA/FIDO changes
17:38:37 <dustymabe> any other topics before we close out the meeting?
17:40:01 <dustymabe> #endmeeting