16:30:30 #startmeeting fedora_coreos_meeting 16:30:30 Meeting started Wed Sep 1 16:30:30 2021 UTC. 16:30:30 This meeting is logged and archived in a public location. 16:30:30 The chair is dustymabe. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:30:30 Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:30:30 The meeting name has been set to 'fedora_coreos_meeting' 16:30:33 #topic roll call 16:30:36 .hi 16:30:37 dustymabe: dustymabe 'Dusty Mabe' 16:31:34 .hi 16:31:35 davdunc: davdunc 'David Duncan' 16:32:33 .hello siosm 16:32:34 travier: siosm 'Timothée Ravier' 16:32:37 .hello miabbott 16:32:38 miabbott: miabbott 'Micah Abbott' 16:32:43 .hello sohank2602 16:32:44 skunkerk: sohank2602 'Sohan Kunkerkar' 16:33:08 #chair davdunc travier miabbott skunkerk 16:33:08 Current chairs: davdunc dustymabe miabbott skunkerk travier 16:34:30 * dustymabe trying to find meeting logs from last week 16:34:32 I don't see them 16:34:34 one sec 16:34:45 .hi 16:34:45 bgilbert: bgilbert 'Benjamin Gilbert' 16:34:53 .hello2 16:34:54 jaimelm_: Sorry, but user 'jaimelm_' does not exist 16:35:02 .hello2 16:35:03 jaimelm: jaimelm 'Jaime Magiera' 16:35:50 .hi 16:35:52 anthr76[m]: Sorry, but user 'anthr76 [m]' does not exist 16:36:40 #chair bgilbert 16:36:40 Current chairs: bgilbert davdunc dustymabe miabbott skunkerk travier 16:36:44 #chair jaimelm 16:36:44 Current chairs: bgilbert davdunc dustymabe jaimelm miabbott skunkerk travier 16:36:47 * jcajka is around lurking 16:36:48 #chair anthr76[m] 16:36:48 Current chairs: anthr76[m] bgilbert davdunc dustymabe jaimelm miabbott skunkerk travier 16:36:53 #topic Action items from last meeting 16:36:58 * jlebon will open a ticket and look for volunteers to investigate "Optimal LUKS Encryption Sector Size" 16:37:00 * jlebon will open a ticket and look for volunteers to investigate "Libvirt Modular Daemons" 16:37:21 #info jlebon opened #936 and #935 16:37:38 +anthr76[m] you need to specify your Fedora account 16:37:48 #topic meeting items 16:38:24 this is a reminder that if you want to discuss a particular meeting item here please add the `meeting` label to the ticket in https://github.com/coreos/fedora-coreos-tracker/issues - if you can't add a label (no permissions) then just comment in the issue and we'll add the label to the ticket 16:38:32 on to tickets now 16:38:44 #topic Stale UEFI boot entry left behind after reprovisioning 16:38:49 #link https://github.com/coreos/fedora-coreos-tracker/issues/946 16:38:58 bgilbert: want to intro this one? 16:39:09 sure 16:39:43 I think we've been implicitly assuming for a long time that RHCOS doesn't configure UEFI boot variables at all, and just uses the fallback boot behavior 16:39:45 .hello2 16:39:46 walters: walters 'Colin Walters' 16:40:10 but in fact, shim configures a boot variable on first boot. so we're gradually working through the consequences of that. 16:40:27 #chair walters 16:40:27 Current chairs: anthr76[m] bgilbert davdunc dustymabe jaimelm miabbott skunkerk travier walters 16:40:58 if you reprovision a UEFI machine frequently, as we encourage users to do when their Ignition config changes, we'll end up leaving old UEFI boot variables sitting around. 16:41:19 and maybe eventually fill up flash with stale UEFI vars. 16:41:48 so, it seems like you actually do *want* to use the fallback boot path, but without the fallback behavior we've got set up by default 16:42:11 zodbot: #halp 16:42:38 welcome pjones :) 16:42:49 pjones: that's my preference. as an image-based distro, we just sort of assume that we can plop down on a disk and successfully boot. 16:43:07 i'd phase it like "we intentionally are trying to have the image install path be a glorified dd" 16:43:12 right 16:43:16 we could teach coreos-installer to deal with UEFI boot vars, but that doesn't help with image-based launches (e.g. AWS with UEFI support, Packet, etc.) 16:43:22 which is itself a consequence of wanting cloud and bare metal to work as symmetrically as possible 16:43:28 ^ 16:43:40 the reason we can't reuse an existing variable is that the GUID of the EFI boot partition is randomized on every image build, and that GUID is included in the boot var. 16:43:44 right, you'd rather just not make the variables in the first place 16:43:47 yup 16:44:00 do image based launches (AWS, etc) reprovision? or just start a new instance? 16:44:03 is this like a flag we could set on shim? 16:44:04 is it possible to just not include /boot/efi/EFI/fedora/*.CSV in your images? 16:44:18 that's where fallback is getting the information from 16:44:25 pjones: would that disable the behavior? I actually haven't looked 16:44:48 it should, yes. removing fallback.efi would also do it currently, but that particular file will go away in the future. 16:45:18 What are the reasons for the current fallback behaviour? Is it mostly to make sure we boot the same thing we booted the first time? or to get a nice EFI entry maybe? 16:45:26 pjones: perfect 16:45:37 ok seems like bootupd should gain a flag to skip installing that 16:45:38 if it's that easy then let's do that 16:45:51 travier: it's intended to repair a system when you've either moved the disk or for some reason the variables have been clobbered 16:45:52 well - i have at least one question :) 16:45:58 I was about to go down the road of "maybe we should stop randomizing the ESP GUID" but that obviates it 16:46:29 we're still in a situation where we're violating the GPT spec by not randomizing it, but at least we wouldn't take a dependency on that behavior 16:46:41 bgilbert: right; that would mean you get the same variable every time and we'd detect that and keep it, but you'd rather just not create it. 16:46:49 pjones: yup 16:47:24 ok let me circle the conversation back (and ask a new question) if pointed discussion has ceased 16:47:31 dustymabe: go for it 16:48:06 current proposal is to delete /boot/efi/EFI/fedora/*.CSV in our bootimages (for RHCOS path would be slightly different I assume) 16:48:18 what are other possible uses of `/boot/efi/EFI/fedora/*.CSV` ? 16:48:32 fallback is the only thing that should be using it. 16:48:46 IOW, could/would new functionality be added in the future that we will be missing because we are nuking the file 16:48:51 literally the only data in it is the data to create that variable 16:49:14 ok - in that case let me make one request as part of this 16:49:38 #link https://github.com/rhboot/shim/blob/main/README.fallback 16:49:46 let's update our code to nuke the file, but also check the contents of the file. If the contents change we fail and re-evaluate the safety of the operation 16:49:49 cc bgilbert ^^ 16:50:00 (note that we would stop installing it instead of removing it) 16:50:26 If you plan to do that, you may want to note that it is encoded ads UCS-2 LE 16:50:27 dustymabe: that feels overly cautious to me 16:50:28 as 16:50:42 but seriously nothing should ever change it, so I think that's overkill. 16:50:43 same to me given the content and purpose here 16:50:46 travier: technical detail.. we would be "removing it" most likely in a post script during image creation - but yes, it would never get installed 16:50:48 dustymabe: we do already have CI for UEFI and UEFI+SB 16:50:49 seam too catious 16:51:01 like... the last time the contents changed was 15 fedora releases ago 16:51:26 maybe a few more than that actually. 16:51:30 * bgilbert would defer to pjones :-) 16:51:32 ehh - doesn't seem too cautious to me - if the contents of the file changed wouldn't we want to know? 16:51:48 dustymabe: if bootupd skips installing it, then it won't be installed at all 16:52:03 as walters said, it's bootupd installing that on image build so it's teaching bootupd to not install it 16:52:23 I see 16:52:24 and it will also drop out of the update payload 16:52:27 dustymabe: nah, it's mostly just a caption for the boot entry 16:52:35 like literally the contents are: 16:52:40 and hence, in-place bootupd updates will remove it 16:52:40 shimx64.efi,Fedora,,This is the boot entry for Fedora 16:53:21 yeah, just feels like if that file ever changed (in the sources we pull from) we'd want to know. 16:53:40 What's the cost of checking? 16:53:46 (I wouldn't think much) 16:54:00 dustymabe: I understand the need to be cautious but what would that give us? 16:54:06 jaimelm: it encodes the idea that the answer is important 16:54:16 we can sha256sum the file and check if it changes 16:54:26 travier: the assumption right now is that the file will never change because it has one use 16:54:45 what if it gains another use (for whatever reason anyone could fathom, I can't predict the future) 16:55:06 now FCOS is more of a snowflake compared to the rest of Fedora 16:55:08 it would seem cleaner to have an explicit way to disable this behavior instead (some new stamp/config file?) but, eh 16:55:36 in the overall space of things that might break in our boot setup, this is a tiny piece 16:55:59 walters: tbh I'm not against making the shim package have a shim-cloud variant that just doesn't include it 16:56:04 it feels to me that if this creates an issue in the future this will be easy to revert and we will see it coming 16:56:07 #proposed we will drop the shim fallback CSV from new releases 16:56:15 and then you all could rely on that package having the desired behavior without focusing on the implementation details. 16:56:18 travier: what if we don't know it created an issue? 16:56:20 pjones: oh, interesting 16:56:35 what if FCOS is silently behaving different from the rest of Fedora because of this delta 16:56:38 would that help? 16:56:54 pjones: nah it's OK, just fleshing out things 16:57:02 Then we would be even more of a snowflake 16:57:08 as that package would be only used by us 16:57:27 would it? or would Cloud also find it useful? 16:57:36 (personally I want IoT and other things to derive from FCOS in which case there's only "coreos-like" and "yum + anaconda like") 16:58:17 we know IoT is investigating coreos-installer because it matches the use case well, and that's going to have the same issue 16:58:27 right 16:58:30 #proposed we'll drop the /boot/efi/EFI/fedora/*.CSV file which will prevent the creation of a new variable every time we reprovision FCOS on a machine 16:58:33 We kind of need it for other things. Like, it'd be a better fit for local media live images and installer images than what we're doing now as well. Anything that's booting an static image instead of install+updates+updates+... 16:58:49 is my proposal technically sound? 16:59:00 i.e. is the problem the "creation of a new variable every time we reprovision FCOS on a machine" 16:59:02 (I am open to naming suggestions as well, if -cloud is too specific and inaccurate) 16:59:28 pjones: if you're willing to add the subpackage, I think we'd gratefully accept 16:59:34 +1 16:59:46 "-imagebased"? 16:59:47 bikeshed bikeshed shim-transient 16:59:55 -dontcreatethatvariableyo 16:59:59 heh 16:59:59 if it's useful for something else then that's a good idea 17:00:04 -imagebased is good 17:00:11 -package-separated-variable 17:00:49 +1 to proposed, independent of mechanism 17:01:05 ack 17:01:42 and once we're not adding boot vars, we could possibly randomize partition GUIDs at first boot 17:01:48 ^^^ 17:02:01 violating the spec seems trivial, but it's not 17:02:14 let me get this straight.. 17:02:20 we already randomize FS UUIDs and the disk GUID; not sure why we're ignoring partition GUIDs right now 17:02:28 the extra variable(s) create extra boot entries ? 17:02:46 in the UEFI boot menu, not in GRUB 17:03:00 I see 17:03:04 I think I have seen this before 17:03:32 not sure if the spec mandates that entries be kept/ignored if the partition can't be found 17:03:34 to say this another way, the issue isn't seen with Anaconda because it will usually keep an existing ESP? 17:03:36 pjones: you could call it -direct as it will directly boot the next EFI binary and not perform other setup 17:04:18 on the RHCOS side I think we've been saved by the policy against updating bootimages 17:04:52 ok i'm going to drop a new proposed (slight tweak) 17:05:24 #proposed we'll drop the /boot/efi/EFI/fedora/*.CSV file which will prevent the creation of a new UEFI boot entry (variable stored in nvram/flash) every time we reprovision FCOS on a machine 17:05:41 and actually let me update it further 17:05:46 *every time FCOS is reprovisioned 17:06:00 there we go 17:06:09 #proposed we'll drop the /boot/efi/EFI/fedora/*.CSV file (or pick up a new shim subpackage which does the same) which will prevent the creation of a new UEFI boot entry (variable stored in nvram/flash) every time we reprovision FCOS on a machine 17:06:27 #proposed we'll drop the /boot/efi/EFI/fedora/*.CSV file (or pick up a new shim subpackage which does the same) which will prevent the creation of a new UEFI boot entry (variable stored in nvram/flash) every time FCOS is reprovisioned 17:06:39 perfect 17:06:41 +1 17:06:42 ish 17:06:45 ack 17:07:04 +1 17:07:34 pjones: many thanks for taking part in this discussion. 17:07:43 thank you pjones 17:07:44 +1 (though I'd call it the "shim fallback .CSV file") but eh, it will be clear from context 17:08:04 agreed thanks! 17:08:05 pjones: thank you! 17:08:28 #agreed we'll drop the shim fallback .CSV file (/boot/efi/EFI/fedora/*.CSV) OR pick up a new shim subpackage which does the same, which will prevent the creation of a new UEFI boot entry (variable stored in nvram/flash) every time FCOS is reprovisioned 17:08:33 walters: updated ^^ 17:08:40 +1 17:08:46 +1 17:08:50 well I learned something new today :) 17:08:57 ok one more question for my clarity 17:09:36 +1 17:09:37 this is really only a problem for people using `coreos-installer` correct (i.e. re-using same machine versus provisioning new one) 17:09:40 +1 to proposed 17:10:04 "provisioning new one" mostly makes sense in the virtual machine context 17:10:20 dustymabe: I think this mostly impacts bare metal installations 17:10:40 that's what I thought 17:10:42 ok moving on 17:10:45 well...that's the thing, by doing a "dd style reprovision" the goal is to make bare metal feel more like cloud 17:10:57 #topic support package overlays in live boot environments 17:11:02 #link https://github.com/coreos/fedora-coreos-tracker/issues/941 17:11:25 ok this one has come up several times lately.. more and more people using a "live environment" and wanting to layer some stuff on top 17:11:44 i'm wondering if we can come up with a plan to fix it sometime soonish 17:12:40 i think "plan" is just "teach rpm-ostree to skip writing a new deployment and directly make changes live" right? 17:13:05 walters: not sure - sounds reasonable to me? 17:13:22 though it would be nice for `rpm-ostree status` to reflect the mutations that were made 17:13:27 ^^ 17:13:44 that would be quite cool 17:14:39 walters: would that be possible in the "teach rpm-ostree to skip writing a new deployment and directly make changes live" plan? 17:14:49 sure 17:15:08 cool. 17:15:29 in the same way that `rpm-ostree install -A` works and shows state in status 17:15:35 i guess the only other thing left is to see where this sits on importance and if anyone wants to work on it 17:16:52 i'll update the ticket with some of the discussion of "plan" -- will move on to next ticket soon 17:17:18 When you "soonish", what are you thinking? 17:17:28 s/you/you say/g 17:17:39 jaimelm: next month? 17:17:43 in the next month 17:17:52 Seems like there are lots of pending FCOS tweaks 17:18:08 indeed 17:18:30 It would be helpful if there was a table, with scoring, to know what is what 17:18:42 but that's me as a community member 17:18:44 we have the hackmd we update periodically 17:18:59 we could have a GH project board 17:19:06 ^^ 17:19:12 "discuss", "decided + waiting for implementation", "in progress", etc. 17:19:24 i'm not opposed 17:19:50 :) 17:19:57 That would be good. Lots of stuff flies by and I see some things fall between the cracks, only to (re)appear at a meeting down the road. 17:20:18 jaimelm: yeah, us too. :-/ 17:20:32 i don't think a project board is going to prevent stuff falling through the cracks 17:20:36 it won't 17:20:55 let me move on to the next ticket for now while we have some time left 17:20:55 but it's true that the tracker is an enormous pile of stuff right now 17:20:58 +1 17:21:06 #topic tracker: Fedora 35 changes considerations 17:21:10 #link https://github.com/coreos/fedora-coreos-tracker/issues?q=is%3Aissue+is%3Aopen+label%3AF35-changes 17:21:12 I'm happy to work WITH someone who tighter in the community to get some better tracking 17:21:37 any updates for any of our F35 Changes tickets that still need discussion/investigation ? 17:22:01 still have to do mine, sorry 17:22:16 no worries travier - we'll get an update next week 17:23:01 travier: you opened #938 - did you want to take that one as well? 17:24:42 jlebon is out this week 17:24:58 only other ticket is the third party software mechanism one 17:25:05 walters: any update on that one? https://github.com/coreos/fedora-coreos-tracker/issues/932 17:25:46 Well an obvious thing to double check is - are we planning to include `fedora-third-party` in our manifest? 17:26:04 doubtful (isn't it about flatpaks?) 17:26:23 ahh - it's about yum repos too 17:27:13 (this is less relevant for docker/podman because the model for those strongly conflates "name of software" with "where to get software") 17:27:59 walters: is it safe to say we probably wouldn't ship the `fedora-third-party` package ? 17:28:24 I think we should not ship it 17:28:40 agree 17:28:42 I'll try to make #938 happen 17:29:03 though it might be good to create a FAQ entry about it 17:29:16 ^^ 17:29:30 i.e. we don't ship `fedora-third-party` because CONTAINERS, but you can enable it by 1. 2. 3. 17:29:35 We could, but it's only really targeted at Workstation use cases 17:29:51 well, it works for server too 17:30:04 travier: yeah, if it pulls in graphical deps then it's a definite no go 17:30:46 #proposed we don't plan to ship the fedora-third-party package because the desired use is that people run software in containers on Fedora CoreOS 17:30:49 ack/nack ^^ 17:30:56 ack 17:30:56 +1 17:31:12 mmm, well, flatpak is much like (podman/docker) containers too 17:31:53 If the group settles on a FAQ, I'll volunteer to write it 17:32:01 we've over btw 17:32:02 i.e. you want me to specify non-flatpak containers 17:32:03 I'd say it's more we don't plan to ship fedora-third-party because podman/docker don't need it 17:32:42 walters: how about "desired use is that people run software in containers and not install extra packages on the host" 17:32:51 and we don't ship flatpak and anyone who wants to install packages non-default yum repos can add them via ignition 17:32:52 I think dustymabe has a good angle though, because it reinforces the overall concept of FCOS 17:34:01 #proposed we don't plan to ship the fedora-third-party package because the desired use is that people run software in podman or docker containers and not install extra packages on the host 17:34:06 I think that should do ^^ 17:34:09 I'm only bikeshedding this a bit because it gets into some deep philosophical/technical bits that are worth elaborating on to ensure we're all on common ground 17:34:36 that's good 17:34:37 walters: most recent proposed sound good ? 17:34:38 but, sure +1 and we can move on 17:34:38 ack 17:34:46 anyone else ? 17:34:57 ack 17:35:02 +1 17:35:07 I think I might volunteer to run the next meeting. 17:35:11 * jaimelm pulls out a whip 17:35:14 :) 17:35:29 nice jaimelm 17:35:44 #action jaimelm to write documentation on fedora-third-party rpm for our FAQ 17:35:52 #agreed we don't plan to ship the fedora-third-party package because the desired use is that people run software in podman or docker containers and not install extra packages on the host 17:35:59 #topic open floor 17:36:04 sorry about going over 17:36:10 * dustymabe cowers in front of jaimelm 17:36:17 heh 17:36:30 #info we're trying really hard to ship aarch64 artifacts in next week's round of releases 17:36:32 people with toddlers have tight time tables :) 17:36:49 anything else before we close? 17:36:55 .hello anthr76 17:36:56 * jaimelm has nothing else 17:36:58 anthr76[m]: anthr76 'Anthony Rabbito' 17:37:00 hi anthr76[m] 17:37:30 Excited and grateful for the aarch64 work. That is all. Just finished moving so hope I can help out soon. 17:37:38 +1 17:38:10 anthr76[m]: try out development images over at https://builds.coreos.fedoraproject.org/browser?stream=testing-devel&arch=aarch64 17:38:31 #endmeeting