16:32:22 <dustymabe> #startmeeting fedora_coreos_meeting
16:32:22 <zodbot> Meeting started Wed Apr  5 16:32:22 2023 UTC.
16:32:22 <zodbot> This meeting is logged and archived in a public location.
16:32:22 <zodbot> The chair is dustymabe. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:32:22 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:32:22 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:32:25 <dustymabe> #topic roll call
16:32:29 <dustymabe> .hi
16:32:29 <bgilbert> .hi
16:32:29 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:32:32 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:32:35 <ravanelli> .hi
16:32:36 <zodbot> ravanelli: ravanelli 'Renata Ravanelli' <renata.ravanelli@gmail.com>
16:32:54 <travier> .hello siosm
16:32:56 <zodbot> travier: siosm 'Timothée Ravier' <travier@redhat.com>
16:32:58 <jbrooks> .hello jasonbrooks
16:32:59 <zodbot> jbrooks: jasonbrooks 'Jason Brooks' <jbrooks@redhat.com>
16:33:01 <c4rt0> .hello c4rt0
16:33:03 <zodbot> c4rt0: c4rt0 'Adam Piasecki' <c4rt0gr4ph3r@gmail.com>
16:33:06 <jlebon> .hello2
16:33:06 <marmijo[m]> .hello marmijo
16:33:06 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:33:09 <zodbot> marmijo[m]: marmijo 'Michael Armijo' <marmijo@redhat.com>
16:34:54 <dustymabe> #chair bgilbert ravanelli travier jbrooks c4rt0 jlebon marmijo[m]
16:34:54 <zodbot> Current chairs: bgilbert c4rt0 dustymabe jbrooks jlebon marmijo[m] ravanelli travier
16:35:33 <dustymabe> reminder as always.. If you have something you'd like to discuss during the meeting please add the meeting label (or you can send me a message and I'll try to get it added to the meeting)
16:36:07 <dustymabe> #topic Action items from last meeting
16:36:14 <dustymabe> #info there were no action items from laste meeting
16:36:30 <dustymabe> #topic Fedora CoreOS talks for DevConf.cz & DevConf.us 2023
16:36:36 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1437
16:36:57 <dustymabe> #info the CFP submission deadline for Devconf.us was extended to April 10th
16:37:24 <dustymabe> it's not looking like I'll be able to go to devconf.cz - maybe devconf.us will work out
16:37:46 <dustymabe> i'll probably try to submit the same session that @travier and I did for devconf.cz to the US version
16:37:59 <travier> 👍
16:38:43 <dustymabe> might be good if we submit a lab/workshop too
16:38:49 <dustymabe> it was fairly well attended last time
16:39:47 <dustymabe> ok new topic
16:40:01 <spresti[m]> .hello spresti
16:40:02 <zodbot> spresti[m]: spresti 'Steven Presti' <spresti@redhat.com>
16:40:05 <dustymabe> #chair spresti[m]
16:40:05 <zodbot> Current chairs: bgilbert c4rt0 dustymabe jbrooks jlebon marmijo[m] ravanelli spresti[m] travier
16:40:14 <dustymabe> #topic Add a "Boot to HD" entry to ISO's boot menu
16:40:22 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1453
16:40:30 <dustymabe> #chair jmarrero
16:40:30 <zodbot> Current chairs: bgilbert c4rt0 dustymabe jbrooks jlebon jmarrero marmijo[m] ravanelli spresti[m] travier
16:40:38 <dustymabe> jlebon: want to intro this one?
16:42:04 <dustymabe> ok I can.
16:42:28 <dustymabe> looks like the user just wants the ability to boot from the hard drive at the interactive menu when booting from the ISO
16:43:12 <jlebon> sorry! yup, you got it :)
16:43:33 <jlebon> tagged it, since I thought this was an interesting one to discuss here
16:43:36 <dustymabe> seems like a simple enough request.. the biggest hurdle is probably making sure it works in grub and syslinux
16:44:06 <dustymabe> on a related note: https://github.com/coreos/fedora-coreos-tracker/issues/1231
16:45:09 <jlebon> ideally we can chainload into the regular grub so we don't have to worry about mdraid support and such
16:45:45 <dustymabe> jlebon: I used to do this with isolinux/pxelinux a while back
16:45:59 <jlebon> it's kind of a weird use case though, because it implies that on every reboot they'd have to catch the prompt
16:46:04 <jlebon> and that doesn't really work well with automatic updates
16:46:07 <dustymabe> there was literally like a boot 0x80 instruction or something that you could use to say "boot from HD"
16:46:15 <dustymabe> not sure if there is something similar in GRUB or not
16:47:07 <jlebon> so i think maybe a cleaner approach is some flag you set where the ISO knows to auto-pivot if it's installed
16:47:30 <jlebon> but not sure how feasible that is from isolinux
16:48:00 <jlebon> but you'd do something like `coreos-installer iso customize --prefer-disk` or something
16:48:02 <dustymabe> yeah. I think the key here is this should be dead simple and best effort
16:48:06 <bgilbert> jlebon: as I understand the use case, they're not trying to leave the CD in indefinitely
16:48:21 <bgilbert> it's just so they can test that the installed system worked before removing the CD
16:48:30 <dustymabe> i.e. if it doesn't work for someone's particular setup.. well, we tried
16:48:34 <bgilbert> dustymabe: +1
16:48:40 <jlebon> bgilbert: yeah, that's way more reasonable
16:48:56 <jlebon> the other case is probably not worth supporting
16:49:03 <dustymabe> https://wiki.syslinux.org/wiki/index.php?title=Config#LOCALBOOT
16:49:05 <jlebon> but ISO things are fun to think about :)
16:49:50 <bgilbert> +1 to not supporting the fancier case.  the firmware will have boot order settings.
16:50:19 <dustymabe> we can can do this with isolinux pretty easy.. the question is "is there an equivalent to `localboot 0x80` in GRUB
16:50:28 <dustymabe> would take some investigation
16:51:04 <dustymabe> anyone with any thoughts on a proposed for this?
16:51:28 <jlebon> hmm, for grub i think we can search for the boot label and then use `configfile`
16:51:44 <bgilbert> dustymabe: https://www.gnu.org/software/grub/manual/grub/html_node/chainloader.html#chainloader maybe
16:52:19 <bgilbert> jlebon: configfile isn't quite the same semantics, since it'd assume an FCOS system
16:53:23 <dustymabe> ok how much of this should I put in the proposed? also should we include any context on priority of such a feature?
16:53:30 <jlebon> bgilbert: true. didn't think about whether we wanted to support non-CoreOS
16:54:31 <bgilbert> jlebon: we should probably use consistent semantics between the two bootloaders in any event
16:54:35 <dustymabe> here's what I have for now:
16:54:37 <dustymabe> #proposed We think this would be a nice feature to have, but we want the implementation to be dead simple (easy to maintain) and best-effort.
16:55:20 <jlebon> +1
16:55:35 <bgilbert> s/dead/very/?
16:57:01 <dustymabe> #proposed We think this would be a nice feature to have, but we want the implementation to be very simple (easy to maintain) and best-effort.
16:57:04 <bgilbert> +1
16:58:14 <dustymabe> anyone else :)
16:58:27 <marmijo[m]> Ack
16:58:34 <dustymabe> #agreed proposed We think this would be a nice feature to have, but we want the implementation to be very simple (easy to maintain) and best-effort.
16:58:57 <dustymabe> I'll update the issue and also put out a call to see if anyone wants to work on it.
16:59:17 <dustymabe> #topic Machines upgraded from old FCOS releases use bootloaders denylisted in newer UEFI dbx
16:59:24 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1452
17:00:11 <dustymabe> ok I can intro this
17:00:31 <dustymabe> We recently added extended upgrade tests that go back and test if we can get all the way up to date from older versions of FCOS
17:00:45 <dustymabe> as part of this it shook out a few problems that we've been looking into
17:01:00 <dustymabe> some of them are ones we knew about (but maybe had forgotten a little)
17:01:02 <dustymabe> others are new
17:01:53 <dustymabe> In this case we can't secureboot any systems older than F34 in COSA right now
17:02:32 <dustymabe> @bgilbert did some investigation on top of this since it has some implications for existing users
17:02:46 <dustymabe> bgilbert: anything you want to add?
17:03:14 <bgilbert> there are a couple pieces of investigative work that should be done, if someone is interested in digging in
17:03:42 <dustymabe> A) Investigate: in a FCOS VM with old UEFI firmware, upgraded from a very old FCOS release, and with an old bootloader, does fwupdmgr update correctly refuse to update dbx? What about on a system installed with boot_device.mirror?
17:03:44 <bgilbert> the fundamental issue is the interaction between the Secure Boot signatures on shim/GRUB and the UEFI dbx database that denylists certain signatures
17:03:55 <dustymabe> B) Investigate: in a FCOS VM with old UEFI firmware, upgraded from a very old FCOS release, and with a bootloader updated via bootupd, does fwupdmgr update correctly update dbx? What about on a system installed with boot_device.mirror?
17:03:58 <bgilbert> ^
17:04:43 <bgilbert> as a Well-Behaved OS we're supposed to update the firmware's dbx database with new versions from the UEFI powers-that-be
17:05:12 <bgilbert> because if we don't, an attacker who wants to bypass Secure Boot could use a known-compromised bootloader to chainload malware etc
17:05:26 <bgilbert> since that bootloader won't have been denylisted in the firmware.
17:06:03 <bgilbert> we ship fwupd, and users could manually use it to update dbx today.  fwupd is supposed to check that the EFI System Partition has no bootloaders that use an old signature
17:06:08 <bgilbert> (since that would brick the OS)
17:06:36 <bgilbert> so the questions are: if there's an old bootloader (because the user hasn't manually updated it with bootupd), does fwupd correctly refuse to update dbx?
17:06:46 <bgilbert> if there isn't an old bootloader, does fwupd successfully update dbx?
17:06:59 <bgilbert> (given FCOS's quirks about the EFI partition, e.g. not mounting it by default)
17:07:06 <bgilbert> (and having multiple independent ones on RAID systems)
17:07:13 <bgilbert> so that's part 1.
17:07:14 <dustymabe> bgilbert: if someone new wanted to use this as an opportunity to learn would you be a good mentor there? or are you looking for someone to come in and own it end-to-end?
17:08:21 <bgilbert> part 2 is: what can we do to close this gap in our practices?  we normally aim for "secure by default".  ideally we'd update dbx automatically, which therefore requires updating the bootloader automatically
17:08:39 <bgilbert> and we don't do the latter because bootupd arguably isn't safe enough yet against bad bootloader updates etc. that could brick the system
17:09:04 <bgilbert> so automatic updates are potentially a lot of work.  we could better document this, and make it the user's problem, but that's not a great compromise
17:09:15 <dustymabe> bgilbert: +1
17:09:16 <jlebon> right. so we have to first get bootupd in a place where we can enable it by default, and then look at dbx updates
17:09:18 <dustymabe> thanks for all the context
17:09:21 <bgilbert> dustymabe: I could help mentor there, yeah
17:09:39 <jlebon> i wonder, is fwupd dbx updates automated in the rest of Fedora?
17:09:54 <dustymabe> IIUC the end goal is to get regular bootloader and dbx updates
17:10:18 <dustymabe> and (unfortunately) that's going to require two different tools (bootupd and fwupd). Am I reading all of that correctly?
17:10:22 <walters> other Fedora editions don't automate updates (AFAIK) but gnome-software at least does expose it
17:10:36 <bgilbert> walters: that's my impression as well
17:11:21 <bgilbert> dustymabe: yeah
17:11:40 <dustymabe> bgilbert: ok. so basically we need to:
17:11:46 <dustymabe> 1 - do the investigations you mentioned
17:11:58 <dustymabe> 2 - get bootupd in a state where we are comfortable with enabling it automatically
17:12:11 <dustymabe> 3 - also enable fwupd updates by default too?
17:12:26 <travier> > so the questions are: if there's an old bootloader (because the user hasn't manually updated it with bootupd), does fwupd correctly refuse to update dbx?
17:12:27 <jlebon> bgilbert: btw, *excellent* breakdown. makes it much easier to parse.
17:12:28 <travier> yes it does
17:12:36 <travier> we have this "issue" in Silverblue
17:12:41 <dustymabe> bgilbert++
17:12:44 <bgilbert> jlebon: thanks :-)
17:12:56 <travier> https://github.com/fedora-silverblue/issue-tracker/issues/355
17:13:09 <jlebon> dustymabe: maybe not all firmware and just dbx updates?
17:13:41 <dustymabe> jlebon: correct
17:14:11 <bgilbert> travier: in general it does.  I'd like to confirm that it does on FCOS specifically
17:14:29 <dustymabe> bgilbert: yeah, because we leave the ESP unmounted (don't think we do that on SB?)
17:14:30 <travier> enableing firware updates by default is tricky as some fireware updates pause devices, etc.
17:14:44 <jlebon> there's also the RAID1 question
17:14:46 <dustymabe> yeah, no to all FW updates by default
17:15:03 <travier> well, it does in Silverblue & Kinoite which are quite close to FCOS on this front
17:15:18 <dustymabe> oh, the ESP is unmounted on SB a K ?
17:15:22 <bgilbert> yeah, I wouldn't argue for updating all firmware by default
17:15:32 <jlebon> it's mounted in SB
17:15:50 <dustymabe> right
17:15:53 <travier> hum, indeed, good catch
17:16:02 <jlebon> anaconda adds it to /etc/fstab
17:16:25 <travier> so I retract my statement :)
17:16:27 <travier> we need to test it
17:16:28 <bgilbert> maybe we should make this a kola test, actually.  we do change ESP handling every now and then
17:16:47 <bgilbert> so --qemu-firmware would have uefi-secure-ancient or such
17:17:06 <jlebon> bgilbert: OOC, did CL automatically update the dbx?
17:17:26 <bgilbert> jlebon: CL never enabled SB
17:17:35 <jlebon> ahhh :)
17:17:37 <travier> fwupd did not exists at the time AFAIK either
17:18:03 <dustymabe> ok.. so I think here (and over the past few weeks of various investigations) have established that we need to really close the gaps here with regards to bootloader and dbx updates. @bgilbert has called out some things for us to investigate as part of making progress towards that and we also need to work on getting bootupd to a state where it can be enabled automatically too
17:18:14 <travier> +1
17:18:22 <dustymabe> what's the best way to proceed here? i.e. anything we can take away in a #proposed/#agreed ?
17:18:50 <dustymabe> anyone with an eye for detail that wants to learn from an OS expert and help us move the needle on this?
17:20:02 <jlebon> this will likely require multiple people over a long time tbh
17:20:09 <bgilbert> +1 jlebon
17:20:40 <jlebon> i'm interested to help with this... at some point
17:20:44 <bgilbert> the investigations in dustymabe's step 1 are bounded, so it's good material for a volunteer
17:20:45 <dustymabe> right, which is why we've probably not fully closed the gap here before now (level of effort is high)
17:20:58 <jlebon> the investigation part is definitely good for someone who wants to learn
17:21:02 <bgilbert> step 2 is bootupd, which is a non-trivial piece of work I think
17:21:19 <bgilbert> and then the fwupd enablement I hope is pretty straightforward
17:21:51 <bgilbert> for this meeting, maybe we should get a consensus on the goal
17:22:03 <bgilbert> do we want bootloader autoupdate?  do we want dbx autoupdate?
17:22:21 <bgilbert> once we have the step 1 investigation done, we could theoretically add some docs and call it good
17:22:30 <dustymabe> bgilbert: I think some version of both, yes
17:22:53 <dustymabe> i.e. maybe we do bootloader and dbx updates at well defined moments in time versus on every update
17:22:55 <jlebon> bgilbert: i.e. leave it to users to manually update?
17:23:10 <bgilbert> jlebon: in principle, yes
17:23:18 <jlebon> or i guess we could do like we do for bootupd and document a systemd unit with caveats
17:23:58 <dustymabe> I think the problem with the "docs" approach is there are cases where your system won't boot
17:24:23 <dustymabe> my thinking in the past was that worst case your system just wouldn't be as secure
17:24:35 <bgilbert> dustymabe: which cases?
17:24:56 <dustymabe> but the recent issues have made it evident that if you don't update your bootloader then at some point your system will no longer boot
17:25:04 <bgilbert> assuming fwupd correctly protects against old bootloaders, the only case should be "new machine with old install image", which we don't support anyway
17:25:11 <jlebon> obviously, this is a problem in RHCOS/OCP too (and other rpm-ostree-based products). i can definitely see this being prioritized soon-ish as a result.
17:25:17 <dustymabe> bgilbert: this is the primary one https://github.com/coreos/fedora-coreos-tracker/issues/1441
17:25:27 <bgilbert> oh, you mean in the broader sense
17:25:33 <dustymabe> bgilbert: right
17:25:57 <dustymabe> i.e. if we leave it to documentation and someone manually executing something then there will come a day where their machine won't boot
17:26:06 <dustymabe> might go for a really long time, but it will happen
17:26:40 <c4rt0> I need to drop earlier today. As always - it was nice to be a small part of your discussion ;) Thanks all!
17:26:47 <jlebon> i mean, as long as we don't have bootupd automated, we'll have to keep looking out for this and doing barrier releases like we did, which obviously isn't a great place to be in
17:26:50 <bgilbert> CL had one of those too and it was a major fire
17:27:12 <dustymabe> c4rt0: 👋
17:27:29 <dustymabe> i've been warned by jforbes that we're going to have some turbulence soonish if we don't start doing it automatically
17:27:52 <dustymabe> :)
17:27:57 <dustymabe> ok let me try to draw up a proposed
17:28:09 <bgilbert> okay, so if we feel we'll need to do the bootupd work, I think we should likely autoupdate dbx too
17:29:08 <jlebon> given that non-ostree systems do update their EFIs routinely, FCOS/RHCOS are the odd ones out, which puts them more at risk
17:30:08 <jlebon> I should've said non-ostree Fedora-based systems*
17:30:22 <dustymabe> #proposed We've identified that we'd ultimately like to regularly update both the bootloader and dbx. For now there are some preliminary investigations that we can do to determine the scope and challenges related to the dbx updates and there is a fallback feature we'd need to investigate and implement in bootupd.
17:30:48 <dustymabe> btw - is there a better issue for us to group this work under? Not sure if https://github.com/coreos/fedora-coreos-tracker/issues/1452 is the right place
17:31:14 <dustymabe> There I was just mostly trying to document the symptom and cause, but the feature work related to the fix should maybe go in a better defined issue.
17:31:17 <jlebon> dustymabe: what's the fallback feature bit about? maybe keep it more generic?
17:31:37 <bgilbert> +1 jlebon
17:31:47 <dustymabe> jlebon: "fallback feature" == if my bootloader update is foobar then I can boot the old one and get back into my system to recover?
17:32:00 <dustymabe> ideas on more generic wording?
17:32:09 <bgilbert> dustymabe: +1 to a separate issue for feature work
17:32:47 <jlebon> dustymabe: maybe "and there are bootloader update safety features we'd like to investigate and implement in bootupd"
17:32:48 <travier> we need to reach out / sync with bootloader folks about that work
17:32:49 <dustymabe> bgilbert: do we want 2 issues (one for bootloader autoupdates and one for dbx autoupdates)?
17:33:00 <bgilbert> dustymabe: I think so
17:33:07 <bgilbert> jlebon: +1
17:33:21 <dustymabe> #proposed We've identified that we'd ultimately like to regularly update both the bootloader and dbx. For now there are some preliminary investigations that we can do to determine the scope and challenges related to the dbx updates and there are bootloader update safety features we'd like to investigate and implement in bootupd"
17:33:36 <dustymabe> #proposed We've identified that we'd ultimately like to regularly update both the bootloader and dbx. For now there are some preliminary investigations that we can do to determine the scope and challenges related to the dbx updates and there are bootloader update safety features we'd like to investigate and implement in bootupd.
17:34:06 <jlebon> ack
17:34:14 <bgilbert> +1
17:34:24 <dustymabe> any opposed?
17:34:55 <dustymabe> #agreed We've identified that we'd ultimately like to regularly update both the bootloader and dbx. For now there are some preliminary investigations that we can do to determine the scope and challenges related to the dbx updates and there are bootloader update safety features we'd like to investigate and implement in bootupd.
17:35:21 <dustymabe> anyone want to break up things into new issues? if not I'll try
17:35:39 <dustymabe> #topic open floor
17:35:48 <dustymabe> anything for open floor today?
17:36:30 <dustymabe> #info F38 final freeze is now in effect, we'll start producing `next` releases every week in order to get rapid feedback from users on our "readiness" for F38 GA.
17:36:35 <jlebon> dustymabe: i can help with filing stuff
17:36:45 <jlebon> e.g. i can do the bootupd one
17:37:09 <dustymabe> #action jlebon to open a new issue related to the "regular bootloader updates" feature
17:37:57 <dustymabe> #action dustymabe to open a new issue related to the "regular dbx updates" feature
17:38:08 <dustymabe> anything else for open floor ?
17:38:49 <dustymabe> #endmeeting