16:29:54 #startmeeting fedora_coreos_meeting 16:29:54 Meeting started Wed Aug 5 16:29:54 2020 UTC. 16:29:54 This meeting is logged and archived in a public location. 16:29:54 The chair is dustymabe. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:29:54 Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:29:54 The meeting name has been set to 'fedora_coreos_meeting' 16:29:58 #topic roll call 16:30:02 .hello2 16:30:02 dustymabe: dustymabe 'Dusty Mabe' 16:30:13 .hello2 16:30:13 cyberpear: cyberpear 'James Cassell' 16:30:20 .hello2 16:30:20 darkmuggle: darkmuggle 'None' 16:30:28 .hello2 16:30:29 bgilbert: bgilbert 'Benjamin Gilbert' 16:30:41 .hello sohank2602 16:30:42 skunkerk: sohank2602 'Sohan Kunkerkar' 16:30:50 .hello2 16:30:51 jdoss: jdoss 'Joe Doss' 16:32:05 .hello2 16:32:06 nasirhm: nasirhm 'Nasir Hussain' 16:32:29 #chair cyberpear darkmuggle bgilbert skunkerk jdoss nasirhm 16:32:29 Current chairs: bgilbert cyberpear darkmuggle dustymabe jdoss nasirhm skunkerk 16:33:06 .hello2 16:33:07 lucab: lucab 'Luca Bruno' 16:33:15 #chair lucab 16:33:15 Current chairs: bgilbert cyberpear darkmuggle dustymabe jdoss lucab nasirhm skunkerk 16:33:17 welcome all :) 16:33:20 #topic Action items from last meeting 16:33:30 * bgilbert to summarize discussion in the ticket 16:33:43 I think this was for ticket #586 16:34:00 .hello2 16:34:03 jlebon: jlebon 'None' 16:34:17 #chair jlebon 16:34:17 Current chairs: bgilbert cyberpear darkmuggle dustymabe jdoss jlebon lucab nasirhm skunkerk 16:34:20 .hello2 16:34:21 lorbus: lorbus 'Christian Glombek' 16:34:28 #chair lorbus 16:34:28 Current chairs: bgilbert cyberpear darkmuggle dustymabe jdoss jlebon lorbus lucab nasirhm skunkerk 16:34:33 bgilbert: should I re-action that? 16:34:46 whoops 16:34:46 yes 16:35:05 #action bgilbert to summarize discussion from last week's meeting in ticket https://github.com/coreos/fedora-coreos-tracker/issues/586 16:35:20 #topic coreos/enhancements repo 16:35:21 #link https://github.com/coreos/fedora-coreos-tracker/issues/586 16:36:06 dustymabe: Sorry, sent it in the wrong topic. 16:36:29 We have created a new repo for us to have high level and well summarized design discussions for larger changes to *COS 16:36:46 This repo is intended to be lower volume than fedora-coreos-tracker 16:36:51 nasirhm: no problem 16:36:53 #undo 16:36:53 Removing item from minutes: 16:37:02 #link https://github.com/coreos/enhancements/ 16:37:35 It will allow us to have well thought out change proposals and also a record of them. 16:37:43 just because tracker got clogged? 16:37:53 cyberpear: not exactly 16:37:59 dustymabe: Thank You for undoing it, today learnt meetbot has undo. 16:38:36 cyberpear: we have a lot of issues and PRs in a lot of places with a lot of discussion in them 16:38:47 maybe better store of decisions, better docs than old tickets 16:38:49 cyberpear: it's more or less "we got an RFE in coreos-tracker, or we came up with a proposed large change, here is a summary of it and the proposed solution" 16:38:49 cyberpear: I believe the initial idea is to keep the *COS enhancements ticket there and use the fcos tracker for FCOS. 16:38:56 cyberpear: and it's easy to miss high-level design discussions that show up e.g. in the middle of a PR 16:39:05 sounds good to me. just should announce widely 16:39:06 so this provides a way to centralize them 16:40:17 i guess there'll be some overlap with https://github.com/coreos/fedora-coreos-tracker/blob/master/Design.md. we could probably move that there 16:40:19 cyberpear: +1. we should probably send a coreos@ (not status) email 16:41:09 jlebon: yeah, definitely some overlap with the early Design.md PRs 16:41:18 similar idea 16:41:29 not sure if we should move it or not (losing history) 16:42:05 yeah, true 16:42:57 #info We have created a new repo for us to have high level and well summarized design discussions for larger changes to *COS. It is intended to be lower volume than fedora-coreos-tracker with concrete proposals and good succinct descriptions of the problem that is trying to be solved. https://github.com/coreos/enhancements/ 16:43:02 ^^ does that look good ? 16:43:09 +1 16:43:29 +1 16:43:43 might try to add a little more context in the email we send out 16:43:55 does anyone disagree we should send an email ? 16:44:37 seems fine to me 16:44:45 gives a chance for people to subscribe 16:44:55 clarification "send an email to the coreos@" list 16:45:19 +1 16:46:01 #action dustymabe to send an email to coreos@ about the coreos/enhancements repo 16:46:30 #topic K8s CoreDns not starting on 32.20200629.3.0 (Pods on master can't talk to the apiserver) 16:46:36 #link https://github.com/coreos/fedora-coreos-tracker/issues/574 16:46:43 any chance we have dghubble today? 16:47:36 It looks like there was an issue opened upstream for kubespray, no input there yet 16:47:50 dghubble: is working around with Ignition since he controls the whole stack 16:48:38 we'd need to talk to coreos flannel upstream (are they active?) to see what they would want to do maybe? 16:48:57 and also the flannel rpm in Fedora 16:49:19 anyone with a proposed path forward on this particular issue? 16:50:53 looks like we'll wait. if anyone wants to start a conversation with the flannel rpm folks in Fedora that would be great (a PR might be best) 16:50:56 not specifically, the flannel bug has been open since sometime. A quick enhancement there would be to at least have the unit in the repo. 16:51:38 right. maybe we can poke on that bug 16:51:41 * cverna is a bit late but still waives 16:51:50 #chair cverna 16:51:50 Current chairs: bgilbert cverna cyberpear darkmuggle dustymabe jdoss jlebon lorbus lucab nasirhm skunkerk 16:52:00 Hey cverna o/ 16:52:00 I'll move on to the next topic 16:52:30 #topic Response to CVE-2020-10713 (GRUB 2 Boot Hole) 16:52:38 #link https://github.com/coreos/fedora-coreos-tracker/issues/587 16:52:52 So, there was a CVE :) 16:53:20 That you need root level access to exploit IIRC. 16:53:50 right, you need privileged access to the machine IIUC so it's not an "end of the world bug" 16:54:20 as walters said in the issue no one should try to fix this bug in a panic 16:54:23 agreed. Put off "fixing" this one until we're 100% sure we won't have breakage 16:54:49 once we do have a fix. there are two proposed options 16:54:56 only deadline is when vendors start pushing firmware updates with the Fedora grub2 in the dbx database 16:54:57 1. have users reprovision 16:55:16 2. provide some manual steps they can run to update the bootloader 16:55:35 colin is working on a tool called bootupd, which can handle boot loader updates in the future 16:55:47 so +1 for the future :) 16:55:47 prob want both options TBH 16:56:07 2 sounds nice to have for users treating their systems as pets, and 1 should always be an option, IMO 16:56:09 jlebon: do you know how complicated those manual steps would be ? 16:56:31 cyberpear: yeah we get 1. for free because that already just works 16:56:48 dustymabe: i'd think it would just be copying a couple of files from /usr to /boot but not sure 16:56:50 As it's a security update, I'm a +1 to have users provision. 16:57:07 cyberpear: that shouldn't happen. it will, but it shouldn't. 16:57:27 cyberpear: when it does, we might have some leverage to make it /un/-happen 16:57:37 pjones: which thing shouldn't happen? 16:57:45 and welcome pjones :) 16:57:46 firmware updates adding us to dbx 16:58:14 ahh interesting, i think I just assumed that would happen for any affected bootloader 16:58:21 I'd expect at least new systems shipping after XYZ date would include the vulnerable grub2 signatures/hashes 16:58:41 MS has told us that they're changing their logo guidelines to specifically ban firmware updating dbx past the 2016.08.09 update; that's an /installed/ OS's job. 16:58:54 interesting 16:58:59 that's very good to know 16:59:15 pjones: so the question is, would we add ourselves to dbx ? 16:59:17 haha 16:59:23 dustymabe: absolutely yes 16:59:39 but then we'd be ok 16:59:45 so, all systems shipping will be known-broken from Secure Boot standpoint, modulo the installed OS 16:59:49 because at the same time we updated dbx we'd also be using a newer bootloader 17:00:09 * cyberpear steps away for another meeting 17:00:29 Thanks for joining cyberpear 17:00:35 okay so we're changing dbxtool so that it doesn't apply just whatever it finds, but rather knows which thing is the current recommendation, 17:01:14 good to know 17:01:19 which means we can add the new dbx stuff without /applying/ it, and instruct admins that if they're really high value targets, they should apply the new dbx on a single machine of each class across their fleet and make sure it works before rolling it out to the full fleet, 17:01:27 and then quite some time after introducing it, we can move the default 17:01:55 and once we have that, this problem basically only affects re-installs using old media 17:02:00 so.. what do you think our recommendation to our users should be at this time ? 17:02:23 our users at this time is if you are a very high value target, you should investigate applying the new dbx once we add it to the fedora packages. 17:02:32 we had discussed an option to give them manual steps to run 17:02:37 pjones: in our case, that should interlock with actually having an updated bootloader. will there be a mechanism for that? 17:02:48 or would we hold back the default until we have a mechanism for bootloader updates 17:03:56 the reality of the situation is that all of the OS's security features and mitigations are between an attacker and this attack surface, and the kaspersky grub2 build that doesn't check signatures right from ~2 years ago is in this same set of dbx updates 17:04:07 nobody is burning 0 days to get to the point where they can use this; if they were, they'd have already done it. 17:04:24 bgilbert: a mechanism for which thing exactly? 17:05:09 FCOS doesn't currently update GRUB on upgrades, so we'd need to hold back the new dbx as a default in FCOS until we've done so 17:05:16 right. 17:05:18 if the Fedora dbx default won't change for a long time, it's moot 17:05:22 so right now the only mechanism I have is package deps? 17:05:52 I don't think the rpm people are particularly open to my ideas of making db/dbx entries into synthetic runtime provides/requires 17:06:24 so the fedora dbxtool will get an update in the next week or so, but it won't apply the new dbx by default 17:06:33 it'll just be present on disk 17:06:49 +1 17:07:05 right, and you mentioned changing that default in future 17:07:29 do we need to be concerned about the timing of that vs. us fixing our bootloader-update problem 17:08:01 +1 17:08:44 and when we finally enable it by default, we'll make that version of dbxtool conflict with shim-x64 < 15.15 , which will conflict with grub2 < 2.04-27 17:09:02 bgilbert: if you could fix your issue in the next 6 months or so that'd be great? and we should keep in touch about it. 17:09:10 pjones: +1 17:09:14 I'm not going to roll that dbxtool out without checking in 17:09:19 cool 17:09:28 but at some point it will need to be done 17:09:36 yup 17:09:44 hmm 17:10:09 let me throw out a timeline and help me fill in the details 17:10:32 1. next week we ship an updated dbx, it doesn't apply it to the system, just exists on disk 17:10:34 * pjones is on his millionth version of this conversation, so if you have any more questions I probably already have at least vague answers 17:11:03 2. the user in this case is passive (doesn't manually do anything) 17:11:04 We appreciate you pjones! 17:11:09 aww 17:11:33 3. Fedora 35 is here, dbxtool now applies a new recommended default dbx that disables the grub2 from fedora 32 17:11:35 pjones++ 17:11:36 nasirhm: Karma for pjones changed to 1 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 17:11:41 4. the user never updated their bootloader 17:11:51 does their system stop booting? 17:12:05 well, they shouldn't be able to install that dbxtool version 17:12:08 did we solve the "bootloader update by automated tool" problem in there somewhere? 17:12:23 pjones: what stops them from installing that dbxtool version?: 17:12:29 of course I'm still thinking about that in non-fcos terms 17:12:45 and when we finally enable it by default, we'll make that version of dbxtool conflict with shim-x64 < 15.15 , which will conflict with grub2 < 2.04-27 17:12:57 i don't think rpm versions matter here 17:13:07 do they ? 17:13:16 well I would hope you all don't generate updates that are built from a packageset that can't verify? 17:13:22 if I'm on Fedora 32 and I update to Fedora 33 does the bootloader get updated ? 17:13:35 on efi on non-fcos, yes 17:13:51 walters: jlebon: does rpm-ostree update packaged files in /boot/efi? 17:14:09 super dumb question (sorry): does this CVE only affect UEFI ? 17:14:20 dbxtool only affects EFI. 17:14:25 bgilbert: it's under /usr/lib/ostree-boot. /boot isn't updated post install 17:14:25 ok cool 17:14:58 jlebon: so that represents a difference between us and traditional fedora ? 17:15:08 the grub CVE effects non-uefi in as much as if you're e.g. netbooting then a) you should probably not be using any protocol other than https, and b) you better trust the admins of the machine you're getting grub.cfg from 17:15:09 correct 17:15:15 because in that usage it's still a root hole 17:15:29 pjones: +1 17:15:37 but in any other usage BIOS simply does not enforce any protections against the Secure Boot threat model at all 17:15:49 jlebon: could we use the same mechanism that traditional fedora does ? 17:15:57 i.e. why do we need bootupd ? 17:16:18 bgilbert: it's under /usr/lib/ostree-boot. /boot isn't updated post install 17:16:35 right, why not? 17:16:47 i can't speak for walters, but IIUC the reason is that you can't update the bootloader atomically, because it's FAT 17:17:04 right, so you'll need to figure out some answer to that before you include dbxtool-10-1 (or whatever version we call it) in upgrades 17:17:14 and in general it's not something we'd want to do on every update 17:17:43 ok, for some reason I thought bootupd was solving a problem for traditional systems as well, not just OSTree 17:17:47 guess I was wrong 17:17:49 there's a lot of backstory in https://github.com/ostreedev/ostree/pull/1873 17:18:10 jlebon++ 17:18:11 jlebon: I don't think being on FAT *should* stop you there, but it'll take some work to avoid it because that filesystem has a lot of stupid. I'm definitely able to help. 17:18:23 (I have, sadly, hacked on FAT before) 17:18:36 pjones: thank you so much for explaining this at a high level for us 17:18:43 pjones++ 17:18:44 bgilbert: Karma for pjones changed to 2 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 17:18:53 dustymabe: no worries, I'm here to help 17:18:57 pjones: ack, thanks! the tracker issue is https://github.com/coreos/fedora-coreos-tracker/issues/510 17:19:02 dustymabe: also in case it isn't obvious this is why I was blowing you off 2 weeks ago 17:19:27 pjones: no worries :) 17:19:43 you didn't blow me off. You gave me good information to know where to look next. 17:19:50 which is all that is ever needed 17:19:53 :) 17:19:57 look at it as you will ;) 17:20:15 I'll summarize this discussion in the ticket. So much learned here 17:20:31 #topic open floor 17:20:41 * pjones goes away again 17:20:41 look at that - actual time for open floor! 17:20:54 I have one regarding the tutorial docs 17:21:41 The advanced provisioning scenario which provisions a podman container with etcd seems to be failing. 17:22:30 I will try fixing it today and create a fix. 17:23:11 also travier++ , thanks for merging the PR in docs. 17:23:24 nasirhm: +1 - podman has changed some since january so we might need to edit some settings in the systemd unit 17:23:35 nasirhm++ 17:23:38 lucab++ 17:23:41 travier++ 17:23:48 nice work on the documentation 17:23:56 dustymabe: Thanks for providing the pointer. 17:24:09 lucab++ thanks for the reviews on the PRs. 17:24:24 nasirhm: see also https://github.com/coreos/fedora-coreos-docs/pull/75 17:24:41 and dustymabe++ for providing the initial blog post. 17:25:10 podman - systemd integration is a bit in a suboptimal place since sometime 17:26:53 any more topics for open floor ? 17:27:01 * nasirhm sent a long message: < https://matrix.org/_matrix/media/r0/download/matrix.org/YMygNtLmrRwsDTUdgrMDohnJ > 17:27:48 Nothing from my side. Thank You everyone for joining and dustymabe for running the meeting. 17:28:02 #endmeeting