16:29:54 <dustymabe> #startmeeting fedora_coreos_meeting
16:29:54 <zodbot> Meeting started Wed Aug  5 16:29:54 2020 UTC.
16:29:54 <zodbot> This meeting is logged and archived in a public location.
16:29:54 <zodbot> The chair is dustymabe. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:29:54 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:29:54 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:29:58 <dustymabe> #topic roll call
16:30:02 <dustymabe> .hello2
16:30:02 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:30:13 <cyberpear> .hello2
16:30:13 <zodbot> cyberpear: cyberpear 'James Cassell' <fedoraproject@cyberpear.com>
16:30:20 <darkmuggle> .hello2
16:30:20 <zodbot> darkmuggle: darkmuggle 'None' <me@muggle.dev>
16:30:28 <bgilbert> .hello2
16:30:29 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:30:41 <skunkerk> .hello sohank2602
16:30:42 <zodbot> skunkerk: sohank2602 'Sohan Kunkerkar' <skunkerk@redhat.com>
16:30:50 <jdoss> .hello2
16:30:51 <zodbot> jdoss: jdoss 'Joe Doss' <joe@solidadmin.com>
16:32:05 <nasirhm> .hello2
16:32:06 <zodbot> nasirhm: nasirhm 'Nasir Hussain' <nasirhussainm14@gmail.com>
16:32:29 <dustymabe> #chair cyberpear darkmuggle bgilbert skunkerk jdoss nasirhm
16:32:29 <zodbot> Current chairs: bgilbert cyberpear darkmuggle dustymabe jdoss nasirhm skunkerk
16:33:06 <lucab> .hello2
16:33:07 <zodbot> lucab: lucab 'Luca Bruno' <lucab@redhat.com>
16:33:15 <dustymabe> #chair lucab
16:33:15 <zodbot> Current chairs: bgilbert cyberpear darkmuggle dustymabe jdoss lucab nasirhm skunkerk
16:33:17 <dustymabe> welcome all :)
16:33:20 <dustymabe> #topic Action items from last meeting
16:33:30 <dustymabe> * bgilbert to summarize discussion in the ticket
16:33:43 <dustymabe> I think this was for ticket #586
16:34:00 <jlebon> .hello2
16:34:03 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:34:17 <dustymabe> #chair jlebon
16:34:17 <zodbot> Current chairs: bgilbert cyberpear darkmuggle dustymabe jdoss jlebon lucab nasirhm skunkerk
16:34:20 <lorbus> .hello2
16:34:21 <zodbot> lorbus: lorbus 'Christian Glombek' <cglombek@redhat.com>
16:34:28 <dustymabe> #chair lorbus
16:34:28 <zodbot> Current chairs: bgilbert cyberpear darkmuggle dustymabe jdoss jlebon lorbus lucab nasirhm skunkerk
16:34:33 <dustymabe> bgilbert: should I re-action that?
16:34:46 <bgilbert> whoops
16:34:46 <bgilbert> yes
16:35:05 <dustymabe> #action bgilbert to summarize discussion from last week's meeting in ticket https://github.com/coreos/fedora-coreos-tracker/issues/586
16:35:20 <dustymabe> #topic coreos/enhancements repo
16:35:21 <nasirhm> #link https://github.com/coreos/fedora-coreos-tracker/issues/586
16:36:06 <nasirhm> dustymabe: Sorry, sent it in the wrong topic.
16:36:29 <dustymabe> We have created a new repo for us to have high level and well summarized design discussions for larger changes to *COS
16:36:46 <dustymabe> This repo is intended to be lower volume than fedora-coreos-tracker
16:36:51 <dustymabe> nasirhm: no problem
16:36:53 <dustymabe> #undo
16:36:53 <zodbot> Removing item from minutes: <MeetBot.items.Link object at 0x7fb213f10710>
16:37:02 <dustymabe> #link https://github.com/coreos/enhancements/
16:37:35 <dustymabe> It will allow us to have well thought out change proposals and also a record of them.
16:37:43 <cyberpear> just because tracker got clogged?
16:37:53 <dustymabe> cyberpear: not exactly
16:37:59 <nasirhm> dustymabe: Thank You for undoing it, today learnt meetbot has undo.
16:38:36 <bgilbert> cyberpear: we have a lot of issues and PRs in a lot of places with a lot of discussion in them
16:38:47 <cyberpear> maybe better store of decisions, better docs than old tickets
16:38:49 <dustymabe> cyberpear: it's more or less "we got an RFE in coreos-tracker, or we came up with a proposed large change, here is a summary of it and the proposed solution"
16:38:49 <nasirhm> cyberpear: I believe the initial idea is to keep the *COS enhancements ticket there and use the fcos tracker for FCOS.
16:38:56 <bgilbert> cyberpear: and it's easy to miss high-level design discussions that show up e.g. in the middle of a PR
16:39:05 <cyberpear> sounds good to me. just should announce widely
16:39:06 <bgilbert> so this provides a way to centralize them
16:40:17 <jlebon> i guess there'll be some overlap with https://github.com/coreos/fedora-coreos-tracker/blob/master/Design.md. we could probably move that there
16:40:19 <dustymabe> cyberpear: +1. we should probably send a coreos@ (not status) email
16:41:09 <dustymabe> jlebon: yeah, definitely some overlap with the early Design.md PRs
16:41:18 <dustymabe> similar idea
16:41:29 <dustymabe> not sure if we should move it or not (losing history)
16:42:05 <jlebon> yeah, true
16:42:57 <dustymabe> #info  We have created a new repo for us to have high level and well summarized design discussions for larger changes to *COS. It is intended to be lower volume than fedora-coreos-tracker with concrete proposals and good succinct descriptions of the problem that is trying to be solved. https://github.com/coreos/enhancements/
16:43:02 <dustymabe> ^^ does that look good ?
16:43:09 <jdoss> +1
16:43:29 <nasirhm> +1
16:43:43 <dustymabe> might try to add a little more context in the email we send out
16:43:55 <dustymabe> does anyone disagree we should send an email ?
16:44:37 <jlebon> seems fine to me
16:44:45 <jlebon> gives a chance for people to subscribe
16:44:55 <dustymabe> clarification "send an email to the coreos@" list
16:45:19 <skunkerk> +1
16:46:01 <dustymabe> #action dustymabe to send an email to coreos@ about the coreos/enhancements repo
16:46:30 <dustymabe> #topic K8s CoreDns not starting on 32.20200629.3.0 (Pods on master can't talk to the apiserver)
16:46:36 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/574
16:46:43 <dustymabe> any chance we have dghubble today?
16:47:36 <dustymabe> It looks like there was an issue opened upstream for kubespray, no input there yet
16:47:50 <dustymabe> dghubble: is working around with Ignition since he controls the whole stack
16:48:38 <dustymabe> we'd need to talk to coreos flannel upstream (are they active?) to see what they would want to do maybe?
16:48:57 <dustymabe> and also the flannel rpm in Fedora
16:49:19 <dustymabe> anyone with a proposed path forward on this particular issue?
16:50:53 <dustymabe> looks like we'll wait. if anyone wants to start a conversation with the flannel rpm folks in Fedora that would be great (a PR might be best)
16:50:56 <lucab> not specifically, the flannel bug has been open since sometime. A quick enhancement there would be to at least have the unit in the repo.
16:51:38 <dustymabe> right. maybe we can poke on that bug
16:51:41 * cverna is a bit late but still waives
16:51:50 <dustymabe> #chair cverna
16:51:50 <zodbot> Current chairs: bgilbert cverna cyberpear darkmuggle dustymabe jdoss jlebon lorbus lucab nasirhm skunkerk
16:52:00 <nasirhm> Hey cverna o/
16:52:00 <dustymabe> I'll move on to the next topic
16:52:30 <dustymabe> #topic Response to CVE-2020-10713 (GRUB 2 Boot Hole)
16:52:38 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/587
16:52:52 <dustymabe> So, there was a CVE :)
16:53:20 <jdoss> That you need root level access to exploit IIRC.
16:53:50 <dustymabe> right, you need privileged access to the machine IIUC so it's not an "end of the world bug"
16:54:20 <dustymabe> as walters said in the issue no one should try to fix this bug in a panic
16:54:23 <cyberpear> agreed.  Put off "fixing" this one until we're 100% sure we won't have breakage
16:54:49 <dustymabe> once we do have a fix. there are two proposed options
16:54:56 <cyberpear> only deadline is when vendors start pushing firmware updates with the Fedora grub2 in the dbx database
16:54:57 <dustymabe> 1. have users reprovision
16:55:16 <dustymabe> 2. provide some manual steps they can run to update the bootloader
16:55:35 <dustymabe> colin is working on a tool called bootupd, which can handle boot loader updates in the future
16:55:47 <dustymabe> so +1 for the future :)
16:55:47 <jdoss> prob want both options TBH
16:56:07 <cyberpear> 2 sounds nice to have for users treating their systems as pets, and 1 should always be an option, IMO
16:56:09 <dustymabe> jlebon: do you know how complicated those manual steps would be ?
16:56:31 <dustymabe> cyberpear: yeah we get 1. for free because that already just works
16:56:48 <jlebon> dustymabe: i'd think it would just be copying a couple of files from /usr to /boot but not sure
16:56:50 <nasirhm> As it's a security update, I'm a +1 to have users provision.
16:57:07 <pjones> cyberpear: that shouldn't happen.  it will, but it shouldn't.
16:57:27 <pjones> cyberpear: when it does, we might have some leverage to make it /un/-happen
16:57:37 <dustymabe> pjones: which thing shouldn't happen?
16:57:45 <dustymabe> and welcome pjones :)
16:57:46 <pjones> firmware updates adding us to dbx
16:58:14 <dustymabe> ahh interesting, i think I just assumed that would happen for any affected bootloader
16:58:21 <cyberpear> I'd expect at least new systems shipping after XYZ date would include the vulnerable grub2 signatures/hashes
16:58:41 <pjones> MS has told us that they're changing their logo guidelines to specifically ban firmware updating dbx past the 2016.08.09 update; that's an /installed/ OS's job.
16:58:54 <cyberpear> interesting
16:58:59 <cyberpear> that's very good to know
16:59:15 <dustymabe> pjones: so the question is, would we add ourselves to dbx ?
16:59:17 <dustymabe> haha
16:59:23 <pjones> dustymabe: absolutely yes
16:59:39 <dustymabe> but then we'd be ok
16:59:45 <cyberpear> so, all systems shipping will be known-broken from Secure Boot standpoint, modulo the installed OS
16:59:49 <dustymabe> because at the same time we updated dbx we'd also be using a newer bootloader
17:00:09 * cyberpear steps away for another meeting
17:00:29 <nasirhm> Thanks for joining cyberpear
17:00:35 <pjones> okay so we're changing dbxtool so that it doesn't apply just whatever it finds, but rather knows which thing is the current recommendation,
17:01:14 <dustymabe> good to know
17:01:19 <pjones> which means we can add the new dbx stuff without /applying/ it, and instruct admins that if they're really high value targets, they should apply the new dbx on a single machine of each class across their fleet and make sure it works before rolling it out to the full fleet,
17:01:27 <pjones> and then quite some time after introducing it, we can move the default
17:01:55 <pjones> and once we have that, this problem basically only affects re-installs using old media
17:02:00 <dustymabe> so.. what do you think our recommendation to our users should be at this time ?
17:02:23 <pjones> our users at this time is if you are a very high value target, you should investigate applying the new dbx once we add it to the fedora packages.
17:02:32 <dustymabe> we had discussed an option to give them manual steps to run
17:02:37 <bgilbert> pjones: in our case, that should interlock with actually having an updated bootloader.  will there be a mechanism for that?
17:02:48 <bgilbert> or would we hold back the default until we have a mechanism for bootloader updates
17:03:56 <pjones> the reality of the situation is that all of the OS's security features and mitigations are between an attacker and this attack surface, and the kaspersky grub2 build that doesn't check signatures right from ~2 years ago is in this same set of dbx updates
17:04:07 <pjones> nobody is burning 0 days to get to the point where they can use this; if they were, they'd have already done it.
17:04:24 <pjones> bgilbert: a mechanism for which thing exactly?
17:05:09 <bgilbert> FCOS doesn't currently update GRUB on upgrades, so we'd need to hold back the new dbx as a default in FCOS until we've done so
17:05:16 <pjones> right.
17:05:18 <bgilbert> if the Fedora dbx default won't change for a long time, it's moot
17:05:22 <pjones> so right now the only mechanism I have is package deps?
17:05:52 <pjones> I don't think the rpm people are particularly open to my ideas of making db/dbx entries into synthetic runtime provides/requires
17:06:24 <pjones> so the fedora dbxtool will get an update in the next week or so, but it won't apply the new dbx by default
17:06:33 <pjones> it'll just be present on disk
17:06:49 <dustymabe> +1
17:07:05 <bgilbert> right, and you mentioned changing that default in future
17:07:29 <bgilbert> do we need to be concerned about the timing of that vs. us fixing our bootloader-update problem
17:08:01 <nasirhm> +1
17:08:44 <pjones> and when we finally enable it by default, we'll make that version of dbxtool conflict with shim-x64 < 15.15 , which will conflict with grub2 < 2.04-27
17:09:02 <pjones> bgilbert: if you could fix your issue in the next 6 months or so that'd be great?  and we should keep in touch about it.
17:09:10 <bgilbert> pjones: +1
17:09:14 <pjones> I'm not going to roll that dbxtool out without checking in
17:09:19 <bgilbert> cool
17:09:28 <pjones> but at some point it will need to be done
17:09:36 <bgilbert> yup
17:09:44 <dustymabe> hmm
17:10:09 <dustymabe> let me throw out a timeline and help me fill in the details
17:10:32 <dustymabe> 1. next week we ship an updated dbx, it doesn't apply it to the system, just exists on disk
17:10:34 * pjones is on his millionth version of this conversation, so if you have any more questions I probably already have at least vague answers
17:11:03 <dustymabe> 2. the user in this case is passive (doesn't manually do anything)
17:11:04 <jdoss> We appreciate you pjones!
17:11:09 <pjones> aww
17:11:33 <dustymabe> 3. Fedora 35 is here, dbxtool now applies a new recommended default dbx that disables the grub2 from fedora 32
17:11:35 <nasirhm> pjones++
17:11:36 <zodbot> nasirhm: Karma for pjones changed to 1 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:11:41 <dustymabe> 4. the user never updated their bootloader
17:11:51 <dustymabe> does their system stop booting?
17:12:05 <pjones> well, they shouldn't be able to install that dbxtool version
17:12:08 <dustymabe> did we solve the "bootloader update by automated tool" problem in there somewhere?
17:12:23 <dustymabe> pjones: what stops them from installing that dbxtool version?:
17:12:29 <pjones> of course I'm still thinking about that in non-fcos terms
17:12:45 <pjones> <pjones> and when we finally enable it by default, we'll make that version of dbxtool conflict with shim-x64 < 15.15 , which will conflict with grub2 < 2.04-27
17:12:57 <dustymabe> i don't think rpm versions matter here
17:13:07 <dustymabe> do they ?
17:13:16 <pjones> well I would hope you all don't generate updates that are built from a packageset that can't verify?
17:13:22 <dustymabe> if I'm on Fedora 32 and I update to Fedora 33 does the bootloader get updated ?
17:13:35 <pjones> on efi on non-fcos, yes
17:13:51 <bgilbert> walters: jlebon: does rpm-ostree update packaged files in /boot/efi?
17:14:09 <dustymabe> super dumb question (sorry): does this CVE only affect UEFI ?
17:14:20 <pjones> dbxtool only affects EFI.
17:14:25 <jlebon> bgilbert: it's under /usr/lib/ostree-boot. /boot isn't updated post install
17:14:25 <dustymabe> ok cool
17:14:58 <dustymabe> jlebon: so that represents a difference between us and traditional fedora ?
17:15:08 <pjones> the grub CVE effects non-uefi in as much as if you're e.g. netbooting then a) you should probably not be using any protocol other than https, and b) you better trust the admins of the machine you're getting grub.cfg from
17:15:09 <jlebon> correct
17:15:15 <pjones> because in that usage it's still a root hole
17:15:29 <dustymabe> pjones: +1
17:15:37 <pjones> but in any other usage BIOS simply does not enforce any protections against the Secure Boot threat model at all
17:15:49 <dustymabe> jlebon: could we use the same mechanism that traditional fedora does ?
17:15:57 <dustymabe> i.e. why do we need bootupd ?
17:16:18 <bgilbert> <jlebon> bgilbert: it's under /usr/lib/ostree-boot. /boot isn't updated post install
17:16:35 <dustymabe> right, why not?
17:16:47 <jlebon> i can't speak for walters, but IIUC the reason is that you can't update the bootloader atomically, because it's FAT
17:17:04 <pjones> right, so you'll need to figure out some answer to that before you include dbxtool-10-1 (or whatever version we call it) in upgrades
17:17:14 <jlebon> and in general it's not something we'd want to do on every update
17:17:43 <dustymabe> ok, for some reason I thought bootupd was solving a problem for traditional systems as well, not just OSTree
17:17:47 <dustymabe> guess I was wrong
17:17:49 <jlebon> there's a lot of backstory in https://github.com/ostreedev/ostree/pull/1873
17:18:10 <dustymabe> jlebon++
17:18:11 <pjones> jlebon: I don't think being on FAT *should* stop you there, but it'll take some work to avoid it because that filesystem has a lot of stupid.  I'm definitely able to help.
17:18:23 <pjones> (I have, sadly, hacked on FAT before)
17:18:36 <dustymabe> pjones: thank you so much for explaining this at a high level for us
17:18:43 <bgilbert> pjones++
17:18:44 <zodbot> bgilbert: Karma for pjones changed to 2 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:18:53 <pjones> dustymabe: no worries, I'm here to help
17:18:57 <jlebon> pjones: ack, thanks!  the tracker issue is https://github.com/coreos/fedora-coreos-tracker/issues/510
17:19:02 <pjones> dustymabe: also in case it isn't obvious this is why I was blowing you off 2 weeks ago
17:19:27 <dustymabe> pjones: no worries :)
17:19:43 <dustymabe> you didn't blow me off. You gave me good information to know where to look next.
17:19:50 <dustymabe> which is all that is ever needed
17:19:53 <dustymabe> :)
17:19:57 <pjones> look at it as you will ;)
17:20:15 <dustymabe> I'll summarize this discussion in the ticket. So much learned here
17:20:31 <dustymabe> #topic open floor
17:20:41 * pjones goes away again
17:20:41 <dustymabe> look at that - actual time for open floor!
17:20:54 <nasirhm> I have one regarding the tutorial docs
17:21:41 <nasirhm> The advanced provisioning scenario which provisions a podman container with etcd seems to be failing.
17:22:30 <nasirhm> I will try fixing it today and create a fix.
17:23:11 <nasirhm> also travier++ , thanks for merging the PR in docs.
17:23:24 <dustymabe> nasirhm: +1 - podman has changed some since january so we might need to edit some settings in the systemd unit
17:23:35 <dustymabe> nasirhm++
17:23:38 <dustymabe> lucab++
17:23:41 <dustymabe> travier++
17:23:48 <dustymabe> nice work on the documentation
17:23:56 <nasirhm> dustymabe: Thanks for providing the pointer.
17:24:09 <nasirhm> lucab++ thanks for the reviews on the PRs.
17:24:24 <lucab> nasirhm: see also https://github.com/coreos/fedora-coreos-docs/pull/75
17:24:41 <nasirhm> and dustymabe++ for providing the initial blog post.
17:25:10 <lucab> podman - systemd integration is a bit in a suboptimal place since sometime
17:26:53 <dustymabe> any more topics for open floor ?
17:27:01 * nasirhm sent a long message:  < https://matrix.org/_matrix/media/r0/download/matrix.org/YMygNtLmrRwsDTUdgrMDohnJ >
17:27:48 <nasirhm> Nothing from my side. Thank You everyone for joining and dustymabe for running the meeting.
17:28:02 <dustymabe> #endmeeting