16:29:10 <ajeddeloh> #startmeeting fedora_coreos_meeting
16:29:10 <zodbot> Meeting started Wed Oct  9 16:29:10 2019 UTC.
16:29:10 <zodbot> This meeting is logged and archived in a public location.
16:29:10 <zodbot> The chair is ajeddeloh. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:29:10 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:29:10 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:29:18 <ajeddeloh> #topic roll call
16:29:21 <slowrie> .hello2
16:29:22 <ajeddeloh> .hello2
16:29:22 <zodbot> slowrie: slowrie 'Stephen Lowrie' <slowrie@redhat.com>
16:29:26 <zodbot> ajeddeloh: ajeddeloh 'Andrew Jeddeloh' <andrew.jeddeloh@redhat.com>
16:29:30 <ajeddeloh> @chair slowrie ajeddeloh
16:29:35 <ajeddeloh> #chair slowrie ajeddeloh
16:29:35 <zodbot> Current chairs: ajeddeloh slowrie
16:29:44 <ajeddeloh> I can type, I swear...
16:30:00 <strigazi> .hello2
16:30:00 <zodbot> strigazi: strigazi 'Spyros Trigazis' <strigazi@gmail.com>
16:30:28 <ajeddeloh> #chair strigazi
16:30:28 <zodbot> Current chairs: ajeddeloh slowrie strigazi
16:31:45 <bgilbert> .hello2
16:31:46 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:31:51 <ajeddeloh> #chair bgilbert
16:31:51 <zodbot> Current chairs: ajeddeloh bgilbert slowrie strigazi
16:31:53 <geoff-> geoff-: Geoff Levand <geoff@infradead.org>
16:32:04 <ajeddeloh> #chair geoff-
16:32:04 <zodbot> Current chairs: ajeddeloh bgilbert geoff- slowrie strigazi
16:32:23 <jlebon> .hello2
16:32:24 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:32:39 <ajeddeloh> #chair jlebon
16:32:39 <zodbot> Current chairs: ajeddeloh bgilbert geoff- jlebon slowrie strigazi
16:32:48 <jlebon> small group today :)
16:32:59 <ajeddeloh> yeah
16:33:09 <ajeddeloh> #topic Action items from last meeting
16:33:16 <ajeddeloh> There are none :)
16:33:46 <ajeddeloh> #topic cgroups v2 strategy
16:33:53 <ajeddeloh> #link https://github.com/coreos/fedora-coreos-tracker/issues/292
16:34:19 <ajeddeloh> bgilbert: you filed this and added it to the meeting agenda, you want to introduce it?
16:34:26 <bgilbert> sure
16:34:52 <bgilbert> in summary: Fedora 31 is switching to cgroups v2, in order to push tooling to support it
16:35:12 <bgilbert> presumably that will not go perfectly at first
16:35:42 <bgilbert> so we may want to keep FCOS on cgroups v1 for the initial stable release
16:36:04 <bgilbert> but then, when we do want to switch, that will be a compat break
16:36:14 <strigazi> as a user +1 to that
16:36:35 <ajeddeloh> To what degree can we have both?
16:36:36 <bgilbert> I don't have a concrete question here; just wanted to raise awareness
16:37:02 <bgilbert> ajeddeloh: when I say "v1", technically that's hybrid mode
16:37:02 <slowrie> bgilbert: did you have an initial timeline in mind of when we'd switch to v2?
16:37:04 <ajeddeloh> is it something can be switched at runtime? boottime? not supported both at all?
16:37:08 <slowrie> e.x.: would it wait until f32?
16:37:19 <bgilbert> in which systemd uses v2 and everyone else uses v1
16:37:34 <bgilbert> it's configured via a karg, so boot time but not runtime
16:37:39 <jlebon> what is the nature of the break? presumably when we switch, it means v2 support in tools have matured, right?
16:37:47 <bgilbert> but only one mode per boot
16:37:56 <ajeddeloh> gotcha
16:38:12 <jlebon> so would it only actually affect users who interact with cgroups directly?
16:38:25 <bgilbert> slowrie: no timeline in mind.  I'm guessing v1 support will not rot very quickly because of the various LTS distros
16:38:58 <bgilbert> jlebon: I've heard that the cgroup layout is exposed directly to some containers, but I don't know details
16:39:44 <bgilbert> also, presumably, any custom user scripts (or management tooling in a privileged container) that work with cgroups directly
16:40:37 <jlebon> gotcha (heh, even cosa queries cgroupfs directly)
16:41:06 <bgilbert> if it's not plausible to ship with v2, I'd guess we should wait a while
16:41:13 <bgilbert> before switching
16:41:22 <ajeddeloh> Can you think of anything we can do now to make the transition easier when it comes?
16:41:45 <bgilbert> I don't know of anything, but I also don't know the implications very well
16:42:57 <ajeddeloh> Does anyone think we should switch to v2 immediately?
16:44:21 <ajeddeloh> I'll take that silence as a "no"?
16:44:27 <bgilbert> +1
16:44:56 <jlebon> well
16:45:34 <jlebon> do we want more clarity of how broken things would be?
16:45:49 <ajeddeloh> Thoughts on assigning an action item to investigate if there's anything we can do now to help later?
16:45:53 <jlebon> the change proposal lists consumers that will need to be adapted: https://fedoraproject.org/wiki/Changes/CGroupsV2#Scope
16:46:30 <jlebon> so the intention is that those tools do support v2 when it gets rolled out
16:46:57 <bgilbert> honestly, I think I'd rather not take the risk
16:47:04 <brtknr> is there a meeting here?
16:47:08 <ajeddeloh> yes
16:47:11 <bgilbert> the migration from CL/AH to FCOS will already be pretty substantial
16:47:24 <ajeddeloh> bgilbert++
16:47:24 <zodbot> ajeddeloh: Karma for bgilbert changed to 11 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
16:47:25 <bgilbert> and every speed bump we add will make it harder
16:47:27 <brtknr> CL?
16:47:33 <bgilbert> CoreOS Container Linux
16:47:38 <brtknr> ah okay
16:47:46 <jlebon> bgilbert: you can use that argument for justifying breaking right from the start :)
16:47:57 <jlebon> instead of later down the road
16:48:22 <bgilbert> true
16:48:31 <brtknr> dustymabe: im having issues with flannel on coreos... without selinux disabled, flannel daemonset is not able to create /run/flannel/subnet.env
16:48:33 <brtknr> attempting to apply this: https://github.com/coreos/flannel/blob/master/Documentation/kube-flannel.yml
16:48:35 <brtknr> the same works fine on fedora atomic 29
16:48:41 <bgilbert> brtknr: please wait for open floor
16:48:46 <bgilbert> also, dustymabe is out today
16:48:50 <brtknr> posting this as an example of migration issues^
16:49:06 <brtknr> sorry i was copying from log
16:49:42 <jlebon> bgilbert: not saying we should start with v2, but it sounds like we should discuss with people that have a better grasp of what it'll be like?
16:50:05 <bgilbert> jlebon: Dan Walsh has already suggested that we should start with v1 :-)
16:50:16 <bgilbert> but yes, more info wouldn't hurt
16:50:18 <jlebon> oh?  missed that :)
16:50:40 <ajeddeloh> We should ask him if there's anything we can do now to make migration in the future easier
16:50:53 <jlebon> anyway, didn't mean to belabour this
16:51:02 <ajeddeloh> it would also be ideal if users nodes that started on v1 don't update to v2 automatically
16:51:22 <bgilbert> right
16:52:27 <ajeddeloh> anyone want to take action items to ask DW if there's anything we can do to help migration or draft a plan to prevent auto-switching on update?
16:53:38 <ajeddeloh> I can take one if no one is interested
16:54:28 <bgilbert> +1
16:54:49 <jlebon> +1 :)
16:55:08 <ajeddeloh> #action ajeddeloh to draft a plan to prevent auto-switching from cgroups v1 to v2 on update
16:55:48 <jlebon> i can ping dan on the issue
16:55:54 <jlebon> re. migration
16:56:22 <ajeddeloh> #action jlebon to ask Dan Walsh if there's anything FCOS can do now to make migration to v2 easier in the future
16:56:39 <ajeddeloh> anyone have anything else for this topic?
16:56:48 <bgilbert> not from my end
16:57:13 <ajeddeloh> #topic disk encryption
16:57:20 <ajeddeloh> #link https://github.com/coreos/fedora-coreos-tracker/issues/287
16:57:39 <ajeddeloh> I think this was left on from last week, does anyone have anything new or should we move on?
16:58:01 <jlebon> +1, don't think there's more on this topic right now
16:58:58 <ajeddeloh> #topic FCOS as Kubernetes / OKD node
16:59:07 <ajeddeloh> #link https://github.com/coreos/fedora-coreos-tracker/issues/93
16:59:21 <ajeddeloh> same with this one; does anyone have anything new?
17:00:51 <jlebon> doesn't sound like it
17:01:01 <ajeddeloh> #topic open floor
17:01:41 <jlebon> i've got a topic
17:02:25 <jlebon> while debugging automated OSTree signing with releng and robosignatory devs issues, the question of which key to sign with came up
17:02:49 <jlebon> i think we discussed this, but never actually reached a conclusion
17:03:06 <bgilbert> I thought we had decided to use the one corresponding to the Fedora major?
17:03:11 <jlebon> basically, either we use the primary Fedora keys (same as the rest of Fedora), or a CoreOS-specific key
17:03:26 * ajeddeloh doesn't feel strongly
17:03:30 <jlebon> did we? https://github.com/coreos/fedora-coreos-tracker/issues/187 doesn't list that
17:03:38 <jlebon> but yeah, i'm in favour of using the same keys as well
17:04:15 <bgilbert> it was months ago, in an OOB discussion
17:04:25 <bgilbert> there might be a Google doc somewhere :-(
17:04:29 <bgilbert> or I might be misremembering
17:04:30 <jlebon> the downside is that we'll have to be a bit careful during major version transitions
17:04:36 <bgilbert> yeah
17:05:14 <jlebon> (though in practice it's not as much an issue because the N+1 key is shipped in release N)
17:05:19 <jlebon> anyway, we can move on :)
17:05:31 <bgilbert> I strongly suspect we'll end up signing a release with the wrong key
17:05:51 <bgilbert> as you say, it won't actually matter, but still
17:05:57 <jlebon> right, i think there's different ways we can mitigate this risk
17:06:23 <jlebon> one, we could teach robosignatory to understand the FCOS versioning scheme and use the right key from that
17:07:07 <jlebon> but also, we'll probably want to write up an SOP for major version transitions
17:07:20 <bgilbert> yep
17:11:21 <ajeddeloh> so sounds like we'll sign with the Fedora keys?
17:11:59 <bgilbert> it gives us key rotation for free
17:12:02 <bgilbert> and one less thing to manage
17:12:08 <jlebon> +1
17:12:09 <ajeddeloh> sgtm
17:12:21 <bgilbert> if someone tries to update across the gap, There Will Be Problems
17:12:22 <slowrie> +1
17:12:31 <ajeddeloh> #info FCOS will be signed with the Fedora signing keys
17:12:40 <bgilbert> e.g. F30 -> F32
17:12:43 <bgilbert> so we should figure out that part
17:12:56 <slowrie> yeah, at least from CL we have seen people launching extremely old versions and updating as a common pattern
17:13:06 <ajeddeloh> cant we just disallow that in cincinatti
17:13:25 <bgilbert> yeah, maybe we retroactively add an update barrier?
17:13:35 <bgilbert> as part of the SOP
17:14:07 <jlebon> N+1 is also shipped actually, so for this specific problem, we'd only need a barrier every other release
17:14:15 <jlebon> s/N+1/N+2/
17:14:28 <ajeddeloh> adding a major barrier seems like a good idea anyway
17:14:39 <ajeddeloh> less edges to worry about
17:15:32 <slowrie> Adding barriers for jumps that wouldn't have the signing keys & also adding a docs page strongly recommending launching versions within a certain reasonable timeframe date from current to prevent needing multiple successive updates would probably be a good thing
17:15:43 <ajeddeloh> yeah
17:16:05 <ajeddeloh> update barriers kinda provide a "stick" to prevent people from launching ancient images
17:16:13 <bgilbert> #info rebase SOP should ensure there's a Cincinnati update barrier so the client always has the right signing keys, plus corresponding docs
17:16:24 <ajeddeloh> like if you want to launch a 4 yr old image, you'll have to reboot 8 times
17:16:24 <bgilbert> #undo
17:16:24 <zodbot> Removing item from minutes: INFO by bgilbert at 17:16:13 : rebase SOP should ensure there's a Cincinnati update barrier so the client always has the right signing keys, plus corresponding docs
17:16:45 <bgilbert> #action bgilbert to file a tracker bug about ensuring update barriers across major versions to ensure signing key availability
17:17:46 <ajeddeloh> anyone have anything else for open floor?
17:19:13 <strigazi> i'm good, i have a question for the channel after.
17:21:09 <ajeddeloh> I'll close out the meeting in 60 seconds unless someone has something else
17:22:33 <ajeddeloh> #endmeeting