16:29:29 <dustymabe> #startmeeting fedora_coreos_meeting
16:29:29 <zodbot> Meeting started Wed Feb 12 16:29:29 2020 UTC.
16:29:29 <zodbot> This meeting is logged and archived in a public location.
16:29:29 <zodbot> The chair is dustymabe. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:29:29 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:29:29 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:29:35 <dustymabe> #topic roll call
16:29:39 <dustymabe> .hello2
16:29:39 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:29:46 <cyberpear> .hello2
16:29:50 <zodbot> cyberpear: cyberpear 'James Cassell' <fedoraproject@cyberpear.com>
16:31:22 <miabbott> .hello miabbott
16:31:23 <zodbot> miabbott: miabbott 'Micah Abbott' <miabbott@redhat.com>
16:31:46 <kaeso[m]> .hello lucab
16:31:46 <zodbot> kaeso[m]: lucab 'Luca Bruno' <lucab@redhat.com>
16:32:23 <dustymabe> #chair cyberpear miabbott kaeso[m]
16:32:23 <zodbot> Current chairs: cyberpear dustymabe kaeso[m] miabbott
16:32:32 * dustymabe wonders where bgilbert and jlebon are
16:33:24 <lorbus> .hello2
16:33:25 <zodbot> lorbus: lorbus 'Christian Glombek' <cglombek@redhat.com>
16:33:28 <dustymabe> will get started in a moment
16:33:38 <dustymabe> mnguyen_: around ?
16:33:47 <dustymabe> #chair lorbus
16:33:47 <zodbot> Current chairs: cyberpear dustymabe kaeso[m] lorbus miabbott
16:34:24 <jlebon> .hello2
16:34:25 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:34:26 <dustymabe> #chair jlebon
16:34:26 <zodbot> Current chairs: cyberpear dustymabe jlebon kaeso[m] lorbus miabbott
16:34:46 <dustymabe> #topic Action items from last meeting
16:34:52 <dustymabe> * dustymabe to work with x3mboy on FCOS infographic
16:34:54 <dustymabe> * dustymabe to open a tracker ticket where we discuss major changes for
16:34:56 <dustymabe> the f32 rebase, including fedora proposed changes for f32
16:34:58 <dustymabe> * cyberpear to file tracker ticket for FCCT in the initramfs
16:35:00 <dustymabe> * jlebon to investigate if latest moby-engine upstream has any cgroups
16:35:02 <dustymabe> v2 support
16:35:04 <dustymabe> * mnguyen_ will ask moby-engine maintainer(s) about a rebase to latest
16:35:06 <dustymabe> upstream release for f32
16:35:08 <dustymabe> * dustymabe to work with ignatenkobrain and koji devs to try to drive
16:35:10 <dustymabe> rust packaging to a better state
16:35:12 <dustymabe> lots of action items in there
16:35:14 <dustymabe> I'll rattle off updates for mine real quick
16:35:19 <dustymabe> re-actioning one
16:35:25 <dustymabe> #action dustymabe to work with x3mboy on FCOS infographic
16:35:56 <jlebon> re. moby-engine
16:35:59 <dustymabe> #info dustymabe created #372 to discuss the f32 rebase and major changes we'd like to make
16:36:04 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/372
16:36:30 <jlebon> support for cgroupsv2 was added to containerd. the basics were added to moby-engine recently. but there's still a bunch of things left to implement
16:36:40 <dustymabe> #info dustymaber opened a releng ticket to try to start a discussion about making rust packaging experience better: https://pagure.io/releng/issue/9229
16:36:45 <dustymabe> #link https://pagure.io/releng/issue/9229
16:36:55 <jlebon> it should be in the next 19.03.6 release IIUC
16:37:11 <dustymabe> jlebon: do you mind putting that in an #info?
16:37:18 <jlebon> #info support for cgroupsv2 was added to containerd. the basics were added to moby-engine recently. but there's still a bunch of things left to implement
16:37:52 <jlebon> wrt switching to cgroupsv2 in FCOS, i think we need to wait a bit more for it to be fully ironed out in moby-engine
16:38:06 <dustymabe> jlebon: hold that thought. we'll discuss it in a bit
16:38:16 <ignatenkobrain> Related to last item, I spoke to tkopecek about missing functionality and he said that it probably won't happen for 1.21 :(
16:38:37 <dustymabe> ignatenkobrain: FYI we're discussing it right now over in #fedora-meeting-2
16:38:41 <ignatenkobrain> https://pagure.io/koji/issue/1998
16:38:43 <dustymabe> ignatenkobrain: thanks for the update, though
16:39:22 <dustymabe> ok cyberpear mnguyen_ - you both had action items from last week
16:39:43 <cyberpear> I opened the ticket as discussed
16:40:06 <mnguyen_> i emailed oliveria but have not received a response yet
16:40:12 <cyberpear> https://github.com/coreos/fedora-coreos-tracker/issues/371
16:40:57 <dustymabe> #info mnguyen_ emailed moby-engine maintainer for an update, but no response yet
16:41:22 <dustymabe> #info cyberpear opened a ticket to discuss including fcct in the initramfs #371
16:41:29 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/371
16:42:06 <dustymabe> ok moving on to tickets
16:42:28 <dustymabe> #topic F32: should we switch to cgroups v2
16:42:33 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/373
16:42:49 <dustymabe> there is some good information in the ticket that was added since last week
16:44:54 <kaeso[m]> all the signs seem to point to "not yet"?
16:45:38 <dustymabe> kaeso[m]: yes I think it's safe to say we should probably be conservative and not yet push to v2
16:45:51 <dustymabe> though I don't necessarily want to wait forever.
16:46:10 <dustymabe> for example we have three major components that we know of (podman, kubernetes, docker)
16:46:17 <jlebon> but note that the switchover doesn't have to correspond to a major rebase
16:46:40 <dustymabe> jlebon: you're right, but the major rebase is probably when the new moby-engine would come in that supports cgroups v2
16:46:40 <jlebon> if it's deemed ready enough in e.g. may, then we could switch then
16:47:35 <dustymabe> jlebon: can you copy here what you said earlier about upstream moby?
16:48:05 <jlebon> it should be in the next 19.03.6 release IIUC
16:48:32 <kaeso[m]> just asking about gut feelings, are we thinking about dead-ending nodes born on v1 or are we aiming at an in-place upgrade v1->v2?
16:48:33 <jlebon> wrt switching to cgroupsv2 in FCOS, i think we need to wait a bit more for it to be fully ironed out in moby-engine
16:49:04 <dustymabe> kaeso[m]: I think we were going to let nodes started on v1 stay on v1
16:49:13 <dustymabe> i.e. if you just keep upgrading you stay on v1
16:49:20 <dustymabe> if you deploy new you get v2
16:49:38 <jlebon> we should look at how upstream will handle this too
16:50:05 <dustymabe> kaeso[m]: WDYT about that strategy?
16:50:54 <kaeso[m]> I'm not sure I'm a fan of having both living in parallel
16:50:55 <lorbus> fwiw OKD doesn't care about the default too much, as we'll use Ignition to configure the FCOS nodes as needed (we'll stick to v1 for now)
16:51:50 <dustymabe> kaeso[m]: but they could already be living in parallel if someone configured their node for v2 now
16:53:05 <kaeso[m]> dustymabe: sure but that's a manual customization
16:53:38 <dustymabe> #proposed while the container ecosystem is maturing, both moby (docker) and kubernetes aren't yet to a place where we're comfortable moving to cgroups v2 by default for F32 based FCOS, we'll consider again for F33
16:54:08 <kaeso[m]> s/for F33/later on/
16:54:10 <jlebon> dustymabe: "consider again at a later time"  ?
16:54:33 <dustymabe> #proposed while the container ecosystem is maturing, both moby (docker) and kubernetes aren't yet to a place where we're comfortable moving to cgroups v2 by default for F32 based FCOS, we'll consider again at a later time
16:54:48 <jlebon> kaeso[m]: what do you mean by parallel though? at the cluster level, or on the node itself?
16:54:54 <jlebon> +1
16:55:30 <dustymabe> any opposed to the above proposal?
16:55:48 <jlebon> kaeso[m]: at the cluster level, like lorbus mentioned, it's likely going to be set at a higher level anyway
16:56:02 <kaeso[m]> jlebon: on the node, across the population of FCOS nodes
16:57:02 <lorbus> dustymabe: +1 to proposal
16:57:53 <jlebon> ack for me too
16:57:56 <dustymabe> #agreed while the container ecosystem is maturing, both moby (docker) and kubernetes aren't yet to a place where we're comfortable moving to cgroups v2 by default for F32 based FCOS, we'll consider again at a later time
16:58:25 <dustymabe> #topic Include wireguard-tools package in FCOS
16:58:32 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/362
16:59:05 <dustymabe> ok so we've got some answers to questions after talking with the maintainers of both wireguard and also the networkmanager implementation of wireguard
16:59:38 <jdoss> .hello2
16:59:39 <zodbot> jdoss: jdoss 'Joe Doss' <joe@solidadmin.com>
17:00:01 <dustymabe> it looks like all maintainers agree that if you are using wireguard you'll want the tools on the system to inspect connections
17:00:36 <dustymabe> considering that fact, and the fact that the binary is tiny and doesn't bring in any other deps, should we include it?
17:00:57 <jdoss> +1 for including :)
17:02:19 <jlebon> i think kaeso[m]'s concern wasn't about the binary itself, but the fact that if we include it without making sure the NM workflow works, we'll have people using wg directly
17:02:45 <jdoss> Is that a bad thing?
17:02:56 <dustymabe> jlebon: of course, we want to make sure the NM flow workflow works
17:03:36 <jlebon> making NM keyfiles the canonical way to set up networking via ignition helps, yes. vs. having systemd units run `wg`
17:04:24 <kaeso[m]> can we maybe say "likely yes" but let it soak so that we first get 1) kernel 2) NM 3) docs?
17:04:47 <dustymabe> so let me pick this apart a bit
17:05:06 <dustymabe> I completely agree that we want the NM workflow to be a "happy path" to success here
17:05:32 <dustymabe> but I think whether we include the tools or not is kind of independent of NM working or not
17:05:46 <dustymabe> if you agree with that, then the real question is mostly "timeing"
17:05:51 <dustymabe> sigh, and I can't spell
17:06:53 <dustymabe> does anyone disagree with that?
17:07:28 <cyberpear> Sounds good to me
17:07:33 <jdoss> From a users perspective, not having `wg` when it is in the kernel makes it really hard to use. Even if we are using NM when it has full support for WireGuard how is an operator going to troubleshoot an issue without it?
17:07:33 <dustymabe> i.e. since both maintainers have said "you want the tools installed", that means even if someone *is* using NM, you still want the tools installed
17:08:01 <kaeso[m]> I don't disagree with that, but if "custom wg script" is the only thing that works than I'd rather prefer a bit of delay to debugging creative scripts
17:08:21 <jdoss> But that isn't on FCOS but the users that are making them?
17:08:53 <dustymabe> ok so let's move to the next step: not "if", but "when"
17:09:31 <dustymabe> so I think we've agreed that we do want to include the tools in FCOS, but.. we need the new kernel first
17:09:57 <dustymabe> and we want to actually test out the NM path (and hopefully see if the outstanding work is critical to the model we want to operate in)
17:10:47 <jlebon> jdoss: do i understand correctly then that until 5.6, one would have to literally do `rpm-ostree override replace kernel...` ?
17:10:58 <jlebon> or just load the kmod?
17:11:10 <jdoss> Yea, or use atomic-wireguard or dusty's stuff.
17:11:25 <jdoss> I think it can wait until 5.6 hits.
17:12:04 <dustymabe> kaeso[m]: WDYT about the not "if", but "when" proposal above ?
17:12:09 <jlebon> ok yeah, agreed. the barrier to get this working is already so high until then, that not having `wg` isn't really a concern when one could just do `rpm-ostree install wg` to fool around
17:12:25 <kaeso[m]> dustymabe: by outstanding work you mean https://gitlab.freedesktop.org/NetworkManager/NetworkManager/issues/358?
17:12:56 <kaeso[m]> dustymabe: I was confused by that, as it seems to be a RFE to nmcli, which is not in our flow
17:12:57 <dustymabe> kaeso[m]: possibly - basically we need to evaluate if that's required for our "happy path" or not
17:13:14 <dustymabe> or any other work that is required for our happy path
17:13:41 <jdoss> I think having it by default when it comes with the 5.6 kernel is important for end users. If WireGuard is enabled in the kernel out of the box, wg should be too.
17:14:16 <dustymabe> i.e. we probably want to come up with a user story for what we want the experience to look like for a FCOS user (i.e. using ignition) and then see if it works the way we want it to
17:14:27 <dustymabe> and identify gaps and take those back as feature requests
17:15:09 <kaeso[m]> dustymabe: that looks like a better plan to me
17:15:32 <dustymabe> so that is the ideal thing for us to do, but if we fail to do that we also shouldn't keep it out because of our lack of ability to flesh out all of those use cases
17:15:48 <dustymabe> so my proposal is something like this:
17:16:02 <dustymabe> - yes, we will include wireguard tools (eventually)
17:16:25 <dustymabe> - we'll investigate a "happy path" for how we'd recommend using wireguard with NM on FCOS
17:16:49 <dustymabe> - we'll open issues to track missing features that block the "happy path" from being used
17:17:08 <dustymabe> - we won't block the tools in FCOS after a certain amount of time has passed and the "happy path" doesn't exist yet
17:17:34 <dustymabe> i.e. a contingency
17:18:01 <jlebon> this SGMT
17:18:11 <jdoss> Yep sounds good to me too.
17:18:20 <jdoss> Thanks for taking the time on this folks.
17:18:35 <kaeso[m]> is the "yes" conditional to "if there is still a usecase for them"?
17:18:38 <miabbott> +1
17:19:17 * jlebon has to drop off early -- see y'all!
17:19:19 <dustymabe> kaeso[m]: IOW if the wireguard-tools are obsolete or something?
17:19:22 <kaeso[m]> that is, are we looking at `wg` as a debug tool or as something that we want people to build their things on?
17:19:48 * lorbus has to drop as well, see y'all soon!
17:19:58 <dustymabe> kaeso[m]: at this point i'm basically condering `wg` and `ip` to be analogous
17:20:22 <dustymabe> so replace `wg` with `ip` in your question and what is your answer?
17:20:28 <kaeso[m]> dustymabe: no, just to spell out what are the usecases that require `wg` on the host
17:20:45 <jdoss> It can be used for both IMO. Why would FCOS want to dictate it's useage to users?
17:21:48 <dustymabe> I haven't used wg specifically, but like `ip`, I would say: configuring connections, investigating connection state
17:22:16 <dustymabe> so in our "happy path" we wouldn't use `wg` to configure connections, but we would for investigating connection state
17:22:23 <kaeso[m]> dustymabe: ip (like iptables) are widely used by softwares and scripts to configure the host networking
17:23:24 <dustymabe> kaeso[m]: what's your point?
17:24:55 <kaeso[m]> dustymabe: my point is that we approached this backward and we failed so far to spell out usecases and intended usages
17:25:42 <dustymabe> how about "investigating connection state" ?
17:26:32 * dustymabe notes we are running low on time
17:27:50 <kaeso[m]> that's a point, but so we are boiling this down to "please include this network debugging tool"
17:28:26 <jdoss> generating key as well. I would like to generate keys on the nodes at the come up and register keys in hashicorp vault.
17:28:35 <jdoss> as they come up*
17:29:05 <dustymabe> jdoss: we've yet to define our "happy path". It probably won't include generating the keys on the host during bringup
17:29:26 <dustymabe> but I don't think we should stop users from doing that if they want to generate the keys and register it with some other service on bringup
17:29:44 <jdoss> The quick path for that use case is use `wg`
17:30:09 <dustymabe> kaeso[m]: yes, it's basically a utlity, useful for debugging/insepcting the wireguard connections
17:30:18 <kaeso[m]> jdoss: that's a better one IMHO. And if that is the intended usage, we should make sure it's actually possible with NM+wg
17:30:18 <dustymabe> and also for generating keys
17:30:24 <dustymabe> so like `ip` and `openssl`
17:31:27 <dustymabe> kaeso[m]: some users will choose to not use NM, but we're not here to stop them from that. My understanding is that you just wanted to make sure we had the "happy path" working before people re-invented the wheel a million times
17:31:42 <dustymabe> is that correct?
17:33:02 <kaeso[m]> yes, but now we are heading to a different story which is "we need wg in order to bypass NM"
17:33:33 <dustymabe> ehh, I think that's mostly tangential
17:33:41 <jdoss> I would agree
17:34:02 <dustymabe> i.e. if they wanted wg they could deliver it as a binary with ignition to /usr/local/bin/ and execute it anyway
17:34:39 <dustymabe> so the main use case is network inspection of a wireguard configured interface that was brought up via ignition
17:34:44 <dustymabe> sorry, not ignition, NM
17:35:11 <dustymabe> but ancillary use cases would be generating keys (that could maybe be used by NM)
17:35:17 <kaeso[m]> yes
17:35:24 <dustymabe> or even bringing up the connections using wg itself
17:35:35 <dustymabe> though that wouldn't be what we document
17:36:12 <dustymabe> kaeso[m]: does any of this help ?
17:36:30 <kaeso[m]> I don't want to drag this forever and I don't have a strong opinion in either directions, I'm just trying to have actual reasons written in the ticket
17:36:48 <dustymabe> ok, I can enumerate what we discussed
17:36:56 <dustymabe> in the ticket if that would help?
17:38:39 <kaeso[m]> yep
17:38:40 <dustymabe> #action dusty to add agreed upon proposal to the ticket and also some points from the discussion about use cases and what we prefer as the "happy path" vs what we don't encourage users to do
17:38:55 <dustymabe> #topic open floor
17:39:00 <dustymabe> sorry we're running late everyone
17:39:22 <dustymabe> #info we have a new testing and stable release that are rolling out now over the next few days, please let us know if you hit any issues
17:39:50 <dustymabe> anybody with anything else for open floor
17:40:27 <dustymabe> #endmeeting