16:31:19 #startmeeting fedora_coreos_meeting 16:31:19 Meeting started Wed Sep 2 16:31:19 2020 UTC. 16:31:19 This meeting is logged and archived in a public location. 16:31:19 The chair is dustymabe. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:31:19 Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:31:19 The meeting name has been set to 'fedora_coreos_meeting' 16:31:24 #topic roll call 16:31:26 .hello2 16:31:27 cyberpear: cyberpear 'James Cassell' 16:31:28 .hello2 16:31:30 dustymabe: dustymabe 'Dusty Mabe' 16:31:31 .hello2 16:31:32 .hello miabbott 16:31:33 aoei: aoei 'Joanna Janet Zaitseva-Doyle' 16:31:35 .hello2 16:31:37 miabbott: miabbott 'Micah Abbott' 16:31:40 slowrie: slowrie 'Stephen Lowrie' 16:32:12 .hello2 16:32:13 bgilbert: bgilbert 'Benjamin Gilbert' 16:32:39 .hello2 16:32:40 lucab: lucab 'Luca Bruno' 16:32:49 #chair cyberpear aoei miabbott slowrie bgilbert lucab 16:32:49 Current chairs: aoei bgilbert cyberpear dustymabe lucab miabbott slowrie 16:33:44 nice to see everyone :) 16:33:48 jdoss: around today? 16:34:25 .hello sohank2602 16:34:26 skunkerk: sohank2602 'Sohan Kunkerkar' 16:34:50 .hello2 16:34:50 jdoss: jdoss 'Joe Doss' 16:35:01 #chair skunkerk jdoss 16:35:01 Current chairs: aoei bgilbert cyberpear dustymabe jdoss lucab miabbott skunkerk slowrie 16:35:16 #topic Action items from last meeting 16:35:24 we have no action items from last meeting \o/ 16:35:45 I will take this opportunity to thank cverna for running the meeting and everyone for participating and making it so productive 16:35:53 cverna++ 16:35:53 miabbott: Karma for cverna changed to 23 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 16:36:08 Here is a link to the working session document from last meeting 16:36:12 #link https://hackmd.io/aVJiL9DUQpCDSafSBcs6ZQ?view 16:36:58 #topic tracker: Fedora 33 rebase work 16:37:04 #link https://github.com/coreos/fedora-coreos-tracker/issues/609 16:37:43 this is a PSA: please help us identify any potential issues with F33 that we need to consider when migrating FCOS to it. 16:38:01 We'll (hopefully) be moving the next stream over in the next cycle or two. 16:38:17 ENDOFPSA 16:39:06 systemd-resolvd is one I'll be looking into shortly 16:39:17 anything else to bring up before we move to the next ticket? 16:39:45 #info please help us identify any potential issues with F33 that we need to consider when migrating FCOS. 16:39:52 There was some bugs with DNS over TLS in F32 that I hope get resolved in the systemd version in F33 16:40:07 in resolved specifically 16:40:09 #info we'll hopefully switch the `next` stream to F33 in the next cycle or two 16:40:27 jdoss: if you could help us verify those get fixed that would be great 16:40:42 yea I will track down the version in systemd-resolved 16:40:50 if you've got links to issues put them in the ticket 16:40:55 will do 16:40:58 jdoss++ 16:41:27 Should we have a cgroups v2 debate for f33 now or later? 16:41:37 container DNS can have issues if you're running systemd-networkd/systemd-resolvd without network manager - but I ran into this problem on Arch so idk if it also crops up on Fedora 16:42:41 aoei: 👍 16:42:49 we'll push the cgroups v2 discussion off til next time 16:43:02 #topic Discuss adding systemd-networkd to FCOS 16:43:10 #link https://github.com/coreos/fedora-coreos-tracker/issues/610 16:43:23 which is a pointer to 16:43:28 #link https://github.com/coreos/fedora-coreos-config/pull/574 16:44:08 There is a lot of discussion in the ticket, but for the sake of discussion today 16:44:13 Let's separate this discussion into two topics: 16:44:20 1. existance of systemd-networkd binaries on the system 16:44:22 2. higher level support/tooling for systemd-networkd throughout the rest of the stack 16:44:44 they intermingle a bit, but I think it's worth separating them out 16:44:52 So we'll start with 1. 16:45:04 Currently the systemd-networkd binaries are part of the systemd rpm. Since the binaries aren't a part of a separate RPM that we can just exclude we remove the binaries during the compose process for Fedora CoreOS. This was because our target for configuring networking has been NetworkManager. 16:45:24 So. The question is should we change this? 16:45:38 Options: 16:45:40 A "We stop removing the binaries during compose" 16:45:42 B "We ask the systemd team to break them out into a separate subpackage, which can be layered" 16:45:44 other options? 16:45:55 ENDOFWALLOFTEXT 16:45:57 * lorbus joins late 16:46:09 is A && B and option? 16:46:27 B 16:46:29 in general, if we delete files from a package during compose, that seems to suggest a problem with the packaging 16:46:29 I think the short term A would unblock people that want to use networkd and it could help people move from CL/Flatcar. 16:46:33 lorbus: sure, but it wouldn't give us benefit to do the extra work 16:47:02 bgilbert: perhaps 16:47:05 agreed, always prefer a package split. there's no "override restore" command to get nuked files back 16:47:45 I agree, but not only suggests it a packaging prob, it also makes it very hard for people who do want to use it to install it 16:47:56 just for completeness, are there other options other than A or B ? 16:48:24 so I can actually pick up B since I've mingled with sd packaging before anyway 16:48:35 lorbus++ 16:48:35 bgilbert: Karma for lorbus changed to 7 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 16:49:39 I couldn't specifically think of any other options, but just wanted to check.. 16:49:50 Going once.. twice on "any other options than A or B?" 16:50:26 Sold 16:50:33 ok A and B are our two options 16:50:43 lorbus: you're suggesting A && B ? 16:50:51 B implies A anyway, no? 16:51:06 * lorbus puts the kebap down 16:51:17 yes kinda, but doing B obsoletes A in a way 16:51:19 bgilbert: umm I think maybe I should explain A and B a bit more 16:51:33 yeah. let me go into more detail 16:51:35 bgilbert: not necessarily though; we could leave the binaries excluded until systemd splits the package and then drop that piece 16:51:39 and the networkd binaries never land 16:51:53 A. would be just us "stop removing binaries from systemd package" 16:52:07 slowrie: that's essentially what I plan to do 16:52:15 even leaving the exclude should have no effect once those files aren't part of the package. 16:52:20 if we subsequently did B and didn't also include systemd-networkd subpackage - then the binaries would effectively get removed again 16:53:04 they wouldn't land in the image, but folks will be able to layer networkd on top -- which is not possible today 16:53:22 so option A is "systemd-networkd binaries are included in the host" and option B is "systemd-networkd binaries can be layered easily" 16:53:32 I have feeling that a FAQ entry like "Can I rpm-ostree install networkd and expect it to work?" is also warranted for B 16:53:47 lucab: yeah that leans into topic 2. a bit 16:54:12 If I layer systemd-networkd do I risk my updates breaking? 16:54:18 so.. do my clarifications on A and B help understanding ? 16:54:38 so the cleanest solution is just do B, allowing layering, but not adding new things to distro, IMO 16:54:48 jdoss: we're working on making package layering more reliable - coming hopefully in the next few weeks 16:55:11 cyberpear: +1 16:55:30 Because if I can't do updates B is isn't really ideal. 16:56:04 jdoss: yeah I know what you're saying. For that particular package I would not be worried about updates being less reliable with it 16:56:17 considering the work we're doing that should be rolling out in the next few weeks 16:56:24 fair enough 16:56:39 though.. obviously networkd isn't part of our stack we test everything against, so there's that 16:57:32 I hear some takers for B 16:57:39 anyone want to advocate for A? 16:57:50 well, I like A 16:57:54 me too. 16:58:01 because it sounds like you were being a bit naughty by removing the binaries in the first place 16:58:03 and A fixes that 16:58:05 dustymabe: why not? I guess it will have a strict versioned dependency on systemd, which is in the base set 16:58:23 (why not worried about that package, I mean) 16:58:40 lucab: I'm just saying that when we boot FCOS in CI on umpteen platforms, we use NetworkManager to bring up the networking 16:58:47 we don't have networkd coverage 16:58:48 but B also means you don't have the issue of wanting to remove binaries from a package 16:58:53 A increases image size for a dep not uses by default - my vote is -1 16:59:16 You are talking about a very very small size lorbus 16:59:25 i am not certain of the details (so i could be spectacularly wrong), but adding the networkd binaries back in and telling users "here's your binary, you have to figure out how to make it work with FCOS" isn't going to be a good experience 16:59:43 so i'm tepid on including them again 16:59:46 re A as a permanent solution, we generally avoid shipping code that isn't required to operate the OS. I'd be surprised if networkd even _works_ on FCOS. 17:00:03 re A as a temporary solution, anyone who uses networkd will break once we move it to a subpackage. 17:00:06 miabbott: Anyone that uses networkd in Fedora Workstation or Server understands the experience they are going to endure. 17:00:22 yes. I don't propose we have any short term solutions to this problem 17:00:47 we don't want to add back and then remove any functionality IMO. If we add it back we add it back, but then we don't remove it 17:01:24 A doesn't sound great if stop-gap til B 17:02:32 I'm torn. 17:02:37 We also have users that went from CL to Flatcar because networkd was not a thing in FCOS. dustymabe talked to one last week with me about his experience. I am sure there are other users that would move to FCOS if they could drop their networkd files and move on with their lives after the network is online via NetworkManager. 17:03:05 (Joe Gooche couldn't make today he got double booked in meetings at work) 17:03:09 C move the binaries to a temporary location in a post process script, until the package is split 17:03:50 cyberpear: does that help? 17:04:08 I don't feel that helps much. 17:04:17 jdoss: the networkd->NetworkManager switch has been always a factor in the FCOS migration. we've known that was going to add friction for users, and that we'd probably lose users over it. it's not ideal, but that's where we are. 17:04:45 bgilbert: and I am saying it doesn't have to be that way if you just put the binaries back. 17:05:00 well.. I think that leads us into topic 2. 17:05:05 it's not that simple, the network stack interacts with a whole lot of stuff we're doing, including in the initramfs 17:05:09 so let's table 1. briefly 17:05:14 jdoss: and I'm saying that if the goal is to have a functioning network, that won't help. 17:05:22 2. higher level support/tooling for systemd-networkd throughout the rest of the stack 17:05:45 we've got tooling at different layers that assume NetworkManager 17:05:46 Fedora Workstation and Server both boot with NetworkManger as the default with systemd networkd binaries in place. What do you we lose here? 17:05:51 how does that change if networkd gets involved 17:06:07 for one thing, neither of those do networking in the initramfs by default 17:06:18 for CoreOS the initramfs is a huge focus point 17:06:33 i.e. Fedora Server and Fedora Workstation don't do networking in the initramfs by default 17:06:39 I want to use networkd after initramfs. 17:07:09 so in jdoss' case he would be fine with NM bringing up network in the initramfs and laying down networkd configuration and disabling NetworkManager 17:07:13 now actually networkd was created partially because NetworkManager was seen as too big to land in the initramfs (which...well, now that we have statically linked go/rust in there is totally moot =) ) 17:07:15 then the real root just uses networkd 17:07:51 or really just let NM do its thing and manage my WG tunnels with networkd specifically =) 17:08:31 any other parts of the stack we don't think will work? 17:08:42 the 'networking in initramfs' part was the big one to me 17:09:01 obviously OKD assumes some things, so you wouldn't be able to use networkd with that, but I don't think jdoss was planning to 17:09:09 DO. Packet. c-i --copy-network. maybe some stuff with ip= kargs, I haven't kept track of how that works. 17:09:18 (it's funny how this huge discussion basically boils down to "wireguard support in NM is lacking/unusable") 17:09:45 I think that's a specific example from jdoss.. others wanted networkd specifically 17:10:02 lucab: you are minimizing things here. there are CL/Flatcar users that haven't moved to FCOS because of the lack of networkd 17:10:08 bgilbert: good point about `--copy-network` 17:10:35 either way (whether we choose A or B) I agree with Luca we should have a FAQ entry 17:11:33 do we want to circle back to 1. ? 17:12:45 A "networkd is shipped in the base layer, We stop removing the binaries during compose," 17:13:05 B "networkd is not shipped in the base layer, We ask the systemd team to break them out into a separate subpackage, which can be layered" 17:13:21 B 17:13:47 I think A has the benefit of maybe bringing some users back on board, but without people actually trying it and bumping into limitations, we won't know 17:14:04 dustymabe: I'm concerned that that approach is a trap 17:14:23 B has the benefit of it staying out of the base layer, but also the negative of depending on another team to accept the proposed change 17:14:24 we don't actually want people to migrate from CL to FCOS+networkd 17:14:30 I feel we are tacking on all this extra stuff to A when if you look at it from what it actually does to the image itself it is allowing for users to use networkd if they want. FCOS as a whole doesn't have to do much beyond that if we don't want to. 17:14:30 I'll abstain from the vote but I will note that even if A is chosen I think it should be clear that we have no intention to support networkd so if it's just broken it's broken 17:14:51 bgilbert: yea but you are not managing their deployments. How would you know what is best for their usecases? 17:14:59 I vote B 17:14:59 bgilbert: i would phrase it as "we prefer users to use FCOS with networkmanager" 17:15:04 jdoss: we're not managing their deployments, but we are managing their OS :-) 17:15:12 jdoss: I think you're missing that people using it are going to file bugs, and it's not clear that the community would step up and fix those bugs 17:15:27 walters: yep. that's the 2. topic. 17:15:28 and I agree with bgilbert that doing it for the users is a trap. design something good and the users will come 17:15:30 yeah, there's 100% going to be an expectation over time that we make things work 17:15:34 a FAQ saying use NetworkManager sounds reasonable. 17:16:04 right. Is a good "disclaimer" good enough?" 17:16:14 not having networkd in the base makes it obvious that it's not supported 17:16:34 lorbus++ 17:16:45 lorbus: +1 17:17:03 one downside of B is that we don't have good support for streamlining package layering right now (that was one of the planning topics we didn't get to) 17:17:25 hopefully that will be all streamlined in the future so that the first time you ever boot into the real root, you already have your layered packages 17:17:37 If it is in the networking section of the docs IMO would be good enough. People use Workstation, Cloud, and Server with NM enabled and networkd managing things. I did on 500+ VMs without issues. 17:18:10 and if I had bugs with networkd I opened issues on the systemd github issues not BZ 17:18:16 lorbus: yeah though we could add a drop-in that adds an `ExecStartPre=echo networkd is not supported by the FCOS team; please see https://doc.fedoraproject.org/coreos/networkd.html` or something in the unit 17:18:30 walters: +1 17:18:37 walters: +1 17:18:47 walters: is that really sufficient visibility though? 17:19:10 drop-in: ExecStartPre=/bin/false 17:19:28 bgilbert: and people who want to use it remove the dropin? 17:19:34 override it, yeah 17:19:52 and if they do that they probably read the docs right above it that told them that they probably shouldn't? 17:20:06 IMO that'd feel like another barrier users would have to work around, instead of layering the package like they would do with any non-base package 17:20:09 if they do that, we can assume that they did 17:20:17 I like it 17:20:38 I'd still prefer that we move to a subpackage 17:20:50 bgilbert: same here 17:21:00 yup 17:21:06 My biggest point for A is if we did nothing but put the binaries back you give users the choice to use it. It might bring more users over from CL/Flatcar and maybe the would see the advantages to switch to NM. It makes FCOS more inclusive to people that loved CL but didn't make the switch for whatever reason. Maybe that reason was networkd being totally unsupported. 17:21:32 Dusty talked to such a user last week on a video chat. 17:21:35 Maybe. It's a compelling argument 17:22:05 Though, hopefully the majority of people are using DHCP and just don't care 17:22:11 but when you care, you really care 17:22:22 dustymabe: which argument are you referring to? 17:22:39 how about A with a disclaimer drop-in that must be overridden, pending implementation of B? 17:22:40 bgilbert: the "bring more people from CL" argument 17:22:50 cyberpear: I thought about that 17:23:04 I don't find it compelling. bringing users from CL to a half-broken FCOS isn't great for those users. 17:23:06 cyberpear: it still won't mean that users systems that get upgraded won't lose the binaries 17:23:14 I get wanting to limit the packages on FCOS. I totally do. I appreciate it. I think FCOS is the best thing since sliced FOSS bread. 17:23:34 🤗 17:23:53 bgilbert: How would it be half broken? Just that they can't use networkd in initramfs? 17:24:09 jdoss: that's definitely one piece 17:24:25 networkd will not, in general, work, due to the integrations already mentioned and whatever integrations we add in the future. 17:24:40 *NM integrations 17:25:06 jdoss: I think everyone understands the benefit of bringing over more CL users etc; what's a bit harder to see is the cost of an additional option. To elaborate, network handling is in at least: afterburn, coreos-installer, coreos-assembler, fedora-coreos-config (in multiple places, e.g. the chrony bits I mentioned, and NM config propagation) and I can barely keep the connections between all those straight in 17:25:07 my head myself, and they're all quite important for bare metal installs for example 17:25:18 walters++ 17:25:18 bgilbert: Karma for walters changed to 2 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 17:25:34 oh now Zodbot gives out points. 17:25:42 jdoss++ 17:25:48 out of luck. 17:25:49 a recurring theme in internal meetings is that the FCOS networking glue is so complex that no one can keep it all in their heads 17:25:52 it only works once per release cycle 17:25:54 with spec 3 networkd support has been explicitly removed. I think it's only fair to reflect that in the reference implementation of an OS using it, which FCOS arguably is 17:26:28 lorbus: that was for a slightly different reason, it was actually just sugar around files 17:26:37 It's too bad that Networking has to be so hard. 17:26:43 my fear is basically a year from now multiple vocal users hit some networkd bug around something in this stack but no one shows up to fix it, or if they do it significantly complicates the links between these 17:26:57 ok we're bumping up into time 17:27:06 vote? 17:27:10 walters: IMO if they have a networkd bug they should bark up systemd's tree 17:27:24 I really liked the A options talked about with the dropins 17:27:41 no it's about things like https://github.com/coreos/fedora-coreos-config/blob/testing-devel/overlay.d/05core/usr/lib/dracut/modules.d/15coreos-network/coreos-copy-firstboot-network.sh 17:27:52 jdoss: maybe they should, but they won't. same as how we're being pushed to make overlays work even when they're explicitly non-supported O:-) 17:28:00 which, I guess is potentially out of scope if we're only doing networkd in the real root 17:28:08 walters: Ahh OK thanks I was confused on what you were talking about 17:28:14 I think I'd vote for B, and to keep away from option A 17:28:15 but that just opens up huge cans of worms around things like which DHCP client identifier is picked 17:28:45 ok let's see if we can get some resolution 17:29:09 It seems like the most favored A option is to have a dropin that disables networkd that a user has to explicitly remove 17:29:26 The most favored B option is to have systemd-networkd into its own subpackage that can be layered 17:29:46 do we have any signal from the systemd team about B? 17:29:47 and hopefully in the future package layering will be a bit more integrated with fcct/ignition 17:29:59 walters: nope, we need to reach out to see how feasible it is 17:30:00 +1 for A. I will even write the networkd disclaimers and documentation myself. 17:30:10 i.e. would it be "no" or is it "we'd accept patches" or "we don't know, maybe but ugh" 17:30:20 walters: right, we'd need to find out 17:30:36 and disable dropins with guidance from the team on the PRs. 17:30:38 walters: I've worked with them in the past and they've been happy reviewing adapting patches for far. 17:30:43 I'll open that convo with them 17:30:53 can we get votes for A and B in this form: 17:31:04 A: +1 B: +1 17:31:12 vote vote vote! November 3! 17:31:13 I could get behind B if A is a fallback if we get a stonewall from systemd on the package splits. 17:31:26 jdoss: I think that is reasonable 17:31:29 A: +/-0 B: +1 17:31:31 hmm I feel like there's a C: touch base with systemd about B and return based on that 17:31:31 and if updates work! 17:31:32 ok I'll do a proposed then 17:31:55 maybe that C is idea to see if B is avail and then revisit? 17:31:57 if it's a hard "no" then...well, we'd need to re-discuss 17:32:06 walters: +1 17:32:14 walters: agreed 17:32:32 #proposed we'll reach to the systemd team to see how they feel about making a systemd-networkd subpackage. If they refuse or are not interested we will explore option A (including systemd-networkd in the base layer) but with a dropin that disalbes it by default. 17:32:36 (doesn't mean we can't vote now on A/B though based on the premise that it'd be "patches accepted") 17:32:49 dustymabe: +1 17:32:54 dustymabe: +1 17:32:56 but yeah +1 to that 17:33:01 dustymabe: +1 17:33:14 B/C: +1 17:33:22 hopefully my #agreed will fix all typoes :) 17:33:59 cyberpear: lucab: slowrie: skunkerk: aoei: thoughts ? 17:34:23 +1 17:34:30 dustymabe: as I mentioned earlier I don't really have an opinion other than that we don't have the resources to support networkd if it's completely foobar and that's not a great UX 17:34:42 s,foobar,fubar, 17:34:44 +1 to reaching out to systemd 17:34:48 slowrie: yep. noted 17:34:58 slowrie: I use foobar too 17:35:07 #agreed we'll reach out to the systemd team to see how they feel about making a systemd-networkd subpackage. If they refuse or are not interested we will explore option A (including systemd-networkd in the base layer) but with a dropin that disables it by default. 17:35:15 seems an obvious "fix" especially given Minimization 17:35:27 yeah I agree with reaching out to systemd, A seems non-idea but making it a very explicit user choice is a sane way to do it, if it has to be done 17:35:35 with all the warnings and caveats 17:35:40 that was a great discussion. Thanks everyone for bringing some passion and also keeping things constructive and coming up with creative ideas. 17:35:48 #topic open floor 17:35:55 anyone with anything for a brief open floor? 17:36:04 nope 17:36:05 It's lunch time; gonna drop now. Thanks all 17:36:32 #info jdoss helped us get our twitter account re-activated https://twitter.com/fedoracoreos 17:37:05 thanks jdoss for that 17:37:19 jdoss++ 17:37:19 lorbus: Karma for jdoss changed to 5 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 17:37:25 also there is a new `next` stream release out today with podman 2.0.6 17:37:38 if all goes well we'll move to podman 2.x in next week's testing release 17:37:52 * aoei is already using podman 2.0.6 17:37:54 You are welcome. Just told my CEO thanks again for making the intro to his contact at Twitter. 17:37:58 #info There is a new `next` stream release out today with podman 2.0.6. If all goes well we'll move to podman 2.x in next week's testing release. 17:38:19 can't wait to try OKD on cgroups v2 w/ podman 2.x now... 17:38:44 jdoss++ for Twitter account and networkd discussion 17:38:44 cyberpear: Karma for jdoss changed to 6 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 17:38:47 lorbus: I have some ignition foo to kick into cgroups v2 if you want to test. 17:38:55 :) - let's talk about cgroups v2 next time 17:39:16 jdoss: neat! I'd give it a try, for sure! 17:39:19 Thanks everyone for the discourse on networkd. I deeply appreciate your time. 17:39:22 dropping as well. Thank you for hosting, Dusty! And thank you all for participating, see y'all next week :) 17:39:25 will close things out in 60 seconds unless we have anything else 17:39:28 I'll send you a gist 17:39:32 jdoss: we deeply appreciate your time as well! 17:39:50 jdoss: awesome, ty! 17:39:57 and I second what Dusty said :) 17:40:11 #endmeeting