16:30:39 #startmeeting fedora_coreos_meeting 16:30:39 Meeting started Wed Feb 5 16:30:39 2020 UTC. 16:30:39 This meeting is logged and archived in a public location. 16:30:39 The chair is dustymabe. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:30:39 Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:30:39 The meeting name has been set to 'fedora_coreos_meeting' 16:30:43 #topic roll call 16:30:45 .hello2 16:30:46 bgilbert__: Sorry, but you don't exist 16:30:48 .hello2 16:30:49 dustymabe: dustymabe 'Dusty Mabe' 16:30:50 .hello2 16:30:52 bgilbert: bgilbert 'Benjamin Gilbert' 16:31:46 .hello2 16:31:49 walters: walters 'Colin Walters' 16:31:51 .hello2 16:31:52 jlebon: jlebon 'None' 16:32:46 #chair bgilbert walters jlebon cyberpear 16:32:46 Current chairs: bgilbert cyberpear dustymabe jlebon walters 16:32:50 .hello2 16:32:51 cyberpear: cyberpear 'James Cassell' 16:33:17 is kaeso or red_beard or miabbott around today? 16:33:30 .hello mnguyen 16:33:31 mnguyen_: mnguyen 'Michael Nguyen' 16:33:56 also ignatenkobrain is going to try to join us but he is AFK right now and trying to make it back to one 16:34:02 #chair mnguyen_ ignatenkobrain 16:34:02 Current chairs: bgilbert cyberpear dustymabe ignatenkobrain jlebon mnguyen_ walters 16:34:06 * miabbott in another meeting 16:35:04 #topic Action items from last meeting 16:35:10 * dustymabe to work with x3mboy on FCOS infographic 16:35:29 I picked this up from bgilbert - I still need to reach out to x3mboy to move this forward 16:35:31 will do 16:35:35 #action dustymabe to work with x3mboy on FCOS infographic 16:35:51 #topic next stream 16:36:00 so, we have testing, we have stable 16:36:16 we are about to branch for f32 in Fedora 16:36:32 I'm thinking we should soon create the next branch 16:36:49 agreed 16:36:56 and also collect a list of "big changes" that we'd like to implement in the major bump 16:37:21 +1 to next. dustymabe, do we have big changes? 16:37:23 for example: should we migrate to cgroups v2 by default for next 16:37:49 that represents a decision we made to differ from the rest of fedora 16:38:10 so we need to either decide to keep it as a delta, or move to v2 16:38:33 and we should probably look at things like 'docker support' etc. 16:38:41 to see where it is for v2 16:39:05 so just refreshing my memory from https://github.com/coreos/fedora-coreos-tracker/blob/master/Design.md#release-streams, next is actually a tricky thing 16:39:24 .hello redbeard 16:39:25 red_beard: redbeard 'Brian 'redbeard' Harrington' 16:39:37 .hello lucab 16:39:37 kaeso[m]: lucab 'Luca Bruno' 16:39:55 #chair red_beard kaeso[m] 16:39:55 Current chairs: bgilbert cyberpear dustymabe ignatenkobrain jlebon kaeso[m] mnguyen_ red_beard walters 16:39:56 is `next` a stream which supports auto-updates? 16:40:02 kaeso[m]: yes 16:40:03 we've defined the bodhi turn on as the switchover; before that we'd be tracking the same release as testing/stable, but with the subtle kernel thing 16:40:22 jlebon: correct 16:40:43 jlebon: bodhi turn on is the same time as branching right? 16:41:08 oh wait, no. that's at beta freeze 16:41:11 dustymabe: oh? 16:41:16 * dustymabe has this written down 16:41:24 2020-02-11 Branch Fedora 32 from Rawhide (Rawhide becomes future F33) 16:41:26 2020-02-11 Post-branch Freeze 16:41:28 2020-02-14 Post-branch Un-Freeze 16:41:30 2020-02-25 Bodhi activation point Beta Freeze (*) 16:41:32 2020-03-17 Beta Release Target date 16:41:34 2020-04-07 Final Freeze (*) 16:41:36 2020-04-21 Fedora 31 Final Release (GA) Target date 16:41:38 gleaned from https://fedoraproject.org/wiki/Releases/32/Schedule 16:41:52 close enough i suppose 16:42:15 jlebon: yeah we could probably start doing it at `Post-branch Un-Freeze` 16:42:31 because they freeze long enough to get something building 16:42:31 ack 16:42:39 either way those are minor details I think 16:42:57 so - any other large changes we should track or make for possible next ? 16:43:09 switching to NM in the initramfs maybe? 16:43:12 right, just wasn't sure if bodhi turn on was actually much farther down the road 16:43:33 for "big changes", could we add fcct to the initrd so folks can just pass a .fcc rather than an .ign (and skip the intermediate step for the user)? IIRC, it's 16:43:45 the original idea was that the Bodhi activation point represented a certain level of stability. I don't think the exact landmark matters too much though 16:44:26 cyberpear: you know that is a question I got asked a lot at the lab we did at devconf 16:44:42 at least 3 or 4 people raised their hand and asked me why the extra step was needed 16:44:57 the binary keeps growing, though... currently at 9.3M 16:45:21 there are some reasons, but I think it's worth considering just including fcct in the initramfs and processing it into ignition before passing it to ignition 16:45:22 I'd like to see us try education first before giving up on the separate step 16:45:23 but I think the convenience tradeoff is probably worth it 16:45:47 offline validation does provide a lot of value, and we haven't given the ecosystem a lot of time to adapt yet 16:45:48 (assuming that fcc is the canonical spec folks should target) 16:46:33 bgilbert: yeah, I see both sides 16:46:46 coming from a cloud-init world, it definitely seems awkward though 16:46:59 I can't remember if there is already an issue for this or not 16:47:03 dustymabe: yeah. but we're trying to change the world, not become it :-) 16:47:08 cyberpear: would you create one if there isn't one already 16:47:20 will do 16:47:38 (fwiw, I'm strongly against fcct in the initramfs) 16:47:57 dustymabe: more broadly: I'm not super in favor of the "big changes at rebase" model 16:48:23 the theory is that users shouldn't care about the Fedora major, so I'd think we'd do most changes in the rolling-release model 16:48:47 the major rebase is part of a rolling release though 16:48:59 right? 16:49:01 sure, just not sure why we would couple those things 16:49:16 (except for things where we need a new version of a package that comes in with the rebase) 16:49:33 bgilbert: for me it's mostly because that is when we'd get new tools (like docker) that support the thing 16:49:39 ah, yeah 16:49:53 fwiw they can happen whenever is convenient after the rebase; don't need to be on day zero 16:49:55 so for example "switch to cgroups v2" -> I'd prefer it happen on rebase 16:50:07 indeed it would be better not to bundle many major changes together, otherwise it's harder to debug and revert single things 16:50:12 "include fcct in initramfs" -> could happen any time as it's a feature add 16:50:40 also switching to NM in initramfs could happen anytime 16:51:35 but in general there probably are going to be a collection of changes (some we decide, some we inherit from Fedora) at a major version rebase 16:51:55 we should probably collect those somewhere and think about them and also communicate the changes somewhere (release notes?) 16:52:30 does that seem like a good idea? 16:52:59 maybe we should reframe as "are there any changes we *have* to think about/deal with during the rebase?" 16:53:16 makes sense to have a tracker ticket at least. I'm +1 to minimizing changes at the rebase point to aid debugging, but you're right that some will be unavoidable 16:53:22 then discuss changes we want to do that may or may not align with the rebase 16:53:34 jlebon: right. it's just important the discussion takes place 16:53:41 "what do we need to do" + "what changes does the rebase unblock" 16:54:07 ok so in favor of adding a tracker ticket for this discussion? 16:54:17 +1 16:54:42 #action dustymabe to open a tracker ticket where we discuss major changes for the f32 rebase, including fedora proposed changes for f32 16:54:49 good with that ^^ ? 16:55:25 there is one more action item I can think of 16:55:42 does anyone want to volunteer to do 2 things: 16:56:00 #action cyberpear to file tracker ticket for FCCT in the initramfs 16:56:03 - reach out to the moby-engine maintainer(s) in fedora and ask about a rebase to latest upstream release for f32 16:56:14 - check latest moby-engine upstream to see about support for cgroups v2 16:57:26 if no one volunteers I have to make cyberpear do it 16:57:36 i can investigate the latter 16:57:36 that'll teach him to come to a community meeting 16:57:50 * cyberpear doesn't care about moby... 16:57:55 #action jlebon to investigate if latest moby-engine upstream has any cgroups v2 support 16:58:00 cyberpear: /me jokes 16:58:10 :P 16:58:22 mnguyen_: any chance you can take the first item? "reach out to the moby-engine maintainer(s) in fedora and ask about a rebase to latest upstream release for f32" ? 16:58:45 sure 16:59:01 FWIW there's some work/progress on OKD with cgroups v2 happening in #openshift-dev slack 16:59:13 🙌 16:59:48 #action mnguyen_ will ask moby-engine maintainer(s) about a rebase to latest upstream release for f32 17:00:03 ok on to a few other items 17:00:23 #topic Include wireguard-tools package in FCOS 17:00:34 #link https://github.com/coreos/fedora-coreos-tracker/issues/362 17:01:18 jdoss (wireguard userspace maintainer in fedora) reached out to ask about inclusion in FCOS 17:01:30 .hello2 17:01:31 jdoss: jdoss 'Joe Doss' 17:01:35 * jdoss waves 17:01:55 I'm guessing the thoughts there are that wireguard is going to become pretty pervasive and it would be nice to be able to easily configure VPN networks by dropping in some files using ignition 17:02:17 * bgilbert knows nothing about wireguard 17:02:24 jdoss: is it completely configurable via config files in /etc? 17:02:45 You can just configure it via ip or systemd-networkd 17:03:08 but the wg binary allows you to interact with it better 17:03:11 why not using a container ? (wild guess) 17:03:26 jdoss: in other words, if I write some config files during provisioning, will they be picked up by some systemd unit and everything will magically work? 17:03:37 without needing to run commands by hand 17:03:47 Having to ship a container for a single binary is a lot of work for such a small command. 17:03:51 (assuming wg-tools is installed) 17:04:37 how is NM support for this? 17:04:47 a container with only the wg command ? 17:05:07 kaeso[m]: https://blogs.gnome.org/thaller/2019/03/15/wireguard-in-networkmanager/ 17:05:27 There is work on a NM plugin but most use the built in systemd template wg-quick to configure it. 17:05:27 jdoss: from looking at the spec file quickly, i presume it brings no additional deps, right? 17:05:44 No additional deps 17:06:30 jdoss: do you know if the wireguard NM plugin requires wireguard-tools ? 17:06:45 Somewhat related: in OpenShift 4 today we ended up adding openvswitch to the host, but most of the control logic comes from containers 17:07:10 I am not sure on that one. I only really care about using the wg command via wg-quick or via systemd-networkd 17:07:54 right, though https://github.com/coreos/fedora-coreos-tracker/issues/24 led us to NM 17:08:14 so I think it would be worth investigating a few things 17:08:23 then I'd say we can simply explore the "write NM profile via Ignition" path and document it 17:08:34 - does NM in fcos today support wireguard (according the blog it was added in NM 1.16) 17:08:52 ^^ that investigation requires a new kernel, of course 17:09:12 - does NM in fcos need wireguard tools package or not 17:09:25 - do we care about supporting more than one way to configure wireguard? 17:09:55 any other good questions to answer during the investigation? 17:10:43 is the NM support a separate plugin RPM? If so, do we ship it by default in the OS image? 17:11:00 *plan to ship 17:11:00 kaeso[m]: a cursory `dnf search` doesn't show anything 17:11:05 wireguard tools has all the things you need to generate keys and set configurations on interfaces via the CLI. Having it makes life with WireGuard better. It's a small binary with no deps. 17:11:27 jdoss: which makes me think it would be required by the NM plugin 17:11:44 right? 17:11:45 Most likely. We call wg to generate keys for usage in systemd-networkd 17:12:00 jdoss: in general we discourage doing things from the command-line 17:12:16 bgilbert: but scripting we encourage, right? 17:12:25 i.e. automation 17:12:25 Right but we would call it via ignition on boot 17:12:40 sure 17:13:04 ok I asked 3 questions above 17:13:08 I'm +1 to supporting WG without needing a container, but if it turns out that NM can do everything itself, maybe we consider not shipping the tools 17:13:34 bgilbert: that sounds reasonable 17:13:50 having to pull down a container that has it would be cumbersome for usage with systemd-networkd. 17:13:53 so we lean towards shipping the tools, but let's see what the investigation turns up first 17:14:00 jdoss: we don't ship networkd 17:14:08 that's a bummer :( 17:14:32 I thought it just came as part of deal with systemd. My bad. 17:14:42 yeah, we explicitly exclude it 17:15:03 we're opinionated :-/ 17:15:10 right, we're trying to create a well trod path by keeping everyone on the same road 17:15:39 it also decreases our support burden, which makes the project more likely to succeed in the long term 17:16:02 ok I'll poke these questions into the ticket and we'll see about the investigation part 17:16:12 jdoss: any idea when that kernel will make it into fedora? 17:16:28 early mid summer. 17:16:50 but who knows. It's going to be this year unless something changes. 17:17:04 +1 17:17:08 I'm not sure I understand what is the scenario where we need key generation on the host (as opposed to pre-generated and injected via Ignition) 17:18:10 Even without the NM stuff for you can use the wg-quick script to manage wg interfaces that comes with the wireguard-tools package. 17:18:49 kaeso[m]: we generate keys on the fly and ship them back to register into our internal management VPC. 17:19:33 yeah I understand both scenarios (doing it before provisioning and doing it during provisioning) 17:19:56 oh well, let's take other discussion to the ticket 17:20:05 moving on to the next topic 17:20:11 I guess the TL;DR from someone who has packaged wireguard for 3+ years would be "not having the tools on whatever hardware you are trying to use WireGuard on would be painful" 17:20:24 I will follow up on the ticket. Thanks for the time folks. :) 17:20:37 I'm a bit wary of encouraging users to build a provisioning flow based on on-host binaries instead of containerized utilities 17:20:44 thanks jdoss 17:20:53 #topic metal installer requires re-fetching metal image (fulliso) 17:20:59 ok, this was an interesting one 17:21:39 I think at a high level people are asking for being able to ship the disk image contents as part of the "installer 17:21:48 " ISO image they boot on a bare metal machine 17:22:15 #link https://github.com/coreos/fedora-coreos-tracker/issues/352 17:22:26 oops, thanks bgilbert 17:23:17 any insights from the discussion there? Should we discuss if there is a feature here we want to implement? 17:23:31 yeah, it's unintuitive that the iso installer requires internet access 17:24:08 i think we should spend some time investigating smart ways to ship the metal image without duplicating the whole thing 17:24:50 jlebon: i.e. by re-using the ostree from the squashfs ? 17:25:03 exactly 17:25:40 could the squashfs just contain one file (the disk image) and then loopback mount that to boot into the live environment? 17:26:00 there'd be no point to squashfs then 17:26:05 does the live-iso squashfs contains the full ostree repo? Or is it just a rendered checkout of the specific commit? 17:26:16 or, well...kind of i guess you'd get compression but 17:26:17 walters: the compression. Fedora does that now. 17:26:19 yeah 17:26:36 so my idea isn't a horrible one? 17:27:03 as a general rule loopback mounts aren't something you want in a production workflow 17:27:19 ahh, and the live ISO image is meant to be used for more than just installs 17:27:36 FWIW the current installer behavior is somewhat intentional: stale ISOs will still install the current OS release, which by policy is what we want people to do. 17:27:39 the redownload _is_ obnoxious though 17:28:36 bgilbert: true, we could have it still default to downloading the latest 17:28:42 right 17:28:54 but support an option to pull from the ISOfs 17:29:04 or use the local image if it _is_ the latest 17:29:11 bgilbert++ 17:29:12 it's not quite just about speed but ease of operating disconnected; which speaking of https://github.com/openshift/enhancements/pull/201 is related a bit 17:29:25 that means we don't plan to support offline flows? 17:29:48 FCOS supports offline today, you'd just have to download the metal image too and host it 17:29:50 offline assumes a local mirror right now, which is awkward 17:30:01 (airgapped, I guess, not really offline) 17:30:15 yeah airgapped is a better term 17:30:31 or just do like keithy and plug in another device containing the image 17:30:50 so does it seem reasonable to investigate the "use the metal image for the live ISO" approach ? 17:30:51 but yeah, clearly want better experience there :) 17:31:48 one thing I could imagine is `coreos-install embed-image` or something; basically given a USB stick, let me inject the Ignition and the metal image 17:33:00 there was also a sub-request in the ticket, which was asking to inject arbitrary files (like OCI images) into the installer 17:33:03 hmm, yeah that's similar to a suggestion i had, though targeting an actual device would make it easier 17:33:12 i.e. customizing the install, before disk-imaging 17:33:40 but I think we should open a separate ticket for that feature 17:33:44 maybe it makes sense to provide a tool for rebuilding the ISO 17:33:51 Ignition embedding was designed to be simple so it can run anywhere 17:33:59 bgilbert: +1 17:34:04 but if you're doing something fancy, maybe you don't mind needing an ISO generation tool 17:34:14 (as a dependency) 17:34:19 right, it doesn't have any complex requirements right now, which i like 17:34:33 .hello2 17:34:34 ignatenkobrain: ignatenkobrain 'Igor Gnatenko' 17:34:50 I'd be happier if 1) we start thinking in terms of "live-ISO" instead of "installer" 2) we try to not to stray into the "custom Linux distro building" toolkit 17:34:50 #chair ignatenkobrain 17:34:50 Current chairs: bgilbert cyberpear dustymabe ignatenkobrain jlebon kaeso[m] mnguyen_ red_beard walters 17:35:30 but if we do find a cheap way to embed the metal image into the live ISO, then i think we should do that vs requiring a separate step for installs 17:35:31 I promised to answer questions how we are going to ship rust packages... 17:35:39 i think a better term here might be "regenerating" or "refreshing" - "rebuild" implies more than that 17:35:46 ignatenkobrain: right, one second and I'll switch the topic 17:36:10 jlebon: would it be the default or something opt-in? 17:36:25 #proposed it would be worth investigating if we could embed the metal image into the live ISO to be re-used for offline installs for #352 17:36:35 walters: ideally the default if it's cheap enough 17:36:35 ack/nack ^^ 17:36:47 so we just keep shipping just one live ISO 17:36:52 jlebon: +1 17:36:58 what about making the live iso image BE the metal image ala isohybrid, but w/ actual partitions? 17:37:45 * dustymabe notes short time 17:37:52 bgilbert: can you hang around for discussion about rust? 17:37:56 dustymabe: yup 17:38:01 cyberpear: that would break diskless i think 17:38:08 I'll add some of this discussion to the ticket 17:38:12 #info fcct in initrd ticket https://github.com/coreos/fedora-coreos-tracker/issues/371 17:38:23 #topic rust packaging in Fedora 17:38:31 sorry about the meeting going over folks 17:38:43 if you need to leave please do, as we don't want to hold you up for other things 17:38:50 welcome ignatenkobrain 17:39:03 thanks for joining us (from the rust packaging SIG) 17:39:37 so this past week bgilbert was going through and going to try to build the new version of rust-coreos-installer for f31 17:39:47 and he asked about creating a branch in dist-git for f32 17:39:50 oops for f31 17:40:01 I remember this not being how it's done any longer 17:40:20 as we build in a side tag from master and then manually tag over into the candidate tag 17:40:45 can you give us a summary of the current state of rust packagin in Fedora and maybe what the next steps are for making it self service for packagers? 17:40:52 So since we have moved away fr modules, I was the only person who is able to build rust packages in stable Fedora. That is due to the fact that sidetag admins can't unblock/block packages. Unfortunately the patch I came up with for sidetag plugin was not good for other people. And recently Koji devs started to bring sidetag plugin into a core Koji. I've asked them again to implement this functionality at 17:40:52 https://pagure.io/koji/pull-request/1956. 17:40:52 We'll see how that goes. So in the meantime, you can ping me to build those packages or ask particular rpms produced by my script unblocked in sidetag. 17:41:27 https://gist.github.com/ignatenkobrain/f2529a3f9e34848fa63587db94089a0f 17:41:46 Here is the way I build in stable branches given that rawhide build is done 17:41:57 ignatenkobrain: so the koji devs are working on the problem (i.e. I see that is a pull request by one of them) ? 17:42:46 So only being able to block-unblock packages in side tag should be needed. Also not entirely sure if requesting f31 branch actually adds a package into Koji, but definitely should. 17:43:43 ignatenkobrain: is the ultimate goal for someone to be able to push to the release branch (i.e. f31) and do `fedpkg build` ? 17:43:54 just like is done for non-rust packages in fedora? 17:43:54 Well, they just copied existing code into a Koji... Not sure if they are really working on implementing this functionality 17:44:32 https://pagure.io/sidetag-koji-plugin/pull-request/9 17:44:32 This was the PR to implement functionality 17:45:16 Not really. Unless fedpkg developers will implement creation of side tag and all those things I put into a gist 17:45:39 But at least after they implement this thing, you should be able to easily consume script and build from master branch into f31/f32 17:45:50 After you already built rawhide pkg 17:46:17 ignatenkobrain: right, but we should probably push for parity here, which means it would be good to engage fedpkg to see if we can do that 17:46:58 i.e. we shouldn't add the extra burden on the packager to know all of this, right? 17:47:05 Sure, I got quite a lot of work at $dayjob, so I can't contribute to Fedora much lately :( 17:47:27 no worries, to be clear I wasn't asking you specifically to do it 17:47:30 Yes, this part definitely should be in fedpkg 17:48:05 ignatenkobrain: ok one or two more questions 17:48:06 all of this needs to be nuked from orbit and replaced with a system that is based on PRs to git 17:48:45 1. the koji PRs you referenced, that just enables people to run https://gist.github.com/ignatenkobrain/f2529a3f9e34848fa63587db94089a0f on their own, correct? 17:48:47 walters: how's that related? 17:49:04 If we would use obs instead of Koji, yes 17:49:27 dustymabe: no, it's not enough 17:50:26 ok, what else is missing? 17:51:42 dustymabe: functionality from https://pagure.io/sidetag-koji-plugin/pull-request/9 17:52:01 right, so we need https://pagure.io/koji/pull-request/1956 and https://pagure.io/sidetag-koji-plugin/pull-request/9 ? 17:52:21 yes, and deployed that in fedora infra 17:52:54 ok, cool I'll try to help drive some of this forward with the team. Do you have any time set up with them already to discuss the future here? 17:53:26 not really, koji devs are usually not possible to catch by any means :) 17:53:52 #action dustymabe to work with ignatenkobrain and koji devs to try to drive rust packaging to a better state 17:54:00 * dustymabe curses bgilbert for loving rust 17:54:02 :) 17:54:03 ignatenkobrain: thanks, this is super-helpful 17:54:11 actually I want to learn rust myself 17:54:14 \o/ 17:54:19 #info to build Rust packages in stable Fedora, build in rawhide and then ping ignatenkobrain 17:54:20 * dustymabe thanks bgilbert for forcing him to learn rust 17:54:38 dustymabe: one thanks + one curse = ? 17:54:38 thanks ignatenkobrain for coming and giving us some insight 17:54:51 bgilbert: :) 17:54:58 #topic open floor 17:55:02 sorry we're way over time 17:55:05 ignatenkobrain: and thanks for manually holding the Fedora Rust ecosystem together while waiting for automation :-) 17:55:18 any critical open floor items to bring up real quick? 17:55:38 #info we did a testing and stable release that started rollout on monday - haven't heard of any issues 17:55:40 btw, if there are some old crates, ping me and I'll add you to the rust-sig so you can update them yourselves ;) 17:55:55 ignatenkobrain: +1 17:56:05 i nominate bgilbert as tribute 17:56:12 oops, did I say that outloud 17:56:20 np, I'm used to it 17:56:27 no, no I didn't. I typed it 17:56:30 :) 17:56:38 will end meeting in 60s 17:57:40 #endmeeting