16:30:14 <jlebon> #startmeeting fedora_coreos_meeting
16:30:14 <zodbot> Meeting started Wed Jun 16 16:30:14 2021 UTC.
16:30:14 <zodbot> This meeting is logged and archived in a public location.
16:30:14 <zodbot> The chair is jlebon. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:30:14 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:30:14 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:30:18 <jlebon> #topic roll call
16:30:25 <lucab> .hello2
16:30:28 <zodbot> lucab: lucab 'Luca Bruno' <lucab@redhat.com>
16:30:35 <lorbus> .hi
16:30:35 <zodbot> lorbus: lorbus 'Christian Glombek' <cglombek@redhat.com>
16:30:37 <darkmuggle> .hello2
16:30:41 <zodbot> darkmuggle: darkmuggle 'None' <me@muggle.dev>
16:30:44 <jdoss> .hi
16:30:44 <cyberpear> .hi
16:30:44 <zodbot> jdoss: jdoss 'Joe Doss' <joe@solidadmin.com>
16:30:46 <bgilbert> .hi
16:30:47 <zodbot> cyberpear: cyberpear 'James Cassell' <fedoraproject@cyberpear.com>
16:30:50 <jlebon> #chair lucab lorbus darkmuggle jdoss cyberpear bgilbert
16:30:50 <zodbot> Current chairs: bgilbert cyberpear darkmuggle jdoss jlebon lorbus lucab
16:30:50 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:31:06 <jaimelm> ./hello2
16:31:10 <jaimelm> .hello2
16:31:11 <zodbot> jaimelm: jaimelm 'Jaime Magiera' <jaimelm@umich.edu>
16:31:30 <jlebon> #chair jaimelm
16:31:30 <zodbot> Current chairs: bgilbert cyberpear darkmuggle jaimelm jdoss jlebon lorbus lucab
16:31:35 <walters> .hello2
16:31:36 <zodbot> walters: walters 'Colin Walters' <walters@redhat.com>
16:31:43 <jbrooks> .hello jasonbrooks
16:31:44 <zodbot> jbrooks: jasonbrooks 'Jason Brooks' <jbrooks@redhat.com>
16:32:38 <jlebon> #chair walters jbrooks
16:32:38 <zodbot> Current chairs: bgilbert cyberpear darkmuggle jaimelm jbrooks jdoss jlebon lorbus lucab walters
16:32:55 <jlebon> let's wait one more minute :)
16:33:06 <walters> #chair 💺
16:33:06 <zodbot> Current chairs: bgilbert cyberpear darkmuggle jaimelm jbrooks jdoss jlebon lorbus lucab walters 💺
16:33:54 <jlebon> alrighty, let's begin!
16:33:59 <jlebon> thanks all for coming
16:34:02 <jlebon> #topic Action items from last meeting
16:34:10 <jlebon> * dustymabe to add butance config to #840 to show how to enable
16:34:11 <jlebon> systemd-oomd
16:34:40 <jlebon> #info dustymabe added butace config in https://github.com/coreos/fedora-coreos-tracker/issues/840#issuecomment-859094939
16:35:07 <jlebon> ahh heh, did a *different* typo from dustymabe
16:35:09 <jlebon> #undo
16:35:09 <zodbot> Removing item from minutes: INFO by jlebon at 16:34:40 : dustymabe added butace config in https://github.com/coreos/fedora-coreos-tracker/issues/840#issuecomment-859094939
16:35:16 <jlebon> #info dustymabe added butane config in https://github.com/coreos/fedora-coreos-tracker/issues/840#issuecomment-859094939
16:35:24 <jlebon> and that's all!
16:35:46 <jlebon> i didn't retag that issue for this meeting, but is there anything anyone wants to bring up on the oomd subject before we move on?
16:36:45 <jlebon> ok cool, let's move on to ticket items
16:37:01 <jlebon> #topic 2021: Revisit SwapOnZram
16:37:04 <jlebon> #link https://github.com/coreos/fedora-coreos-tracker/issues/859
16:37:20 <jlebon> anyone want to introduce this topic?
16:37:43 <jlebon> lucab perhaps?
16:37:52 <walters> i think this just needs someone to assign themselves right?
16:38:02 <skunkerk> .hello sohank2602
16:38:03 <zodbot> skunkerk: sohank2602 'Sohan Kunkerkar' <skunkerk@redhat.com>
16:38:05 <jdoss> I am doing some testing for Dusty this week with my workloads.
16:38:18 <jlebon> #chair skunkerk
16:38:18 <zodbot> Current chairs: bgilbert cyberpear darkmuggle jaimelm jbrooks jdoss jlebon lorbus lucab skunkerk walters 💺
16:38:43 <jlebon> walters: my understanding is we were to discuss whether to re-enable now
16:39:04 <jlebon> or you mean as a spike to investigate before we do so?
16:39:06 <lucab> I'm not sure about this one, I haven't been paying attention to zram but I don't think things changes in the higher levels yet
16:39:41 <lucab> (i.e. anybody using k8s would need to opt-out of this)
16:39:44 <jlebon> dusty pointed out there's acceptance for having this be supported in k8s finally
16:40:07 <jlebon> but until then, we can either: keep it disabled, or enable it and require k8s distros to disable it
16:41:21 <jlebon> and then ideally once k8s support is there, they'd just be able to stop disabling it
16:41:47 <lucab> well yeah, the plan is place, but the timing is:
16:41:49 <walters> ok so this is similar to oomd
16:42:13 <lucab> > It will take a minimum of 3 release cycles to "graduate". I am targeting the 1.22 release [August 2021] for alpha support.
16:42:51 <lucab> so even in the best case, we are really speaking about mid-2022
16:43:15 <jdoss> I think if users can easily opt into using it until k8s gets its act together that is a pretty good balance before enabling it by default.
16:44:02 <jdoss> It would be nice to not have to layer a package to use it at least and have it just turned off.
16:44:06 <jlebon> feels like this and systemd-oomd basically comes down to whether we default to the single node case or the k8s case
16:44:13 <jlebon> either way someone is going to have to do some work
16:44:46 <jlebon> at least in the k8s case it's just part of the distro glue and users themselves don't really have to care
16:45:07 <jdoss> https://github.com/coreos/fedora-coreos-tracker/issues/840#issuecomment-859094939 is what Dusty wants me to test out this week.
16:45:35 <jlebon> jdoss: ack thanks. were you planning to test combinations with zram too?
16:45:43 <jdoss> Yep, I am going to do both.
16:45:58 <jdoss> I have a few nodes with Datadog running so I can get graphs.
16:46:01 <jlebon> +1
16:46:11 <lucab> the zram-generator-defaults RPM is just  /usr/lib/systemd/zram-generator.conf
16:48:07 <jlebon> i'm not sure how to move forward on this. i think we could debate a bit more on what we optimize for, but happy to move on too for now
16:48:26 <jdoss> Punt to next week and lets get some tests going?
16:48:31 <jaimelm> ^^
16:48:40 <jlebon> ack WFM
16:48:57 <jlebon> #topic New Package Request: NetworkManager-wifi
16:49:03 <jlebon> #link https://github.com/coreos/fedora-coreos-tracker/issues/862
16:49:07 <jaimelm> Down the road, it would be hepful to have a sense of how much single node vs k8s use there is of FCOS.
16:49:16 <walters> i feel like this will be much less abstract when we add e.g .`/test okd-e2e` as an optional thing to our repo (and integrate typhoon testing etc.)
16:49:21 <jlebon> bgilbert: i'm going to pick on you
16:49:39 <bgilbert> sure
16:49:53 <bgilbert> to summarize, you mean?
16:50:00 <jlebon> "bgilbert, I choose you!"  /me throws bgilbert pokeball
16:50:01 <jlebon> yeah
16:50:33 <bgilbert> the Raspberry Pi 4 has onboard WiFi and it would be nice to support that board well
16:50:45 <bgilbert> as a cheap way to get bare-metal clusters running for experimentation
16:50:59 <bgilbert> (this presumes aarch64 support of course)
16:51:20 <bgilbert> so the question is, should we add the NetworkManager-wifi package and its small number of dependencies to the base
16:51:32 <bgilbert> it's unfortunate that jlebon picked me, because I'm opposed :-D
16:52:04 <jdoss> The usecase of FCOS on RPi4 is pretty cool.
16:52:14 <jlebon> bgilbert: now argue for the opposite side :)  heh j/k
16:52:14 <bgilbert> adding the packages to the base is relatively straightforward, but supporting WiFi in the initramfs is not.  and FCOS doesn't make much sense without Ignition.
16:52:38 <bgilbert> in particular, WiFi has no defaults, so there's no reasonable "just use DHCP"-like path
16:52:45 <jaimelm> Yeah, it seems to break the model in that regard
16:53:06 <bgilbert> the ticket argues that we can ship the packages but not try to support wifi in the initramfs
16:53:07 <jdoss> yeah how would you tell it what SSID to join, we'd have to add stuff to the Ignition spec?
16:53:18 <bgilbert> jdoss: no, that'd be a circular dependency
16:53:18 <lucab> "just hop into the first open network available" sounds like a great default :)
16:53:23 <darkmuggle> I think that supporting the initramfs is a fools errand
16:53:24 <jdoss> oh boy
16:53:27 <jaimelm> ha
16:53:36 <bgilbert> I'd expect that if we didn't support it in the initrd, we'd rapidly get requests to do so
16:53:38 <jaimelm> f security, we're gonna connect!
16:53:48 <jlebon> hmm, in the RPi case, i think it's more likely one would be image the SD card anyway, right? so then the ignition config would be baked in
16:53:54 <darkmuggle> the counter argument is that supporting it in the initramfs is not required for boot
16:53:58 <jdoss> could write the configs out ya
16:54:20 <bgilbert> we could document a model where users install over wired and then switch to WiFi, or embed the Ignition config as jlebon said
16:54:25 <walters> my vote is document how to do it (though this ties into https://github.com/coreos/fedora-coreos-tracker/issues/401 )
16:54:41 <walters> no the problem is it's not just networking in the initramfs, it's how you get the package onto the system right?
16:54:56 <jlebon> yeah agreed
16:55:04 <jaimelm> +1
16:55:05 <bgilbert> walters: ah, yeah, good point
16:55:19 <bgilbert> but also: we're mainly a server OS aimed at production, and this is a corner case.  I'd love to just document that WiFi is unsupported and move on.
16:55:33 <bgilbert> honestly our networking stack is waaaay too complicated already.
16:55:35 <jdoss> +1 to bgilbert's Document with wired and switch to wifi or read baked configs on boot.
16:55:39 <lucab> specifically for RPi4, do we also have firmwares to consider in the picture?
16:55:44 <jlebon> obviously, you'd just serialize the whole RPM into the Ignition config
16:55:52 <jaimelm> +1 bgilbert and walters
16:56:06 <jaimelm> document and let the edge case folks work it out themself
16:56:18 <walters> eh, in this corner case I think it's a bit more like "drop the rpm in /boot after running coreos-installer"
16:56:21 <jlebon> (/s in case that wasn't obvious)
16:56:39 <jdoss> lol
16:56:39 <bgilbert> jlebon: not 100% obvious, no :-P
16:57:10 * bgilbert adds Butane support for embedding RPM contents
16:57:13 <jlebon> personally leaning towards documenting it as well
16:57:26 <jdoss> stay on target bgilbert stay on target
16:57:40 <jlebon> it doesn't seem worth our time, esp. since this heavily overlaps with IoT
16:57:57 <jaimelm> ^^
16:58:10 <jbrooks> yeah, that's what I'm thinking, the IoT overlap
16:58:10 <bgilbert> there are three flavors of "document": 1. ship wifi package and document switching to it after install, 2. document layering packages in afterward, 3. document not to do that
16:58:21 <bgilbert> which one are folks suggesting?
16:58:24 <jbrooks> And you could rebase from IoT to fcos, I imagine
16:58:37 <bgilbert> I assume option 2 but want to make sure
16:58:38 <jlebon> bgilbert: was thinking 2
16:58:38 <walters> 2
16:58:58 <jlebon> there's non-IoT reasons to need this
16:59:05 <walters> but it may not even be "document" so much as something like a FAQ entry that links to that issue?
16:59:09 <jlebon> heck, I have a desktop in the basement on WiFi
16:59:26 <jdoss> I like making it easy for folks to do basics so I vote 1 as a better path but 2 is fine for now I guess.
16:59:42 <bgilbert> we can always add features; removing them is hard
17:00:00 <jbrooks> Would the extra size for the package be onerous?
17:00:20 <bgilbert> the ticket says 1.8 MB
17:00:28 <bgilbert> ...today.
17:00:43 <jbrooks> Doesn't seem bad
17:00:54 <bgilbert> seems like the kind of thing that might pull in more deps in future
17:00:58 <jlebon> jbrooks: multiply that by every package which goes through our tracker :)
17:01:18 <bgilbert> #proposed we will document how to layer in WiFi packages for installations that need them
17:01:44 <skunkerk> +1
17:01:45 <jaimelm> ack
17:01:53 <jlebon> +1
17:01:55 <jdoss> +1
17:01:58 <jbrooks> And you'd write the config to disk and drop the rpm in boot, and it'd layer on first boot, something like that?
17:02:03 <darkmuggle> +/-1
17:02:09 <jaimelm> heh
17:02:18 <bgilbert> darkmuggle: so +0?
17:02:19 <jlebon> jbrooks: eventually yes, but currently you'd have to script the install
17:02:37 <jlebon> will add something about this in https://github.com/coreos/fedora-coreos-tracker/issues/681
17:02:46 <jbrooks> Documenting seems like a good start, and if there's outcry or something we could revisit
17:02:58 <jdoss> maybe this documentation can help us see the usage and we can revisit option 1 if there is a huge demand for the WiFis
17:03:04 <jaimelm> ^^
17:03:13 <jbrooks> +1
17:03:14 <jdoss> Give me the WiFis
17:03:16 <bgilbert> jbrooks jdoss: yeah
17:03:17 <jaimelm> Usage and need would be good to gauge
17:03:19 <darkmuggle> yeah,  I generally see both sides of the arguments. But WIFI is going to get more ubiquitous. And getting developer adoption by having Wifi support seems generally good.
17:03:32 <jdoss> darkmuggle: +1
17:03:50 <darkmuggle> I'm in favor having the packages in the aarch64 base
17:03:58 <jdoss> I am pro any features that bring us more users.
17:04:17 <darkmuggle> I think that aarch64 is a special case because of the RPi
17:04:29 <bgilbert> jdoss: I am not, necessarily.  FCOS tries to be focused on particular use cases.
17:04:30 <jlebon> i'm really curious about the reach of FCOS on aarch64 once we support it given IoT
17:04:44 <bgilbert> we don't have the capacity to support everything
17:04:48 <jaimelm> generally, but there's effort in maintaining, documenting,etc. There has to be some selectivity.
17:04:54 <jdoss> Mo ~money~ users. Mo problems.
17:04:57 <jlebon> bgilbert: +1
17:05:15 <bgilbert> #agreed we will document how to layer in WiFi packages for installations that need them
17:05:21 <jdoss> \o/
17:05:28 <jaimelm> "I would like FCOS to run on my Speak-n-Spell"
17:05:37 <jdoss> lol yess!
17:05:46 <jlebon> ok, moving on!
17:05:49 <bgilbert> Platform Request: 8051
17:06:01 <jaimelm> jlebon: sorry. continue.
17:06:06 <jlebon> #info Support converting the live ISO into a minimal live ISO
17:06:12 <jlebon> #link https://github.com/coreos/fedora-coreos-tracker/issues/868
17:06:21 <jlebon> #link https://github.com/coreos/coreos-installer/pull/559
17:07:36 <jlebon> so the summary here is: some environments really need CoreOS installation to happen via a virtual media
17:08:03 <jdoss> When they say virtual media does that mean like mounting the ISO over IPMI?
17:08:04 <jlebon> and there are sometimes restrictions on the ISO mounted, or performance issues
17:08:11 <jlebon> jdoss: yeah, exactly
17:08:19 <jaimelm> yeah, which would be fantastic.
17:08:56 <jlebon> this is a real problem which the Assisted Installer team had to solve. so they have code which remaster the ISO
17:09:13 <walters> (OpenShift Assisted Installer to clarify)
17:09:40 <jlebon> this is a proposal to fold this concept into our artifacts so that (1) it simplifies the AI workflow, (2) everyone else in the same situation can benefit
17:10:24 <jdoss> Sounds like a win. Are there any downsides for us?
17:10:50 <jlebon> the way it works roughly is: there's data in the live ISO, at negligible space cost, to describe how to generate a minimal ISO from itself
17:11:04 <jlebon> `coreos-installer iso uproot` uses this data to generate the minimal ISO
17:11:23 <jlebon> the approach is cross-architecture, and provides bit-for-bit reproducibility so that we could sanity-check the artifact at build time
17:11:33 <jlebon> (or ship it eventually if we wanted to)
17:11:58 <jlebon> jdoss: main downside IMO is just increased complexity
17:12:00 <bgilbert> jlebon: briefly, how is that data generated?
17:12:34 <jlebon> bgilbert: basically osmet, but adapted for ISO9660
17:13:15 <jlebon> builds a table of file -> offset, then xzpacks the residue (e.g. partition table)
17:13:44 <walters> (tangentially related we should be migrating from xz -> zstd)
17:13:45 <jlebon> the trippy part is that that file is itself embedded in the live ISO
17:14:02 <jlebon> we pre-allocate a block of data for it
17:15:40 <jlebon> is anyone *opposed* to this?
17:16:11 <bgilbert> mildly disquieted, but no
17:16:38 <jlebon> bgilbert: fair. do you want to elaborate? just the complexity part or something else?
17:16:56 <bgilbert> haven't looked at the PR yet, but the incremental complexity doesn't sound too bad
17:17:18 <bgilbert> OTOH the ISO is gradually becoming an artifact from a futuristic civilization
17:17:31 <jaimelm> heh, that is true
17:17:32 <walters> haha
17:17:33 * cyberpear suggests updating #topic
17:17:36 <jlebon> hehe
17:17:48 <bgilbert> #topic Support converting the live ISO into a minimal live ISO
17:18:05 <jlebon> whoops, thanks!
17:18:06 <cyberpear> 🎉
17:18:26 <jlebon> #info jlebon forgot to change topic earlier. actual discussion starts at previous #info
17:19:10 <jlebon> bgilbert: you can look at the code, but prepare for many hacks :)
17:19:57 <jlebon> #proposed there is overall consensus for pursuing the `coreos-installer iso uproot` approach
17:19:59 <bgilbert> it's an ISO with an MBR and GPT, with holes for embedding Ignition configs and kernel arguments
17:20:11 <jlebon> #undo
17:20:11 <zodbot> Removing item from minutes: INFO by jlebon at 17:18:26 : jlebon forgot to change topic earlier. actual discussion starts at previous #info
17:20:37 <bgilbert> it contains the PXE artifacts, residue for generating the bare-metal artifacts, and will contain residue for creating a smaller version of itself
17:20:47 <bgilbert> for the record :-)
17:21:08 <bgilbert> also, bootloaders for MBR, UEFI, and ISO boots
17:21:15 <jlebon> #proposed there's some concerns about the ISO's increased complexity, though there is overall consensus for pursuing the `coreos-installer iso uproot` approach
17:21:21 <dustymabe> .hi
17:21:22 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
17:21:30 * jdoss waves at Dusty
17:21:33 <jlebon> bgilbert: noted
17:21:35 <jlebon> #chair dustymabe
17:21:35 <zodbot> Current chairs: bgilbert cyberpear darkmuggle dustymabe jaimelm jbrooks jdoss jlebon lorbus lucab skunkerk walters 💺
17:22:15 <jlebon> bgilbert: in your mind, is the problem this is solving something worth tackling a different way, or not tackling at all in our scope?
17:22:27 <bgilbert> jlebon: if the first clause of the #proposed is based on my comments, we can strike it
17:22:58 <bgilbert> jlebon: hurm.  I'm comfortable with the AI continuing to remaster the image themselves, but the IPMI restrictions are worth considering
17:23:16 <bgilbert> we could reconsider just shipping a minimal artifact
17:23:25 <bgilbert> since it's minimal, the size shouldn't be enormous
17:23:32 <bgilbert> 100 MB or so?
17:23:40 <jlebon> 89M for FCOS currently
17:24:03 <bgilbert> jlebon: wdyt?
17:24:30 <bgilbert> my point wasn't that we shouldn't have the complexity, but just that we keep an eye on the level of complexity we do have
17:25:04 <jlebon> bgilbert: worth considering.  even so, it's really nice that you can extract it from the live ISO instead of re-downloading. but if we're OK to ship, it's not obvious we need that feature immediately
17:25:35 <bgilbert> if someone is looking for a minimal ISO, how do we feel about the UX of requiring them to generate it?
17:25:49 <bgilbert> presumably they'd have to know FCOS well enough to go looking for docs
17:25:55 <bgilbert> rather than just noticing that the artifact doesn't exist
17:26:08 <jlebon> agreed
17:26:19 <jlebon> we're nearing the hour.  let's keep chatting upstream about that possibility?
17:26:42 <bgilbert> jlebon: sgtm.  you're more engaged with this issue and I'm comfortable deferring to you on it
17:26:53 <bgilbert> but we can discuss more
17:27:22 <jlebon> bgilbert: ack thanks. i think if we're willing to ship the added artifact, a lot (but not all) of the motivation drops away
17:27:33 <jlebon> ok cool, let's move on then
17:27:39 <jlebon> #topic Open Floor
17:28:05 <jlebon> anyone has anything there?
17:28:06 <cyberpear> feels like re-implementing jigdo (but in r0everse?). IIRC, other distros only shipped jigdo for some of the ISOs and never shipped the ISOs themselves.
17:29:03 <lucab> (this ^ jigdo comment just threw walters in a loop)
17:29:16 <walters> heh
17:29:23 <jaimelm> jigdo. suddenly the early 2000s flash before my eyes.
17:29:40 <dustymabe> just caught up on meeting reading
17:29:56 <jlebon> huh, interesting they wouldn't ship the ISO whole as well
17:30:03 * jlebon waves at dustymabe
17:30:17 <dustymabe> i like not shipping another artifact (keeping confusion down) - people who need the minimal ISO will ask or otherwise find the docs for how to create it
17:30:25 <walters> as William Gibson said, "The future is already here – it's just not evenly distributed." - some of the concerns of jigdo and this are still valid, even though here I have a gigabit internet connection
17:30:31 <dustymabe> just $.02
17:30:39 <bgilbert> jlebon: trying to reduce mirror storage requirements AIUI
17:31:08 <jlebon> dustymabe: let's add that to the ticket
17:31:18 <dustymabe> jlebon: yeah I can add that comment to the ticket
17:31:24 <bgilbert> yeah, I guess the minimal ISO isn't usable as is
17:31:24 <jlebon> ok, will close in 30s since we're over now :)
17:31:37 <bgilbert> since it needs a rootfs_url karg
17:31:49 <dustymabe> walters: considering the rehydrator - i wonder if we could get the same functionality just all using coreos-installer and embedded osmets
17:32:09 <jlebon> #endmeeting