16:31:14 <dustymabe> #startmeeting fedora_coreos_meeting
16:31:14 <zodbot> Meeting started Wed May  6 16:31:14 2020 UTC.
16:31:14 <zodbot> This meeting is logged and archived in a public location.
16:31:14 <zodbot> The chair is dustymabe. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:31:14 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:31:14 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:31:20 <dustymabe> #topic roll call
16:31:23 <cyberpear> .hello2
16:31:24 <zodbot> cyberpear: cyberpear 'James Cassell' <fedoraproject@cyberpear.com>
16:31:27 <dustymabe> .hello2
16:31:30 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:31:56 <jlebon> .hello2
16:32:00 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:32:04 <skunkerk> .hello sohank2602
16:32:05 <zodbot> skunkerk: sohank2602 'Sohan Kunkerkar' <skunkerk@redhat.com>
16:32:26 <lucab> .hello2
16:32:27 <zodbot> lucab: lucab 'Luca Bruno' <lucab@redhat.com>
16:34:04 <dustymabe> #chair cyberpear skunkerk lucab jlebon
16:34:04 <zodbot> Current chairs: cyberpear dustymabe jlebon lucab skunkerk
16:34:41 <dustymabe> #topic Action items from last meeting
16:34:51 <dustymabe> no action items from last meeting \o/
16:35:04 <dustymabe> #topic agenda for this meeting
16:35:36 <dustymabe> #info if you'd like to bring up things to discuss in these meetings please add a `meeting` label to the ticket or add a comment to an issue to ask a meeting label be added to the ticket
16:35:37 <cyberpear> new ignition release to unblock OKD (spec3 doesn't work OpenStack; spec2 does)
16:35:55 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues
16:36:00 <cyberpear> (if that's the correct thing that's blocking; I think it is)
16:36:07 <bgilbert> .hello2
16:36:07 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:36:27 <bgilbert> if we're proposing meeting topics, I'd like to discuss https://github.com/coreos/fcct/issues/76#issuecomment-624448723 if we have time
16:36:37 <dustymabe> bgilbert: +1
16:36:45 <dustymabe> jlebon: did we want to bring up https://github.com/coreos/fedora-coreos-tracker/issues/390 ?
16:37:02 <jlebon> dustymabe: yeah, let's do that
16:37:11 <dustymabe> walters: FYI ^^
16:37:22 <dustymabe> cyberpear: is there an issue associated with your proposed topic ?
16:38:09 <cyberpear> https://github.com/openshift/okd/issues/127 I think
16:39:20 <dustymabe> ok one sec while I put things together
16:39:49 <dustymabe> #topic [next] auto-updates are disabled on 32.20200416.1.0
16:39:55 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/473
16:40:00 <dustymabe> this is an FYI topic
16:40:37 <dustymabe> #info if you are currently running the next stream you will need to manually change a setting to enable auto updates. Workaround provided in https://github.com/coreos/fedora-coreos-tracker/issues/473#issuecomment-623817298
16:41:08 <dustymabe> I have proposed text for a coreos-status mailing list post: details in: https://github.com/coreos/fedora-coreos-tracker/issues/473#issuecomment-623837085
16:41:43 <bgilbert> dustymabe: are you looking for proposed edits?  where should they go?
16:41:46 <bgilbert> that hackmd is read-only to me
16:42:19 <dustymabe> bgilbert: yeah looking for proposed edits, though I don't think hackmd has a good suggestion/comment ability
16:43:00 <dustymabe> I can open it up for edits but would kind of like to control it a bit since the communication will come from me
16:43:02 <bgilbert> sure
16:43:14 <dustymabe> maybe just add suggestions on a separate line with your name in front of it
16:43:31 <bgilbert> +1
16:43:46 <dustymabe> ok it should be editable now
16:44:13 <dustymabe> shall we move on to the next topic ?
16:44:47 <lucab> dustymabe: draft text looks good, I would move command inline at the bottom
16:45:13 <dustymabe> lucab: you mean the workaround commands ?
16:45:24 <lucab> dustymabe: yep
16:45:39 <dustymabe> ok yeah let's chat after meeting and I can see about doing that
16:45:46 <dustymabe> #topic new ignition feature needed for Openstack+OKD
16:45:48 <dustymabe> #link https://github.com/openshift/okd/issues/127
16:45:56 <dustymabe> cyberpear: this is you
16:46:19 <bgilbert> I just submitted Ignition 2.3.0 builds to koji
16:46:39 <cyberpear> https://github.com/coreos/ignition/issues/973
16:46:44 <dustymabe> cyberpear: is the only problem here just getting a new ignition version?
16:46:46 <cyberpear> great!
16:47:15 <cyberpear> yeah, then some FCOS with it included
16:47:36 <dustymabe> cool.. looks like it's already in progress then
16:47:42 <cyberpear> 👍
16:48:02 <cyberpear> thanks! that's all from me
16:48:13 <dustymabe> #info new ignition 2.3.0 has been released and is in the process of being added to FCOS for the next round of releases
16:48:38 <dustymabe> and when I say `next` I mean the one in 2 weeks
16:48:46 <cyberpear> 👍
16:48:50 <dustymabe> but will be in testing devel sooner than that
16:49:00 <dustymabe> well that one was easy.. thanks cyberpear bgilbert
16:49:16 <dustymabe> cyberpear++
16:49:19 <dustymabe> bgilbert++
16:49:19 <zodbot> dustymabe: Karma for bgilbert changed to 1 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
16:49:24 <dustymabe> #topic filetranspiler in fcct
16:49:26 <dustymabe> #link https://github.com/coreos/fcct/issues/76#issuecomment-624448723
16:49:42 <cyberpear> bgilbert++
16:49:42 <zodbot> cyberpear: Karma for bgilbert changed to 2 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
16:50:21 <bgilbert> that comment link is a proposal that I wanted to get people's thoughts on
16:50:40 <bgilbert> walters ^
16:50:44 <jlebon> bgilbert: what does "relative to files-dir" mean?
16:50:50 <cyberpear> how many LOC does it add?
16:51:09 <bgilbert> cyberpear: not that many, I suspect
16:51:17 <bgilbert> jlebon: -d/--files-dir is added as part of https://github.com/coreos/fcct/pull/99
16:51:33 <bgilbert> it's a command-line arg giving the root for file inclusions
16:51:37 <cyberpear> MachineConfig vs Ignition is also interesting. are they the same?
16:52:00 <jlebon> ahh gotcha
16:52:02 <bgilbert> cyberpear: out of scope for fcct, unfortunately, as is Ignition spec 2 support
16:52:22 <bgilbert> jlebon: it's off by default, so users don't inadvertently leak information from the host running fcct
16:52:23 <cyberpear> except MC happens after the fact?
16:52:39 <bgilbert> gonna defer to someone else on the MC questions :-)
16:52:51 <jlebon> bgilbert: right yeah, makes sense
16:53:02 <bgilbert> so the question is, does the proposal seem to meet the need
16:53:18 <bgilbert> I'd like to stabilize FCC spec 1.1.0 soon for Ignition spec 3.1.0
16:53:22 <dustymabe> bgilbert: seems like it to me
16:53:27 <bgilbert> and it might make sense to include this
16:53:36 <jlebon> yup, looks like a good proposal to me!
16:53:53 <dustymabe> bgilbert: one possible added feature
16:54:16 <dustymabe> add an option for fcct to generate an single fcct file given a tree
16:54:39 <dustymabe> previously fcct configs were contained within a single file
16:54:51 <dustymabe> this will enable them to be defined by a file+environment
16:54:58 <bgilbert> dustymabe: I don't follow
16:55:18 <bgilbert> the idea here is that a tree is included by some (as few as 2) lines in an existing FCC
16:55:27 <dustymabe> bgilbert: correct
16:55:55 <bgilbert> is the proposal to have fcct generate the FCC?
16:56:11 <dustymabe> i think what I'm proposing is that we have the ability to distill the fcct back into a single file (for sharing)
16:57:06 <dustymabe> not a big deal, we can discuss further in the ticket
16:57:09 <bgilbert> and the single file would be an FCC rather than Ignition config?
16:57:13 <dustymabe> bgilbert: right
16:57:22 <bgilbert> that... would be a significant change to how fcct works
16:57:44 <dustymabe> ahh ok. we can discuss that as a future feature then
16:57:52 <bgilbert> +1
16:57:58 <dustymabe> next topic
16:58:11 <walters> sharing via "fcct in git repo" seems like a better thing to encourage
16:58:27 <dustymabe> #topic
16:58:29 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/390
16:58:38 <dustymabe> jlebon: want to take this one?
16:58:57 <jlebon> yup, sure
16:59:01 <bgilbert> #topic Publish the initrd and rootfs images separately for PXE booting
16:59:08 <bgilbert> #link https://github.com/coreos/fedora-coreos-tracker/issues/390
16:59:11 <bgilbert> #link https://github.com/coreos/fedora-coreos-tracker/issues/390
16:59:41 <jlebon> so last time we talked about this, we kinda left it at a point where more investigation was needed to try to get to the bottom of the slowdown issues
16:59:54 <dustymabe> thanks bgilbert - sorry my copy/paste failed me on that one
17:00:02 <bgilbert> +1
17:00:31 <jlebon> and while we don't have as much detailed information as i would've liked on that problem, it seems like more and more people are hitting it
17:01:12 <jlebon> relatedly, walters has a proposal in https://github.com/coreos/fedora-coreos-tracker/issues/390 to change how the live ISO is implemented
17:02:16 <jlebon> those are somewhat independent, but it's good to think about the live issues together since they do share some common problems
17:02:58 <dustymabe> can we get a quick summary of what the options are?
17:03:39 <jlebon> it's essentially (1) do we provide the rootfs as a separate artifact, and if so, (2) what kind of artifact is it exactly and how is it plumbed into FCOS
17:04:15 <jlebon> there are options for (2), but seems like we should decide on (1) first
17:04:39 <dustymabe> if we don't provide the rootfs as a separate artifact it implies keeping it embedded in the initramfs?
17:04:40 <cyberpear> basically, fetching stuff in the PXE stage is slow, but having the initrd do it is much faster
17:05:43 <jlebon> dustymabe: i suppose so, but not necessarily. one could imagine providing a tool that rebuilds it from one of our other artifacts (not saying we want that of course)
17:05:47 <cyberpear> (why exactly eludes me, but apparently it's common between ipxe and traditional TFTP)
17:06:01 <dustymabe> jlebon: right
17:06:15 <dustymabe> so it's almost like 1 and 2 *could* be independent then
17:06:33 <dustymabe> so maybe let's discuss 2 first
17:07:16 <dustymabe> what are those options for (2) that are on the table?
17:07:48 <jlebon> well, the main problem we're trying to solve is slow boot speeds, and that won't be resolved unless we shrink the initrd, which does imply a second artifact or pointer of some kind
17:07:59 <jlebon> right, so
17:08:13 <jlebon> some things we discussed were:
17:08:27 <jlebon> - just publish the rootfs as a cpio
17:08:38 <jlebon> i should label these
17:08:42 <jlebon> (a) just publish the rootfs as a cpio
17:09:19 <jlebon> (b) keep publishing the fat initrd, but provide a tool to split out the rootfs
17:09:48 <jlebon> (c) related to the above, just publish the offset at which to split the fat initrd
17:10:22 <jlebon> (d) publish the rootfs as a squashfs
17:10:35 <dustymabe> b and c imply a "stage2" mechanism, correct? a could imply a stage2 depending on how we required people to use it
17:11:09 <jlebon> right, all 4 options require a stage2 mechanism to solve the slow boot problem
17:11:42 <jlebon> the advantage of (a) is that it's more similar to what we're already doing
17:12:06 <jlebon> see https://github.com/coreos/fedora-coreos-tracker/issues/390#issuecomment-588323611 -- a PXE config could just provide multiple initrds to keep the status quo
17:12:09 <dustymabe> is (a) the rootfs squashfs inside a cpio archive ?
17:12:22 <jlebon> right
17:12:43 <dustymabe> so (a) (b) and (c) all are basically getting to the same end point, just different ways to do it
17:13:00 <dustymabe> (d) is publishing the squashfs (not in cpio archive)
17:13:26 <jlebon> right, yup. in which case, you'd feed it as a stage2 karg or something
17:13:40 <jlebon> it's also easier to understand what the heck it is as an artifact
17:13:52 <dustymabe> right
17:14:17 <dustymabe> so at a highlevel we can ask ourselves: "do we prefer squashfs in cpio, or just squashfs?"
17:14:33 <dustymabe> the negative is that combinding initrds in PXE won't work any longer if we go away from cpio
17:15:26 <dustymabe> any opinions here?
17:16:06 <jlebon> there's also this related ticket: https://github.com/coreos/fedora-coreos-tracker/issues/399
17:16:21 <jlebon> (about factory reset capabilities)
17:16:56 <bgilbert> from a UX perspective I like publishing it as a cpio archive
17:17:00 <bgilbert> from a code perspective it's messier
17:17:56 <jlebon> right, UX wise I prefer (a) as well. it just seems slightly odd to publish as an artifact. but meh...
17:18:22 <cyberpear> I'd avoid the extra layer without significant benefit
17:18:25 <dustymabe> (a) gives us the opportunity to decide later what we want to do about (1)
17:18:45 <bgilbert> cyberpear: the benefit is that we're not forcing anyone to use the karg indirection
17:18:54 <cyberpear> fair enough
17:19:05 <dustymabe> since (a) (b) and (c) are all rootfs.squashfs.cpio
17:19:30 <dustymabe> though if we provide just rootfs.squashfs - could we possibly use that for other things in the future related to osmet ?
17:20:10 <cyberpear> I suppose any tool could learn to unpack first; not sure what the RAM impact is
17:20:28 <jlebon> OTOH, it's not hard to take a squashfs and wrap it into a CPIO
17:20:54 <jlebon> but there are semantics there that would have to be taught, like what filename it should be
17:21:08 <bgilbert> jlebon, the cpio tool is mildly painful
17:21:27 <dustymabe> if we think about this from a "no manual steps" perspective I think (a) wins
17:21:48 <dustymabe> since our build tools automate creating the root.squashfs.cpio and our stage2 code would automate unpacking it
17:22:24 <dustymabe> for (b) the manual step would be if a PXE user wanted to keep existing functionality and combine initrds using PXE
17:22:31 <jlebon> cyberpear: that's a good point re. RAM usage. cpio is streaming, so we should be able to pipe it through without downloading upfront
17:22:32 <dustymabe> sorry for (d)
17:23:36 <jlebon> right, yeah
17:23:48 <dustymabe> ok should I summarize?
17:24:00 <dustymabe> or are there any other points to be made?
17:24:30 <jlebon> yup, go ahead
17:24:42 <cyberpear> given streaming, probably fine w CPIO for downloads, and with ISO, it's also not loaded twice?
17:25:41 <dustymabe> In order to solve the problem related to "slow PXE" we think it would be best to provide a mechanism to load the rootfs via a CPIO archive that is optionally fetched via HTTPS in the initramfs.
17:26:00 <dustymabe> We haven't yet made a decision about how the user gets that CPIO archive
17:26:31 <dustymabe> or maybe it's implied that we'll be shipping a "thin" initramfs from this point forward
17:27:02 <dustymabe> basically the above statements says "we need to support a stage2 mechanism"
17:27:35 <jlebon> the question is whether we're ok breaking existing setups that rely on the fat initrd and have them tweak their PXE configs
17:27:40 <dustymabe> "we need to support a stage2 mechanism that loads a rootfs.squashfs.cpio"
17:27:55 <jlebon> (and IMO, yes i think it's acceptable)
17:28:09 <jlebon> right, i think we're agreed we need a stage2
17:28:12 <dustymabe> jlebon: I think I'm OK with it given appropriate communication to the community
17:28:51 <dustymabe> the nice thing is the new proposal still gives them an opportunity to have the same behavior by specifying two initrds
17:28:59 <jlebon> so... let's just agree on stopping to ship the fat initrd and starting to ship the base initrd + the rootfs cpio?
17:29:25 <dustymabe> +1 - and the ISO will include the rootfs.squashfs.cpio
17:29:30 <walters> do we need to deprecate the old path and maybe add a sleep(60) and some visible logs for a bit?
17:29:56 <bgilbert> at the minimum, we'll need to ratchet cosa
17:30:02 <dustymabe> is there a way to deprecate the old path?
17:30:20 <bgilbert> maybe start this only in `next`, then `testing`, then `stable`, with a coreos-status post warning people
17:30:28 <jlebon> from the booted system, there's no discernable difference between one or two initrds
17:30:42 <walters> well
17:31:02 <walters> ratcheting cosa makes sense as a general rule, but I think we also need to start using tagged releases or so for stable FCOS builds
17:31:06 <jlebon> i guess we could change the location of the rootfs to tell them apart?
17:31:13 <bgilbert> walters: sure, fair enough
17:31:22 <dustymabe> walters: +1
17:31:24 <jlebon> yeah, we definitely need that
17:31:33 <jlebon> https://github.com/coreos/fedora-coreos-streams/issues/64
17:32:06 <bgilbert> and an explicit failure msg with instructions if we can't find the rootfs
17:33:01 <dustymabe> #proposed In order to solve the problem related to "slow PXE" we think it would be best to provide a "stage2" mechanism to load the rootfs via a CPIO archive that is optionally fetched via HTTPS in the initramfs. We will now split the current "fat" initramfs into an initrd+ rootfs.squashfs.cpio that can be used.
17:33:19 <bgilbert> +1
17:33:33 <jlebon> +1
17:33:34 <bgilbert> oh, also, we should consider version-locking the rootfs
17:33:35 <dustymabe> slight edit "into two artifacts"
17:33:47 <cyberpear> +1
17:33:54 <bgilbert> though I guess the initrd might not know the FCOS version
17:34:13 <jlebon> re. the ISO, thoughts on just having it use the same stage2 mechanism? that way, it ships the cpio too, and users can explode it and re-use it as is for PXE
17:34:30 <dustymabe> jlebon: yeah except the stage2 just finds an existing file
17:34:38 <jlebon> exactly
17:34:43 <dustymabe> that would be my desire
17:34:45 <bgilbert> I'm not wild about encouraging users to disassemble the ISO, but sure
17:35:05 <dustymabe> any other ack/nack on the #proposed?
17:35:13 <dustymabe> walters: how does the proposal sound?
17:35:22 <jlebon> bgilbert: isn't that a common pattern though?
17:35:38 <dustymabe> jlebon: it is at least in RH land.. maybe not outside
17:35:45 <bgilbert> jlebon: I sure hope not
17:35:56 <jlebon> ahhh sure   /me takes off RH-coloured glasses
17:36:14 <walters> dustymabe: SGTM, I had already commented in the ticket with that I think
17:36:16 <cyberpear> certainly common to mount an ISO for both pxe and repos access
17:36:30 <dustymabe> bgilbert: for example here is what I would often do in the past https://dustymabe.com/2019/01/04/easy-pxe-boot-testing-with-only-http-using-ipxe-and-libvirt/
17:36:44 <dustymabe> i.e. grab the ISO. copy files out, serve PXE from there
17:36:51 <bgilbert> yeah, I know it's a thing.  on s390x apparently that assumption is baked in pretty deeply
17:37:02 <bgilbert> but I don't think we should encourage for FCOS
17:37:21 <dustymabe> #agreed In order to solve the problem related to "slow PXE" we think it would be best to provide a "stage2" mechanism to load the rootfs via a CPIO archive that is optionally fetched via HTTPS in the initramfs. We will now split the current "fat" initramfs into two artifacts (an initrd + a rootfs.squashfs.cpio) that can be used.
17:37:27 <bgilbert> constrains the ISO implementation too much
17:37:30 <walters> i think part of this is that for RHEL the "download gigantic DVD with everything" is convenient for mirroring , which covers not just PXE but also other stuff like GCC
17:37:47 <dustymabe> #topic open floor
17:37:50 <walters> (veering slightly offtopic I have been thinking about doing something similar for OpenShift that includes the release image containers etc.)
17:37:52 <dustymabe> sorry for going over time today (again)
17:37:55 <jlebon> related to this is https://github.com/coreos/fedora-coreos-tracker/issues/221
17:38:15 <bgilbert> now that F32 is out, should we start signing new upstream release artifacts with the F32 key?
17:38:17 <bgilbert> ignition-validate etc.
17:38:23 <walters> i think the mirroring concern is less for FCOS because there's no subscription gate
17:38:24 <dustymabe> bgilbert: yep
17:38:44 <jlebon> yup, makes sense
17:38:46 <bgilbert> +1
17:39:17 <dustymabe> any other topics for open floor ?
17:39:50 <dustymabe> #endmeeting