16:29:42 <dustymabe> #startmeeting fedora_coreos_meeting
16:29:42 <zodbot> Meeting started Wed May  8 16:29:42 2019 UTC.
16:29:42 <zodbot> This meeting is logged and archived in a public location.
16:29:42 <zodbot> The chair is dustymabe. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:29:42 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:29:42 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:29:46 <dustymabe> #topic roll call
16:29:49 <dustymabe> .hello2
16:29:50 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:30:02 <kaeso> .hello lucab
16:30:03 <zodbot> kaeso: lucab 'Luca Bruno' <lucab@redhat.com>
16:30:04 <ajeddeloh> .hello2
16:30:06 <zodbot> ajeddeloh: ajeddeloh 'Andrew Jeddeloh' <andrew.jeddeloh@redhat.com>
16:30:30 <slowrie> .hello2
16:30:31 <zodbot> slowrie: slowrie 'Stephen Lowrie' <slowrie@redhat.com>
16:31:23 <jlebon> .hello2
16:31:24 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:31:36 <mnguyen_> .hello mnguyen
16:31:37 <zodbot> mnguyen_: mnguyen 'Michael Nguyen' <mnguyen@redhat.com>
16:31:37 <lorbus> .hello2
16:31:39 <zodbot> lorbus: lorbus 'Christian Glombek' <cglombek@redhat.com>
16:31:59 <ksinny> .hello sinnykumari
16:32:00 <zodbot> ksinny: sinnykumari 'Sinny Kumari' <ksinny@gmail.com>
16:32:49 <lorbus> dustymabe \o/
16:32:59 <lorbus> hope all's well with fam!! =)
16:33:05 <dustymabe> lorbus: woot
16:33:19 <dustymabe> #chair kaeso ajeddeloh slowrie jlebon mnguyen_ lorbus ksinny
16:33:19 <zodbot> Current chairs: ajeddeloh dustymabe jlebon kaeso ksinny lorbus mnguyen_ slowrie
16:33:22 <rfairley> .hello2
16:33:23 <zodbot> rfairley: rfairley 'None' <rfairley@redhat.com>
16:33:24 <dustymabe> #chair rfairley
16:33:24 <zodbot> Current chairs: ajeddeloh dustymabe jlebon kaeso ksinny lorbus mnguyen_ rfairley slowrie
16:35:26 <dustymabe> #info Please add items to the meeting agenda at https://public.etherpad-mozilla.org/p/20190508-FCOS-Meeting
16:35:36 <dustymabe> ^^ if you've got anything else to add please do!
16:35:51 <dustymabe> i've been out for a month and have missed a lot so can't wait to get all caught back up
16:36:04 <dustymabe> hopefully none of it is too redundant
16:36:21 <dustymabe> #topic Action items from last meeting
16:36:25 <dustymabe> * bgilbert to pick a name for zincinnati
16:36:27 <dustymabe> * ajeddeloh to remove selinux tools that need pytho as discussed last
16:36:29 <dustymabe> meeting and poke the xfsprogs ticket
16:36:40 <dustymabe> I think bgilbert is at RH summit today but I can give an update
16:37:24 <dustymabe> #info we are going with zincati for the name - https://github.com/coreos/fedora-coreos-tracker/issues/83#issuecomment-490548168
16:37:35 * ajeddeloh did both, xfsprogs is the last bit of python and we're just waiting on them to pick a name; there's been some activity in the ticket too, so we're not forgotten or anything
16:38:32 <dustymabe> #info from ajeddeloh: only remaining python holdout is xfsprogs and we're close to a solution there
16:38:57 <dustymabe> on to tickets?
16:39:10 <ajeddeloh> +1
16:39:13 <dustymabe> #topic Root on raid and other complex devices
16:39:17 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/94
16:39:53 <ajeddeloh> So this is not urgent or anything, but I want to start thinking about it
16:40:38 <ajeddeloh> but the question is: if we have an immutable initramfs, how do we specify how to find the root filesystem if its non-trivial
16:41:04 <ajeddeloh> rootfs meaning the block device with it
16:41:29 <ajeddeloh> with CL we only allowed root-on-raid and only one level deep
16:41:53 <ajeddeloh> (one level of types of special devices)
16:42:34 <ajeddeloh> we did that by allocating a partition type guid to mean "this is a raid device that contains root or other raid devices that contain root" and should be started in the initramfs
16:42:54 <ajeddeloh> I don't know if that's the best solution though
16:43:55 <ajeddeloh> we also looked at the dracut `rd.auto` flag but decided it did far too much and would end up starting more raid arrays than needed which is bad since those might be intended to be started later with different configuration
16:44:40 <dustymabe> ajeddeloh: regarding `immutable initramfs` - that's important because otherwise we could just embed an mdadm.conf ?
16:44:57 <ajeddeloh> or other configuration, yes
16:45:34 <ajeddeloh> I'm using RAID as an example, but I want to talk about it generally so we arrive at a solution that works for things like luks, lvm, etc
16:45:43 <kaeso> ajeddeloh: our root+var split makes things a bit more complex here compared to CL, right?
16:46:14 <ajeddeloh> kaeso: I don't think so
16:46:28 <dustymabe> ajeddeloh: are we "creating" the raid using ignition?
16:46:34 <ajeddeloh> yes
16:47:03 <dustymabe> could we create the raid using ignition and then add kernel args (for next boot) that tells dracut to assemble that specific raid device ?
16:47:18 <ajeddeloh> kaeso: IIRC all of the special /var stuff happens after mounting the rootfs at /sysroot
16:47:42 <ajeddeloh> jlebon: correct me if I
16:47:45 <ajeddeloh> m wrong
16:48:15 <kaeso> ajeddeloh: but we may need to find and assemble both in initramfs I guess
16:48:36 <ajeddeloh> dustymabe: We looked at dracut's handling of that with CL and didn't really like it. It's a giant pile of bash to discover raid devices
16:48:38 <jlebon> the populating /var stuff happens after mounting yes
16:49:02 * ajeddeloh hasn't thought about using kernel args
16:49:53 <ajeddeloh> one other thought would be to have "bootmap" file in /boot that describes the path to the rootfs and have users write that
16:50:02 <ajeddeloh> though that's less than frictionless
16:50:23 <ajeddeloh> that's similar to the encryption solution kaeso was working on back in the CL days
16:50:42 <kaeso> indeed
16:51:02 <ajeddeloh> encrpytion is especially tricky since there's a significant amount of metadata about where/how to get keys from
16:51:14 <ajeddeloh> probably too much for kcmdline
16:51:26 <ajeddeloh> kaeso: that was in /boot, yes
16:51:31 <dustymabe> could we do something like rd.md.uuid=421a005c:9f585d66:2f941de8:e6ee6f49 ?
16:51:53 <dustymabe> so no `rd.auto` magic, but specifically tell it what we want done
16:52:04 <ajeddeloh> dustymabe: what if I have luks on raid 0 on raid 1, that'd get messy
16:52:29 <kaeso> the /boot thing does not require fiddling with kargs and a new deployment
16:52:56 <kaeso> on the downside, it is not versioned/coupled to the rest of ostree versioning
16:53:12 <ajeddeloh> one downside of the /boot thing is it's more configuraton the user needs to write; Ignition does not have any idea of what the rootfs is, and thus can't generate it
16:53:25 <dustymabe> ajeddeloh: the whole point of the initramfs is to find the root device, mount it, and switch root, right?
16:53:42 <ajeddeloh> dustymabe: well originally, yes
16:54:14 <ajeddeloh> I don't like the term "find" really, it implies any way of getting to it is ok, which I disagree with
16:54:46 <dustymabe> I just don't understand why what we're trying to do is unique
16:54:47 <ajeddeloh> Users should be able to concretely know how their initramfs is mounting the rootfs imo
16:55:11 <ajeddeloh> dustymabe: its unique because on mutable distros you just generate an initramfs with custom configuration
16:56:19 <ajeddeloh> also generally people don't treat initramfs's very strictly; if it finds the rootfs somehow and mounts it somehow, that's fine by most distro's standards (hence rd.auto)
16:57:13 <ajeddeloh> I think we have an opportunity to do something much better defined and much stricter, which fits with the whole "be correct or die" theme
16:57:49 <jlebon> you can regenerate the initrd on FCOS too (rpm-ostree initramfs --enable), but yeah the first boot problem is tricky
16:58:10 <dustymabe> jlebon: but the first boot is when the raid would be created so not a problem?
16:58:29 <ajeddeloh> well the first boot problem is actually not tricky, since Ignition needs to create/start things anyway
16:59:14 <ajeddeloh> although we could make tear eveything back down then let the normal initramfs machinery bring it back up just to be sure (not sure if that's worth it)
16:59:47 <jlebon> ok, i might be misunderstanding the problem space. i haven't played with raid much
17:00:31 <ajeddeloh> re: generating a new initramfs: I don't know how I feel about that
17:00:36 <dustymabe> ajeddeloh: i'd argue if you modify the root filesystem on first boot then I'd let ignition reboot the machine
17:00:48 <ajeddeloh> with CL it wasn't possible since we signed the kernel+initramfs
17:00:50 <dustymabe> i.e. don't worry about tearing things down
17:01:38 <ajeddeloh> I think I'd prefer to tear things back down to rebooting, though that's getting a bit off topic
17:01:56 <dustymabe> tabled
17:02:08 <dustymabe> ok so let's talk about this scenario
17:02:21 <dustymabe> luks on raid
17:02:51 <dustymabe> we could have rd.luks.uuid= and rd.md.uuid=
17:02:58 <dustymabe> on kernel cmdline
17:03:54 <ajeddeloh> but how do you specify how to get the key
17:03:56 <kaeso> dustymabe: the luks configuration I was sketching had way too many knobs to sanely fit into kcmdline
17:04:32 <ajeddeloh> we were going to support cloud key servers, i think vanilla https, etc IIRC
17:04:39 <dustymabe> ajeddeloh: so you're referring to an /etc/crypttab ?
17:05:15 <ajeddeloh> well not really, pretend /etc/crypttab doesn't exist since it's basically just for generators
17:05:33 <ajeddeloh> and handles more than just root
17:06:11 <dustymabe> ok - this disucssion is running a bit long
17:06:26 <dustymabe> i'll add these comments to the ticket
17:06:28 <ajeddeloh> agreed, mostly just wanted to bring it up to get people thinking
17:06:32 <ajeddeloh> +1
17:06:49 <dustymabe> 1 - I think we should come up with specific scenarios we'd like to support (as a result of this also ones we don't care to support)
17:07:02 <ajeddeloh> I actually disagree with that
17:07:20 <ajeddeloh> I think we should solve the problem in the general case
17:07:26 <dustymabe> 2 - would be nice if we don't invent something new here, but if we find things we'd like to support that can be solved in a simple way then maybe we can do that
17:08:09 <dustymabe> ajeddeloh: that goes a bit against out opinionated position
17:08:15 <dustymabe> *our
17:08:35 <ajeddeloh> I don't think so
17:08:43 <dustymabe> i think part of delivering an OS like FCOS is that we can limit the supported options
17:09:05 <ajeddeloh> we should build it such that we can extend it to other things (e.g. lvm) if we want, but limit it initially
17:09:25 <ajeddeloh> i.e. the solution should allow any number of root-on-X things
17:09:37 <ajeddeloh> but we should limit the number of Xs we supprt
17:09:50 <ajeddeloh> but the solution shouldn't be hardcoded to specific set of X
17:10:10 <dustymabe> i think we might be saying close to the same thing
17:10:22 <dustymabe> i'm just saying that let's find out what the set of X is that we care about right now
17:10:37 <ajeddeloh> yeah, from the end users perspective I agree, from a development standpoint I disagree
17:10:38 <dustymabe> and work backwards to see what our options are
17:10:44 <kaeso> nesting them (raid-lvm-luks) also gets tricky, so some kind of limiting is pretty much necessary
17:11:22 <kaeso> ajeddeloh, FYI https://github.com/coreos/coreos-cryptagent/blob/master/Documentation/devel/config.md
17:11:22 <dustymabe> kaeso: yeah, but i'd say we'd need to understand why someone would want to do that (i.e. for each X what is the benefit)
17:11:42 <ajeddeloh> I'm not opposed to that as long as we don't build specific solutions
17:11:50 <ajeddeloh> if that makes sense
17:12:12 <dustymabe> :)
17:12:34 <dustymabe> move on to next ticket?
17:12:36 <ajeddeloh> +1
17:12:57 <dustymabe> #topic open floor: status update on releng tags (coreos-pool and continuous tags)
17:12:58 <kaeso> yup
17:13:28 <dustymabe> we now have koji tags that we can use in FCOS for controlling the content in the repos we pull from
17:13:47 <dustymabe> here is one example where there is a yum repo created from the tag:
17:14:07 <dustymabe> https://kojipkgs.fedoraproject.org/repos-dist/coreos-pool/
17:14:20 <dustymabe> rfairley: ^^ FYI I tagged the afterburn rpm into that repo earlier today
17:14:24 <ksinny> \o/
17:15:06 <dustymabe> so now we'll need to build tooling on top of this to leverage our existing design for our streams implementation
17:15:08 <rfairley> dustymabe++ thanks
17:15:09 <zodbot> rfairley: Karma for dustymabe changed to 1 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:15:15 <rfairley> can see the module build in there
17:15:38 <rfairley> built*
17:15:44 <dustymabe> rfairley: I'm going to do some experimentation today and possibly add that repo as an input to our compose (i.e. in fcos configs)
17:16:28 <dustymabe> regarding the continuous tag - jlebon, not sure if you ever got to look at packit or not?
17:17:15 <jlebon> dustymabe: hmm, i think it'd be cleaner to tag it into the continuous tag instead of pool
17:17:32 <dustymabe> jlebon: there's a bit of a split here with what we've got
17:17:52 <dustymabe> i essentially think that the `continuous` and `pool` tags are for different things
17:17:54 <jlebon> dustymabe: yup i did, and opened https://github.com/packit-service/packit/issues/264
17:18:16 <jlebon> there's still some work left. notably there's no Packit-as-a-service setup yet (though actually, maybe that changed now)
17:18:42 <dustymabe> continuous is more for CI - running builds of 'closer to upstream' things
17:18:45 <jlebon> one thought i had was whether we could just re-use rdgo to create the srpm, and the feed it to koji ourselves
17:19:02 <dustymabe> and pool is more for stable
17:19:48 <dustymabe> so for example we would have a stream that would be like the FACI stream that rebuilt things from upstream sources
17:19:59 <dustymabe> and the koji tag we'd use for that would be the continuous tag
17:20:22 <jlebon> dustymabe: i agree. but tagging  into pool was originally meant to be automated, not something we'd manually tag things into
17:20:37 <jlebon> otherwise that means the mechanical refs never actually pick up things ahead of time
17:20:44 <dustymabe> jlebon: completely agree: 13:15:05  dustymabe | so now we'll need to build tooling on top of this to leverage our existing design for our streams implementation
17:21:43 <dustymabe> all i'm saying now is that the continuous repo should be used for continous rpm builds and output to a continous stream (for CI) and coreos-pool should be used for our more official builds
17:21:46 <jlebon> dustymabe: what i mean is, it might be cleaner to get it into the compose separately, and let it get auto-tagged just like everything else as it goes into the non-mechanical refs
17:22:41 <dustymabe> jlebon: hmm - not sure what you mean - maybe grab me after the meeting to work out the details
17:22:55 <jlebon> +1
17:23:01 <dustymabe> next topic
17:23:09 <dustymabe> unless anyone has anything else?
17:23:44 <dustymabe> #topic open floor: Super minor flamewar: Ignition uses camelCase, CLCs use snake_case, what should FCCs use?
17:23:49 <dustymabe> ajeddeloh: ??
17:23:55 <ajeddeloh> haha
17:24:23 <ajeddeloh> So yeah, all of the keys (e.g. authorizedSSHKeys) in ignition are camelCase
17:24:33 <ajeddeloh> all of the keys in ct are snake_case
17:24:48 <slowrie> kebab-case (not serious, just to trigger ajeddeloh)
17:25:02 <ajeddeloh> ct actually has to translate between the two, though it's not too difficult
17:25:26 <dustymabe> that sure is not fun
17:25:36 <ajeddeloh> from a user perspective I don't know what's more confusing, FCCs being different from Ignition or FCCs being different from CLCs
17:25:46 <dustymabe> which one would it be harder to change?
17:25:52 <jlebon> slowrie: i actually like dash-case for yaml docs
17:25:56 <jlebon> see e.g. https://github.com/projectatomic/rpm-ostree/pull/1749
17:26:01 <dustymabe> dash-case++
17:26:05 <dustymabe> :)
17:26:44 <ajeddeloh> One thing I _dont_ think we should do is support all of them
17:26:57 <jlebon> it's easier to type out too since you don't have to hold shift :)
17:27:07 <jlebon> ajeddeloh: oh yeah, definitely
17:27:09 <ajeddeloh> fun fact: iirc yaml supports spaces in keys without quotes
17:27:25 <kaeso> (argh)
17:27:26 <dustymabe> yeah we should pick one IMHO
17:28:03 <ajeddeloh> yaml is wild. I don't really have a strong opinion on this one, was bringing it up in case someone else did
17:28:25 <ajeddeloh> if no one does, I'll default to snake_case like CLCs for CL users migrating
17:29:16 <kaeso> I do like snake_case everywhere, but I have the feeling there is a minority of us
17:29:22 <dustymabe> ajeddeloh: dumb question
17:29:30 * jlebon has a slight preference for dash-case overall, but between camelCase and snake_case i'd rather the latter
17:29:37 <dustymabe> could we just migrate everything to the same thing ?
17:30:04 <dustymabe> i.e. would that change involve changing ignition? would that be bad?
17:30:21 <ajeddeloh> nothing would change in Ignition
17:30:32 <ajeddeloh> oh wait I see what you're saying
17:30:36 <kaeso> dustymabe: yes, I don't think that that option is on the table
17:30:47 <ajeddeloh> we could, would be a major version bump, but translatable
17:30:59 <ajeddeloh> would be significant amount of churn though
17:30:59 <dustymabe> ajeddeloh: did we release the final version of ignition 2 yet :)
17:31:05 * ajeddeloh screams
17:31:08 <kaeso> lol
17:31:21 <jlebon> heh
17:31:45 <dustymabe> the other option - if it's easily translatable is to support both forms in ignition and only publish docs on the new form
17:31:58 <ajeddeloh> arg no
17:31:59 <dustymabe> and then we can have them match everywhere
17:32:12 <ajeddeloh> Ignition more than ct should only support one
17:32:23 <slowrie> I'd argue that if we wanted CT & Ignition to match
17:32:26 <slowrie> Then CT should be the one changed
17:32:28 <slowrie> not Ignition
17:32:32 <ajeddeloh> once we have tooling uesrs shouldn't be writing ignition configs by hand
17:32:58 <ajeddeloh> and if we do want to change ignition we should do it with a clean spec version bump
17:33:21 * dustymabe quiets down and leaves with this vote:
17:33:23 <ajeddeloh> like the Ignition spec versioning allows for this, and we should use that instead of making it sloppy accepting both
17:33:30 * dustymabe has a slight preference for dash-case overall, but between camelCase and snake_case i'd rather the latter
17:34:05 <kaeso> seems like current snake_case is good enough
17:34:17 <ajeddeloh> We're about out of time, I'll create a tracker issue and we can all comment there?
17:34:29 <dustymabe> +1
17:34:37 <dustymabe> although I think we mostly figured it out
17:35:03 <dustymabe> closing out meeting soon
17:35:18 <dustymabe> #endmeeting