16:30:12 <dustymabe> #startmeeting fedora_coreos_meeting
16:30:12 <zodbot> Meeting started Wed Oct  5 16:30:12 2022 UTC.
16:30:12 <zodbot> This meeting is logged and archived in a public location.
16:30:12 <zodbot> The chair is dustymabe. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:30:12 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:30:12 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:30:16 <dustymabe> #topic roll call
16:30:18 <bgilbert> .hi
16:30:18 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:30:18 <dustymabe> .hi
16:30:21 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:30:34 <gursewak> .hi
16:30:35 <zodbot> gursewak: gursewak 'Gursewak Singh' <gurssing@redhat.com>
16:30:39 <ravanelli> .hi
16:30:40 <zodbot> ravanelli: ravanelli 'Renata Ravanelli' <renata.ravanelli@gmail.com>
16:31:06 <jlebon> .hello2
16:31:07 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:31:49 <dustymabe> #chair bgilbert gursewak ravanelli jlebon
16:31:49 <zodbot> Current chairs: bgilbert dustymabe gursewak jlebon ravanelli
16:33:04 <dustymabe> welcome all
16:33:07 <dustymabe> #chair jbrooks
16:33:07 <zodbot> Current chairs: bgilbert dustymabe gursewak jbrooks jlebon ravanelli
16:33:13 <davdunc> .hello2
16:33:15 <zodbot> davdunc: davdunc 'David Duncan' <davdunc@amazon.com>
16:33:17 <dustymabe> #chair davdunc
16:33:17 <zodbot> Current chairs: bgilbert davdunc dustymabe gursewak jbrooks jlebon ravanelli
16:33:26 <dustymabe> #topic Action items from last meeting
16:33:31 <dustymabe> Looks like there were no action items from last meeting!
16:33:36 <dustymabe> \o/
16:34:33 <dustymabe> #topic Fedora CoreOS 37 test day
16:34:37 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1225
16:35:16 <dustymabe> Thanks everyone who helped us test F37 based CoreOS for the test day last week
16:35:35 <dustymabe> Thank you to jbrooks for organizing the test day!
16:35:40 <dustymabe> jbrooks++
16:35:45 <jbrooks> .hello jasonbrooks
16:35:46 <zodbot> jbrooks: jasonbrooks 'Jason Brooks' <jbrooks@redhat.com>
16:36:02 <jlebon> jbrooks++
16:36:10 <dustymabe> looks like we didn't find any major issues - but did have some docs updates (always welcome)
16:36:24 <dustymabe> fifofonix also added some new documentation for proxying!
16:36:26 <dustymabe> fifofonix++
16:36:57 <jlebon> nice!
16:37:16 <dustymabe> #info we had a successful test day. No major issues were found with F37 based FCOS on the `next` stream. We had several documentation updates that were proposed and some new documentation that was created as well!
16:37:18 <jbrooks> fifofonix++
16:38:12 <dustymabe> if it turns out that DigitalOcean doesn't work in the end we can blame jdoss
16:38:18 <dustymabe> :)
16:38:38 <dustymabe> #topic Supporting FIPS mode in FCOS
16:38:47 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/302
16:39:06 <dustymabe> jlebon: i'm going to let you drive this topic
16:39:21 <fifofonix> hi
16:39:29 <jlebon> sure
16:39:31 <fifofonix> .hello
16:39:31 <zodbot> fifofonix: (hello <an alias, 1 argument>) -- Alias for "hellomynameis $1".
16:39:54 <dustymabe> #chair fifofonix
16:39:54 <zodbot> Current chairs: bgilbert davdunc dustymabe fifofonix gursewak jbrooks jlebon ravanelli
16:40:53 <jlebon> so FIPS originally used to work in FCOS when we initially added support for it in RHCOS. then it broke at some point. fifofonix mentioned that it works again now in f37. we don't really document it as supported anywhere but some FCOS users have chimed in saying it's a use case they're interested in.
16:41:22 <jlebon> so we need to decide if we want to officially support FIPS mode, and if so, figure out what the UX for it should be
16:41:32 <jlebon> (and actually test it in CI)
16:42:15 <jlebon> for comparison, in RHCOS there's glue code that makes it painless for customers. they just need to enable a knob in the MachineConfig.
16:42:34 <jlebon> some of that glue code should go away I think now that we have Ignition kargs but I suspect we'll still need some
16:42:51 <dustymabe> I don't have a strong opinion here
16:43:04 <bgilbert> if we care about FIPS 140-2 and not just 140-3, then Butane needs to know that FIPS is enabled when configuring disk encryption
16:43:21 <dustymabe> considering it is working now it would be nice to have a test for it, but I don't really know how much time we can spend on it
16:43:37 <bgilbert> the Butane openshift variant handles that with the openshift.fips flag but there's currently no equivalent for FCOS
16:44:15 <fifofonix> well, as a consumer i would be happy to ditch the systemd unit i am using to enable this currently.
16:44:34 <bgilbert> there's an argument that FCOS tries to be modern and so should just support 140-3
16:44:45 <bgilbert> but there's also an argument that FCOS tries to be modern and so should completely ignore FIPS :-)
16:44:49 <walters> does seem like a common butane field makes sense
16:44:53 <jlebon> bgilbert: i.e. with openshift, if the knob is on, Butane auto-enables encryption and adds some extra cryptsetup options?
16:44:58 <bgilbert> jlebon: yup
16:45:04 <bgilbert> well
16:45:04 <jlebon> cool, didn't know that
16:45:08 <bgilbert> well, it doesn't auto-enable encryption
16:45:17 <bgilbert> but it adds the cryptsetup options if encryption is also enabled
16:45:18 <dustymabe> bgilbert: if we were to add "sugar" to enhance the UX then maybe butane could just do the right thing if that "sugar" was enabled
16:45:20 <jlebon> ahh gotcha
16:45:43 <bgilbert> dustymabe: indeed
16:46:09 <bgilbert> on the openshift side we'd have the same problem we have with kernel arguments, which is that there'd be two ways to enable FIPS and we'd probably want to omit one
16:47:24 <bgilbert> in general, since we don't certify for FIPS and have no intention of certifying for FIPS, I'm a bit uncomfortable claiming to support it
16:47:51 <jlebon> bgilbert: i wonder what the overall Fedora stance is there
16:48:03 <dustymabe> yeah, though I do see value in testing upstream something that we need downstream
16:48:07 <bgilbert> dustymabe: indeed
16:48:22 <jlebon> if it's "supported but not officially certified" then matching that seems reasonable
16:48:31 <dustymabe> so for example.. the sugar,etc,documentation could be a nice to have
16:48:41 <dustymabe> but could come with a large disclaimer
16:48:54 * walters has a flashback to https://bugzilla.redhat.com/show_bug.cgi?id=1380866
16:48:57 <dustymabe> but that has its own drawbacks
16:49:37 <jlebon> walters: oh yeah, that's a classic :)
16:50:08 <dustymabe> walters: IOW - that's an example where we caught something early by testing it upstream and saved us pain later?
16:50:45 <walters> dustymabe: in theory...yes?  I think we got that fixed before rhel8
16:50:47 <bgilbert> also also, we've generally been pretty good about making reasonable security tradeoffs in FCOS, and I generally consider FIPS to be snake oil
16:51:00 <dustymabe> bgilbert: :)
16:51:15 <fifofonix> fifofonix tries to find a comparison table of CURRENT, FUTURE, FIPS
16:51:37 <dustymabe> bgilbert: yeah, I don't know if I know enough to have an opinion on that one
16:52:47 <dustymabe> so.. my perspective here: 1) sugar/docs/tests would help us find issues before they land downstream 2) I think we would have to make certain disclaimers about users relying on this 3) not sure how high of a priority it would be to do the work to implement `1)`
16:53:20 <bgilbert> yeah, IMO the main argument is to reduce divergence with downstream
16:53:25 <walters> bgilbert: not going to fully disagree, I personally feel the value in having shared infrastructure and just the bare minimum of "OS boots in fips" is enough to start
16:54:45 <dustymabe> does the `1)` `2)` `3)` reflect the sentiment of most here?
16:54:58 <bgilbert> +1 to those
16:55:02 <bgilbert> though one approach is to not support FIPS sugar in Butane for FCOS
16:55:05 <jlebon> i can't find an official Fedora stance on this, but clearly we do support a crypto-policy called FIPS, and so I think it makes sense to support putting FCOS in that mode too, with the same caveats that would apply to traditional
16:55:36 <bgilbert> reason being: it doesn't help reduce divergence with downstream, since downstream would continue to use the openshift.fips flag for compatibility
16:55:58 <jlebon> i initially didn't think we'd need sugar at all for it and just document `fips=1` as a karg, but hadn't thought about the encryption bit
16:56:15 <dustymabe> bgilbert: IOW - maybe add just a test and not sugar or documentation?
16:56:18 <fifofonix> i'm interested in snakeoil comment, and whether really there should be sugar for enabling FUTURE crypto?
16:56:19 <bgilbert> and it's okay for the UX to be a little rough if we don't really want to encourage use of the feature
16:56:47 <jlebon> bgilbert: but maybe have butane warn about it if the karg is there and an encrypted device is defined
16:56:49 <bgilbert> dustymabe: well, we could add docs, and the docs could say to manually set the LUKS parameters if you want 140-2 compat
16:57:03 <bgilbert> jlebon: hmm, yeah, that's possible
16:57:22 <bgilbert> though the warning would be spurious if you only want 140-3
16:57:33 <jlebon> gotcha
16:57:57 <jlebon> so 140-2 dictates a specific algo to use for disk encryption, but not 140-3?
16:58:11 <bgilbert> jlebon: no, the issue is that 140-2 doesn't allow XTS
16:58:18 <bgilbert> so we have to fall back to CBC
16:59:42 <bgilbert> fifofonix: sugar for enabling FUTURE could be interesting
17:00:00 <jlebon> bgilbert: gotcha. so in that case, probably better to leave it as a documentation item
17:00:35 <jlebon> i think there's an issue for better UX for crypto policies in general
17:00:51 <jlebon> where Butane sugar to manually set the symlinks was discussed
17:01:07 <dustymabe> jlebon: i.e. https://github.com/coreos/fedora-coreos-tracker/issues/607
17:01:36 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/607
17:01:38 <fifofonix> As a first step maybe another PR on docs covering some of: https://discussion.fedoraproject.org/t/strong-crypto-settings-how-best-to-define-apply/22600/15
17:01:42 <jlebon> yeah, I don't think we ever added the docs for that
17:02:32 <dustymabe> anyone want to sign up for pushing that `pending-action` ticket forward?
17:02:47 <walters> (one thing to note here is that in a layering world, the python dependency is OK if one can also remove it after a build is done)
17:03:43 <dustymabe> let's see if we can move forward on this conversation
17:03:46 <bgilbert> fifofonix: re snake oil, basically a really heavyweight certification process is orthogonal to security
17:03:52 <dustymabe> #proposed We do think documentation and tests for FIPS in FCOS would be useful for catching issues before the land downstream, but because we're not confident in the ability to keep FIPs mode working we're not sure about encouraging users to use it. At this time we're also not sure if adding "sugar" to butane for this would be necessary given the above position.
17:04:35 <dustymabe> maybe s/necessary/desired/
17:04:52 <walters> that seems like a non-conclusion?
17:05:04 <bgilbert> fifofonix: the standard has some good rules in it, but also ends up pushing distros to maintain special code paths in order to run less-maintained crypto code implementing obsolete algorithms
17:05:31 <fifofonix> bgilbert: 🙏
17:05:33 <dustymabe> walters: the discussion started off at "should we do anything for this"?
17:05:53 <dustymabe> and I think we agreed that having a test for it would help at least for preventing issues leaking into downstream
17:06:23 <dustymabe> walters: want to attempt a #proposed?
17:06:45 <jlebon> how about "we will add documentation and tests for FIPS in FCOS but will not add sugar for it in Butane at this time" ?
17:06:56 <jlebon> (this is orthogonal to prioritizing it)
17:07:29 <dustymabe> yeah - that second part is the open question
17:07:44 <walters> #proposed adding a fips=1 kernel argument via Ignition becomes the canonical low level API to signal desire for FIPS mode, FCOS will add glue to do the remaining userspace portions on firstboot, and we will add a minimal kola test that verifies this works; no butane sugar yet
17:07:48 <dustymabe> i get caught up when we say `we will` and then have no plans for it to happen anytime soon
17:08:19 <dustymabe> ^^ response to jlebon
17:08:19 <bgilbert> dustymabe: yeah
17:08:36 <jlebon> agreed this is a long-running problem, but at least we declare our intent
17:08:41 <jlebon> +1 to proposed
17:08:49 <cyberpear> +1
17:09:12 <bgilbert> walters: should we mention docs?
17:09:58 <dustymabe> +1 docs would be nice
17:10:42 <walters> #proposed adding a fips=1 kernel argument via Ignition becomes the canonical low level API to signal desire for FIPS mode, FCOS will add glue to do the remaining userspace portions on firstboot, and we will add a minimal kola test that verifies this works; no butane sugar yet, and documentation that briefly describes this state
17:11:14 <jlebon> ack
17:11:16 <dustymabe> +1
17:11:28 <bgilbert> +1
17:12:15 <dustymabe> #chair walters
17:12:15 <zodbot> Current chairs: bgilbert davdunc dustymabe fifofonix gursewak jbrooks jlebon ravanelli walters
17:12:22 <jlebon> i think possible butane sugar to set the crypto-policy is strongly related to this since it'd allow us to drop a chunk of the userspace portion
17:12:37 <dustymabe> any other votes?
17:12:47 <fifofonix> +!
17:12:56 <fifofonix> +1 even
17:13:28 <dustymabe> #agreed adding a fips=1 kernel argument via Ignition becomes the canonical low level API to signal desire for FIPS mode, FCOS will add glue to do the remaining userspace portions on firstboot, and we will add a minimal kola test that verifies this works; no butane sugar yet, and documentation that briefly describes this state
17:13:46 * dustymabe wonders about priority - but maybe a discussion for another time
17:14:02 <dustymabe> #topic OstreeNativeContainerStable proposal
17:14:09 <dustymabe> #link https://fedoraproject.org/wiki/Changes/OstreeNativeContainerStable
17:14:37 <dustymabe> This is a proposal by walters to Fedora
17:14:51 <dustymabe> which has implications on Fedora CoreOS
17:15:06 <walters> (yes, though there are two other names there too, happy to gather more 😄)
17:15:22 <dustymabe> I'm interested in discussing this proposal in the context of this group
17:15:37 <bgilbert> walters: does the switch away from ostree repos substantially increase the download size of updates?
17:15:52 <dustymabe> ^^ that's one bit that has improved
17:16:18 <dustymabe> The containers are now "chunked" mostly based at the RPM level IIUC
17:16:25 <dustymabe> and you only download the chunks that you don't have
17:16:55 <walters> bgilbert: see https://github.com/ostreedev/ostree-rs-ext/issues/69 - you can actually try this today by pulling `quay.io/fedora/fedora-coreos:stable` and then pulling `quay.io/fedora/fedora-coreos:testing-devel` and seeing the shared chunks
17:17:07 <bgilbert> +1
17:17:22 <dustymabe> walters: when you say "pulling" do we need to be more specific?
17:17:32 <dustymabe> i.e. `podman pull` wouldn't do the right thing would it? Or maybe it would
17:17:53 <walters> it works exactly the same in podman (and docker, and kubelet)
17:18:30 <dustymabe> +1
17:19:06 <walters> but we are missing a section on this, actually the delta support was a big concern of Fedora IoT, will flesh that out there
17:19:09 <dustymabe> at a high level I think this proposal is large and has several pieces that aren't necessary to couple. it might be good to break them out into separate proposals
17:19:21 <bgilbert> I'll note for the record that I strongly disagree with https://fedoraproject.org/wiki/Changes/OstreeNativeContainerStable#Encouraging_building_derivative_containers in an FCOS context
17:19:29 <walters> but very briefly, we want https://github.com/containers/image/pull/902
17:20:06 <walters> bgilbert: can you elaborate why?
17:20:28 <bgilbert> it makes sense in RHCOS for a variety of customer reasons ("run all your code in containers" + "use Ignition" is a big shift for some customers)
17:20:44 <jlebon> walters: would it make sense to rework this proposal to only set the stage so people can opt-in and have changing defaults (and changing default behaviours) be a separate proposal, possibly f39?
17:20:55 <jlebon> otherwise, it's quite a lot to take in in one shot
17:21:56 <bgilbert> but for FCOS I think we should try to maintain the core premise that you shouldn't install arbitrary software in the host, and shouldn't use FCOS as an "OS construction kit".  users who want arbitrarily customizable OSes can use other editions; a big part of the FCOS value prop is that we don't do that.
17:22:20 <dustymabe> bgilbert: I think your concern is mostly with how much we encourage people to create derivatives?
17:22:58 <bgilbert> dustymabe: yup, where "create derivatives" is read to include "customize the host software for the user's own purposes"
17:23:45 <jlebon> bgilbert: i think there's some shades there, just like there is today with layering in FCOS. some software make sense to install/layer because they're core host services
17:24:08 <bgilbert> jlebon: there are indeed shades :-)
17:25:01 <walters> one big, big thing here is that it gives an efficient and sane model for "day 2" config changes that aren't "reprovision the machine"
17:25:37 <bgilbert> walters: yes, and that's out of scope for FCOS :-)
17:25:40 <dustymabe> I think some of the nuances of the postivies and negatives of "CoreOS Layering" is covered in https://github.com/coreos/fedora-coreos-tracker/issues/1219
17:26:11 <dustymabe> bgilbert: I'm not sure I 100% agree. I think it's worth exploring our position on that
17:26:31 <bgilbert> walters: I'd rather see us improve the reprovisioning UX instead
17:26:32 <dustymabe> though.. what I do think this discussion is proving. is that the proposal is too large
17:27:09 <dustymabe> for example.. there's switching to container registry + CoreOS Layering + dnf cliwrap + rebranding  - all thrown in there
17:27:30 <dustymabe> all of those could be considered separately IIUC
17:27:58 <dustymabe> well s/IIUC/IMO
17:28:00 <walters> well...dnf cliwrap makes less sense without the rest
17:28:13 <jlebon> bgilbert: i mean, it's out of scope today because we can't really support it with our current tech, so we ended up building our philosophy around that. but if the tech could support it, we might've come to a different conclusion
17:28:57 <walters> dustymabe: I'm kind of trying to actively stop saying "coreos layering" FWIW because it actually has nothing to do with coreos other than it happens to be a place where I'm trying to drive this to happen first
17:29:12 <dustymabe> walters: sorry - what's the new term?
17:29:18 <dustymabe> I'll try to use that
17:29:38 <walters> there's "ostree native container" but...see, I think the actual succinct term should be "dnf image"
17:30:22 <dustymabe> To me "ostree native container" is just an ostree in a container.. but "ostree native container builds" could be what we are referring to as CoreOS Layering today
17:30:46 <jlebon> "ostree container layering" ?
17:30:57 <walters> hmm, I don't quite get what the difference between those two is
17:31:00 <dustymabe> semantics :) jlebon +1
17:31:28 <walters> the current change says: "A top level goal is that users should not need to know or understand ostree (or rpm-ostree); they only need to know containers and most key functionality is exposed via the yum/dnf frontend with a few extensions. "
17:31:47 <dustymabe> walters: "ostree native container" could just be a container we put in a registry that you could pull that didn't support Dockerfile style building on top
17:31:47 <walters> if you need to know ostree to customize the OS, this effort has failed
17:32:06 <bgilbert> jlebon: partially, but I'm not sure completely, e.g. moving back toward imperative doesn't seem great
17:32:20 <bgilbert> (and also, our users signed up for the path we're on, and I'm not wild about dragging them in a different direction)
17:32:27 <walters> dustymabe: true...but the value in doing the former is quite low without the latter
17:33:01 <jlebon> bgilbert: agreed
17:33:01 <dustymabe> still has "distribution" benefits (just container registry to deliver OS updates)
17:33:15 <dustymabe> anyway that's a tangent
17:33:21 <walters> bgilbert: bear in mind one can use any container build system on top of this, you are not required to use Dockerfile - and in fact, i'm going to investigate having the existing `rpm-ostree compose image` learn to build derivative images (we *already have* a declarative interface there)
17:33:48 <bgilbert> walters: +1
17:34:12 <jlebon> where i was driving this before was to a place where you still use Ignition to configure your hosts, but now it's just about whether you apply it to the image or to a specific node
17:34:35 <dustymabe> considering we've discussedf like 4 different topics here.. does anyone disagree that it would be good to break up this proposal (how do you eat a cow? one bite at a time.)?
17:35:15 <walters> dustymabe: note that the current proposal does link to github issues for the individual topics already
17:35:19 <walters> or at least, some of them
17:35:20 <bgilbert> jlebon: yeah, though I've seen discussion lately that deemphasizes Ignition quite a bit
17:35:46 <dustymabe> walters: yes, but it's hard to propose and discuss things as a community and at a project level that have so many subtopics
17:36:02 <dustymabe> kind of like a code review that has 100 commits - sometimes we break those off into smaller reviews
17:36:03 <bgilbert> (note: Ignition has plenty of problems of its own, but it's the UX we've built expectations around)
17:37:07 <bgilbert> dustymabe: +1 to splittin
17:37:09 <bgilbert> g
17:38:33 <jlebon> i see at least the "changing over-the-wire format" as a good chunk we could easily split out
17:38:44 <walters> hmm, splitting how exactly?
17:39:07 <dustymabe> indeed, which has it's own lot of details that will need investigation/discussion
17:39:14 <jlebon> as a separate change proposal
17:40:15 <walters> you're right that technically we could do everything else without changing the default wire transport, but...that seems weird
17:40:40 <dustymabe> or we could change the default wire transport without doing everything else
17:40:44 <dustymabe> they're independent
17:40:49 <walters> (Note that https://github.com/coreos/coreos-assembler/pull/2523 when we flip it on will make the initial pull efficient for derived images too)
17:40:53 <jlebon> yeah, i more meant the latter :)
17:41:16 <dustymabe> #info Summary: there are a lot of subtopics within this proposal and we think it would be useful to split up the proposal into smaller proposals in order to focus discussion and help drive to conclusions and actions.
17:41:31 <dustymabe> I can undo ^^ if needed
17:42:44 <dustymabe> we're out of time already - move to open floor?
17:43:19 <jlebon> it's good to discuss it as a whole too of course to see the big picture
17:43:57 <jlebon> definitely interested to see what the dnf folks and the rest of Fedora think about `dnf image`. that's a bit i don't quite agree with :)
17:43:58 <dustymabe> I'm OK with that too. though might be good at a video session?
17:44:33 <jlebon> to go over how to split it?
17:45:11 <walters> bgilbert: just to have something actually change as a result of this discussion I edited the page to s/Encourage/Support/ in "building derivative containers"
17:45:12 <dustymabe> no.. to talk about the high level goal - here we keep getting caught up in the details, which are important, but gets in the way of the high level outcome discussion I think
17:45:30 <jlebon> dustymabe: that makes sense, yeah
17:45:39 <bgilbert> walters: +1 to "Support", thanks!
17:45:49 <dustymabe> #topic open floor
17:45:55 <dustymabe> anybody with anything for open floor?
17:46:11 <dustymabe> sorry for the meeting running late, though I'm always happy to have a lot of discussion with a lot of participants
17:46:29 <jlebon> walters: should the bullet under https://fedoraproject.org/wiki/Changes/OstreeNativeContainerStable#Detailed_Description be changed too?
17:47:21 <dustymabe> #info Fedora Final Freeze is now in effect and we will start building `next` once a week to increase use/feedbacak of F37 in during this time
17:47:23 <walters> sure, done
17:47:31 <dustymabe> #undo
17:47:31 <zodbot> Removing item from minutes: INFO by dustymabe at 17:47:21 : Fedora Final Freeze is now in effect and we will start building `next` once a week to increase use/feedbacak of F37 in during this time
17:47:34 <dustymabe> #info Fedora Final Freeze is now in effect and we will start building `next` once a week to increase use/feedback of F37 in during this time
17:47:48 <bgilbert> #info The new serial console defaults landed in `next`.  Please test!
17:47:52 <jlebon> walters: +1
17:47:56 <dustymabe> bgilbert++
17:48:04 <bgilbert> #link https://lists.fedoraproject.org/archives/list/coreos-status@lists.fedoraproject.org/message/GHLXX4MXNHUEAXQLK6BZN45IQYHRVQB4/
17:48:33 <jlebon> bgilbert: awesome! really nice to finally see it out
17:48:42 * dustymabe will close out the meeting in 60s if no other discussion occurs
17:49:37 <fifofonix> busy meeting.  thanks for running dustymabe.  ostree native containers fascinating.  scary user optionality.
17:49:41 <dustymabe> #endmeeting