16:30:25 <dustymabe> #startmeeting fedora_coreos_meeting
16:30:25 <zodbot> Meeting started Wed Oct  2 16:30:25 2019 UTC.
16:30:25 <zodbot> This meeting is logged and archived in a public location.
16:30:25 <zodbot> The chair is dustymabe. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:30:25 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:30:25 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:30:34 <dustymabe> #topic roll call
16:30:58 <slowrie> .hello2
16:30:59 <zodbot> slowrie: slowrie 'Stephen Lowrie' <slowrie@redhat.com>
16:31:16 <ajeddeloh> .hello2
16:31:17 <zodbot> ajeddeloh: ajeddeloh 'Andrew Jeddeloh' <andrew.jeddeloh@redhat.com>
16:31:19 <darkmuggle> .hello2
16:31:20 <zodbot> darkmuggle: darkmuggle 'None' <darkarts@utlemming.org>
16:31:55 <strigazi> .hello2
16:31:56 <zodbot> strigazi: strigazi 'Spyros Trigazis' <strigazi@gmail.com>
16:32:05 <jlebon> .hello2
16:32:07 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:32:24 <dustymabe> #chair ajeddeloh darkmuggle strigazi slowrie jlebon walters jbrooks
16:32:24 <zodbot> Current chairs: ajeddeloh darkmuggle dustymabe jbrooks jlebon slowrie strigazi walters
16:32:55 <bgilbert> .hello2
16:32:56 <zodbot> bgilbert: Something blew up, please try again
16:33:01 <ajeddeloh> lol
16:33:04 <jbrooks> .fas jasonbrooks
16:33:04 <zodbot> jbrooks: jasonbrooks 'Jason Brooks' <jbrooks@redhat.com>
16:33:06 <walters> .hello2
16:33:07 <bgilbert> huh
16:33:08 <kaeso[m]> .hello lucab
16:33:10 <zodbot> walters: walters 'Colin Walters' <walters@redhat.com>
16:33:11 <bgilbert> .hello2
16:33:13 <zodbot> kaeso[m]: lucab 'Luca Bruno' <lucab@redhat.com>
16:33:16 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:33:21 <dustymabe> #chair bgilbert kaeso[m]
16:33:21 <zodbot> Current chairs: ajeddeloh bgilbert darkmuggle dustymabe jbrooks jlebon kaeso[m] slowrie strigazi walters
16:33:28 <dustymabe> bgilbert: haha, zodbot must have had an issue
16:33:38 <dustymabe> .hire bgilbert
16:33:38 <zodbot> adamw hires bgilbert
16:34:25 <dustymabe> #topic Action items from last meeting
16:34:25 <bgilbert> .chair bgilbert
16:34:25 <zodbot> bgilbert is seated in a chair with a nice view of a placid lake, unsuspecting that another chair is about to be slammed into them.
16:34:49 <dustymabe> * ajeddeloh to investigate the addition of python in  update-crypto-policies #280
16:35:00 <dustymabe> ajeddeloh: I know there were some updates in the ticket.. want to give a summary?
16:35:04 <ajeddeloh> sure
16:35:30 <ajeddeloh> The things the python script does are all things that don't really make sense on FCOS
16:35:43 <ajeddeloh> i.e. doing configuration
16:36:01 <ajeddeloh> under the hood it's just writing out a bunch of symlinks
16:36:27 <ajeddeloh> I asked if we could split that scripts out from the config and the packagers were open to that
16:36:51 <ajeddeloh> then we can just ship the configs and add some FCCT sugar to do the same thing the script did
16:37:22 <ajeddeloh> (i.e. generating an ign config with those same symlinks)
16:37:57 <ajeddeloh> next step is just breaking out the package and PRing that
16:39:06 <dustymabe> thanks ajeddeloh
16:39:08 <dustymabe> moving on
16:39:27 <dustymabe> #topic FCOS as Kubernetes / OKD node
16:39:32 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/93
16:39:41 <dustymabe> I believe strigazi has an update for us here
16:40:24 <strigazi> In out deployment we were using the atomic cli
16:41:13 <strigazi> replacing all "atomic install" systemd services running podman, I was able to create conformant k8s clusters
16:41:19 <strigazi> with v1.16.0
16:41:47 <strigazi> I took the kubelet unit shamelessly from posseidon
16:42:31 <strigazi> despite the --containerized being dropped, kubelet works fine with selinux enforcing
16:43:04 <dustymabe> strigazi: that's great news
16:43:08 <jbrooks> sweet
16:43:23 <ajeddeloh> woo \o/
16:43:31 <strigazi> the same units worked for us in F29AH and FCOS
16:43:32 <dustymabe> strigazi: does this investigation help OpenStack Magnum move to FCOS sooner?
16:43:49 <strigazi> soon as in this month
16:44:21 <ajeddeloh> I saw something go by re: ignition and heat, but we can save that for open floor
16:44:38 <dustymabe> strigazi: that's great news
16:44:47 <dustymabe> anything else on this topic before we move to the next topic ?
16:44:56 <strigazi> ajeddeloh: I'll be here for open floor
16:45:12 <ajeddeloh> +1
16:45:59 <strigazi> dustymabe: i'm good
16:48:16 <dustymabe> #topic Encryption: All disks are belong to us
16:48:21 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/287
16:48:27 * dustymabe waves at darkmuggle
16:48:41 <darkmuggle> Howdy :)
16:49:11 <dustymabe> so disk encryption :)
16:49:23 <dustymabe> do you want to give us some context for the discussion?
16:49:30 <darkmuggle> the history on this issue is that walters and I have been working on a downstream issue in Red Hat CoreOS
16:49:39 <darkmuggle> Specifically: https://github.com/openshift/enhancements/blob/master/enhancements/automated-policy-based-disencryption.md
16:50:04 <darkmuggle> The core idea is that Red Hat CoreOS in the Openshift context will support encrypted disks.
16:50:38 <darkmuggle> Since the goal is to make FCOS the upstream to RHCOS, there's a lot of dependencies and features that we would like to land in FCOS.
16:51:28 <darkmuggle> For example, walters, has been working on a dracut module copying ostrees into the initrd to allow for a re-formating of the rootfs
16:51:53 <darkmuggle> And in order to offer flexible disk solutions, we're going to need to add support for Ignitiion setting up LUKS volumes.
16:52:12 <walters> (like https://github.com/coreos/ignition/pull/515 )
16:52:19 <darkmuggle> The purpose of that tracker item was to provide a public place to reference the work.
16:52:43 <dustymabe> TL;DR: we'd like encrypted disks in RHCOS, so there will be some supporting work in FCOS. FCOS could consider adopting encrypted disks support as well
16:52:57 <dustymabe> ^^ accurate ?
16:53:05 <darkmuggle> Indeed, that's accurate.
16:53:18 <dustymabe> #info TL;DR: we'd like encrypted disks in RHCOS, so there will be some supporting work in FCOS. FCOS could consider adopting encrypted disks support as well
16:53:40 <dustymabe> has anybody had a chance to consider encrypted disk support for FCOS ?
16:53:44 <ajeddeloh> I think everyone wants encrypted disk support for FCOS
16:53:52 <ajeddeloh> there's basically three parts
16:54:09 <walters> I think the MVP is add `root: tpm: true` to FCCT and have that expand into clevis pinning requiring a tpm2 device
16:54:22 <ajeddeloh> 1) LUKS support in ignition 2) being able to move root 3) being able to find root on subsequent boots
16:54:44 <darkmuggle> I've done the work on 3.
16:54:46 <walters> note my proposed MVP does not require official LUKS in Ignition
16:54:57 <ajeddeloh> darkmuggle: how so?
16:54:58 <dustymabe> ajeddeloh: would we need changes to our existing parition layout to support this ?
16:54:59 <walters> though it'd be good
16:55:04 * ajeddeloh is out of the loop on that
16:55:10 <ajeddeloh> no
16:55:36 <dustymabe> darkmuggle: are there proposed changes to the partition layout on the RHCOS end ?
16:55:41 <ajeddeloh> AIUI the goal is to support it via ignition but not have it encrypted by default
16:55:51 <ajeddeloh> dustymabe: yeah they are
16:55:55 <walters> dustymabe: wasn't proposed, it was merged https://github.com/coreos/coreos-assembler/pull/785
16:56:05 <walters> wasn't *just* proposed I mean
16:56:21 <darkmuggle> To ajeddeloh's question, its fairly simple. Dracut has support to automatically find the LUKS container.
16:56:34 <walters> the proposal is also in the openshift enhancement link above
16:56:36 <ajeddeloh> ah, 3 for FCOS is more general
16:57:05 <walters> isn't 3) just requiring the new filesystem have the label "root" ?
16:57:13 <ajeddeloh> walters: no
16:57:16 <ajeddeloh> https://github.com/coreos/fedora-coreos-tracker/issues/94#issuecomment-517045698
16:57:36 <ajeddeloh> since we need to know which LUKS/RAID/etc devices to start and which not to start
16:57:43 <walters> oh right, ok
16:57:48 <ajeddeloh> and we really shouldn't go around starting things trying to find it
16:58:02 <darkmuggle> ajeddeloh: I don't want to waylay this conversation, but the tl;dr is that if that auto-activation of LUKS is can happen.
16:58:21 <ajeddeloh> darkmuggle: yeah, but we don't want to do that to !root
16:58:27 <walters> today anaconda mostly sets kargs which dracut reads AFAIK
16:58:43 <walters> though there are special things like iSCSI authentication which end up as config files that go in the initramfs
16:59:08 <walters> xref https://github.com/fedora-silverblue/issue-tracker/issues/3
16:59:27 <ajeddeloh> walters: that requires generating a new initramfs, yes?
16:59:29 <walters> (and https://lists.freedesktop.org/archives/systemd-devel/2019-September/043486.html )
17:00:37 * ajeddeloh isn't sure how deep into this rabbit hole we want to go right now
17:00:42 <darkmuggle> there's a couple ways to do this. In the LUKS case we could bind a token in the LUKS header named `activateme` and then use systemd generator that adds the udev rule for cryptsetup
17:00:48 <dustymabe> yeah let's come back up for air
17:00:55 <darkmuggle> +1
17:01:06 <dustymabe> so IIUC - there are two ways to go forward
17:01:31 <dustymabe> in trying to achieve disk encryption for FCOS (which I believe we want to support) we have
17:01:40 <dustymabe> 1. leave the current filesystem partition structure in place
17:01:57 <dustymabe> 2. adopt a empty luks container for root using --luks-rootfs  to create_disk
17:02:10 <dustymabe> i don't know any of the details of which one of those two options are better
17:02:25 <dustymabe> but as FCOS we should probably hash that out at some point and come up with a plan
17:02:31 <walters> 1. is more "lay groundwork for general rootfs changes, including LUKS but also switching filesystems"
17:02:54 <ajeddeloh> I _strongly_ prefer 1
17:03:00 <walters> see https://github.com/coreos/fedora-coreos-tracker/issues/94#issuecomment-535042020
17:03:13 <bgilbert> ajeddeloh: +1
17:03:19 <dustymabe> ajeddeloh: where should we capture future discussion on this?
17:03:21 <darkmuggle> The cost of 2 is that is adds a layer of abstraction that makes it more difficult for the end user.
17:03:28 <walters> but, it does require jlebon's kernel patch, plus probably systemd patches that have landed around the relabeling, etc.
17:03:29 <darkmuggle> I strongly perfer 1 as well.
17:03:33 <ajeddeloh> #94 I assume
17:03:36 <dustymabe> should it be in https://github.com/coreos/fedora-coreos-tracker/issues/287 or in a new issue ?
17:04:22 <dustymabe> ajeddeloh: ok #94 then?
17:04:23 <ajeddeloh> I think #287 should reference #94 and the Ignition encryption ticket
17:05:00 <dustymabe> ok cool. I'll try to add some context to each of the issues
17:05:01 <darkmuggle> 287 is more of a game plan. I like what ajeddeloh is suggesting.
17:05:47 <dustymabe> darkmuggle: IOW 1. is a good longer term plan, but RHCOS needs something shorter term for now so are going with 2. ?
17:06:10 <darkmuggle> That is correct.
17:06:21 <dustymabe> cool. my understanding has been massively improved on this topic :)
17:06:26 <ajeddeloh> high 287 Can be a tracker for LUKS specifically, but thats actually a combination of smaller pieces that are somewhat independent of each other
17:06:49 <dustymabe> ajeddeloh: if we could make a checklist in #287 of the smaller pieces - that would be nice
17:06:55 <ajeddeloh> +1
17:07:02 <darkmuggle> WFM
17:07:09 <dustymabe> bonus points if each checkmark is related to another ticket/work item
17:07:50 <dustymabe> #info for FCOS we want disk encryption. There are a few ways to achieve that goal and we prefer leaving the existing partition structure in place and replacing it on boot if the users asks for encryption
17:08:04 <dustymabe> any objections ^^
17:08:37 <dustymabe> #topic CI: Prow integration
17:08:38 <ajeddeloh> +1
17:08:44 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/263
17:08:52 <dustymabe> jlebon: walters: care to set the stage for this one ?
17:09:41 <walters> i'm a bit tempted to close the issue and try a new one that better captures the multiple CI system angle
17:09:48 <walters> but eh
17:10:35 <dustymabe> we could start with what is prow? how does it help us? are there examples where we can use it ?
17:10:52 <dustymabe> i can take a stab if you'd like?
17:11:23 <walters> there's a whole lot of angles to this but basically we're trying to close down the homu/papr project which rpm-ostree (and ostree) use.   The issue talks about moving to a combination of Prow and CentOS CI (but of these, the idea is to use Prow as "merge policy bot" in particular)
17:11:49 <jlebon> basically, right now we're "spiking" on different CI approaches
17:11:51 <walters> one nice thing about Github though is that CI is like an app; none of this is exclusive with using other CI services
17:12:24 <dustymabe> +1
17:12:37 <dustymabe> so the two options are prow or CentOS CI ?
17:12:50 <walters> no
17:13:29 <jlebon> re. the ticket, i think it's good. it's a good place to ask for feedback before rolling out to more repos
17:13:31 <walters> I guess implicit in this is that it's sort of up to the "owners" of a particular project to decide on this, now "owners" here becomes nebulous of course
17:13:57 <jlebon> i think having a merge bot in cosa would really help us
17:14:03 <dustymabe> what are we deciding on though? which CI strategy to use?
17:14:09 <dustymabe> what are the options?
17:14:30 <jlebon> dustymabe: like we said, we're still exploring :)
17:14:32 <walters> i don't this IRC meeting is about coming to any kind of final conclusions
17:14:47 <jlebon> it might be a combination of multiple services
17:15:20 <jlebon> rpm-ostree is a good test bed because of its testing requirements
17:15:22 <dustymabe> walters: yeah no need for any conclusions. I think it would help a lot to bring some of the rest of us up to speed on those options when they are ready though
17:15:46 <walters> personally while there are actually a lot of things I don't like about Prow (and some things I think are super cool), the ultimate fact is it's maintained by people paid to do so and who will listen to our concerns means I personally am aiming to invest some in it, and it will help us in terms of aligning with the larger OpenShift build system for RHCOS
17:16:02 <dustymabe> so maybe once we're done investigating we add a nice summary to the ticket ?
17:17:12 <jlebon> dustymabe: yeah, i think that makes sense
17:17:29 <walters> regarding other CI systems for example, one thing Github recently introduced is the new "checks" API which allows a CI system to annotate individual lines in a PR; good for "style checkers" and the like, can't do it with a Prow job, so individual repo owners could choose to do that for theirs
17:17:36 <jlebon> but "done investigating" might actually just be "ready to try out things in more repos"
17:18:08 <dustymabe> +1 it'll help when we're ready to try out some things to have a discussion about it. we'll get more input that way
17:18:19 <dustymabe> walters: +1 on style checker
17:18:31 <dustymabe> ready for open floor ?
17:18:39 <ajeddeloh> +1
17:18:43 <jlebon> walters: i wouldn't be surprised if prow eventually grows support for that (or does it have to be a registered "app" ?)
17:18:57 <walters> yeah I think so https://github.com/kubernetes/test-infra/issues/10423
17:19:12 <dustymabe> #topic open floor
17:19:28 <dustymabe> anyone with a topic for open floor?
17:20:21 <dustymabe> #info we added architecture information into our image filenames that are output by our build system https://github.com/coreos/fedora-coreos-tracker/issues/264
17:20:21 <ajeddeloh> o/
17:20:32 <dustymabe> the next Fedora CoreOS release will include that chage
17:20:35 <dustymabe> change*
17:20:39 <dustymabe> ajeddeloh: go for it
17:20:52 <ajeddeloh> strigazi: what's the status on ignition + heat?
17:21:21 <ajeddeloh> i.e. is https://github.com/coreos/ignition/pull/862 necessary still
17:21:38 <strigazi> ajeddeloh: Feilong that opened the issue patched heat to append the creds we need in ignition format.
17:22:23 <ajeddeloh> so does heat still serve a multipart thing or just an Ignition config?
17:22:28 <strigazi> ajeddeloh: for the upcoming openstack release it is not needed. for older releases would be useful
17:23:20 <ajeddeloh> So for new releases heat will just serve an Ignition config over http normally, no multipart?
17:23:28 <strigazi> ajeddeloh: yes
17:23:32 <ajeddeloh> \o/
17:23:47 <dustymabe> so.. what do we do in the interim ?
17:23:52 <ajeddeloh> thoughts on supporting this for older releases?
17:24:00 <strigazi> it will detect that the user passed data are an ingntion config and append a file there.
17:24:05 * ajeddeloh also doesn't have a way to test that PR right now
17:24:28 <strigazi> ajeddeloh: if you have a binary I can test it and let you know
17:24:33 <bgilbert> strigazi: awesome!
17:24:50 <ajeddeloh> strigazi: Yeah I can build you a binary and ship it off to you
17:24:51 <strigazi> or I can just build it
17:24:57 <ajeddeloh> that too
17:25:12 <ajeddeloh> lets sync after this meeting?
17:25:17 <strigazi> +1
17:25:37 <dustymabe> so what do we do about older releases of openstack ?
17:25:52 <dustymabe> i'd love to never support this, but you know environments stay around forever
17:26:13 <ajeddeloh> ^ I feel the same
17:26:41 <dustymabe> let's try this:
17:26:51 <dustymabe> 1. ajeddeloh and strigazi test the proposed patch(PR)
17:27:00 <dustymabe> 2. confirm it is working as expected
17:27:08 <dustymabe> 3. leave PR open until users come asking for it
17:27:13 <strigazi> let's see how it goes? For us (CERN + magnum users I know )it is fine, we keep up with the latest release or cherry-pick aggressively
17:27:21 <bgilbert> dustymabe: +1
17:27:29 <ajeddeloh> +1
17:27:41 <strigazi> +1
17:27:41 <dustymabe> cool
17:27:50 <dustymabe> thanks for that ajeddeloh and strigazi
17:27:54 <ajeddeloh> o/
17:27:55 <dustymabe> any other items for open floor ?
17:28:41 <dustymabe> #info jlebon got a kernel patch accepted upstream to help us support selinux with ignition/dracut: https://lore.kernel.org/selinux/20190912133007.27545-1-jlebon@redhat.com/T/#m5f950ca9fc3ed374cb793fc9aed0435602b1b6ad
17:28:46 <dustymabe> jlebon++
17:28:52 <ajeddeloh> jlebon++
17:29:01 <bgilbert> jlebon++
17:29:01 <zodbot> bgilbert: Karma for jlebon changed to 6 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:29:17 <dustymabe> will end meeting in 30s
17:29:20 <jlebon> :)  so excited for all the cleanups that will follow from this
17:29:51 <dustymabe> #endmeeting