16:31:02 <dustymabe> #startmeeting fedora_coreos_meeting
16:31:02 <zodbot> Meeting started Wed Sep 11 16:31:02 2019 UTC.
16:31:02 <zodbot> This meeting is logged and archived in a public location.
16:31:02 <zodbot> The chair is dustymabe. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:31:02 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:31:02 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:31:06 <dustymabe> #topic roll call
16:31:09 <bgilbert> .hello2
16:31:11 <dustymabe> .hello2
16:31:11 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:31:14 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:32:13 <kaeso[m]> .hello lucab
16:32:14 <zodbot> kaeso[m]: lucab 'Luca Bruno' <lucab@redhat.com>
16:32:50 <dustymabe> #chair bgilbert kaeso[m] jlebon geoff- walters slowrie
16:32:50 <zodbot> Current chairs: bgilbert dustymabe geoff- jlebon kaeso[m] slowrie walters
16:32:53 <jlebon> .hello2
16:32:54 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:32:59 <kaeso[m]> (but I won't be around for very long, busy for dinner)
16:33:02 <slowrie> .hello2
16:33:02 <zodbot> slowrie: slowrie 'Stephen Lowrie' <slowrie@redhat.com>
16:33:15 <dustymabe> kaeso[m]: in that case I'll try to start with one you might have some input on
16:34:29 <red_beard> .hello red_beard
16:34:29 <zodbot> red_beard: Sorry, but you don't exist
16:34:38 <red_beard> .hello redbeard
16:34:39 <zodbot> red_beard: redbeard 'Brian 'redbeard' Harrington' <bharring@redhat.com>
16:35:02 <dustymabe> #chair red_beard
16:35:02 <zodbot> Current chairs: bgilbert dustymabe geoff- jlebon kaeso[m] red_beard slowrie walters
16:35:10 <dustymabe> #topic Action items from last meeting
16:35:19 <dustymabe> it's been a long time since we had a meeting so I'll skip this part
16:35:38 <dustymabe> and skip to meeting tickets
16:35:53 <dustymabe> #topic Ignition: Support multi part mime for user-data
16:36:01 <dustymabe> #link https://github.com/coreos/ignition/issues/849
16:36:18 <mnguyen_> .hello mnguyen
16:36:19 <zodbot> mnguyen_: mnguyen 'Michael Nguyen' <mnguyen@redhat.com>
16:36:25 <bgilbert> this topic should really wait until ajeddeloh gets back
16:36:36 <dustymabe> I added this ticket to our meeting agenda because I was asked about it on the dev list by the folks that produce openstack magnum
16:37:11 <dustymabe> bgilbert: so not worth discussing at all?
16:37:13 <kaeso[m]> yes, I was also holding a bit till Andrew was back
16:37:20 <bgilbert> yeah, let's hold a week
16:37:39 <kaeso[m]> my gut feeling is that the request is reasonable
16:37:40 <bgilbert> he's the one that'll need to be convinced :-)
16:38:00 <bgilbert> I think fundamentally the request misunderstands the purpose of Ignition as the sole source of truth
16:38:08 <bgilbert> but I'll defer to Andrew
16:38:21 <kaeso[m]> that's a fair point
16:38:41 <dustymabe> #info will punt on discussin this until next week when ajeddeloh is back
16:38:46 <dustymabe> #undo
16:38:46 <zodbot> Removing item from minutes: INFO by dustymabe at 16:38:41 : will punt on discussin this until next week when ajeddeloh is back
16:39:00 <dustymabe> #info will punt on discussing coreos/ignition/issues/849  until next week when ajeddeloh is back
16:39:11 <dustymabe> #topic coreos/toolbox v2
16:39:21 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/38
16:39:51 <dustymabe> so basically we're down to the bottom of the discussion in that ticket and mostly splitting hairs on added dependencies IIUC
16:40:40 <dustymabe> I tried to build an FCOS based on rawhide to see if the added deps there would change much from F30, looks like it's about the same
16:41:02 <dustymabe> maybe the changes debarshiray made haven't made it into an rpm build
16:42:05 <kaeso[m]> there was no flatpak release in the meantime, from what I see
16:42:29 <dustymabe> anywho, how do we feel about the current set of pulled in deps
16:42:40 <dustymabe> to me it looks ugly
16:43:05 <bgilbert> it does
16:43:10 <kaeso[m]> a bit, yes, unfortunately
16:43:50 <dustymabe> I just asked debarshiray in the ticket if there is an existing rpm somewhere we can test with
16:44:00 <bgilbert> <disclaimer type="haven't read the ticket">are we in good shape to ship it, and deal with the deps later?</disclaimer>
16:44:07 <dustymabe> maybe that will give us some new information about "added deps"
16:44:15 <bgilbert> we have a variety of unfortunate deps in FCOS already
16:44:36 <dustymabe> bgilbert: i think we're mostly in agreement that we want to ship it, correct?
16:44:43 <kaeso[m]> the dconf in there, I think that brings in a binary which is directly in $PATH
16:44:43 <bgilbert> we want to ship something like it
16:45:08 <bgilbert> we could mask files.  like, say, all the font data
16:45:39 * bgilbert stops talking
16:46:05 <dustymabe> bgilbert: I'd say let's ship toolbox (once we iterate on deps, trying to get it down to an acceptable list) and then replace in the future if we decide to
16:46:20 <bgilbert> sgtm
16:46:49 <bgilbert> let's track unwanted deps in https://github.com/coreos/fedora-coreos-tracker/issues/186 though
16:47:08 <kaeso[m]> I'll add my dconf note directly in the ticket, if I'm not alone on that
16:47:21 <bgilbert> kaeso[m]: +1
16:47:25 <dustymabe> bgilbert: personally I'd like to iterate on the deps in #38 "before" we ship it
16:47:50 <dustymabe> since we've already been working with debarshiray it seems like we might be close ?
16:47:52 <bgilbert> dustymabe: yeah, I meant anything remaining once we ship
16:48:08 <dustymabe> bgilbert: +1
16:49:12 <dustymabe> #info we want to ship something like toolbox and toolbox is something that already exists so we will ship it once we've iterated on the dependencies in #38. Will work with debarshiray to see if there is an updated rpm we can evaluate.
16:49:59 <dustymabe> any objections ^^
16:51:05 <jlebon> +1
16:51:16 <dustymabe> k, next topic
16:51:22 <kaeso[m]> sgtm, but again I don't know the details of this toolbox
16:51:25 <dustymabe> #topic docker group or not
16:51:36 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/2
16:51:46 <dustymabe> this is one of our first issues!
16:52:07 <dustymabe> I think I summarized the current decision at the bottom of the ticket
16:52:13 <dustymabe> So the open question here is whether we do one of these two options:
16:52:21 <dustymabe> - we add a docker group to /etc/group so that usermod -aG docker myusername works?
16:52:28 <dustymabe> - we encourage users to use sudo docker instead
16:52:36 <bgilbert> it sounds as though the current situation is a bug?
16:52:49 <dustymabe> bgilbert: that depends on who you ask
16:52:54 <bgilbert> we have half a group
16:53:05 <dustymabe> It's an architectural artifact
16:53:09 <jlebon> yup, that's an artifact of using nss-altfiles
16:53:18 <dustymabe> And it's not working as people expect it to, which one could call a bug
16:53:33 <bgilbert> okay, so this is a special case of the nss-altfiles problem?
16:53:44 <dustymabe> AFAICT, yes
16:53:53 <bgilbert> we should fix that generally :-)
16:53:56 <dustymabe> jlebon: would systemd-sysusers fix this?
16:54:09 <jlebon> yes
16:54:25 <dustymabe> i think we either need to say we "NEED" systemd-sysusers for stable
16:54:38 <dustymabe> OR we add these groups into /etc/group somehow
16:55:04 <dustymabe> at this point docker group is a special case, but the conversation is more general than that
16:55:26 <jlebon> hmm, if one of the primary motivations for the docker group is convenience, then we should encourage rootless podman for those cases, which is as convenient, but actually secure :)
16:55:33 <bgilbert> people use docker
16:55:58 <dustymabe> yeah, in this case i'm considering that someone has already made their decision
16:56:00 <bgilbert> is there a reason not to ship the group?  to me, the larger question is who's in it by default
16:56:33 <dustymabe> bgilbert: the moby-engine rpm ships it
16:56:42 <bgilbert> +1
16:56:59 <dustymabe> it's only not shipped on our systems because of this unique bug/feature
16:57:07 <dustymabe> depending on how you look at it
16:57:22 <bgilbert> I'm on record as wanting altfiles fixed generally
16:57:34 <dustymabe> +1 people hit this in silverblue too
16:57:36 <bgilbert> if it's infeasible, then yeah, we probably have to explicitly add to /etc/groups
16:57:49 <dustymabe> does anyone not want altfiles fixed?
16:57:54 <bgilbert> heh
16:58:05 <bgilbert> I meant for stable
16:58:13 <dustymabe> another way to phrase that is: "is anyone opposed to systemd-sysusers being implemented?"
16:58:33 <dustymabe> bgilbert: ahh, ok I get it
16:58:39 <dustymabe> new question:
16:58:46 <kaeso[m]> my answer would be yes to both ticket questions: we fix group-file somehow (because the RPM has the group), and we tell people that the group is empty by default and they should `sudo docker`
16:59:11 <dustymabe> kaeso[m]: agree
16:59:15 <dustymabe> ok new questions:
16:59:26 <jlebon> bgilbert: walters already has an initial patch pending for it -- might be feasible to make it a requirement for stable
16:59:33 <dustymabe> - is it feasible to implement systemd-sysusers for stable (remember f30)
16:59:48 <dustymabe> - if not then we should definitely get the `docker` group into `/etc/group`
16:59:57 <bgilbert> (strictly, f31 would work)
17:00:18 <dustymabe> bgilbert: got ya. so you're saying we could do f31 for stable
17:00:44 <bgilbert> dustymabe: that's the current plan.  we should test a Fedora major rebase before declaring FCOS stable
17:01:09 <dustymabe> ok so my two questions remain (minus the caveat)
17:01:14 <jlebon> i think it could make the cut, it just hasn't been prioritized much
17:01:38 <dustymabe> jlebon: would you be able to pick it up if walters wasn't able to ?
17:01:48 <dustymabe> do you think most of the work is capital D done?
17:01:49 <jlebon> we'd need to get it in soon-ish so we gain experience on it and work out the kinks
17:02:51 <jlebon> dustymabe: probably? haven't looked at it in a while, but I know walters is definitely motivated too to drop nss-altfiles :)
17:03:00 <dustymabe> ok
17:03:15 <dustymabe> let me ask a dumb question
17:03:27 <dustymabe> with systemd-sysusers would `usermod -aG docker myusername` work?
17:03:34 <dustymabe> since that's what people are doing/trying
17:03:41 <kaeso[m]> (I'm dropping here, will read logs)
17:03:50 <jlebon> yes, it should
17:03:52 <dustymabe> kaeso[m]: o/
17:03:55 <dustymabe> later
17:04:00 <dustymabe> jlebon: good to know
17:04:13 <dustymabe> ok so we want to try to get that in soon
17:04:39 <jlebon> everything is still in the regular locations, it's basically a declarative way to manage /etc/passwd
17:04:45 <dustymabe> should we hackishly add docker to /etc/group in a quick PR now just to unblock people who are going to try our platform?
17:05:15 <bgilbert> meh.  probably
17:05:32 <dustymabe> i'm +1
17:05:34 <dustymabe> anyone else?
17:05:35 <walters> FWIW I have already been working on using `coretoolbox `for privileged paths too which I think is quite relevant for FCOS
17:05:35 <jlebon> not against
17:06:48 <dustymabe> #info we are going to try to prioritize systemd-sysusers work for FCOS stable, but we will hackishly try to add docker to the /etc/group file to smooth the path for users who are currently using FCOS preview releases
17:07:21 <dustymabe> moving on to next ticket
17:07:33 <jlebon> walters: do you see a path where you upstream to fedora toolbox ?
17:07:40 <dustymabe> #topic Allow creation of immutable directories below root folder
17:07:49 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/270
17:08:14 <dustymabe> this was brought to us by a user who is testing out FCOS preview
17:08:35 <dustymabe> and trying to migrate from CL (/me notes that we'll need docs for this)
17:08:50 <bgilbert> physically impossible on live systems
17:08:59 <dustymabe> they want to create mountpoints at the top of the filesystem hierarchy
17:09:16 <bgilbert> unless we put an overlayfs on /sysroot, which doesn't seem great
17:09:44 <dustymabe> bgilbert: because things are mounted read-only ?
17:09:48 <bgilbert> I think this makes sense as a caveat in the migration guide, and we can catch it in Ignition config translation tooling
17:09:52 <bgilbert> dustymabe: because it's a squashfs
17:09:59 <dustymabe> which is read-only
17:10:01 <bgilbert> yeah
17:10:51 <walters> jlebon: hopefully
17:11:21 <dustymabe> any other technical reasons why this could or could not work?
17:11:39 <dustymabe> Is what I told the user correct?
17:12:02 <bgilbert> dustymabe: link?
17:12:04 <dustymabe> I told him it was failing because we were using OSTree, which is generally correct as I've tried to do this in the past on my atomic host systems
17:12:08 <bgilbert> ah
17:12:22 <jlebon> right
17:12:22 <dustymabe> but in this case he's using ignition to try to create those directories
17:12:35 <dustymabe> I don't know if it is actually failing because of OSTree in his case or because of something in ignition
17:13:06 <jlebon> not making it immutable implies having some e.g. config for rpm-ostree to know what top-level dirs to recreate on new deployments
17:13:20 <dustymabe> jlebon: correct, we've discussed that in an rpm-ostree issue
17:13:34 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/270
17:13:55 <dustymabe> but even if we had that config. would it work early enough such that ignition could use it
17:14:09 <dustymabe> I guess ignition could create the 'config' and then the service would create the mountpoint
17:14:09 <jlebon> sorry, that was badly phrased. / could still be immutable. it's just that it's like this to prevent users being surprised when it's wiped out on the next deployment
17:14:48 <bgilbert> I guess I'm not seeing the issue here.  configs will need to be modified when switching CL -> FCOS.  it shouldn't be a big deal to repoint docker at a different path.
17:14:56 <lorbus> .hello2
17:14:57 <zodbot> lorbus: lorbus 'Christian Glombek' <cglombek@redhat.com>
17:14:57 <dustymabe> jlebon: correct. it was done to prevent unexpected behavior
17:15:13 <jlebon> dustymabe: yeah, we'd either need to teach ignition some ostree stuff, or have more magic goop in the initrd...
17:15:22 <jlebon> bgilbert: +1
17:15:41 <bgilbert> FCOS makes some parts of the tree immutable for good reason.  it doesn't seem worth jumping through a lot of hoops to work around that
17:15:46 <jlebon> FWIW, i like the current model for the reasons stated in https://github.com/projectatomic/rpm-ostree/issues/337#issuecomment-302703626
17:15:50 <dustymabe> ok back to bgilberts point about squashfs
17:16:04 <dustymabe> do you not use squashfs in lives today with CL?
17:16:13 <bgilbert> sure we do, but it's /usr
17:16:32 <dustymabe> I see
17:16:44 <dustymabe> ok
17:17:05 <jlebon> one hack would be to have / be tmpfs, then mount the squashfs somewhere, then bind-mount the subdirs... yuck
17:17:10 <dustymabe> #link https://lists.fedoraproject.org/archives/list/coreos@lists.fedoraproject.org/message/F2D2TMYJFRU3H24RJJNHCN3QGJPACXXU
17:17:19 <dustymabe> ^^ FYI there is the link to the original discussion
17:17:45 <dustymabe> basically they aren't going in greenfield and it's going to be easier for them if they can use /dockerdata everywhere
17:18:03 <dustymabe> that's their reasoning for wanting this to exist in FCOS
17:18:11 <bgilbert> #link https://github.com/coreos/fedora-coreos-tracker/issues/159
17:18:20 <bgilbert> there are _so_ _many_ _things_ which will change
17:18:44 <bgilbert> I'm in favor of being compatible where it makes sense, but it doesn't always make sense
17:18:55 <dustymabe> bgilbert: +1 does that ticket need to be updated to include something for this?
17:19:01 <bgilbert> dustymabe: yup
17:19:07 <dustymabe> can you add it?
17:19:11 <bgilbert> sure
17:19:37 <dustymabe> #action bgilbert to update #159 to add doc item for immutable '/'
17:20:43 <dustymabe> #info while having an immutable top level directory seems like a reasonable ask there are technical reasons why this won't work for fedora coreos and we suggest users to change their mount point to be under one of the writable filesystems
17:21:17 <dustymabe> I'll add a comment to the ticket - we can close it out when the docs get written?
17:21:49 <bgilbert> let's not hold it open for the docs
17:22:00 <dustymabe> ok, yeah I didn't want to seem to harsh on the user :(
17:22:09 <dustymabe> I always struggle with that
17:22:30 <dustymabe> #topic open floor
17:22:33 <bgilbert> understood.  IMO, better to give a clear explanation and a clear "no" :-)
17:22:38 <dustymabe> +1
17:22:54 <dustymabe> anyone with open floor topics
17:23:13 <bgilbert> jlebon: how much does rpm-ostree assume /boot exists?
17:23:25 <dustymabe> FYI: we're really close to having aws replication and also kola AWS testing merged into our pipeline
17:23:52 <bgilbert> rpm-ostreed refuses to start on live systems and I don't have a sense of whether modifying it to run read-only is a major task
17:23:56 <jlebon> bgilbert: s/rpm-ostree/ostree/, then i think quite a bit
17:24:51 <jlebon> right. e.g. libostree scans BLS entries when initializing IIRC
17:25:20 <bgilbert> we could probably fake up that part of /boot...
17:25:46 <jlebon> i don't have a good estimate, but offhand i think it wouldn't be a small task
17:26:06 <jlebon> i haven't been keeping up with your latest patches -- do you have a ticket to track that part?
17:26:24 <bgilbert> not yet, it's in ~/TODO
17:26:30 <bgilbert> can file one though
17:27:03 <jlebon> yeah, let's discuss upstream. definitely something we can make happen
17:27:07 <bgilbert> +1
17:28:48 <jlebon> re. signing, i'm thinking of making it part of cosa instead of a separate helper
17:29:07 <jlebon> e.g. the pipeline would call something like `cosa sign robosignatory`
17:29:15 <dustymabe> ok meeting is running out of time
17:29:24 <dustymabe> closing this out in 30s
17:30:09 <bgilbert> jlebon: +1
17:30:32 <jlebon> btw, related to the signing work: https://github.com/coreos/coreos-assembler/pull/741
17:30:41 <dustymabe> #endmeeting