16:31:02 #startmeeting fedora_coreos_meeting 16:31:02 Meeting started Wed Sep 11 16:31:02 2019 UTC. 16:31:02 This meeting is logged and archived in a public location. 16:31:02 The chair is dustymabe. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:31:02 Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:31:02 The meeting name has been set to 'fedora_coreos_meeting' 16:31:06 #topic roll call 16:31:09 .hello2 16:31:11 .hello2 16:31:11 bgilbert: bgilbert 'Benjamin Gilbert' 16:31:14 dustymabe: dustymabe 'Dusty Mabe' 16:32:13 .hello lucab 16:32:14 kaeso[m]: lucab 'Luca Bruno' 16:32:50 #chair bgilbert kaeso[m] jlebon geoff- walters slowrie 16:32:50 Current chairs: bgilbert dustymabe geoff- jlebon kaeso[m] slowrie walters 16:32:53 .hello2 16:32:54 jlebon: jlebon 'None' 16:32:59 (but I won't be around for very long, busy for dinner) 16:33:02 .hello2 16:33:02 slowrie: slowrie 'Stephen Lowrie' 16:33:15 kaeso[m]: in that case I'll try to start with one you might have some input on 16:34:29 .hello red_beard 16:34:29 red_beard: Sorry, but you don't exist 16:34:38 .hello redbeard 16:34:39 red_beard: redbeard 'Brian 'redbeard' Harrington' 16:35:02 #chair red_beard 16:35:02 Current chairs: bgilbert dustymabe geoff- jlebon kaeso[m] red_beard slowrie walters 16:35:10 #topic Action items from last meeting 16:35:19 it's been a long time since we had a meeting so I'll skip this part 16:35:38 and skip to meeting tickets 16:35:53 #topic Ignition: Support multi part mime for user-data 16:36:01 #link https://github.com/coreos/ignition/issues/849 16:36:18 .hello mnguyen 16:36:19 mnguyen_: mnguyen 'Michael Nguyen' 16:36:25 this topic should really wait until ajeddeloh gets back 16:36:36 I added this ticket to our meeting agenda because I was asked about it on the dev list by the folks that produce openstack magnum 16:37:11 bgilbert: so not worth discussing at all? 16:37:13 yes, I was also holding a bit till Andrew was back 16:37:20 yeah, let's hold a week 16:37:39 my gut feeling is that the request is reasonable 16:37:40 he's the one that'll need to be convinced :-) 16:38:00 I think fundamentally the request misunderstands the purpose of Ignition as the sole source of truth 16:38:08 but I'll defer to Andrew 16:38:21 that's a fair point 16:38:41 #info will punt on discussin this until next week when ajeddeloh is back 16:38:46 #undo 16:38:46 Removing item from minutes: INFO by dustymabe at 16:38:41 : will punt on discussin this until next week when ajeddeloh is back 16:39:00 #info will punt on discussing coreos/ignition/issues/849 until next week when ajeddeloh is back 16:39:11 #topic coreos/toolbox v2 16:39:21 #link https://github.com/coreos/fedora-coreos-tracker/issues/38 16:39:51 so basically we're down to the bottom of the discussion in that ticket and mostly splitting hairs on added dependencies IIUC 16:40:40 I tried to build an FCOS based on rawhide to see if the added deps there would change much from F30, looks like it's about the same 16:41:02 maybe the changes debarshiray made haven't made it into an rpm build 16:42:05 there was no flatpak release in the meantime, from what I see 16:42:29 anywho, how do we feel about the current set of pulled in deps 16:42:40 to me it looks ugly 16:43:05 it does 16:43:10 a bit, yes, unfortunately 16:43:50 I just asked debarshiray in the ticket if there is an existing rpm somewhere we can test with 16:44:00 are we in good shape to ship it, and deal with the deps later? 16:44:07 maybe that will give us some new information about "added deps" 16:44:15 we have a variety of unfortunate deps in FCOS already 16:44:36 bgilbert: i think we're mostly in agreement that we want to ship it, correct? 16:44:43 the dconf in there, I think that brings in a binary which is directly in $PATH 16:44:43 we want to ship something like it 16:45:08 we could mask files. like, say, all the font data 16:45:39 * bgilbert stops talking 16:46:05 bgilbert: I'd say let's ship toolbox (once we iterate on deps, trying to get it down to an acceptable list) and then replace in the future if we decide to 16:46:20 sgtm 16:46:49 let's track unwanted deps in https://github.com/coreos/fedora-coreos-tracker/issues/186 though 16:47:08 I'll add my dconf note directly in the ticket, if I'm not alone on that 16:47:21 kaeso[m]: +1 16:47:25 bgilbert: personally I'd like to iterate on the deps in #38 "before" we ship it 16:47:50 since we've already been working with debarshiray it seems like we might be close ? 16:47:52 dustymabe: yeah, I meant anything remaining once we ship 16:48:08 bgilbert: +1 16:49:12 #info we want to ship something like toolbox and toolbox is something that already exists so we will ship it once we've iterated on the dependencies in #38. Will work with debarshiray to see if there is an updated rpm we can evaluate. 16:49:59 any objections ^^ 16:51:05 +1 16:51:16 k, next topic 16:51:22 sgtm, but again I don't know the details of this toolbox 16:51:25 #topic docker group or not 16:51:36 #link https://github.com/coreos/fedora-coreos-tracker/issues/2 16:51:46 this is one of our first issues! 16:52:07 I think I summarized the current decision at the bottom of the ticket 16:52:13 So the open question here is whether we do one of these two options: 16:52:21 - we add a docker group to /etc/group so that usermod -aG docker myusername works? 16:52:28 - we encourage users to use sudo docker instead 16:52:36 it sounds as though the current situation is a bug? 16:52:49 bgilbert: that depends on who you ask 16:52:54 we have half a group 16:53:05 It's an architectural artifact 16:53:09 yup, that's an artifact of using nss-altfiles 16:53:18 And it's not working as people expect it to, which one could call a bug 16:53:33 okay, so this is a special case of the nss-altfiles problem? 16:53:44 AFAICT, yes 16:53:53 we should fix that generally :-) 16:53:56 jlebon: would systemd-sysusers fix this? 16:54:09 yes 16:54:25 i think we either need to say we "NEED" systemd-sysusers for stable 16:54:38 OR we add these groups into /etc/group somehow 16:55:04 at this point docker group is a special case, but the conversation is more general than that 16:55:26 hmm, if one of the primary motivations for the docker group is convenience, then we should encourage rootless podman for those cases, which is as convenient, but actually secure :) 16:55:33 people use docker 16:55:58 yeah, in this case i'm considering that someone has already made their decision 16:56:00 is there a reason not to ship the group? to me, the larger question is who's in it by default 16:56:33 bgilbert: the moby-engine rpm ships it 16:56:42 +1 16:56:59 it's only not shipped on our systems because of this unique bug/feature 16:57:07 depending on how you look at it 16:57:22 I'm on record as wanting altfiles fixed generally 16:57:34 +1 people hit this in silverblue too 16:57:36 if it's infeasible, then yeah, we probably have to explicitly add to /etc/groups 16:57:49 does anyone not want altfiles fixed? 16:57:54 heh 16:58:05 I meant for stable 16:58:13 another way to phrase that is: "is anyone opposed to systemd-sysusers being implemented?" 16:58:33 bgilbert: ahh, ok I get it 16:58:39 new question: 16:58:46 my answer would be yes to both ticket questions: we fix group-file somehow (because the RPM has the group), and we tell people that the group is empty by default and they should `sudo docker` 16:59:11 kaeso[m]: agree 16:59:15 ok new questions: 16:59:26 bgilbert: walters already has an initial patch pending for it -- might be feasible to make it a requirement for stable 16:59:33 - is it feasible to implement systemd-sysusers for stable (remember f30) 16:59:48 - if not then we should definitely get the `docker` group into `/etc/group` 16:59:57 (strictly, f31 would work) 17:00:18 bgilbert: got ya. so you're saying we could do f31 for stable 17:00:44 dustymabe: that's the current plan. we should test a Fedora major rebase before declaring FCOS stable 17:01:09 ok so my two questions remain (minus the caveat) 17:01:14 i think it could make the cut, it just hasn't been prioritized much 17:01:38 jlebon: would you be able to pick it up if walters wasn't able to ? 17:01:48 do you think most of the work is capital D done? 17:01:49 we'd need to get it in soon-ish so we gain experience on it and work out the kinks 17:02:51 dustymabe: probably? haven't looked at it in a while, but I know walters is definitely motivated too to drop nss-altfiles :) 17:03:00 ok 17:03:15 let me ask a dumb question 17:03:27 with systemd-sysusers would `usermod -aG docker myusername` work? 17:03:34 since that's what people are doing/trying 17:03:41 (I'm dropping here, will read logs) 17:03:50 yes, it should 17:03:52 kaeso[m]: o/ 17:03:55 later 17:04:00 jlebon: good to know 17:04:13 ok so we want to try to get that in soon 17:04:39 everything is still in the regular locations, it's basically a declarative way to manage /etc/passwd 17:04:45 should we hackishly add docker to /etc/group in a quick PR now just to unblock people who are going to try our platform? 17:05:15 meh. probably 17:05:32 i'm +1 17:05:34 anyone else? 17:05:35 FWIW I have already been working on using `coretoolbox `for privileged paths too which I think is quite relevant for FCOS 17:05:35 not against 17:06:48 #info we are going to try to prioritize systemd-sysusers work for FCOS stable, but we will hackishly try to add docker to the /etc/group file to smooth the path for users who are currently using FCOS preview releases 17:07:21 moving on to next ticket 17:07:33 walters: do you see a path where you upstream to fedora toolbox ? 17:07:40 #topic Allow creation of immutable directories below root folder 17:07:49 #link https://github.com/coreos/fedora-coreos-tracker/issues/270 17:08:14 this was brought to us by a user who is testing out FCOS preview 17:08:35 and trying to migrate from CL (/me notes that we'll need docs for this) 17:08:50 physically impossible on live systems 17:08:59 they want to create mountpoints at the top of the filesystem hierarchy 17:09:16 unless we put an overlayfs on /sysroot, which doesn't seem great 17:09:44 bgilbert: because things are mounted read-only ? 17:09:48 I think this makes sense as a caveat in the migration guide, and we can catch it in Ignition config translation tooling 17:09:52 dustymabe: because it's a squashfs 17:09:59 which is read-only 17:10:01 yeah 17:10:51 jlebon: hopefully 17:11:21 any other technical reasons why this could or could not work? 17:11:39 Is what I told the user correct? 17:12:02 dustymabe: link? 17:12:04 I told him it was failing because we were using OSTree, which is generally correct as I've tried to do this in the past on my atomic host systems 17:12:08 ah 17:12:22 right 17:12:22 but in this case he's using ignition to try to create those directories 17:12:35 I don't know if it is actually failing because of OSTree in his case or because of something in ignition 17:13:06 not making it immutable implies having some e.g. config for rpm-ostree to know what top-level dirs to recreate on new deployments 17:13:20 jlebon: correct, we've discussed that in an rpm-ostree issue 17:13:34 #link https://github.com/coreos/fedora-coreos-tracker/issues/270 17:13:55 but even if we had that config. would it work early enough such that ignition could use it 17:14:09 I guess ignition could create the 'config' and then the service would create the mountpoint 17:14:09 sorry, that was badly phrased. / could still be immutable. it's just that it's like this to prevent users being surprised when it's wiped out on the next deployment 17:14:48 I guess I'm not seeing the issue here. configs will need to be modified when switching CL -> FCOS. it shouldn't be a big deal to repoint docker at a different path. 17:14:56 .hello2 17:14:57 lorbus: lorbus 'Christian Glombek' 17:14:57 jlebon: correct. it was done to prevent unexpected behavior 17:15:13 dustymabe: yeah, we'd either need to teach ignition some ostree stuff, or have more magic goop in the initrd... 17:15:22 bgilbert: +1 17:15:41 FCOS makes some parts of the tree immutable for good reason. it doesn't seem worth jumping through a lot of hoops to work around that 17:15:46 FWIW, i like the current model for the reasons stated in https://github.com/projectatomic/rpm-ostree/issues/337#issuecomment-302703626 17:15:50 ok back to bgilberts point about squashfs 17:16:04 do you not use squashfs in lives today with CL? 17:16:13 sure we do, but it's /usr 17:16:32 I see 17:16:44 ok 17:17:05 one hack would be to have / be tmpfs, then mount the squashfs somewhere, then bind-mount the subdirs... yuck 17:17:10 #link https://lists.fedoraproject.org/archives/list/coreos@lists.fedoraproject.org/message/F2D2TMYJFRU3H24RJJNHCN3QGJPACXXU 17:17:19 ^^ FYI there is the link to the original discussion 17:17:45 basically they aren't going in greenfield and it's going to be easier for them if they can use /dockerdata everywhere 17:18:03 that's their reasoning for wanting this to exist in FCOS 17:18:11 #link https://github.com/coreos/fedora-coreos-tracker/issues/159 17:18:20 there are _so_ _many_ _things_ which will change 17:18:44 I'm in favor of being compatible where it makes sense, but it doesn't always make sense 17:18:55 bgilbert: +1 does that ticket need to be updated to include something for this? 17:19:01 dustymabe: yup 17:19:07 can you add it? 17:19:11 sure 17:19:37 #action bgilbert to update #159 to add doc item for immutable '/' 17:20:43 #info while having an immutable top level directory seems like a reasonable ask there are technical reasons why this won't work for fedora coreos and we suggest users to change their mount point to be under one of the writable filesystems 17:21:17 I'll add a comment to the ticket - we can close it out when the docs get written? 17:21:49 let's not hold it open for the docs 17:22:00 ok, yeah I didn't want to seem to harsh on the user :( 17:22:09 I always struggle with that 17:22:30 #topic open floor 17:22:33 understood. IMO, better to give a clear explanation and a clear "no" :-) 17:22:38 +1 17:22:54 anyone with open floor topics 17:23:13 jlebon: how much does rpm-ostree assume /boot exists? 17:23:25 FYI: we're really close to having aws replication and also kola AWS testing merged into our pipeline 17:23:52 rpm-ostreed refuses to start on live systems and I don't have a sense of whether modifying it to run read-only is a major task 17:23:56 bgilbert: s/rpm-ostree/ostree/, then i think quite a bit 17:24:51 right. e.g. libostree scans BLS entries when initializing IIRC 17:25:20 we could probably fake up that part of /boot... 17:25:46 i don't have a good estimate, but offhand i think it wouldn't be a small task 17:26:06 i haven't been keeping up with your latest patches -- do you have a ticket to track that part? 17:26:24 not yet, it's in ~/TODO 17:26:30 can file one though 17:27:03 yeah, let's discuss upstream. definitely something we can make happen 17:27:07 +1 17:28:48 re. signing, i'm thinking of making it part of cosa instead of a separate helper 17:29:07 e.g. the pipeline would call something like `cosa sign robosignatory` 17:29:15 ok meeting is running out of time 17:29:24 closing this out in 30s 17:30:09 jlebon: +1 17:30:32 btw, related to the signing work: https://github.com/coreos/coreos-assembler/pull/741 17:30:41 #endmeeting