16:30:53 #startmeeting fedora_coreos_meeting 16:30:53 Meeting started Wed Sep 5 16:30:53 2018 UTC. 16:30:53 This meeting is logged and archived in a public location. 16:30:53 The chair is dustymabe. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:30:53 Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:30:53 The meeting name has been set to 'fedora_coreos_meeting' 16:30:58 #topic roll call 16:31:09 .hello2 16:31:10 bhavin192: bhavin192 'Bhavin Gandhi' 16:31:17 .hello rfairleyredhat 16:31:18 rfairley: rfairleyredhat 'Robert Fairley' 16:31:30 .hello2 16:31:31 dustymabe: dustymabe 'Dusty Mabe' 16:31:45 .hello2 16:31:46 bgilbert: bgilbert 'Benjamin Gilbert' 16:31:50 .hello sinnykumari 16:31:50 .hello lorbus 16:31:50 ksinny: sinnykumari 'Sinny Kumari' 16:31:53 lorbus[m]: lorbus 'Christian Glombek' 16:31:55 .hello lucab 16:32:00 kaeso: lucab 'Luca Bruno' 16:32:04 .hello2 16:32:05 yzhang: yzhang 'Yu Qi Zhang' 16:32:06 .fas jasonbrooks 16:32:09 jbrooks: jasonbrooks 'Jason Brooks' 16:32:26 .hello2 16:32:27 mskarbek: mskarbek 'None' 16:32:31 .hello2 16:32:34 strigazi: strigazi 'Spyros Trigazis' 16:32:40 yay strigazi 16:32:59 :) 16:33:10 I beat the traffic 16:33:13 #chair bhavin192 rfairley bgilbert ksinny lorbus[m] kaeso yzhang jbrooks mskarbek strigazi 16:33:13 Current chairs: bgilbert bhavin192 dustymabe jbrooks kaeso ksinny lorbus[m] mskarbek rfairley strigazi yzhang 16:33:24 .hello2 16:33:25 slowrie: slowrie 'Stephen Lowrie' 16:33:26 .hello sayanchowdhury 16:33:28 sayan: sayanchowdhury 'Sayan Chowdhury' 16:33:57 .hello mnguyen 16:33:58 mnguyen_: mnguyen 'Michael Nguyen' 16:34:51 #chair mnguyen_ slowrie sayan 16:34:51 Current chairs: bgilbert bhavin192 dustymabe jbrooks kaeso ksinny lorbus[m] mnguyen_ mskarbek rfairley sayan slowrie strigazi yzhang 16:35:42 #topic previous meeting action items 16:35:56 * dustymabe to create a FCOS tracker ticket for toolbox v2 design 16:35:58 discussion 16:36:00 * ajeddeloh to tag closed FCOS issues into design-doc PRs 16:36:02 * dustymabe to add additional discussion to 16:36:04 coreos/fedora-coreos-tracker#35 16:36:47 #info dusty created FCOS tracker for toolbox v2 design https://github.com/coreos/fedora-coreos-tracker/issues/38 16:37:18 #info dusty added additional discusstion to https://github.com/coreos/fedora-coreos-tracker/issues/35 16:38:42 I'll re-action andrew for now 16:39:03 #action ajeddeloh to tag closed FCOS issues into design-doc PRs 16:39:20 I'll move into meeting items 16:39:36 .hello2 16:39:37 ajeddeloh: ajeddeloh 'Andrew Jeddeloh' 16:40:07 * rishi is lurking 16:40:19 #topic Equivalent to system containers from Fedora Atomic in Fedora CoreOS 16:40:21 the CL toolbox is just a script to run a container image with systemd-nspawn, but this issue/38 goes into the creation of an image 16:40:33 #link https://github.com/coreos/fedora-coreos-tracker/issues/37 16:41:51 I think everyone is positive on portable systemd services, we need some experience with it and 16:42:10 iterate a bit on the delivery method 16:42:31 +1 for that 16:43:01 its also still marked experimental 16:43:32 ajeddeloh: we could chat with systemd maintainers to see how mature they think it is 16:43:44 the systemd PS + ostree + Ignition idea sounds real slick 16:43:47 yeah 16:43:57 if we put effort it may graguate from experimental 16:44:22 At least I hope it does :) 16:45:04 Should we try the portable cli in the atomic host? 16:45:18 strigazi: is that what it's called ? 16:45:21 portable cli ? 16:45:30 portablectl 16:45:41 strigazi: is that a separate rpm we need to include ? 16:45:47 or is it there already ? 16:45:51 systemd portable services (not that I'd played with them much) do look like they need an equivilent of "enablement" 16:46:15 dustymabe: I have to check, I don't know 16:46:16 dustymabe: it's a flag on systemd srpm, I don't know if the binary rpm is split 16:46:17 like it looks like right now you need to manually run `portablectl attach` 16:46:36 kaeso: +1 16:47:05 strigazi: if we think this is a possible future for us then we can try to get the appropriate bits into f29AH so you can at least try to prove them out 16:48:21 strigazi: to be clear, those wouldn't help with augmenting the non-containerizable bits (docker, kubelet, etc) 16:49:08 kaeso: we couldn't run kubelet as a systemd portable service? 16:50:02 strigazi: the kubelet --containerized flag is being deprecated as it is broken (from a design PoV) 16:50:25 kaeso: yes, that is true 16:50:31 systemd PS don't have much "containized" bits to them though 16:50:33 at least I've been told 16:50:48 its not a ton more than a chroot 16:51:02 * ajeddeloh also doesn't know much about the kubelet 16:51:52 strigazi: but that was just an example. In general there are software that assumes running in the host, and those can't be easily swapped via containerized services 16:52:52 kaeso: I think where he is coming from is "system containers" so I think "some containerization" probably still satisfies his use case 16:52:59 If systemd PS work well for docker, kubelet, etc I wonder if we could just use that and not ship rpm-ostree (just ostree) 16:53:11 kaeso: ok, do you have any pointers on the deprecation of --containerized ? 16:53:31 strigazi: jbrooks might have a link handy 16:53:43 but mostly look at upstream kube and I think you should see discussion about it 16:53:48 strigazi: I don't have them at hand, no 16:54:20 as ajeddeloh mentioned systemdPS are just a chroot plus a few more bits. I'll have a closer look. 16:54:32 ajeddeloh: that's the same split between rkt and torcx, I don't think it can go away 16:54:34 I don't have a link 16:55:15 does anyone know if we can deliver a PS payload wvia OCI ? 16:55:15 They were talking about it in https://github.com/kubernetes/kubernetes/issues/43708 16:55:23 i.e. an OCI registry ? 16:55:34 kaeso: what do you mean? 16:56:47 jbrooks: also relevant https://github.com/kubernetes/kubernetes/issues/61675 16:57:49 any takeaways from this discussion we should add to the ticket ? 16:58:07 dustymabe: delivering systemdPS with skopeo or similar would be great. it should be possible. 16:58:12 should we investigate running docker, kubelet, etc from a PS? 16:58:18 ajeddeloh: that if you move a piece of software to its own mountns/chroot, you break all its assumptions about files/binaries/services on the host that it gets via runtime probing 16:58:19 ^^ yes 16:58:42 kaeso: ah, otcha 16:58:49 gotcha* 16:59:17 https://public.etherpad-mozilla.org/p/20180905-fcos 16:59:57 ajeddeloh: we already did and it works at a first approximation, but it breaks its fundamental assumptions 17:01:09 I think that PS (or podman) would be fine for most of our needs 17:01:17 for the rest, we have rpm-ostree 17:01:45 kaeso: are you saying we might just be able to use podman itself instead of PS? 17:02:07 and we fix software to be container-friendly, we can move more stuff there 17:02:39 draft of what I'll put in the ticket: https://public.etherpad-mozilla.org/p/20180905-fcos 17:02:56 strigazi: are there any pieces of that that you can sign up for to investigate ? 17:03:16 dustymabe: it may, like rkt did. It's mostly a matter of patterns in writing systemd service units running podman 17:03:27 kaeso: +1 17:03:41 dustymabe: I'm a little lost, should we (or I) put effort on investigating kubelet or docker as a PS? 17:04:27 strigazi: i think it would defintely be useful to know if it worked or not 17:04:33 and if you hit any obvious blockers 17:05:08 ok, I can give it a go. 17:05:22 I can also try to collect all the open tickets on containerized docker/kubelet I remember 17:05:40 cool 17:05:41 kaeso: thanks, that would be great 17:05:42 I updated the ticket 17:05:49 https://github.com/coreos/fedora-coreos-tracker/issues/37#issuecomment-418806643 17:05:54 next topic 17:06:10 #topic How to ship cloud specific bits 17:06:14 #link https://github.com/coreos/fedora-coreos-tracker/issues/41 17:06:43 ajeddeloh: i'll let you lead 17:07:40 Do every cloud will need a different image. Ideally we want them to be as similar as possible and to localize any differences 17:07:47 on CL we had the OEM partition 17:08:08 we decided against that for FCOS (the OEM partition was a bad idea in many ways) 17:08:36 That ticket is mostly "how do we localize the changes" 17:09:07 things that will probably need to change between clouds: base ignition configs, files for those configs, and grub configs 17:09:28 thoughts? 17:09:44 if /boot/ is reliable then it seems reasonable to use that 17:10:17 it doesn't really need to go in /boot, and it feels like /boot should be minimal 17:10:31 also there's always the possibility that we'll need to ship something large for a cloud 17:10:48 yeah, i was thinking these would mostly be non-binary files 17:10:53 probably 17:10:58 *cough* openvmtools *cough* 17:11:01 so for me, the rootfs seems the natural place 17:11:05 well, openvmtools would go in the image anyway 17:11:39 bgilbert: is there a problem with using the rootfs (i.e. the ignition disks "blow away root" case) 17:11:54 i guess not because we'd replace it with something and then ignition-files stage could read whatever is there? 17:12:09 well 17:12:21 there would be Ignition configs in whatever-the-place-is 17:12:31 ajeddeloh: so your question is whether we ship all of them and enable only one based on OEM id, or if we do some advanced ostree trick so that only the right one is in the deployed rootfs? 17:12:34 so if we blow away the rootfs, then fail, then reboot, we wouldn't be able to reread the base config 17:12:41 fair point 17:13:01 kaeso: more the former, not sure I follow the latter 17:13:31 for large things for clouds, can we pull those down from somewhere? 17:14:55 (via Ignition configs :D ) 17:15:00 ajeddeloh: I don't have enough knowledge for that, but I was jut wondering if your question was more complex than what I thought 17:15:41 so basically we should "try rootfs" and fallback to /boot if we hit blockers ? 17:16:20 ajeddeloh: rpm-ostree has a feature to rebuild the initramfs on the client, so I think you'll always need to ship those bits somewhere in the rootfs 17:16:24 if we want to support the reformatting-root case, rootfs seems like more trouble than it's worth 17:16:37 dustymabe: what about the rootfs blown away case 17:16:49 right 17:16:59 kaeso: I'm firmly of the opinion we should not support regenerating the initramfs on the client 17:17:12 ajeddeloh++ 17:17:12 bgilbert: Karma for ajeddeloh changed to 1 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 17:18:11 ajeddeloh: that's probably not a bad idea, but again I don't have the knowledge about usecase it's solving 17:18:20 *which usecase 17:18:26 ok cool ajeddeloh mind updating the ticket with this discussion ? 17:18:34 we can move to open floor for last 10 minutes 17:18:40 but regardless of where we put them, do we have opinions on if we ship everything or just 1 clouds worth (assuming no large things) 17:18:47 dustymabe: sure 17:19:08 ajeddeloh: I prefer everything.. but if we start shipping large files for each cloud I may change my opinion 17:19:14 +1 17:19:28 * dustymabe wishes all clouds were created equal :) 17:19:36 ultimately the grub config will need to differ to set oemid 17:19:48 but if we can minimize the differences that'd be ideal 17:20:43 ajeddeloh: right.. i'd honestly like to be able to "detect" and set the oemid too, but I've been told by multiple people that's a bad idea :) 17:21:00 can't win them all 17:21:07 #topic open floor 17:21:16 anyone with anything for open floor ? 17:21:28 I wanted to highlight some work by walters recently 17:22:06 dustymabe: I think it was me, autodetection was the bad part in that context. Setting it from externally-provided hint is ok. 17:22:53 walters: created https://github.com/coreos/coreos-assembler - which is kinda a "toolbox" for creating ostrees and image artifacts for fedora coreos 17:23:00 it's rough around the edges right now 17:23:14 walters++ 17:23:16 https://github.com/coreos/coreos-assembler/pull/29 is a WIP PR which I've been using 17:24:03 there is also https://github.com/cgwalters/fedora-coreos-config which is where the manifest files are held 17:24:12 #link https://github.com/coreos/coreos-assembler 17:24:17 #link https://github.com/cgwalters/fedora-coreos-config 17:24:23 all of this is open to feedback/change 17:24:30 geoff-: I think has already opened a PR 17:24:34 geoff-++ 17:25:44 any other open floor items ? 17:25:44 walters: would be nice to have a rough: run these steps and build an image guide, even if its crude right now 17:26:13 but super cool nonetheless 17:26:13 ajeddeloh: if you look at the readme in the PR you'll see a very rough guide 17:26:15 it's in the README.md in the PR 17:26:19 ah +1 17:26:41 ajeddeloh: of coures there may be some "i'm running fedora so this just works for me" things 17:26:52 but there's a whole lot of things more; the biggest I'd say is polishing the workflow for "I have a patch/local git for e.g. systemd, use that for build" 17:26:53 so if you hit trouble just comment in the PR 17:27:08 +1 17:27:37 any other topics for open floor ? 17:28:12 I'll try to spend some more time with coreos-assembler this week 17:29:23 cool 17:29:29 awesome, FWIW I completely agree with https://github.com/coreos/fedora-coreos-tracker/issues/9#issuecomment-404920705 17:29:50 ending meeting in 30 seconds 17:30:30 #endmeeting