16:30:20 <miabbott> #startmeeting fedora_coreos_meeting
16:30:20 <zodbot> Meeting started Wed Aug 29 16:30:20 2018 UTC.
16:30:20 <zodbot> This meeting is logged and archived in a public location.
16:30:20 <zodbot> The chair is miabbott. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:30:20 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:30:20 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:30:30 <miabbott> #topic roll call
16:30:34 <slowrie> .hello2
16:30:35 <zodbot> slowrie: slowrie 'Stephen Lowrie' <slowrie@redhat.com>
16:30:45 <bhavin192> .hello2
16:30:46 <zodbot> bhavin192: bhavin192 'Bhavin Gandhi' <bhavin7392@gmail.com>
16:31:00 <miabbott> .hello miabbott
16:31:01 <kaeso> .hello lucab
16:31:01 <zodbot> miabbott: miabbott 'Micah Abbott' <miabbott@redhat.com>
16:31:05 <zodbot> kaeso: lucab 'Luca Bruno' <lucab@redhat.com>
16:31:12 <rfairley> .hello rfairleyredhat
16:31:13 <zodbot> rfairley: rfairleyredhat 'Robert Fairley' <rfairley@redhat.com>
16:31:21 <miabbott> #chair slowrie bhavin192 kaeso rfairley
16:31:21 <zodbot> Current chairs: bhavin192 kaeso miabbott rfairley slowrie
16:31:37 <jbrooks> .fas jasonbrooks
16:31:38 <zodbot> jbrooks: jasonbrooks 'Jason Brooks' <jbrooks@redhat.com>
16:31:46 <dustymabe> .hello2
16:31:47 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:31:57 <miabbott> #chair jbrooks dustymabe
16:31:57 <zodbot> Current chairs: bhavin192 dustymabe jbrooks kaeso miabbott rfairley slowrie
16:32:04 <akshayg96> .hello akshay196
16:32:05 <zodbot> akshayg96: akshay196 'Akshay Gaikwad' <akgaikwad001@gmail.com>
16:32:28 <miabbott> #chair akshayg96
16:32:28 <zodbot> Current chairs: akshayg96 bhavin192 dustymabe jbrooks kaeso miabbott rfairley slowrie
16:32:46 <ksinny> .hello sinnykumari
16:32:47 <zodbot> ksinny: sinnykumari 'Sinny Kumari' <ksinny@gmail.com>
16:32:48 <sayan> .hello sayanchowdhury
16:32:50 <zodbot> sayan: sayanchowdhury 'Sayan Chowdhury' <sayan.chowdhury2012@gmail.com>
16:32:59 <miabbott> #chair ksinny sayan
16:32:59 <zodbot> Current chairs: akshayg96 bhavin192 dustymabe jbrooks kaeso ksinny miabbott rfairley sayan slowrie
16:33:09 <bgilbert> .hello2
16:33:10 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:33:14 <ajeddeloh> .hello2
16:33:15 <zodbot> ajeddeloh: ajeddeloh 'Andrew Jeddeloh' <andrew.jeddeloh@redhat.com>
16:33:36 <miabbott> #chair bgilbert ajeddeloh
16:33:36 <zodbot> Current chairs: ajeddeloh akshayg96 bgilbert bhavin192 dustymabe jbrooks kaeso ksinny miabbott rfairley sayan slowrie
16:33:49 <misc> .hello2
16:33:50 <zodbot> misc: misc 'None' <misc@zarb.org>
16:33:54 <miabbott> we'll get started in about a minute
16:33:57 <miabbott> #chair misc
16:33:57 <zodbot> Current chairs: ajeddeloh akshayg96 bgilbert bhavin192 dustymabe jbrooks kaeso ksinny miabbott misc rfairley sayan slowrie
16:34:37 <miabbott> #topic Action items from last meeting
16:34:45 <miabbott> * ajeddeloh to file ticket regarding ignition and spec versions
16:34:45 <miabbott> * strigazi to file ticket for system containers discussion
16:35:02 <miabbott> ajeddeloh: i think you have a ticket filed?
16:35:07 <ajeddeloh> yup
16:35:14 <miabbott> could you link it here please?
16:35:29 <kaeso> the second is https://github.com/coreos/fedora-coreos-tracker/issues/37 and marked for discussion today
16:35:35 <miabbott> kaeso: thanks
16:35:51 <ajeddeloh> #link https://github.com/coreos/fedora-coreos-tracker/issues/31
16:36:08 <miabbott> thanks!
16:36:11 <kaeso> #link https://github.com/coreos/fedora-coreos-tracker/issues/37
16:36:32 <miabbott> let's get to that issue
16:36:38 <miabbott> #topic Equivalent to system containers from Fedora Atomic in Fedora CoreOS #37
16:37:13 <miabbott> anyone want to take the lead on this?  i'm not familiar with the issue
16:37:46 <dustymabe> strigazi: presetn?
16:38:08 <dustymabe> if he's not hear I can probably speak to this topic
16:38:33 <miabbott> he's been idle for almost 3h
16:38:38 <miabbott> so i'd say go for it dustymabe
16:38:58 <dustymabe> he did file the ticket earlier today so I figured he wanted to speak to the topic himself during the meeting but I'll shoot
16:39:40 <dustymabe> basically i think what he is expressing is a desire to have the host be configurable in the same way system containers allowed flexibility for atomic host in the past
16:40:10 <dustymabe> whether that be using the system container technology or not, i think he doesn't care about as much
16:41:11 <kaeso> this somehow follows the same discussion of rkt/podman/etc
16:41:20 <dustymabe> I know some of us have discussed how we would achieve swapping out various versions of docker/kubernetes from the host
16:41:41 <miabbott> this feels like we'll need to gather some more input before we can make an decisions
16:41:47 <miabbott> any*
16:42:01 <bgilbert> the swapping-out discussion has focused on package overlays so far
16:42:18 <bgilbert> possibly in some less-general sense than the current uses
16:42:29 <dustymabe> bgilbert: indeed - the thought there was that system containers were a bit hard to manage (i.e. separate from the host management tool `rpm-ostree`)
16:42:45 <mnguyen_> .hello mnguyen
16:42:46 <zodbot> mnguyen_: mnguyen 'Michael Nguyen' <mnguyen@redhat.com>
16:42:48 <dustymabe> and package layering would possibly provide what we needed
16:42:56 <bgilbert> and there will of course be no impediment to running systemd-nspawn from a systemd service
16:42:58 <jbrooks> I've had problems w/ system containerized kubelet
16:43:01 <miabbott> #chair mnguyen_
16:43:01 <zodbot> Current chairs: ajeddeloh akshayg96 bgilbert bhavin192 dustymabe jbrooks kaeso ksinny miabbott misc mnguyen_ rfairley sayan slowrie
16:43:08 <kaeso> but system-containers are still containers, right?
16:43:12 <jbrooks> And via package layering, it just works
16:43:20 <dustymabe> kaeso: yes
16:43:47 <jbrooks> There are benefits to system containers, but they require additional attn, beyond what we pay to rpms
16:43:58 <dustymabe> jbrooks: is that because of system containers or because the kubelet isn't really meant to be run containers (I hear they are dropping support for containerized kubelet)
16:44:17 <kaeso> so those and package layering are different topics
16:44:36 <dustymabe> kaeso: well it's mostly in the realm of "host extensibility"
16:44:40 <jbrooks> dustymabe, I think it could probably work, but it needs to be made to work, and there aren't enough interested makers
16:44:53 <dustymabe> jbrooks: i don't disagree with that
16:44:54 <miabbott> another consideration is how the system containers are going to be installed/managed.  atomic CLI is probably going to be phased out and that was the primary tool for working with them
16:45:18 <dustymabe> miabbott: yep. and it's written in python so that presents its own problem set
16:45:30 <jbrooks> If system containers aren't going to be part of RH's game plan, I am less enthusiastic about them. That corp backing is important, imo
16:45:58 <jbrooks> I don't know if it's part of the game plan or not
16:46:05 <kaeso> dustymabe: if they don't run in pid1 namespaces and don't share the rootfs bits, they are in the realm of "containers" I'd say
16:46:47 <dustymabe> kaeso: but if they modify the host in some way and are providing low level system services, they are in the realm of "host extensibility" i would say :)
16:47:05 <dustymabe> add a caveat - and they are not managed by an orchestrator, like kube
16:47:12 <jbrooks> They pretty much always are bind mounting some part of the host
16:47:29 <jbrooks> They're managed by the host's systemd
16:47:39 <kaeso> dustymabe: that's what rkt was doing for kubelet/etcd/flannel on CL
16:48:19 <kaeso> dustymabe: which is why I think we are talking about another container runtime
16:48:29 <dustymabe> yep. so I think there are two questions here
16:48:39 <kaeso> related, I hope that systemd portable services mature in the meanwhile
16:48:40 <jbrooks> they run with runc, which will be there already
16:49:14 <dustymabe> 1. what technology do we want to use for extending the host for things we support (i.e. different versions of docker or kubernetes)
16:49:40 <dustymabe> 2. can we give somebody a valid path to use system containers even if we don't ship the atomic CLI in the host ?'
16:50:18 <yzhang> my take on 2: since the atomic CLI only really consolidated a bunch of other tools to do system containers, its possible to do it all manually without it
16:50:20 <dustymabe> for #2 basically even if "we" don't care that much about system containers strigazi could still use them for their purposes in openstack magnum
16:50:33 <yzhang> is that a good idea though? Probably not?
16:51:03 <dustymabe> is what a good idea ?
16:51:16 <yzhang> Doing the system container workflow manually via a script, etc.
16:51:33 <kaeso> yzhang: that's what bgilbert was saying regarding systemd-nspawn from a systemd service
16:51:44 <miabbott> dustymabe: do you know if strigazi and his team were using atomic CLI to work with the sys containers?
16:52:00 <kaeso> yzhang: with s/systemd-nspawn/runc/
16:52:04 <miabbott> if not, the lack of the CLI might be less of an issue
16:52:24 <dustymabe> miabbott: I believe so
16:52:36 <yzhang> kaeso: gotcha, I joined a bit late so might have missed that bit
16:52:55 <miabbott> #chair yzhang
16:52:55 <zodbot> Current chairs: ajeddeloh akshayg96 bgilbert bhavin192 dustymabe jbrooks kaeso ksinny miabbott misc mnguyen_ rfairley sayan slowrie yzhang
16:53:04 <dustymabe> miabbott: I think mostly we'll have to wait and hear from him - maybe we can discuss next time
16:53:15 <dustymabe> are there any key takeaways we want to write down from our discussion here today ?
16:53:51 <kaeso> #link http://0pointer.net/blog/walkthrough-for-portable-services.html
16:53:54 <miabbott> i think the management question is one...do we need an equivalent atomic CLI or do we let users figure out their own way
16:54:20 <miabbott> (my question assumes we use the sys container paradigm...)
16:55:20 <dustymabe> miabbott: those sound like good questions for the ticket
16:55:24 <dustymabe> any other things ?
16:55:36 <dustymabe> i.e. what have we said today we don't want to say again next week?
16:55:39 <miabbott> the larger question of using sys containers vs systemd portable services vs....
16:56:12 <dustymabe> i'm definitely interested in systemd portable services.. if someone can present that as a viable alternative then I think we mostly get it for free (systemd already in the host)
16:56:26 <dustymabe> maybe we can ask strigazi himself to investigate
16:57:07 <kaeso> dustymabe: they are very young (from v239, latest systemd version)
16:58:07 <dustymabe> kaeso: yeah :(
16:58:25 <dustymabe> ok miabbott do you think you have enough to update the ticket?
16:58:43 <miabbott> lol, i was just going to ask you.  but FIFO, so i'll do it  :)
16:59:15 <miabbott> that was the only 'meeting' issue, so open floor time
16:59:19 <miabbott> #topic open floor
16:59:21 <dustymabe> miabbott: I can try if you think it'll roll off the stack before EOM
16:59:36 <yzhang> I have a quick open floor
16:59:38 <miabbott> dustymabe: no worries, i got it
16:59:40 <yzhang> Regarding https://pagure.io/releng/issue/7698
16:59:49 <miabbott> #link https://pagure.io/releng/issue/7698
16:59:50 <yzhang> Although I'm not sure how many of the interested parties are here
17:00:25 <yzhang> theres also a bit of discussion on https://github.com/openshift/os/issues/78
17:00:58 <miabbott> for those not following the links, this is about a 'toolbox' container
17:01:01 <yzhang> but essentially from what I gather we want some variant of the toolbox container that exists today adapted for FCOS
17:01:07 <yzhang> yep
17:01:37 <yzhang> Anyone have any thoughts on this topic? I'd like to help out making this happen :)
17:02:19 <dustymabe> yzhang: +1
17:02:26 <rubao> +1 for toolbox container
17:02:38 <dustymabe> I know some people from the containers SIG were discussing a "toolbox" container
17:02:45 <dustymabe> cverna: ^^ FYI
17:02:46 <ajeddeloh> Toolbox was never meant for any sort of production use on CL, with no *real* contract about breakage
17:03:07 <ajeddeloh> so I'm not too concerned about doing something now and not being able to change it later
17:03:18 <kaeso> for reference, coreos-toolbox is a plain fedora image with a wrapper script for the bindmounts
17:03:19 <dustymabe> ajeddeloh: is there a short list of things you would change about toolbox if we were going to do toolbox v2 ?
17:03:39 <ajeddeloh> from me: nope. I haven't used it much though
17:03:48 <ajeddeloh> it was mostly just a debugging tool for me
17:03:54 <bgilbert> dustymabe: the tool or the container?
17:04:15 <dustymabe> bgilbert: the tool
17:04:35 <bgilbert> off the top of my head, dropping the rkt dependency, adding a mechanism for command-line arguments, probably changing the mount point for /
17:05:22 <dustymabe> maybe we should start a FCOS tracker issue where we can discuss something like this ?
17:05:28 <kaeso> #link https://github.com/coreos/toolbox/blob/master/toolbox
17:05:33 <ajeddeloh> dustymabe++
17:05:34 <zodbot> ajeddeloh: Karma for dustymabe changed to 12 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:05:36 <yzhang> I think there is one dustymabe: https://pagure.io/ContainerSIG/container-sig/issue/11
17:05:45 <yzhang> wait
17:05:46 <dustymabe> ahh, so there is one over in the container sig repo
17:05:55 <yzhang> Not exactly a FCOS tracker issue
17:05:56 <bgilbert> not exposing systemd-nspawn syntax in the config API
17:06:09 <dustymabe> yeah I don't think the one you link yzhang is exactly what we're talking about
17:06:21 <dustymabe> that link is for a container for "developing packages and such"
17:06:42 <dustymabe> however, swapping out the default container for something else, should be something that's easy to do
17:06:48 <dustymabe> probably
17:07:14 <dustymabe> #action dustymabe to create a FCOS tracker ticket for toolbox v2 design discussion
17:07:53 <yzhang> sounds good
17:08:06 <kaeso> it seems like we are re-inventing another container runtime wrapper, can't we instead have a doc "this is how you run a toolbox container in ${default_fcos_runtime}"?
17:08:47 <dustymabe> somewhat related to this, but also related to host extensibility is a comment I made in a silverblue forum earlier about something called "host contexts": https://discussion.fedoraproject.org/t/rancher-linuxkit-style-systems/332/2?u=dustymabe
17:08:57 <ajeddeloh> kaeso: I at least want like a bash alias or something
17:09:05 <dustymabe> ajeddeloh: yeah I agree
17:09:19 <dustymabe> a very "thin" wrapper with narrow scope isn't a bad thing I don't think
17:11:04 <kaeso> we can't have "a very thin wrapper" and "not exposing $runtime syntax" at the same time, I fear
17:11:37 <dustymabe> kaeso: fair
17:11:49 <miabbott> let's take any additional discussion to the ticket that dustymabe will eventually file
17:11:51 <dustymabe> are there any other topics we'd like to bring up ?
17:12:12 <dustymabe> any thoughts on what we say in https://github.com/coreos/fedora-coreos-tracker/issues/35
17:12:20 <dustymabe> "Should OSTree be used at all?"
17:12:32 <ajeddeloh> I say yes?
17:12:36 <dustymabe> I was thinking about saying
17:12:44 <dustymabe> - OSTree itself is used by several different projects and we like how healthy the community is there.
17:13:20 <dustymabe> - A few Fedora CoreOS contributors are the primary contributors to OSTree and RPM-OSTree and thus we can get new features in and get bug fixes fixed faster than by using something like ...
17:13:32 <dustymabe> what was the name of the updater from chromeos
17:13:46 <ajeddeloh> rpm-ostree's usage within fedora and RH means things are already packages for it
17:13:48 <miabbott> omaha?
17:13:55 <ajeddeloh> update-engine
17:14:01 <dustymabe> update-engine. that's it
17:14:09 <ajeddeloh> or update_engine depending on where you look :P
17:14:27 <miabbott> the submitter of the ticket seems very keen on us using ZFS
17:14:38 <kaeso> I personally think that ajeddeloh's answer nailed it already
17:14:45 <miabbott> heh
17:15:08 <ajeddeloh> but the seperate USR means we've got a bunch of crap like extra symlinks, weird systemd-tmpfiles, etc that we'd have to recreate and manage for FCOS if we used the AB USR approach
17:15:24 <ajeddeloh> oh and we don't overwrite the fallback during upgrade
17:15:30 <dustymabe> kaeso: i agree. I'll just add these few other points and see if we get much back.. then we should be able to close it out and "decided" and add it to the design doc
17:15:44 <ajeddeloh> so we wont hit that weird issue where we had to pause rollouts for the TCP bug
17:16:03 <mskarbek> miabbott: I'm not, I perfectly know that ZFS is not an option :D
17:16:13 <kaeso> miabbott: historically CL was using btrfs for doing those awesome things, but it turned out it wasn't mature enough for that
17:16:23 <miabbott> mskarbek: welcome :)
17:16:28 * ajeddeloh uses btrfs on his daily driver
17:16:35 <dustymabe> mskarbek: \o
17:16:52 * ajeddeloh isn't _too_ concerned about losing his data either
17:17:15 * dustymabe does use it as well - at least on this laptop i'm sitting on
17:17:46 <dustymabe> mskarbek: any counter points you'd like to make while we're in the meeting ?
17:18:49 <kaeso> I haven't used btrfs in a while, but I wouldn't push for that as a replacement for ostree features
17:19:12 <dustymabe> kaeso: yes, it's definitely a different layer
17:19:27 <ajeddeloh> Oh, I have a quick thing: we should do design-doc PRs for the already closed issues. I can take one, but would prefer to not take _all_ of them
17:19:57 <mskarbek> if we are choosing between ostree vs btrfs then ostree definitively
17:20:02 <dustymabe> ajeddeloh: if you want to go through and tag a few of us in each one of them - you can voluntold us :)
17:20:34 <ajeddeloh> +1
17:20:38 <miabbott> #action ajeddeloh to tag closed FCOS issues into design-doc PRs
17:20:38 <dustymabe> mskarbek: :) - i'll respond in ticket. thanks for bringing it up because i think it is worth having these discussions
17:21:05 * ajeddeloh looks and sees there's not as many as he thought
17:21:08 <miabbott> #action dustymabe to add additional discussion to coreos/fedora-coreos-tracker#35
17:21:16 <dustymabe> +1
17:21:43 <miabbott> any other topics for open floor?
17:22:46 <miabbott> ok then...thanks all for attending!
17:22:49 <miabbott> #endmeeting