16:30:30 <dustymabe> #startmeeting fedora_coreos_meeting
16:30:30 <zodbot> Meeting started Wed Sep 18 16:30:30 2019 UTC.
16:30:30 <zodbot> This meeting is logged and archived in a public location.
16:30:30 <zodbot> The chair is dustymabe. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:30:30 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:30:30 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:30:35 <dustymabe> #topic  roll call
16:30:40 <slowrie> .hello2
16:30:40 <red_beard> .hello redbeard
16:30:41 <zodbot> slowrie: slowrie 'Stephen Lowrie' <slowrie@redhat.com>
16:30:41 <dustymabe> .hello2
16:30:44 <zodbot> red_beard: redbeard 'Brian 'redbeard' Harrington' <bharring@redhat.com>
16:30:47 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:30:49 <jdoss> .hello2
16:30:50 <zodbot> jdoss: jdoss 'Joe Doss' <joe@solidadmin.com>
16:30:52 <rishi> .hello2
16:30:54 <zodbot> rishi: rishi 'Debarshi Ray' <debarshir@redhat.com>
16:30:57 * jdoss waves
16:31:07 <kaeso[m]> .hello lucab
16:31:08 <zodbot> kaeso[m]: lucab 'Luca Bruno' <lucab@redhat.com>
16:31:55 <dustymabe> #chair slowrie red_beard rishi kaeso[m] walters jdoss
16:31:55 <zodbot> Current chairs: dustymabe jdoss kaeso[m] red_beard rishi slowrie walters
16:32:10 <strigazi> .hello strigazi
16:32:11 <zodbot> strigazi: strigazi 'Spyros Trigazis' <strigazi@gmail.com>
16:32:20 <mnguyen_> .hello mnguyen
16:32:21 <zodbot> mnguyen_: mnguyen 'Michael Nguyen' <mnguyen@redhat.com>
16:32:30 <ajeddeloh> .hello2
16:32:31 <zodbot> ajeddeloh: ajeddeloh 'Andrew Jeddeloh' <andrew.jeddeloh@redhat.com>
16:32:46 <dustymabe> #chair mnguyen_ strigazi
16:32:46 <zodbot> Current chairs: dustymabe jdoss kaeso[m] mnguyen_ red_beard rishi slowrie strigazi walters
16:32:53 <ksinny> .hello sinnykumari
16:32:54 <zodbot> ksinny: sinnykumari 'Sinny Kumari' <ksinny@gmail.com>
16:33:16 <jlebon> .hello2
16:33:17 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:33:25 <dustymabe> #chair ajeddeloh ksinny jlebon
16:33:25 <zodbot> Current chairs: ajeddeloh dustymabe jdoss jlebon kaeso[m] ksinny mnguyen_ red_beard rishi slowrie strigazi walters
16:34:02 <dustymabe> #topic Action items from last meeting
16:34:10 <dustymabe> * bgilbert to update #159 to add doc item for immutable '/'
16:34:47 <bgilbert> .hello2
16:34:47 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:35:16 <dustymabe> #info bgilbert added ` Immutable root directory (#270)` to the list of items to document in #159
16:35:22 <dustymabe> thanks bgilbert for doing that
16:35:22 <bgilbert> +1
16:36:15 <dustymabe> #topic Ignition: Support multi part mime for user-data
16:36:24 <dustymabe> #link https://github.com/coreos/ignition/issues/849
16:36:44 <dustymabe> we punted on this discussion last week because our good friend ajeddeloh was not around to take part in the discussion
16:36:51 * ajeddeloh waves
16:37:13 <dustymabe> for some background some of our OpenStack Magnum friends (/me waves at strigazi) are evaluating FCOS for use as the base for Openstack Magnum
16:37:22 <dustymabe> they used Fedora Atomic Host in the past
16:37:50 <strigazi> and f29ah atm :)
16:37:51 <dustymabe> I believe the request here is to support mutli-part MIME input for the user-data to an instance
16:38:34 <dustymabe> this would allow for ignition to parse and use some of the user-data and other applications to parse and use other parts of the user-data
16:39:09 <dustymabe> I believe the application in question right now that wants to take advantage of this is openstack heat templates ?
16:39:17 <strigazi> yes
16:39:22 <strigazi> the issue is
16:40:13 <strigazi> heat (the orchestration service like cloud formation), appends some credentials in json format to the user's passed user_data
16:40:24 <ajeddeloh> repeating just so I'm clear exactly what the idea is: the userdata supplied would contain multiple types of userdata, including an ignition config.
16:40:42 <dustymabe> ajeddeloh: that is my understanding - strigazi, is that correct?
16:40:43 <ajeddeloh> ignition would use only the ignition part and ignore the rest
16:40:54 <bgilbert> strigazi: to be completely clear, the user does not supply multipart data, right?  the system does that behind their back.
16:40:55 <strigazi> for our use-case, one igniton config + a json that needs to be written in the fs
16:41:31 <strigazi> bgilbert: unfortunately yes, the user can even pass multipart but the server will add one more part
16:42:01 <bgilbert> strigazi: hmm.  is that use case in scope here?
16:42:32 <ajeddeloh> and each part is standalone? i.e. no splitting an ignition config across parts
16:43:03 <dustymabe> ajeddeloh: yeah I wouldn't think we'd want to support multiple ignition config fragments
16:43:06 <strigazi> if we treat the server (heat) as a user, from ignitions pov it is I guess?
16:43:07 <red_beard> strigazi: i know this has been outlined in the scope of Magnum (hence the concerns around heat).  Would you foresee this having benefits around atoption of FCoS for use with Kolla?
16:43:42 <strigazi> the ignition config would be only one
16:43:52 <bgilbert> strigazi: what I'm asking is: would it be acceptable for us to explicitly reject userdata that contains more than the Ignition config + heat-injected part
16:44:11 <bgilbert> strigazi: the implementation effort isn't the issue.  the issue is the principle that Ignition is the sole source of configuration for the machine
16:44:36 <bgilbert> (imo)
16:44:44 <dustymabe> bgilbert: but does this proposal change that?
16:45:10 <dustymabe> to me this looks like ignitions is still soruce of config for the OS but user-data is being used to provide some info to the app that runs on top
16:45:11 <bgilbert> if users expect to be able to pass N parts, where N-1 parts are input to $OTHER_CONFIGURATION_SYSTEM, yes
16:45:19 <ajeddeloh> There's two parts I think: getting ignition to read multipart to find it's config, and what to do with the second part
16:45:34 <red_beard> bgilbert: to clarify, you mean the idea that 1) FCoS is a component of building distributed systems 2) in a distributed system there should only be a single source of truth for a fact and thus 3) .... (what you just said re: $OTHER_CONFIGURATION_SYSTEM)
16:46:05 <bgilbert> red_beard: that's slightly more general than I was thinking, but yes
16:46:15 <bgilbert> red_beard: the Ignition config should completely describe the customizations to an FCOS system
16:46:20 <ajeddeloh> I think it's reasonable for ignition to know how to read it's userdata out of a multipart thing, but it shouldn't do anything with the other parts
16:46:35 <dustymabe> ajeddeloh: agreed about the "other parts" thing
16:46:55 <bgilbert> strigazi: how is the heat-specific part normally processed now?  does the heat agent read userdata directly?
16:46:57 <ajeddeloh> similar to how ignition reads user data on clouds, but does not read the clouds ssh heys
16:47:14 <strigazi> ajeddeloh: bgilbert let me fetch a small example
16:47:54 <dustymabe> while we wait.. does openstack heat configure the OS? I don't know much about it
16:49:02 <red_beard> dustymabe: it _can_, which makes it worse
16:49:16 <red_beard> because (as with any similar mechanism) if someone can use it, they will use it
16:49:40 <dustymabe> MCO anyone ?
16:49:43 <dustymabe> :)
16:49:59 <dustymabe> we break our own philosophy right there
16:50:22 <bgilbert> yeah, so let's not do it more often :-)
16:50:45 <bgilbert> there are other platform agents that also like to help configure the OS.  which is one of the reasons why, in general, we have a policy against platform agents :-/
16:50:59 <red_beard> +1
16:51:13 <dustymabe> ehh, it's a bit arbitrary to allow it for openshift and not openstack IMO
16:51:23 <dustymabe> strigazi: got that small example?
16:51:35 <bgilbert> dustymabe: users are allowed to run whatever they want in containers
16:51:46 <bgilbert> dustymabe: and talk to their own metadata services, etc
16:51:47 <red_beard> dustymabe: IMHO openshift shouldn't be doing it.  it's an anti-pattern.
16:51:56 <bgilbert> dustymabe: but we don't encourage it
16:51:56 <dustymabe> I don't disagree
16:51:59 <slowrie> red_beard: +1
16:52:27 <strigazi> couldn't find dustymabe couldn't find it, to generate the parts. :( I can post in the issue
16:52:45 <red_beard> but it's also a long way from being my circus or my monkey in the case of RHCOS
16:53:02 <dustymabe> ok so let me see if we can reign this conversation in
16:53:22 <dustymabe> in this case we have a platform that wants to pass some data to an app that is running on top of FCOS via the user-data mechanism
16:53:48 <dustymabe> normally apps could pull their data from a URL somewhere and be fine, but in this case the platform has chosen to re-use user-data
16:53:52 <dustymabe> so we could
16:53:58 <dustymabe> 1. change the platform
16:54:09 <dustymabe> 2. support multi-part mime input and ignore non-ignition input
16:54:41 <dustymabe> does that reflect reality ?
16:54:50 <bgilbert> strigazi: does the heat agent read userdata directly, or is the request for Ignition to copy the other parts out somewhere?
16:54:50 <strigazi> I think yes
16:55:06 <strigazi> bgilbert: the heat-agent does not read user data/
16:55:19 <dustymabe> strigazi: so it wants ignition to write it to a file somewhere ?
16:55:27 <strigazi> dustymabe: yes
16:55:31 <dustymabe> ehh
16:55:39 <dustymabe> so here is where I am
16:55:58 <strigazi> I tried to make ignition write that file.
16:55:59 <dustymabe> I support #2 but not special casing writing out files of other types of MIME data
16:56:00 <bgilbert> strigazi: and it can't just add that to the Ignition config, because it doesn't want to have to understand Ignition + cloud-init + ...
16:56:11 <ajeddeloh> If anything I think that fits more with afterburn, not ignition
16:56:19 <ajeddeloh> dustymabe: ++
16:56:21 <bgilbert> ajeddeloh: oh, hmm
16:56:56 <bgilbert> strigazi: couldn't the heat agent ship something (e.g. another unit) which reads userdata itself?
16:57:01 <bgilbert> *systemd unit
16:57:04 <strigazi> bgilbert: yes, (also developement is bit stale in the project (heat))
16:57:14 <dustymabe> strigazi: you say you tried to make ignition write that file ? what issue did you hit ?
16:57:54 <strigazi> dustymabe: the credentials required by the agent are generated after I do a POST to the server and I can not access them
16:58:20 <dustymabe> so you can't access the contents of the file in order to do the write ? \
16:58:51 <strigazi> so, I pass an ignition json file as user_data, and the server takes that puts it as part together with the required creds
16:58:57 <strigazi> dustymabe: yes
16:59:03 <dustymabe> strigazi: ok
16:59:06 <bgilbert> strigazi: right, the proposal was more that heat itself would append to the Ignition config
16:59:11 <dustymabe> here is a possible solution:
16:59:23 <dustymabe> we implement #2 where ignition completely ignores non ignition bits
16:59:23 <bgilbert> would also be possible for your Ignition config to write a unit which itself curl's userdata
16:59:33 <dustymabe> bgilbert++
16:59:33 <zodbot> dustymabe: Karma for bgilbert changed to 10 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
16:59:38 <dustymabe> that's what I was going to ask
16:59:48 <dustymabe> so we make ignition ignore non ignition MIME data
17:00:05 <dustymabe> and then you can write a unit that will read the parts of the user-data you want and write it to the file you want
17:00:11 <strigazi> would also be possible for your Ignition config to write a unit which itself curl's userdata, that wouldn't work
17:00:21 <bgilbert> why not?
17:00:37 <ajeddeloh> that's also something that could go in the platform base config if it's universally needed (not sure exactly if that would work given the large number of openstack variants)
17:00:58 <strigazi> oh, wait, if ignition reads multipart and takes into account only ignition config, yes, it would work
17:01:20 <kaeso[m]> (altough we can, I wouldn't drag Afterburn into this as the logics seems very specific to the specific agent, so it can be confined there fully)
17:01:21 <dustymabe> so basically we make ignition be more tolerant, but not special case anything
17:01:37 <bgilbert> at a minimum, Ignition should do this only on OpenStack
17:01:44 <ajeddeloh> right
17:01:55 <dustymabe> why?
17:02:02 <bgilbert> to avoid encouraging the practice
17:02:26 <dustymabe> I'd say using user-data to pass info in to apps could be legitimate
17:02:36 <dustymabe> if they don't have any other way to pass information to the app
17:02:47 <dustymabe> but I guess they should embed that in ignition
17:02:48 <bgilbert> it is!  that's what the Ignition files section is for :-)
17:02:49 <bgilbert> yup
17:03:03 <dustymabe> so it's mostly a "platform" problem
17:03:30 <bgilbert> strigazi: maybe this seems strange from your perspective.  the general idea is that, if I have an Ignition config, I can use it to exactly replicate a machine, without any other outside info
17:04:02 <strigazi> bgilbert: it is not, I agree. We just can't get around it
17:04:03 <bgilbert> if a machine is defined by several different things, it's a lot harder to tell what the result will be, or to replicate it
17:04:29 <dustymabe> #proposed to support the openstack heat use case we can make ignition support multi-part mime user-data where ignition will ignore any non ignition parts. users can then parse the other parts and pass that information to openstack heat
17:04:35 <bgilbert> dustymabe: wait one
17:04:39 <dustymabe> oops
17:04:42 <dustymabe> k
17:04:50 <kaeso[m]> there is also the problem of defining a lifecycle for such userdata. If it is only Ignition, we know that changes won't affect a node after the first boot
17:05:02 <strigazi> we can consider that the machine is defiled by igntion only. What happens after is up to the user?
17:05:29 <bgilbert> strigazi: sure.  we encourage users not to change the machine after, but they're free to
17:05:31 <dustymabe> kaeso[m]: good point.. does the openstack heat info change over time?
17:06:23 <strigazi> user_data are immutable in openstack, a change in user_data causes machine replacement
17:06:30 <dustymabe> +1
17:06:33 <ajeddeloh> +1
17:06:38 <bgilbert> and, again, is the issue just the heat-injected credentials?  can we intentionally disallow other user-provided parts?
17:06:51 <strigazi> it would work for us
17:06:56 <bgilbert> (it's always possible to allow more things later.  it's impossible to forbid more things later)
17:07:19 <bgilbert> +1
17:07:38 <dustymabe> should i make any changes to the proposal?
17:07:53 <kaeso[m]> bgilbert: you know that will only drag people to use the heat mime-type for everything else :)
17:08:18 <bgilbert> okay, I'm +1 to allowing multipart for OpenStack only, accept the Ignition config, ignore exactly one instance of the heat-injected part, and fail if we see anything else
17:08:27 <ajeddeloh> +1
17:08:34 <dustymabe> hmm
17:08:48 <dustymabe> so we are special casing specifically the heat-injected part ?
17:08:52 <bgilbert> yes
17:09:08 <bgilbert> dustymabe: the nice thing about being strict is that if we're wrong we get bug reports :-P
17:09:11 <dustymabe> sounds like their platform changes in the future and we have to then change ignition
17:09:50 <dustymabe> I don't typically like hardcoding things for specific platforms in code, but I guess ignition already does that
17:10:14 <dustymabe> honestly if we're going that far then we might as well write the file to the right location
17:10:15 <strigazi> I wouldn't expect changes in heat's side.
17:10:32 <bgilbert> strigazi: is there a well-defined path for the heat part?
17:10:35 <bgilbert> (destination path)
17:10:46 <strigazi> yes, one moment
17:11:01 <dustymabe> this is running long and there is another topic I want to get to
17:11:35 <bgilbert> dustymabe: I'm -0.something to writing the file, but we can discuss that offline if you'd like
17:12:11 <ajeddeloh> Lets discuss the exact details in the ignition ticket
17:12:31 <dustymabe> #proposed to support the openstack heat use case we can make ignition support multi-part mime user-data on the openstack platform only. discussion will continue in the ticket on how targeted to heat the support will be
17:12:53 <dustymabe> ack/nack?
17:12:56 <strigazi> here /var/lib/cloud/data/cfn-init-data
17:13:03 <bgilbert> dustymabe: +1
17:13:05 <ajeddeloh> ack
17:13:35 <dustymabe> #agreed to support the openstack heat use case we can make ignition support multi-part mime user-data on the openstack platform only. discussion will continue in the ticket on how targeted to heat the support will be
17:13:40 <bgilbert> strigazi: thanks for your patience
17:13:49 <dustymabe> #topic coreos/toolbox v2
17:13:55 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/38
17:14:06 <dustymabe> ok we discussed this last week in the meeting
17:14:26 <dustymabe> rishi: has nicely provided us some new rpms to test out to evaluate dependency additions
17:14:48 <dustymabe> walters wanted to further a side discussion about the direction of toolbox
17:14:53 <dustymabe> before moving forward
17:15:06 <dustymabe> walters: can you frame the discussion and give us some background ?
17:15:47 <dustymabe> #info toolbox future direction discussion ticket: https://github.com/debarshiray/toolbox/issues/265
17:16:53 * dustymabe reads that ticket
17:17:22 * rishi is around
17:17:51 <dustymabe> ok so it looks like the discussion is centered around future programming language direction
17:18:02 <kaeso[m]> walters has a good point between the lines there, which is: growing/forking another container runtime is not a good thing
17:18:07 <dustymabe> rishi: you would like to rewrite toolbox from bash into Golang?
17:18:49 <rishi> dustymabe: /usr/bin/toolbox is currently POSIX shell, and POSIX shell is a bit limiting as a programming environment. :)
17:18:51 <dustymabe> and it looks like the link to coretoolbox shows it is written in rust
17:19:15 <rishi> The script has worked surprisingly ok so far - we don't get too many bugs from it, but it does make it hard to hack.
17:19:29 <dustymabe> rishi: +1 - lots of shell code can get unwieldy
17:19:39 <rishi> Yeah, it's more than 2000 lines already.
17:20:04 <dustymabe> rishi: so are you planning to vendor libpod when you rewrite or are you planning to just use Go, but shell out
17:20:14 <dustymabe> the reason I ask is because I think it matters how big toolbox is
17:20:33 <dustymabe> podman is already large on our system and if toolbox grows large it's not as nice IMHO
17:20:41 <rishi> dustymabe: Shelling out from Go, or vendoring in libpod into a Go frontend - both are possible.
17:20:59 <dustymabe> rishi: what do you anticipate will be the size difference between those two options?
17:21:12 <rishi> I, personally, want to keep the door open to re-evaluate and switch to a different approach if one doesn't work out for some reason.
17:21:40 <bgilbert> CL toolbox is 81 lines.  /me wonders what happened
17:21:46 <rishi> But I am a bit biased in favour of Go, simply because the entire OCI ecosystem is Go.
17:22:21 <dustymabe> rishi: regarding go vs rust - I'd say you (and other contributors) are the authors of the code
17:22:22 <rishi> The Toolbox front-end code isn't that big, so asking contributors to learn both Rust and Go (because you will end up poking at Podman sooner or later) seems like a needless hurdle.
17:22:36 <dustymabe> so it's up to you
17:22:54 <dustymabe> unless you think that merging with colin's POC would give you some new contributors
17:22:56 <jlebon> +1 to just shelling out. ISTM like toolbox should just be a thin layer on top of existing tools
17:22:59 <rishi> It seems wise to stay aligned with Podman and friends.
17:23:16 <rishi> dustymabe: Regarding size ...
17:23:34 <rishi> Maybe mheon can chime in. I see that /usr/bin/podman is 51M (!) today.
17:23:48 <mheon> we've looked into reducing that, but Go does not make it easy
17:24:09 <rishi> No idea how much of that we could cut out - the Toolbox frontend would be simpler than Podman in some sense.
17:24:14 <mheon> static linking plus an ecosystem of heavily interdependent libraries are not friendly to binary size
17:24:22 <dustymabe> :)
17:24:35 <dustymabe> ok we're running short on time. I think I have a proposal
17:24:43 <bgilbert> I haven't been following toolbox work.  anyone have a little bit of background on why this is an issue?
17:24:47 <bgilbert> again, CL toolbox is tiny
17:24:58 <bgilbert> is it growing additional features?
17:25:33 <rishi> bgilbert: There's a desire to unify the SB and COS toolboxes.
17:25:54 <rishi> The SB toolbox has a few more things, since it's mostly a development tool.
17:26:25 <dustymabe> #proposed regarding golang vs rust that is up to the author of toolbox. we do care about size, however, and thus if Golang is chosen we prefer that we don't vendor libpod, but rather either shell out or use the varlink API to manage containers.
17:26:53 <rishi> jlebon: There are some drawbacks to shelling out though.
17:27:37 <rishi> Handling errors becomes tricky. Can't do fancy progress bars and what not. Plus the fact that Podman can change underneath.
17:27:49 <kaeso[m]> dustymabe: "about size and exploding the number of container-runtime implementations"
17:28:09 <rishi> The idea of being in charge of the full stack and QE:ing it in one shot, and being able to cherry-pick a libpod fix and release quick update, seems tempting.
17:28:35 <mheon> kaeso[m]: as a libpod developer, i'm not terribly concerned by that as long as they use libpod - Podman is mostly intended to be a very thin layer over the libpod library
17:28:53 <rishi> But then that comes at the cost of a massive binary. If the binary size is a real deal breaker for CoreOS, then I guess we'd have to ditch vendoring.
17:28:59 <mheon> kaeso[m]: changes to the frontend aren't particularly significant in my eyes, so long as the backend code remains the same
17:29:21 <dustymabe> rishi: I'd really prefer podman not vendor libpod - shared maintenance of podman would be better IMO
17:30:06 <dustymabe> it's better if you help maintain podman than maintain things yourself.. a lot less work
17:30:18 <dustymabe> WDYT?
17:30:24 <rishi> dustymabe: Well, the libpod/cmd/podman doesn't seem to be a lot. The real issue is the binary size, which I totally understand.
17:30:33 <dustymabe> +1
17:30:43 <dustymabe> ok we're over time
17:30:47 <rishi> *the libpod/cmd/podman code
17:30:51 <dustymabe> kaeso[m]: you had a proposed change to my proposal?
17:31:14 <walters> I’m AFK at the moment, will read scrollback  a bit later
17:31:43 <kaeso[m]> dustymabe: yes, just a small addition about duplicated/vendored logic
17:32:15 <dustymabe> #proposed regarding golang vs rust that is up to the author of toolbox. we do care about size and exploding the number of container-runtime implementations. If Golang is chosen we prefer that we don't vendor libpod, but rather either shell out or use the varlink API to manage containers.
17:32:31 <dustymabe> better?
17:32:40 <kaeso[m]> SGTM
17:32:46 <dustymabe> ack/nack?
17:33:20 <ajeddeloh> ack
17:33:31 <dustymabe> no nacksss..
17:33:36 <jlebon> +1
17:33:38 <dustymabe> #agreed regarding golang vs rust that is up to the author of toolbox. we do care about size and exploding the number of container-runtime implementations. If Golang is chosen we prefer that we don't vendor libpod, but rather either shell out or use the varlink API to manage containers.
17:33:47 <dustymabe> #topic open floor
17:33:57 <dustymabe> anything important for open floor?
17:34:00 <jdoss> https://github.com/coreos/mantle/pull/1046 merge me!
17:34:05 <dustymabe> i'll close out in 30 seconds if not
17:34:14 <dustymabe> jdoss: :)
17:34:16 <dustymabe> that's on me
17:34:24 <dustymabe> will look at it soon I promise this time
17:34:27 <jdoss> :) Thanks!
17:34:28 <strigazi> thanks bgilbert dustymabe ajeddeloh everyone
17:34:42 <dustymabe> #endmeeting