16:30:01 <bgilbert> #startmeeting fedora_coreos_meeting
16:30:01 <zodbot> Meeting started Wed Apr  3 16:30:01 2019 UTC.
16:30:01 <zodbot> This meeting is logged and archived in a public location.
16:30:01 <zodbot> The chair is bgilbert. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:30:01 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:30:01 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:30:03 <bgilbert> #topic roll call
16:30:10 <slowrie> .hello2
16:30:11 <zodbot> slowrie: slowrie 'Stephen Lowrie' <slowrie@redhat.com>
16:30:15 <bgilbert> .hello2
16:30:18 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:30:21 <jlebon> .hello2
16:30:21 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:30:41 <geoff-> geoff-: Geoff Levand <geoff@infradead.org>
16:30:42 <dustymabe> .hello2
16:30:43 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:30:54 <ajeddeloh> .hello2
16:30:55 <zodbot> ajeddeloh: ajeddeloh 'Andrew Jeddeloh' <andrew.jeddeloh@redhat.com>
16:31:00 <jlebon> one day, you will get my name right, zodbot! either that, or i'll just change my legal name
16:31:06 <mnguyen_> .hello mnguyen
16:31:07 <zodbot> mnguyen_: mnguyen 'Michael Nguyen' <mnguyen@redhat.com>
16:31:09 <jlebon> dustymabe: howdy! :)
16:31:11 * slowrie waves at dustymabe
16:31:12 <ksinny> .hello sinnykumari
16:31:13 <zodbot> ksinny: sinnykumari 'Sinny Kumari' <ksinny@gmail.com>
16:31:21 <jbrooks> .fas jasonbrooks
16:31:22 <zodbot> jbrooks: jasonbrooks 'Jason Brooks' <jbrooks@redhat.com>
16:31:29 <mnguyen_> dusty!
16:31:41 <smooge> dusty!!
16:31:47 <ksinny> dusty!!!
16:31:53 <red_beard> dusty!!!!
16:31:54 <ajeddeloh> dusty!!!!
16:32:00 <bgilbert> dusty!!!!!!!!!!!11
16:32:03 <bgilbert> :-P
16:32:04 <dustymabe> how many can we go!!
16:32:13 <bexelbie> dusty^dusty
16:32:19 * dustymabe waves at the beautiful faces in the community
16:32:21 * bexelbie had to join the party
16:32:42 <bgilbert> #chair slowrie jlebon geoff- dustymabe ajeddeloh mnguyen_ ksinny jbrooks
16:32:42 <zodbot> Current chairs: ajeddeloh bgilbert dustymabe geoff- jbrooks jlebon ksinny mnguyen_ slowrie
16:32:43 * smooge gives him the 'aren't you supposed to be doing something else?' look
16:33:07 <dustymabe> don't worry my real baby is close by - has the hiccups right now
16:33:14 <kaeso> .hello lucab
16:33:15 <zodbot> kaeso: lucab 'Luca Bruno' <lucab@redhat.com>
16:34:00 <bgilbert> #chair kaeso
16:34:00 <zodbot> Current chairs: ajeddeloh bgilbert dustymabe geoff- jbrooks jlebon kaeso ksinny mnguyen_ slowrie
16:34:01 <jbrooks> dustymabe, awesome, congrats
16:34:16 <bgilbert> #topic Action items from last meeting
16:34:22 <bgilbert> none!
16:34:31 <bgilbert> #topic reboot coordination: locksmith successor
16:34:39 <bgilbert> #link https://github.com/coreos/fedora-coreos-tracker/issues/3
16:34:46 <rfairley> .hello2
16:34:47 <zodbot> rfairley: rfairley 'None' <rfairley@redhat.com>
16:35:01 <bgilbert> #chair rfairley
16:35:01 <zodbot> Current chairs: ajeddeloh bgilbert dustymabe geoff- jbrooks jlebon kaeso ksinny mnguyen_ rfairley slowrie
16:35:06 <kaeso> juicy details at https://github.com/coreos/fedora-coreos-tracker/issues/3#issuecomment-465953659
16:35:54 <kaeso> we used devconf.cz time for some further brainstorming
16:36:26 <kaeso> we ended up in a model where rpm-ostreed take care of deploying updates
16:36:54 <kaeso> some on-host client takes care of bridging update hints cincinnati<->rpm-ostreed
16:37:38 <red_beard> so the on host client can act in situations where you need a reboot but not attached to an update?
16:37:47 <kaeso> and some containerized component optionally takes care of keeping cluster-wide reboot locks
16:38:34 <kaeso> red_beard: uh? no, this is only scoped for updates
16:39:11 <red_beard> ok.  just clarifying since the github issue nor the topic clarify that
16:39:22 <kaeso> ack
16:39:50 <kaeso> in short, that means taking parts of https://github.com/coreos/locksmith
16:40:16 <kaeso> moving them to a container
16:40:38 <kaeso> and having a protocol to perform locking/unlocking
16:41:28 <kaeso> I started sketching that at https://github.com/lucab/exp-locksmith2
16:41:29 <dustymabe> kaeso: would we include the container in Fedora CoreOS ?
16:41:48 <dustymabe> or have that be an add on with docs ?
16:42:06 <kaeso> dustymabe: likely not, depends on the default on-host client reboot strategy
16:42:38 <kaeso> CL one currently is "single node, immediate reboot if nobody is logged in"
16:42:49 <kaeso> I think it makes sense sticking to that default
16:43:20 <ajeddeloh> +1 don't see a strong reason to change it
16:43:22 <jlebon> right, the container is only needed for cluster-aware updates, so i guess it'd be part of the cluster install procedure?
16:43:35 <kaeso> this container covers the case "I already have an etcd3 cluster"
16:43:59 <dustymabe> kk
16:44:21 <bgilbert> for ease of conversion, we might tell migrating CL users to run an instance of the container on every node
16:44:37 <bgilbert> but I don't think that'd be the recommended model, and we still wouldn't ship the container
16:44:46 <kaeso> it can run in a local podman/systemd-nspawn, yup
16:45:07 <kaeso> but the whole setup needs configuration for the specific cluster anyway
16:45:35 <kaeso> on this "containerized locksmith" side, things that I think are next on the roadmap
16:45:57 <kaeso> * picking a name, and moving to the coreos orga
16:46:23 <lorbus> .hello2
16:46:24 <zodbot> lorbus: lorbus 'Christian Glombek' <cglombek@redhat.com>
16:46:30 <kaeso> * review the locking/unlocking protocol
16:46:37 * lorbus messed up the time change
16:47:05 <kaeso> and from there on is mostly code-review and incremental changes
16:47:30 <bgilbert> #chair lorbus
16:47:30 <zodbot> Current chairs: ajeddeloh bgilbert dustymabe geoff- jbrooks jlebon kaeso ksinny lorbus mnguyen_ rfairley slowrie
16:47:48 <kaeso> I only sketched the lock-manager-server part, it likely needs some CLI too
16:48:58 <kaeso> also, I was not very familiar with original locksmith nor with etcd3 go-client, so at least a pass for code feedback is needed
16:49:13 <kaeso> and that's it I guess
16:49:32 <bgilbert> thanks for the update, kaeso!
16:49:42 <bgilbert> any other comments before moving on?
16:49:46 <kaeso> the consuming side of this is at https://github.com/coreos/fedora-coreos-tracker/issues/83#issuecomment-467016083
16:50:02 <kaeso> but I don't have any meaningful summary for that yet
16:50:03 <red_beard> .hello2
16:50:07 <zodbot> red_beard: Sorry, but you don't exist
16:50:22 <bgilbert> red_beard: .hello <fas_id> also works
16:50:30 <red_beard> .hello redbeard
16:50:31 <zodbot> red_beard: redbeard 'Brian 'redbeard' Harrington' <bharring@redhat.com>
16:50:35 <bgilbert> #chair red_beard
16:50:35 <zodbot> Current chairs: ajeddeloh bgilbert dustymabe geoff- jbrooks jlebon kaeso ksinny lorbus mnguyen_ red_beard rfairley slowrie
16:51:03 <bgilbert> that's all the meeting tickets; please add additional topics here: https://public.etherpad-mozilla.org/p/20190403-FCOS-Meeting
16:51:13 <kaeso> red_beard: was your initial question somehow related to "config change then reboot" (without upgrades)?
16:52:26 <red_beard> kaeso: it was more about general maintenance tasks and troubleshooting.  things like needing to do a reboot to test a DKMS build (as a single example)
16:52:45 <red_beard> (honestly i'd prefer to never be in the state of doing that with a cluster, but i can foresee needing it)
16:53:12 <kaeso> red_beard: I see
16:54:00 <bgilbert> #topic Inline file contents in Ignition spec 3: in or out
16:54:25 <kaeso> I think we are not taking any kind of cluster-wide change propagation at this point
16:55:06 <ajeddeloh> So some folks (mostly from the OpenShift side) have been asking for Ignition to support inlining file contents without data urls
16:55:25 <ajeddeloh> this is more convienent, but does encourage hand written configs
16:55:48 <red_beard> to avoid having to do the encoding of contents to a data URI?
16:55:52 <kaeso> does that means base64 without the `data:` prefix?
16:55:59 <ajeddeloh> red_beard: yes
16:56:19 <ajeddeloh> kaeso: I think we'd want it as a seperate field, and not base64'd
16:56:28 <ajeddeloh> The use case is textual configs
16:57:00 <bgilbert> it doesn't enable any new functionality over and above data URIs, correct?
16:57:13 <ajeddeloh> one minor advantage is it would make FCOS CT's format purely Ignition spec + sugar, rather than mangled-ignition-spec + sugar
16:57:20 <ajeddeloh> bgilbert: correct
16:57:37 <ajeddeloh> its something we could add in spec 3.1 or 4.0 if we wanted
16:57:38 <kaeso> ajeddeloh: so it's a JSON unicode string?
16:57:39 <bgilbert> we've generally tried to keep the Ignition spec orthogonal
16:57:40 <red_beard> i think that (while convenient) it invites a lot of potential for human error and ambiguity in parsing the information.
16:57:52 <ajeddeloh> kaeso: yes
16:58:07 <bgilbert> and the OpenShift configs in question are machine-generated anyway, yes?
16:58:08 <slowrie> red_beard: +1
16:58:14 <ajeddeloh> bgilbert: yes
16:58:44 <bgilbert> then... I don't see it
16:58:50 <ajeddeloh> sounds like the consensus is to not. Does anyone have a strong reason to include it
16:58:56 <kaeso> I'm not use about how much better ergonomics would be
16:59:24 <kaeso> for binaries, it still needs encoding
16:59:41 <jlebon> bgilbert: not all ignition configs are machine-generated though, right? some OCP4 users will be writing ignition by hand
16:59:52 <kaeso> and for textual, it still needs JSON escaping
17:00:07 <bgilbert> jlebon: they really really should be.
17:00:14 <bgilbert> people do write configs by hand
17:00:19 <jlebon> bgilbert: the reality is that they won't all be :(
17:00:33 <bgilbert> but I'd view that as 100% a failure to provide better tools so they won't have to :-)
17:00:50 <bgilbert> so it's a short-term problem, is what I'm getting at
17:00:57 <jlebon> gotcha
17:01:25 <kaeso> ajeddeloh: I think the reason for b64 data: is exactly to avoid encoding/escaping gotchas
17:02:16 <ajeddeloh> +1 sounds like it's out
17:02:29 <ajeddeloh> misc thought on CT
17:02:47 <ajeddeloh> we should build it such that we have a base that is purely the mangled ignition spec
17:02:57 <ajeddeloh> then pluggable sugar on top of that
17:03:19 <kaeso> ajeddeloh: mind detailing a bit more your "mangled" concern?
17:03:21 <ajeddeloh> like a distro independent base and the distro "modules"
17:03:27 <ajeddeloh> sure
17:03:29 <jlebon> i.e. different keys that aren't defined in the top-level of the spec?
17:03:32 <slowrie> kaeso: likely inline files, local files, etc
17:03:42 <ajeddeloh> slowrie: yup
17:03:51 <ajeddeloh> the clc spec follows the ignition spec
17:04:02 <ajeddeloh> except changes bits like adding "inline" and "local"
17:04:27 <ajeddeloh> when translating we need to do a dance to convert that bit vs relying on the automatic bits
17:04:41 <kaeso> I see
17:04:45 <ajeddeloh> especially for handling plumbing line numbers
17:05:58 <bgilbert> anything else on this topic?
17:06:30 <bgilbert> #topic Artifact storage for FCOS pipeline
17:06:38 <kaeso> it's tiring from the dev point-of-view, but I think it's the right choice to keep the two non-aligned and have bridging logic
17:06:58 <kaeso> (done, move on)
17:07:26 <jlebon> right now because of #reasons, we're limited to how many builds we can store in the CentOS CI artifact server. thus we employ aggressive pruning
17:07:41 <jlebon> also, downloading from the artifact server is very slow
17:08:14 <jlebon> we should push out our builds to a bucket in $cloud so we can have faster speeds and start building history
17:08:57 <jlebon> which will be useful for testing lots of things like the promotion process and the cincinnati client/server
17:09:49 <slowrie> ideally it'd use the same / an api compatible service as whatever the production buckets would end up being so that any tools we create/update to work on developer setups could be applied to prod workloads
17:10:27 <bgilbert> slowrie: +1
17:10:28 <jlebon> yeah, agreed
17:10:34 <bgilbert> though presumably that'll evolve a bit
17:11:01 <dustymabe> i'm not opposed to this suggestion - would like for the progress we make on it to be useful when we start to do actual releases too
17:11:24 <dustymabe> IOW - can we collaborate with releng ?
17:11:25 <jlebon> we should ask fedora folks about what our options are if we can reuse the same cloud storage
17:12:01 <kaeso> dustymabe: I think the topic came up because we have a bootstrapping cycle to break for auto-updates, before making real releases
17:12:12 <slowrie> Do we have an understanding of what service will be used to store production bits?
17:12:35 <slowrie> i.e.: if it'll ultimately be gcs or s3 we can set it up initially to just store in that service and evolve the bucket layout structure to match later
17:13:07 <kaeso> (that is, released OS <-> cincinnati service <-> cincinnati client in released OS)
17:13:15 <dustymabe> slowrie: we *should* be able to use s3 I believe
17:13:59 <jlebon> ok, let's reword this topic as: "figure out storage strategy for FCOS Preview, get access to said storage, redirect CCI pipeline to it right away so we have something to play with and iterate on"
17:14:02 <dustymabe> it shouldn't be too hard to confirm that
17:14:10 <dustymabe> jlebon: +1
17:14:39 <bgilbert> jlebon: want to file a tracker issue?
17:14:45 <kaeso> it's also a good exercise to iron out details, right now I think we are missing the basearch label somewhere in the artifacts URL
17:14:57 <jlebon> bgilbert: sure thing, will do
17:15:17 <bgilbert> #action jlebon to file tracker issue: figure out storage strategy for FCOS Preview, get access to said storage, redirect CCI pipeline to it
17:15:21 <ksinny> It will be nice to have a ticket for this issue and we can add releng/fedora-infra folks to see what is feasible
17:15:28 <bgilbert> ksinny: +1
17:15:30 <ksinny> ah, already actioned
17:16:28 <bgilbert> cool, anything else here?
17:16:56 <jlebon> kaeso: hmm, that's a good point. this speaks to a higher level question re. coreos-assembler and multi-arch if we want to be able to share "build trees"
17:17:18 <jlebon> could you file an issue against coreos-assembler maybe?
17:17:24 * jlebon checks if there isn't already one
17:17:42 <kaeso> jlebon: I'm not sure I got the last part of that
17:17:50 <dustymabe> jlebon: i'd like to 'share build trees' if possible
17:18:14 <kaeso> jlebon: are you concerned with a single repo for arm/x86/etc
17:18:26 <kaeso> versus multiple dedicated ones
17:18:31 <kaeso> at ostree level?
17:18:42 <jlebon> kaeso: basically whether to make the `builds/$buildid` dir hold artifacts for all archs instead of completely split ones
17:19:01 <jlebon> i don't see an issue yet for it in a quick scan, so it'd be really good to bring
17:19:05 <jlebon> up
17:19:29 <kaeso> ah ok, then I understood something else
17:19:38 <kaeso> I'll open that
17:19:52 <jlebon> at the ostree level, if we want to share with the main fedora ostree repo, we kinda have to host it all in there
17:21:33 <bgilbert> #topic Open Floor
17:21:50 <kaeso> #action kaeso open a ticket on coreos-assembler to sort out remote bucket layout for multi-arch artifacts
17:24:05 <bgilbert> seems like we already covered the things
17:24:09 <kaeso> my little open floor bit
17:24:41 <kaeso> should we start naming and sketching the format for the next "ct"?
17:25:04 <kaeso> I realised this morning we may need that for general OS docs too
17:25:20 <ajeddeloh> For the core ct format I'd like to follow ignition's spec
17:25:22 <slowrie> kaeso: my assumption was that after we finished up with the initial ignition spec 3.0.0 work that was the next major work choice
17:25:26 <ajeddeloh> at least roughly
17:26:13 <ajeddeloh> like we did with CLCs
17:26:57 <kaeso> I think in the immediate term I'm just concerned with naming and letting the ball roll, more than the actual spec
17:27:24 <slowrie> I'm still partial to FCC :)
17:27:30 <bgilbert> was about to say
17:27:35 <bgilbert> FCC seems nice
17:28:09 <bgilbert> or a town in Massachusetts
17:28:35 * ajeddeloh wonders what's so special about MA
17:28:53 <kaeso> do we already have a name bikeshedding ticket?
17:28:57 <bgilbert> convenient to dracut maintainers, presumably
17:29:05 <bgilbert> kaeso: not AFAIK
17:29:15 <bgilbert> want to file?  :-P
17:29:22 <slowrie> #129 is `CT for Fedora CoreOS`
17:29:31 <slowrie> and has some naming suggestions in it
17:29:32 <bgilbert> sure
17:30:24 <kaeso> yay, let's keep that and try to pick a name till next week
17:31:11 <bgilbert> cool.  we're at time; I'll close the meeting in 30s or so
17:31:58 <bgilbert> #endmeeting