16:30:01 <bgilbert> #startmeeting fedora_coreos_meeting 16:30:01 <zodbot> Meeting started Wed Apr 3 16:30:01 2019 UTC. 16:30:01 <zodbot> This meeting is logged and archived in a public location. 16:30:01 <zodbot> The chair is bgilbert. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:30:01 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:30:01 <zodbot> The meeting name has been set to 'fedora_coreos_meeting' 16:30:03 <bgilbert> #topic roll call 16:30:10 <slowrie> .hello2 16:30:11 <zodbot> slowrie: slowrie 'Stephen Lowrie' <slowrie@redhat.com> 16:30:15 <bgilbert> .hello2 16:30:18 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net> 16:30:21 <jlebon> .hello2 16:30:21 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com> 16:30:41 <geoff-> geoff-: Geoff Levand <geoff@infradead.org> 16:30:42 <dustymabe> .hello2 16:30:43 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com> 16:30:54 <ajeddeloh> .hello2 16:30:55 <zodbot> ajeddeloh: ajeddeloh 'Andrew Jeddeloh' <andrew.jeddeloh@redhat.com> 16:31:00 <jlebon> one day, you will get my name right, zodbot! either that, or i'll just change my legal name 16:31:06 <mnguyen_> .hello mnguyen 16:31:07 <zodbot> mnguyen_: mnguyen 'Michael Nguyen' <mnguyen@redhat.com> 16:31:09 <jlebon> dustymabe: howdy! :) 16:31:11 * slowrie waves at dustymabe 16:31:12 <ksinny> .hello sinnykumari 16:31:13 <zodbot> ksinny: sinnykumari 'Sinny Kumari' <ksinny@gmail.com> 16:31:21 <jbrooks> .fas jasonbrooks 16:31:22 <zodbot> jbrooks: jasonbrooks 'Jason Brooks' <jbrooks@redhat.com> 16:31:29 <mnguyen_> dusty! 16:31:41 <smooge> dusty!! 16:31:47 <ksinny> dusty!!! 16:31:53 <red_beard> dusty!!!! 16:31:54 <ajeddeloh> dusty!!!! 16:32:00 <bgilbert> dusty!!!!!!!!!!!11 16:32:03 <bgilbert> :-P 16:32:04 <dustymabe> how many can we go!! 16:32:13 <bexelbie> dusty^dusty 16:32:19 * dustymabe waves at the beautiful faces in the community 16:32:21 * bexelbie had to join the party 16:32:42 <bgilbert> #chair slowrie jlebon geoff- dustymabe ajeddeloh mnguyen_ ksinny jbrooks 16:32:42 <zodbot> Current chairs: ajeddeloh bgilbert dustymabe geoff- jbrooks jlebon ksinny mnguyen_ slowrie 16:32:43 * smooge gives him the 'aren't you supposed to be doing something else?' look 16:33:07 <dustymabe> don't worry my real baby is close by - has the hiccups right now 16:33:14 <kaeso> .hello lucab 16:33:15 <zodbot> kaeso: lucab 'Luca Bruno' <lucab@redhat.com> 16:34:00 <bgilbert> #chair kaeso 16:34:00 <zodbot> Current chairs: ajeddeloh bgilbert dustymabe geoff- jbrooks jlebon kaeso ksinny mnguyen_ slowrie 16:34:01 <jbrooks> dustymabe, awesome, congrats 16:34:16 <bgilbert> #topic Action items from last meeting 16:34:22 <bgilbert> none! 16:34:31 <bgilbert> #topic reboot coordination: locksmith successor 16:34:39 <bgilbert> #link https://github.com/coreos/fedora-coreos-tracker/issues/3 16:34:46 <rfairley> .hello2 16:34:47 <zodbot> rfairley: rfairley 'None' <rfairley@redhat.com> 16:35:01 <bgilbert> #chair rfairley 16:35:01 <zodbot> Current chairs: ajeddeloh bgilbert dustymabe geoff- jbrooks jlebon kaeso ksinny mnguyen_ rfairley slowrie 16:35:06 <kaeso> juicy details at https://github.com/coreos/fedora-coreos-tracker/issues/3#issuecomment-465953659 16:35:54 <kaeso> we used devconf.cz time for some further brainstorming 16:36:26 <kaeso> we ended up in a model where rpm-ostreed take care of deploying updates 16:36:54 <kaeso> some on-host client takes care of bridging update hints cincinnati<->rpm-ostreed 16:37:38 <red_beard> so the on host client can act in situations where you need a reboot but not attached to an update? 16:37:47 <kaeso> and some containerized component optionally takes care of keeping cluster-wide reboot locks 16:38:34 <kaeso> red_beard: uh? no, this is only scoped for updates 16:39:11 <red_beard> ok. just clarifying since the github issue nor the topic clarify that 16:39:22 <kaeso> ack 16:39:50 <kaeso> in short, that means taking parts of https://github.com/coreos/locksmith 16:40:16 <kaeso> moving them to a container 16:40:38 <kaeso> and having a protocol to perform locking/unlocking 16:41:28 <kaeso> I started sketching that at https://github.com/lucab/exp-locksmith2 16:41:29 <dustymabe> kaeso: would we include the container in Fedora CoreOS ? 16:41:48 <dustymabe> or have that be an add on with docs ? 16:42:06 <kaeso> dustymabe: likely not, depends on the default on-host client reboot strategy 16:42:38 <kaeso> CL one currently is "single node, immediate reboot if nobody is logged in" 16:42:49 <kaeso> I think it makes sense sticking to that default 16:43:20 <ajeddeloh> +1 don't see a strong reason to change it 16:43:22 <jlebon> right, the container is only needed for cluster-aware updates, so i guess it'd be part of the cluster install procedure? 16:43:35 <kaeso> this container covers the case "I already have an etcd3 cluster" 16:43:59 <dustymabe> kk 16:44:21 <bgilbert> for ease of conversion, we might tell migrating CL users to run an instance of the container on every node 16:44:37 <bgilbert> but I don't think that'd be the recommended model, and we still wouldn't ship the container 16:44:46 <kaeso> it can run in a local podman/systemd-nspawn, yup 16:45:07 <kaeso> but the whole setup needs configuration for the specific cluster anyway 16:45:35 <kaeso> on this "containerized locksmith" side, things that I think are next on the roadmap 16:45:57 <kaeso> * picking a name, and moving to the coreos orga 16:46:23 <lorbus> .hello2 16:46:24 <zodbot> lorbus: lorbus 'Christian Glombek' <cglombek@redhat.com> 16:46:30 <kaeso> * review the locking/unlocking protocol 16:46:37 * lorbus messed up the time change 16:47:05 <kaeso> and from there on is mostly code-review and incremental changes 16:47:30 <bgilbert> #chair lorbus 16:47:30 <zodbot> Current chairs: ajeddeloh bgilbert dustymabe geoff- jbrooks jlebon kaeso ksinny lorbus mnguyen_ rfairley slowrie 16:47:48 <kaeso> I only sketched the lock-manager-server part, it likely needs some CLI too 16:48:58 <kaeso> also, I was not very familiar with original locksmith nor with etcd3 go-client, so at least a pass for code feedback is needed 16:49:13 <kaeso> and that's it I guess 16:49:32 <bgilbert> thanks for the update, kaeso! 16:49:42 <bgilbert> any other comments before moving on? 16:49:46 <kaeso> the consuming side of this is at https://github.com/coreos/fedora-coreos-tracker/issues/83#issuecomment-467016083 16:50:02 <kaeso> but I don't have any meaningful summary for that yet 16:50:03 <red_beard> .hello2 16:50:07 <zodbot> red_beard: Sorry, but you don't exist 16:50:22 <bgilbert> red_beard: .hello <fas_id> also works 16:50:30 <red_beard> .hello redbeard 16:50:31 <zodbot> red_beard: redbeard 'Brian 'redbeard' Harrington' <bharring@redhat.com> 16:50:35 <bgilbert> #chair red_beard 16:50:35 <zodbot> Current chairs: ajeddeloh bgilbert dustymabe geoff- jbrooks jlebon kaeso ksinny lorbus mnguyen_ red_beard rfairley slowrie 16:51:03 <bgilbert> that's all the meeting tickets; please add additional topics here: https://public.etherpad-mozilla.org/p/20190403-FCOS-Meeting 16:51:13 <kaeso> red_beard: was your initial question somehow related to "config change then reboot" (without upgrades)? 16:52:26 <red_beard> kaeso: it was more about general maintenance tasks and troubleshooting. things like needing to do a reboot to test a DKMS build (as a single example) 16:52:45 <red_beard> (honestly i'd prefer to never be in the state of doing that with a cluster, but i can foresee needing it) 16:53:12 <kaeso> red_beard: I see 16:54:00 <bgilbert> #topic Inline file contents in Ignition spec 3: in or out 16:54:25 <kaeso> I think we are not taking any kind of cluster-wide change propagation at this point 16:55:06 <ajeddeloh> So some folks (mostly from the OpenShift side) have been asking for Ignition to support inlining file contents without data urls 16:55:25 <ajeddeloh> this is more convienent, but does encourage hand written configs 16:55:48 <red_beard> to avoid having to do the encoding of contents to a data URI? 16:55:52 <kaeso> does that means base64 without the `data:` prefix? 16:55:59 <ajeddeloh> red_beard: yes 16:56:19 <ajeddeloh> kaeso: I think we'd want it as a seperate field, and not base64'd 16:56:28 <ajeddeloh> The use case is textual configs 16:57:00 <bgilbert> it doesn't enable any new functionality over and above data URIs, correct? 16:57:13 <ajeddeloh> one minor advantage is it would make FCOS CT's format purely Ignition spec + sugar, rather than mangled-ignition-spec + sugar 16:57:20 <ajeddeloh> bgilbert: correct 16:57:37 <ajeddeloh> its something we could add in spec 3.1 or 4.0 if we wanted 16:57:38 <kaeso> ajeddeloh: so it's a JSON unicode string? 16:57:39 <bgilbert> we've generally tried to keep the Ignition spec orthogonal 16:57:40 <red_beard> i think that (while convenient) it invites a lot of potential for human error and ambiguity in parsing the information. 16:57:52 <ajeddeloh> kaeso: yes 16:58:07 <bgilbert> and the OpenShift configs in question are machine-generated anyway, yes? 16:58:08 <slowrie> red_beard: +1 16:58:14 <ajeddeloh> bgilbert: yes 16:58:44 <bgilbert> then... I don't see it 16:58:50 <ajeddeloh> sounds like the consensus is to not. Does anyone have a strong reason to include it 16:58:56 <kaeso> I'm not use about how much better ergonomics would be 16:59:24 <kaeso> for binaries, it still needs encoding 16:59:41 <jlebon> bgilbert: not all ignition configs are machine-generated though, right? some OCP4 users will be writing ignition by hand 16:59:52 <kaeso> and for textual, it still needs JSON escaping 17:00:07 <bgilbert> jlebon: they really really should be. 17:00:14 <bgilbert> people do write configs by hand 17:00:19 <jlebon> bgilbert: the reality is that they won't all be :( 17:00:33 <bgilbert> but I'd view that as 100% a failure to provide better tools so they won't have to :-) 17:00:50 <bgilbert> so it's a short-term problem, is what I'm getting at 17:00:57 <jlebon> gotcha 17:01:25 <kaeso> ajeddeloh: I think the reason for b64 data: is exactly to avoid encoding/escaping gotchas 17:02:16 <ajeddeloh> +1 sounds like it's out 17:02:29 <ajeddeloh> misc thought on CT 17:02:47 <ajeddeloh> we should build it such that we have a base that is purely the mangled ignition spec 17:02:57 <ajeddeloh> then pluggable sugar on top of that 17:03:19 <kaeso> ajeddeloh: mind detailing a bit more your "mangled" concern? 17:03:21 <ajeddeloh> like a distro independent base and the distro "modules" 17:03:27 <ajeddeloh> sure 17:03:29 <jlebon> i.e. different keys that aren't defined in the top-level of the spec? 17:03:32 <slowrie> kaeso: likely inline files, local files, etc 17:03:42 <ajeddeloh> slowrie: yup 17:03:51 <ajeddeloh> the clc spec follows the ignition spec 17:04:02 <ajeddeloh> except changes bits like adding "inline" and "local" 17:04:27 <ajeddeloh> when translating we need to do a dance to convert that bit vs relying on the automatic bits 17:04:41 <kaeso> I see 17:04:45 <ajeddeloh> especially for handling plumbing line numbers 17:05:58 <bgilbert> anything else on this topic? 17:06:30 <bgilbert> #topic Artifact storage for FCOS pipeline 17:06:38 <kaeso> it's tiring from the dev point-of-view, but I think it's the right choice to keep the two non-aligned and have bridging logic 17:06:58 <kaeso> (done, move on) 17:07:26 <jlebon> right now because of #reasons, we're limited to how many builds we can store in the CentOS CI artifact server. thus we employ aggressive pruning 17:07:41 <jlebon> also, downloading from the artifact server is very slow 17:08:14 <jlebon> we should push out our builds to a bucket in $cloud so we can have faster speeds and start building history 17:08:57 <jlebon> which will be useful for testing lots of things like the promotion process and the cincinnati client/server 17:09:49 <slowrie> ideally it'd use the same / an api compatible service as whatever the production buckets would end up being so that any tools we create/update to work on developer setups could be applied to prod workloads 17:10:27 <bgilbert> slowrie: +1 17:10:28 <jlebon> yeah, agreed 17:10:34 <bgilbert> though presumably that'll evolve a bit 17:11:01 <dustymabe> i'm not opposed to this suggestion - would like for the progress we make on it to be useful when we start to do actual releases too 17:11:24 <dustymabe> IOW - can we collaborate with releng ? 17:11:25 <jlebon> we should ask fedora folks about what our options are if we can reuse the same cloud storage 17:12:01 <kaeso> dustymabe: I think the topic came up because we have a bootstrapping cycle to break for auto-updates, before making real releases 17:12:12 <slowrie> Do we have an understanding of what service will be used to store production bits? 17:12:35 <slowrie> i.e.: if it'll ultimately be gcs or s3 we can set it up initially to just store in that service and evolve the bucket layout structure to match later 17:13:07 <kaeso> (that is, released OS <-> cincinnati service <-> cincinnati client in released OS) 17:13:15 <dustymabe> slowrie: we *should* be able to use s3 I believe 17:13:59 <jlebon> ok, let's reword this topic as: "figure out storage strategy for FCOS Preview, get access to said storage, redirect CCI pipeline to it right away so we have something to play with and iterate on" 17:14:02 <dustymabe> it shouldn't be too hard to confirm that 17:14:10 <dustymabe> jlebon: +1 17:14:39 <bgilbert> jlebon: want to file a tracker issue? 17:14:45 <kaeso> it's also a good exercise to iron out details, right now I think we are missing the basearch label somewhere in the artifacts URL 17:14:57 <jlebon> bgilbert: sure thing, will do 17:15:17 <bgilbert> #action jlebon to file tracker issue: figure out storage strategy for FCOS Preview, get access to said storage, redirect CCI pipeline to it 17:15:21 <ksinny> It will be nice to have a ticket for this issue and we can add releng/fedora-infra folks to see what is feasible 17:15:28 <bgilbert> ksinny: +1 17:15:30 <ksinny> ah, already actioned 17:16:28 <bgilbert> cool, anything else here? 17:16:56 <jlebon> kaeso: hmm, that's a good point. this speaks to a higher level question re. coreos-assembler and multi-arch if we want to be able to share "build trees" 17:17:18 <jlebon> could you file an issue against coreos-assembler maybe? 17:17:24 * jlebon checks if there isn't already one 17:17:42 <kaeso> jlebon: I'm not sure I got the last part of that 17:17:50 <dustymabe> jlebon: i'd like to 'share build trees' if possible 17:18:14 <kaeso> jlebon: are you concerned with a single repo for arm/x86/etc 17:18:26 <kaeso> versus multiple dedicated ones 17:18:31 <kaeso> at ostree level? 17:18:42 <jlebon> kaeso: basically whether to make the `builds/$buildid` dir hold artifacts for all archs instead of completely split ones 17:19:01 <jlebon> i don't see an issue yet for it in a quick scan, so it'd be really good to bring 17:19:05 <jlebon> up 17:19:29 <kaeso> ah ok, then I understood something else 17:19:38 <kaeso> I'll open that 17:19:52 <jlebon> at the ostree level, if we want to share with the main fedora ostree repo, we kinda have to host it all in there 17:21:33 <bgilbert> #topic Open Floor 17:21:50 <kaeso> #action kaeso open a ticket on coreos-assembler to sort out remote bucket layout for multi-arch artifacts 17:24:05 <bgilbert> seems like we already covered the things 17:24:09 <kaeso> my little open floor bit 17:24:41 <kaeso> should we start naming and sketching the format for the next "ct"? 17:25:04 <kaeso> I realised this morning we may need that for general OS docs too 17:25:20 <ajeddeloh> For the core ct format I'd like to follow ignition's spec 17:25:22 <slowrie> kaeso: my assumption was that after we finished up with the initial ignition spec 3.0.0 work that was the next major work choice 17:25:26 <ajeddeloh> at least roughly 17:26:13 <ajeddeloh> like we did with CLCs 17:26:57 <kaeso> I think in the immediate term I'm just concerned with naming and letting the ball roll, more than the actual spec 17:27:24 <slowrie> I'm still partial to FCC :) 17:27:30 <bgilbert> was about to say 17:27:35 <bgilbert> FCC seems nice 17:28:09 <bgilbert> or a town in Massachusetts 17:28:35 * ajeddeloh wonders what's so special about MA 17:28:53 <kaeso> do we already have a name bikeshedding ticket? 17:28:57 <bgilbert> convenient to dracut maintainers, presumably 17:29:05 <bgilbert> kaeso: not AFAIK 17:29:15 <bgilbert> want to file? :-P 17:29:22 <slowrie> #129 is `CT for Fedora CoreOS` 17:29:31 <slowrie> and has some naming suggestions in it 17:29:32 <bgilbert> sure 17:30:24 <kaeso> yay, let's keep that and try to pick a name till next week 17:31:11 <bgilbert> cool. we're at time; I'll close the meeting in 30s or so 17:31:58 <bgilbert> #endmeeting