16:30:01 <bgilbert> #startmeeting fedora_coreos_meeting
16:30:01 <zodbot> Meeting started Wed Apr 17 16:30:01 2019 UTC.
16:30:01 <zodbot> This meeting is logged and archived in a public location.
16:30:01 <zodbot> The chair is bgilbert. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:30:01 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:30:01 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:30:06 <bgilbert> #topic roll call
16:30:08 <geoff--> geoff-: Geoff Levand <geoff@infradead.org>
16:30:15 <slowrie> .hello2
16:30:16 <zodbot> slowrie: slowrie 'Stephen Lowrie' <slowrie@redhat.com>
16:30:18 <kaeso> .hello lucab
16:30:19 <zodbot> kaeso: lucab 'Luca Bruno' <lucab@redhat.com>
16:30:31 <jlebon> .hello2
16:30:32 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:31:27 <jligon> .hello2
16:31:28 <zodbot> jligon: jligon 'Jeff Ligon' <jligon@redhat.com>
16:31:30 <bgilbert> .hello2
16:31:31 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:32:19 <yzhang> .hello2
16:32:20 <zodbot> yzhang: yzhang 'Yu Qi Zhang' <jzehrarnyg@gmail.com>
16:32:52 <rfairley> .hello2
16:32:53 <zodbot> rfairley: rfairley 'None' <rfairley@redhat.com>
16:32:56 <mnguyen_> .hello mnguyen
16:33:00 <zodbot> mnguyen_: mnguyen 'Michael Nguyen' <mnguyen@redhat.com>
16:33:22 <ksinny> .hello sinnykumari
16:33:23 <zodbot> ksinny: sinnykumari 'Sinny Kumari' <ksinny@gmail.com>
16:34:05 <bgilbert> #chair geoff- slowrie kaeso jlebon jligon yzhang rfairley mnguyen_ ksinny
16:34:05 <zodbot> Current chairs: bgilbert geoff- jlebon jligon kaeso ksinny mnguyen_ rfairley slowrie yzhang
16:34:21 <bgilbert> #info please add items to the meeting agenda at https://public.etherpad-mozilla.org/p/20190417-FCOS-Meeting
16:34:35 <bgilbert> #topic Action items from last meeting
16:34:42 <bgilbert> * jlebon to investigate packit
16:34:46 <bgilbert> * ksinny to request pool tag(s) once stream implementation PR is finalized
16:34:50 <bgilbert> * bgilbert to create coreos/airlock
16:35:03 <bgilbert> #info bgilbert created https://github.com/coreos/airlock
16:35:38 <ksinny> #info ksinny will request for tags after this meeting
16:36:02 <bgilbert> #action ksinny to request pool tags
16:36:34 <jlebon> re. packit: i opened up https://github.com/packit-service/packit/pull/265 to tell upstream we're interested in being early adopters
16:36:40 <jlebon> i also opened https://github.com/packit-service/packit/issues/264 to discuss our use-case
16:37:12 <red_beard> .hello redbeard
16:37:13 <zodbot> red_beard: redbeard 'Brian 'redbeard' Harrington' <bharring@redhat.com>
16:37:15 <jlebon> TL;DR is that Packit is not quite ready yet, but it should do what we want it to do in the future
16:37:52 <bgilbert> #info jlebon engaging with Packit upstream
16:37:55 <bgilbert> #chair red_beard
16:37:56 <zodbot> Current chairs: bgilbert geoff- jlebon jligon kaeso ksinny mnguyen_ red_beard rfairley slowrie yzhang
16:38:17 <bgilbert> #topic Finalize artifact storage strategy
16:38:18 <bgilbert> #link https://github.com/coreos/fedora-coreos-tracker/issues/169
16:38:52 <bgilbert> jlebon, want to summarize?
16:39:01 <jlebon> we had a productive mtg with releng this morning about this
16:39:46 <jlebon> consensus is for an S3 bucket for FCOS artifacts that we can use
16:40:28 <jlebon> there are still things to be figured out in the periphery, e.g. related to the OSTree commits, and how to actually expose the bucket to clients
16:41:08 <jlebon> one suggestion was a CNAME from $subdomain.coreos.fedoraproject.org, though we're also discussing a redirector
16:41:33 <jlebon> but at least getting an S3 bucket will unblock us in the short term
16:41:39 <jlebon> https://pagure.io/fedora-infrastructure/issue/7719
16:42:55 <jlebon> bgilbert: i'm not sure if you wanted to dive into any specific bit in particular
16:43:13 <bgilbert> nope, that was it
16:43:15 <red_beard> jlebon: related to that can we make sure that if we need to have multiple files related to a single "oem" (e.g. a kernel and separate initram fs) that we correlate them in the meta.json file or something similar?
16:43:47 <bgilbert> red_beard: we're going to have a JSON endpoint that points people to the correct artifacts to use
16:43:48 <red_beard> as a part of the general storage strategy for how we push builds out?  this makes it so that if i wanted to sync a single platform i can easily query that out
16:44:04 <red_beard> :thumbs_up:
16:44:13 <bgilbert> my preference is for the bucket not to be listable at all, though we may not be able to go that far
16:44:21 <jlebon> red_beard: check out the images[] array in http://artifacts.ci.centos.org/fedora-coreos/prod/builds/latest/meta.json
16:44:49 <red_beard> well, that's why i ask. i saw meta.json and the concern about the bucket being listable
16:45:20 <bgilbert> red_beard: https://github.com/coreos/fedora-coreos-tracker/issues/98 gives a good sense of my thinking there
16:45:29 <red_beard> that way meta.json can be used to only expose the needed assets while making it easier to sync (something that was a PITA since the get-go)
16:45:34 <bgilbert> I don't think meta.json is the right level of abstraction for users
16:45:40 <kaeso> jlebon: yeah, though it isn't exactly aligned with what he's asking. Installer files are grouped by filename only.
16:45:56 <red_beard> perfect.  i'll shut up and review #98
16:46:09 <jlebon> red_beard, kaeso: gotcha :)
16:47:11 <bgilbert> thanks for the update jlebon!
16:47:33 <bgilbert> red_beard: +1, ping me if you still have concerns
16:47:39 <bgilbert> #topic Implement tooling for release streams
16:47:48 <bgilbert> #link https://github.com/coreos/fedora-coreos-tracker/issues/137
16:47:55 <bgilbert> #link https://github.com/coreos/fedora-coreos-tracker/blob/master/stream-tooling.md
16:48:31 <bgilbert> I went ahead and merged the doc, but we can still update it if there are concerns
16:48:46 <bgilbert> I'll attempt to summarize:
16:49:03 <bgilbert> we'll have the production streams that are already in the design (stable, testing, next)
16:49:43 <bgilbert> we'll have development streams, testing-devel and next-devel.  those will be nightly snapshot builds.  they're equivalent to Container Linux's "master" branch: development snapshots of what will become the next testing/next release
16:49:53 <bgilbert> there's no stable-devel because that's called "testing" :-)
16:50:19 <bgilbert> we'll have mechanical streams, which are robotic snapshots of rawhide/bodhi-updates/etc.
16:50:38 <bgilbert> the development and mechanical streams will be in a separate ostree repo from prod to avoid confusion
16:50:52 <bgilbert> and each stream will correspond to an ostree ref and a Git branch
16:51:17 <bgilbert> that is, a Git branch of some repo, perhaps fedora-coreos-config.  we won't branch _everything_
16:51:45 <bgilbert> that repo will store lockfiles pinning the exact package NEVRs that went into the build
16:52:12 <bgilbert> and there will be various automation to help maintain the lockfiles and treefiles.  e.g., the lockfiles in the mechanical branches will be automatically maintained.
16:52:52 <bgilbert> NEVRs pinned in the dev and prod streams will automatically be tagged into a new "coreos-pool" koji tag
16:53:23 <bgilbert> so we can control our own garbage-collection lifecycle.  when we do an official release, we'll also tag into a second koji tag with a longer retention period.
16:53:47 <bgilbert> I think those are the highlights.  questions/concerns?
16:54:32 <kaeso> bgilbert: what does "annex" is "ostree repo" context mean there?
16:55:18 <kaeso> I mean is it just an arbitrary name/label or are some technical details that I don't know?
16:55:20 <bgilbert> it's the name I picked for the second ostree repo for non-prod streams
16:55:21 <jlebon> bgilbert: would be nice to keep the development branch in the same repo so that the testing-devel -> testing push is more obvious (and same for next-devel/next)
16:55:45 <bgilbert> kaeso: arbitrary label
16:55:51 <bgilbert> jlebon: same git repo or same ostree repo?
16:55:53 <kaeso> bgilbert: ack, I guessed so but was unsure
16:56:02 <jlebon> bgilbert: same git repo
16:56:27 <bgilbert> yup, I was assuming all the streams use the same git repo
16:56:32 <bgilbert> I guess that didn't actually make it into the doc
16:56:50 <bgilbert> well, it sorta did
16:56:58 <bgilbert> first paragraph of https://github.com/coreos/fedora-coreos-tracker/blob/master/stream-tooling.md#how-will-the-package-list-be-maintained
16:57:06 <jlebon> ohh whoops, i misread "separate ostree repo" up there with "separate git repo" :)
16:57:15 <bgilbert> jlebon: +1
16:57:42 <bgilbert> I'm increasingly concerned about the complexity of the next stream
16:58:07 <bgilbert> it switches upstreams between branched and testing, and kernels between its upstream and rawhide
16:58:20 <kaeso> bgilbert: it sounds like we don't anymore anything like `repo` to maintain multiple repos in sync, correct?
16:58:30 <bgilbert> and that'll ripple down into, at least, some of the automatic mirroring between Git branches
16:59:27 <bgilbert> I think that complexity is actually necessary to get us the bake time we need, but I wanted to at least point it out
16:59:44 <bgilbert> kaeso: I think that's correct?
17:00:04 <bgilbert> kaeso: coreos-assembler and fedora-coreos-config should probably still be mostly in sync
17:00:15 <bgilbert> kaeso: and the streams repo, if different
17:00:17 <slowrie> bgilbert: do mechanical streams function have automatic updates?
17:01:11 <bgilbert> slowrie: good point, we haven't talked about Cincinnati for non-prod streams
17:01:40 <bgilbert> kaeso: but I wasn't thinking we'd actually use a sync tool like `repo`
17:02:14 <bgilbert> slowrie: mechanical streams should almost certainly have Cincinnati disabled
17:02:23 <jlebon> yeah, i think the crucial bits are coreos-assembler and fedora-coreos-config. hopefully, there's nothing else to sync
17:02:30 <bgilbert> they're completely uncurated.  you can sync them with rpm-ostree directly
17:03:03 <bgilbert> the development streams... likewise, I think?
17:03:25 <slowrie> bgilbert: agreed, just wanted to make sure that was the expectation otherwise any rawhide based streams could potentially be very problematic
17:03:26 <bgilbert> except that then we're not really testing Cincinnati upgrades outside of official releases
17:03:30 <bgilbert> slowrie: yeah
17:04:00 <kaeso> we can either turn-off cincinnati client-side for those streams, or just not source those releases into cincinnati update-graph
17:04:25 <bgilbert> for the record, CL development builds can't update themselves at all
17:04:34 <kaeso> jlebon: coreos-assembler should come from a container image, though
17:05:31 <kaeso> bgilbert: I don't know the details for CL dev, how do we disable auto-updates there? url/channel setting?
17:06:18 <bgilbert> developer channel.  they check into CoreUpdate, but CoreUpdate has updates disabled on that channel
17:06:43 <bgilbert> anything else on this topic?
17:06:57 <jlebon> kaeso: yeah. i think we can (1) mark the cosa git commit we used in CI when pushing, then (2) have a corresponding tag in the cosa imagestream in OCP with which to do the build?
17:08:03 <kaeso> bgilbert: I see. I'd rather disable them client-side to even avoid hitting the cincinnati server
17:08:14 <bgilbert> kaeso: +1
17:08:32 <bgilbert> we need to do that anyway to prevent rpm-ostree from bailing out when users try to update directly
17:08:39 <jlebon> kaeso: so basically, we always run a prod build with the same exact cosa that built its devel and passed CI
17:09:13 <kaeso> jlebon: +1
17:09:20 <bgilbert> #topic Implement tooling for release streams
17:09:24 <bgilbert> #link https://github.com/coreos/fedora-coreos-tracker/issues/137
17:09:41 <bgilbert> #undo
17:09:41 <zodbot> Removing item from minutes: <MeetBot.items.Link object at 0x7f7e6379f8d0>
17:09:43 <bgilbert> #undo
17:09:43 <zodbot> Removing item from minutes: <MeetBot.items.Topic object at 0x7f7e6323d250>
17:09:51 <bgilbert> #topic Produce live PXE images
17:09:56 <bgilbert> #link https://github.com/coreos/fedora-coreos-tracker/issues/105
17:10:30 <bgilbert> I wanted to raise this here because of a recent OOB discussion about use cases
17:10:48 <bgilbert> summarized here:
17:10:48 <bgilbert> #link https://github.com/coreos/fedora-coreos-tracker/issues/105#issuecomment-481967168
17:10:59 <bgilbert> Container Linux has several PXE use cases
17:11:01 <bgilbert> some of which are obscure
17:11:21 <bgilbert> the usual one is PXE to a live system, running CL from RAM
17:11:48 <bgilbert> that approach can be used to run the installer, or to actually run the OS in production
17:11:51 <red_beard> FYI, funny enough i'm writing up some of these in my response to issues/#98
17:12:09 <red_beard> specifically about running live + diskless
17:12:20 <bgilbert> red_beard: we absolutely want to support the live + diskless case
17:12:47 <bgilbert> that case implies running Ignition on every boot, because every boot is fresh
17:13:19 <bgilbert> the OS _cannot_ update itself in this case; it simply reboots (via some sort of out-of-band scheduling) and the PXE server hands out a new OS image
17:13:38 <bgilbert> but there is also a second case:
17:13:41 <red_beard> yup, that's what i'm calling out (which is why the OEM metadata is critical)
17:14:15 <bgilbert> red_beard: +1, please go ahead and post your comment so we make sure we have it recorded :-)
17:14:52 <bgilbert> in CL, /etc and /var are both on the root partition (by default), but the OS is in /usr
17:15:03 <bgilbert> and we allow live PXE + persistent root.
17:15:31 <bgilbert> so your configuration is persistent, but OS upgrades still happen out-of-band, as in the diskless case.
17:15:56 <bgilbert> in this configuration, Ignition may or may not run every boot, depending on how clever the PXE server is.
17:16:05 <bgilbert> (generally it will run every boot)
17:16:37 <bgilbert> the proposal for FCOS is to defer, or skip, supporting the second case.
17:17:02 <bgilbert> you'd still be able to have live PXE + persistent data volumes, e.g. /var
17:17:35 <bgilbert> but not a persistent root, and in particular, not a persistent /etc.  and thus, you'd need to run Ignition on every boot, since there'd be no other way to populate /etc.
17:18:20 <bgilbert> the proximate reason for deferring this functionality is that, without OS-managed updates, we'd have to specially handle the rpm-ostree three-way merge of /etc on updates
17:18:38 <bgilbert> that wasn't a problem in CL because we never updated the user's /etc once it was created
17:19:17 <bgilbert> there are ways to do this, of course.  but a) we'd like to manage complexity for the preview release, and b) it's not clear how many people use live PXE + persistent root on CL.
17:19:23 <bgilbert> so it may turn out not to be an important use case.
17:19:57 <bgilbert> tl;dr: anyone who needs live PXE + persistent root in FCOS, please speak up
17:19:59 <bgilbert> here or in the ticket
17:20:14 <kaeso> red_beard: is your diskless scenario completely diskless?
17:21:42 <red_beard> for one of our former customers it was.  In another case they would generate a LUKS key at boot time, setup /var/lib/docker as an encrypted volume, but then intentionally not persist the key
17:21:50 <red_beard> that way they could avoid scrubbing on reboot
17:22:11 <bgilbert> nice
17:22:26 <red_beard> saved on SSD write cycles and speed up reprovisioning
17:23:31 <kaeso> red_beard: ack, seems fine. What was the other concern on platform metadata?
17:25:27 <red_beard> it was around being able to discover all of the components for an OEM
17:25:42 <red_beard> (i'm still writing my novella and cross-citing prior art)
17:25:43 <red_beard> :D
17:25:46 <bgilbert> red_beard: +1
17:25:58 <bgilbert> #info we're planning not to initially support live PXE with a persistent root partition in FCOS.  if you use this functionality on CL, please speak up in https://github.com/coreos/fedora-coreos-tracker/issues/105
17:26:18 <bgilbert> #topic Open Floor
17:28:02 <bgilbert> anyone?
17:28:57 <jligon> i got nothing today
17:29:45 <ksinny> #info Requested for coreos-pool and coreos-release tag https://pagure.io/releng/issue/8294
17:30:07 <bgilbert> ksinny++
17:30:09 <zodbot> bgilbert: Karma for sinnykumari changed to 11 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:30:22 <jlebon> ksinny++
17:30:23 <zodbot> jlebon: Karma for sinnykumari changed to 12 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:30:37 <bgilbert> cool
17:30:39 <bgilbert> thanks for coming, all!
17:30:41 <bgilbert> #endmeeting