16:30:11 <lorbus> #startmeeting fedora_coreos_meeting
16:30:11 <zodbot> Meeting started Wed May 15 16:30:11 2019 UTC.
16:30:11 <zodbot> This meeting is logged and archived in a public location.
16:30:11 <zodbot> The chair is lorbus. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:30:11 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:30:11 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:30:18 <lorbus> #topic roll call
16:30:21 <slowrie_> .hello2
16:30:21 <kaeso> .hello lucab
16:30:21 <zodbot> slowrie_: Sorry, but you don't exist
16:30:24 <zodbot> kaeso: lucab 'Luca Bruno' <lucab@redhat.com>
16:30:25 <jlebon> .hello2
16:30:27 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:30:28 <ajeddeloh> .hello2
16:30:30 <zodbot> ajeddeloh: ajeddeloh 'Andrew Jeddeloh' <andrew.jeddeloh@redhat.com>
16:30:31 <slowrie> .hello2
16:30:33 <zodbot> slowrie: slowrie 'Stephen Lowrie' <slowrie@redhat.com>
16:30:54 <ksinny> .hello sinnykumari
16:30:55 <zodbot> ksinny: sinnykumari 'Sinny Kumari' <ksinny@gmail.com>
16:31:17 <lorbus> #chair slowrie lucab jlebon ajeddeloh ksinny
16:31:17 <zodbot> Current chairs: ajeddeloh jlebon ksinny lorbus lucab slowrie
16:31:52 <lorbus> #info Please add items to the meeting agenda at https://public.etherpad-mozilla.org/p/20190515-FCOS-Meeting
16:32:07 <bgilbert> .hello2
16:32:08 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:32:27 <lorbus> #chair bgilbert
16:32:27 <zodbot> Current chairs: ajeddeloh bgilbert jlebon ksinny lorbus lucab slowrie
16:32:35 <mnguyen_> .hello mnguyen
16:32:39 <zodbot> mnguyen_: mnguyen 'Michael Nguyen' <mnguyen@redhat.com>
16:32:47 <dustymabe> .hello2
16:32:48 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:33:03 <dkliban> .hello2
16:33:05 <zodbot> dkliban: dkliban 'Dennis Kliban' <dkliban@redhat.com>
16:33:12 <lorbus> #chair mnguyen dustymabe
16:33:12 <zodbot> Current chairs: ajeddeloh bgilbert dustymabe jlebon ksinny lorbus lucab mnguyen slowrie
16:33:22 <lorbus> #chair dkliban
16:33:22 <zodbot> Current chairs: ajeddeloh bgilbert dkliban dustymabe jlebon ksinny lorbus lucab mnguyen slowrie
16:33:43 <red_beard> .hello redbeard
16:33:44 <zodbot> red_beard: redbeard 'Brian 'redbeard' Harrington' <bharring@redhat.com>
16:34:02 <dkliban> i am not sure if i used the correct command. this is my first time attneding this meeting
16:34:11 <lorbus> #chair redbeard
16:34:11 <zodbot> Current chairs: ajeddeloh bgilbert dkliban dustymabe jlebon ksinny lorbus lucab mnguyen redbeard slowrie
16:34:22 <lorbus> dkliban: I think you're good :)
16:34:24 <mrguitar> .hello mrguitar
16:34:25 <zodbot> mrguitar: mrguitar 'Ben Breard' <mrguitar.net@gmail.com>
16:34:41 <lorbus> #chair mrguitar
16:34:41 <zodbot> Current chairs: ajeddeloh bgilbert dkliban dustymabe jlebon ksinny lorbus lucab mnguyen mrguitar redbeard slowrie
16:34:43 <geoff-> geoff-: Geoff Levand <geoff@infradead.org>
16:36:06 <lorbus> say hello to zod, geoff- :)
16:36:29 <lorbus> #topic Action items from last meeting
16:37:23 <lorbus> #undo
16:37:23 <zodbot> Removing item from minutes: <MeetBot.items.Topic object at 0x7f7ebf6a35d0>
16:37:28 <lorbus> there are none :)
16:38:01 <lorbus> there is nothing on the tracker either, unless somebody wishes to revisit
16:38:07 <lorbus> "Root on raid and other complex devices"
16:38:26 <lorbus> https://github.com/coreos/fedora-coreos-tracker/issues/94 ^
16:38:32 <ajeddeloh> We talked about that last week, I forgot to remove the meeting label
16:38:39 <lorbus> ack
16:38:52 <ajeddeloh> if someone has something ne we can talk about it again, but I don't have anything new
16:39:14 <lorbus> should we move to Open Floor straight away then?
16:39:23 <kaeso> we did followup and closed the FCCT casing
16:39:42 <kaeso> #link https://github.com/coreos/fedora-coreos-tracker/issues/176#issuecomment-491986978
16:39:57 <bgilbert> lorbus: +1
16:40:06 <ajeddeloh> tl;dr: snake_case
16:40:11 <bgilbert> #topic Open Floor
16:40:14 <lorbus> alrighty, thanks kaeso!
16:40:33 <lorbus> we have to things on the agenda:
16:40:46 <lorbus> #topic No Python in FCOS now!
16:40:58 <bgilbert> \o/
16:40:58 <lorbus> #link https://github.com/coreos/fedora-coreos-tracker/issues/92
16:41:15 <jlebon> very nice!
16:41:24 <mrguitar> +1
16:41:31 <lorbus> whoohee!
16:41:41 <mnguyen_> woot
16:41:43 <kaeso> we were discussing in private a followup, we should add relevant 'exclude's in the treefile
16:41:46 <lorbus> awesome work ksinny! cookie part is open
16:41:53 <lorbus> ksinny++
16:41:54 <zodbot> lorbus: Karma for sinnykumari changed to 1 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
16:41:59 <ksinny> Added it just for info, don't have much to discuss. Basically all Python deps packges which were in FCOS has been taken care of :)
16:42:19 * mrguitar wipes tear from eye. thank you
16:42:20 <ksinny> kaeso: right
16:42:47 <kaeso> ksinny: do you want to self-action that yourself for both py2 and py3?
16:42:51 <mnguyen_> ksinny++
16:43:01 <ksinny> kaeso: sure
16:43:19 <kaeso> ksinny++
16:43:20 <zodbot> kaeso: Karma for sinnykumari changed to 2 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
16:43:35 <lorbus> I wonder how much smaller our images are now
16:43:46 <jlebon> ksinny: see also https://github.com/projectatomic/rpm-ostree/issues/1768
16:43:48 <ksinny> #action  add relevant 'exclude's in the treefile to make sure we don't accidently pull in Python dependent packages in futture
16:43:54 <ksinny> #undo
16:43:54 <zodbot> Removing item from minutes: ACTION by ksinny at 16:43:48 : add relevant 'exclude's in the treefile to make sure we don't accidently pull in Python dependent packages in futture
16:44:00 <jlebon> also ksinny++
16:44:01 <zodbot> jlebon: Karma for sinnykumari changed to 3 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
16:44:04 <ksinny> #action ksinny to add relevant 'exclude's in the treefile to make sure we don't accidently pull in Python dependent packages in futture
16:44:07 <lorbus> really great we've finally achieved this :)
16:44:19 <jlebon> ksinny: the functionality doesn't exist yet :)
16:44:33 <jlebon> though we *could* just spam exclude= in all the repo files for now
16:45:19 <kaeso> jlebon: ah, I read yesterday suggestion as if the treefile feature was already there
16:45:23 <jlebon> basically `exclude=python2 python3` should do it
16:45:39 <ksinny> jlebon: ah, thanks for the info. Will it be easier to implement or nees lots of changes?
16:45:49 <ksinny> needs*
16:45:54 <ajeddeloh> misc thought: could we add tests in cosa to basically grep through the full package list and fail if it finds python?
16:46:11 <jlebon> ksinny: it shouldn't be very hard to implement
16:46:14 * ksinny is willing to try to implement that feature in rpm-ostree
16:46:29 <ajeddeloh> wouldn't exclude succeed and silently remove python
16:46:52 <ksinny> jlebon: awesome, will create an issue and then we can discuss the implementation detail there
16:46:58 <ajeddeloh> I'd think we'd want to fail hard rather than subtly suceed
16:47:36 <jlebon> ajeddeloh: exclude= essentially removes python from the pool of available pkgs. adding exclude= to the manifest though could have more powerful "assert-like" semantics
16:48:11 <dustymabe> i think colin said it would fail with a depsolver error if some package required it but it couldn't be installed
16:48:28 <ajeddeloh> oh, ok, great! nvm then
16:48:47 <dustymabe> we'd have to verify that is true once we have the feature in place
16:49:11 <kaeso> is server-sive and client-side symmetrical on this?
16:49:15 <kaeso> *side
16:49:35 <dustymabe> meaning could someone `rpm-ostree install python3` ?
16:49:47 <kaeso> that is, if we fill in "exclude"s, do we also want to stop client-side installs?
16:49:51 <kaeso> dustymabe: that, yes
16:50:02 <walters> no
16:50:05 <jlebon> hmm it won't no
16:50:23 <ksinny> that shouldn't impact client side since we will be defining exclude in manifest file while running rpm-ostree compose
16:50:48 <walters> (the server and client side stuff uses separate config formats for...historical reasons; really the only "data" passed between them is the rpmdb)
16:51:18 <walters> another good example of this is that disabling docs server side doesn't disable them client side
16:51:35 <jlebon> kaeso: would we actually want that anyway though? once you go pkg-layering it's up to you what's acceptable
16:52:22 <kaeso> jlebon: I was reading the GH ticket you posted and got confused. I don't think we want that, no.
16:52:33 <lorbus> do we have a follow-up ticket for this already that we can link to now?
16:53:17 <lorbus> otherwise, let's revisit next time. all ok to move on?
16:53:29 <ksinny> jlebon: I missed https://github.com/projectatomic/rpm-ostree/issues/1768 link earlier, just saw while looking at scrollback :)
16:53:44 <kaeso> #link https://github.com/projectatomic/rpm-ostree/issues/1768
16:53:48 <lorbus> #link https://github.com/projectatomic/rpm-ostree/issues/1768
16:53:54 <lorbus> #undo
16:53:54 <zodbot> Removing item from minutes: <MeetBot.items.Link object at 0x7f7e69b75250>
16:54:37 <lorbus> #topic Pulp and OSTree support
16:54:39 <lorbus> #info possibly related to https://github.com/coreos/fedora-coreos-tracker/issues/169
16:54:55 <dkliban> I put this on here ...
16:55:11 <dkliban> I work on Pulp and we have a plugin for managing OSTree repositories
16:55:21 <lorbus> (sorry everyone, I seem to experience a bit of a Matrix-lag)
16:55:31 <lorbus> dkliban: go! :)
16:56:03 <dkliban> This OSTree plugin for PUlp 2 currently lets users mirror OSTree repos and copy branches of those repos from one repo to another in Pulp
16:57:05 <dkliban> We are now starting work on porting the OSTree plugin to Pulp 3. As we start this work, I was hoping that we could work close with FCOS to make sure that the features we are delivering are actually valuable
16:57:29 <dkliban> so as I was looking through the issue tracker I notied that there is some discussion about managing build artifacts
16:57:50 <dkliban> i thought that Pulp 3 could be a solution for managing OSTree build artifacts
16:58:32 <dkliban> Pulp 2 does not support any kind of upload use case for OSTreee, but I would like to chnage that for Pulp 3
16:59:11 <dkliban> i discussed this with jlebon yesterday and he suggested that i bring this up here today
16:59:31 <jlebon> dkliban: note there's Silverblue and IoT which also use vanilla OSTree
16:59:34 <lorbus> dkliban++ 100% thanks for bringing this up!
16:59:35 <zodbot> lorbus: Karma for dkliban changed to 1 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:00:08 <dustymabe> flatpaks can also be distributed that way
17:00:15 <jlebon> easy OSTree mirroring i think is super useful
17:00:36 <dkliban> jlebon: but is there a need for content producers to be able to upload also?
17:01:01 <kaeso> does fedora-infra already hosts a Pulp 3 instance somewhere?
17:01:08 <kaeso> *host
17:01:10 <dkliban> kaeso: not yet
17:01:49 <jlebon> dkliban: yeah "uploading" OSTrees has always been tricky. if you can control the target host on which it is hosted, you can at least `ostree pull` from there. but otherwise, we've mostly relied on rsync
17:02:12 <dkliban> jlebon: is this due to the many number of files?
17:02:15 <dustymabe> for fedora infra we've got a master ostree repo set up today
17:02:50 <jlebon> dkliban: mostly due to the lack of a "smart server". i.e. it's just files hosted over HTTP
17:02:59 <dkliban> dustymabe: do you rsync to that repo?
17:03:28 <dustymabe> dkliban: we use NFS mounts and an `ostree pull-local`
17:03:54 <dustymabe> to sync between ostree repos (we have a compose repo and a prod repo)
17:04:44 <dkliban> gotcha
17:05:30 <dkliban> so far it seems to me that we should continue to focus on mirroring
17:05:46 <dkliban> and in that case, how does cincinati come into play?
17:05:47 <lorbus> #info work on porting the Pulp 2 OSTree plugin to Pulp 3 has begun
17:06:44 <jlebon> dkliban: yeah, that's a good question. i think we're still figuring out where Cincinnati gets its data from (right kaeso?) for FCOS
17:07:51 <dustymabe> dkliban: is anyone else asking you for the feature work you are proposing ?
17:08:08 <jlebon> (aside: how did mirroring work for CL? could people point the update agent at a local mirror?)
17:08:17 <kaeso> cincinnati only cares about ostree commit-hash, so where the repo lives should not be relevant to it
17:08:31 <dkliban> dustymabe: this is for Foreman (Satellite 6)
17:09:32 <bgilbert> jlebon: for update payloads, there wasn't really a good way to do mirroring
17:09:49 <dkliban> dustymabe: we initially thought that since atomic host was going away that we would drop support for OSTree, but we are not realizing that we will want to let users manage OSTree content so they can have local copies of Fedora Siliverblue repo and Fedora CoreOS repo
17:09:55 <bgilbert> jlebon: might be possible if you work at it, but we never tried to support it
17:10:25 <dustymabe> dkliban: ok - yeah it should be pretty easy to do the mirror work (as you mentioned already done in pulp2)
17:10:34 <jlebon> kaeso: it seems like there are two use cases: one where clients keep contacting the upstream cincinnati server, but download from a local mirror, and another where you'd want to set up your own cincinnati server for your own cadence/rollout strategies?
17:10:44 <kaeso> I think FCOS is anyway different than CL one
17:10:48 <dustymabe> I wouldn't spend time adding new features unless someone requests it
17:11:08 <kaeso> on FCOS there a matrix of custom-cincinnati and custom-ostree-repo
17:11:14 <dkliban> dustymabe: i agree ... that's why i wanted to talk to you all to see if you are interested in these features
17:11:23 <kaeso> and all the 4 combinations are client-side configs
17:11:27 <jlebon> kaeso: right, agreed
17:12:31 <lorbus> looking at the time, lets move on to the next topic
17:12:36 <dkliban> alright ... i think i got the answers i wanted out of this
17:12:44 <dkliban> i ageee ... let's go on to the next topic
17:12:58 <lorbus> thanks again for brining this up dkliban!
17:12:59 <jlebon> dkliban: thanks again for bringing up the subject!
17:13:05 <lorbus> #topic etcd-wrapper, flannel-wrapper equivilents?
17:13:07 <dustymabe> thanks dkliban
17:13:14 <kaeso> jlebon: if there is some more details which seems unclear, let's discuss that out of band
17:13:27 <ajeddeloh> basvdlei brought this up in IRC the other day
17:13:31 <lorbus> ajeddeloh that's yours I guess?
17:13:44 <ajeddeloh> do we want to ship these wrappers in fcos?
17:14:11 <ajeddeloh> on CL they're used as sort of a "happy path" for making setting up etcd and flannel
17:14:39 * ajeddeloh wonders if they can literally be copied over and used as is
17:15:09 <kaeso> likely not
17:15:14 <dustymabe> does it make it super easy to join a cluster ?
17:15:16 <kaeso> they all use rkt inside
17:15:21 <jlebon> links?
17:15:25 <ajeddeloh> https://github.com/coreos/coreos-overlay/tree/master/app-admin/etcd-wrapper
17:15:34 <slowrie> was going to say, don't those use stage 1 flys?
17:15:37 <ajeddeloh> https://github.com/coreos/coreos-overlay/tree/master/app-admin/flannel-wrapper
17:15:40 <lorbus> #link https://github.com/coreos/coreos-overlay/tree/master/app-admin/etcd-wrapper
17:15:48 <kaeso> but even if we switch to podman, I don't think we have to
17:16:05 <lorbus> #link https://github.com/coreos/coreos-overlay/tree/master/app-admin/flannel-wrapper
17:16:53 <bgilbert> can't speak to the flannel part
17:16:56 <dustymabe> yeah - could easily add them in the future if we find it necessary
17:16:56 <bgilbert> etcd-wrapper sort of encourages an etcd on every node, which isn't really the right thing
17:17:25 <kaeso> on CL, they were part of bigger story of "cluster orchestration"
17:17:39 <kaeso> together with kubelet-wrapper (and fleet?)
17:18:27 <bgilbert> we don't have a really coherent story there yet, but it'll look different than on CL
17:18:32 <ajeddeloh> bgilbert: would you prefer to not have an etcd-wrapper?
17:18:39 <bgilbert> I think so?
17:18:49 <ajeddeloh> or just document "hey don't do this on all your nodes"
17:18:53 <bgilbert> "run this container" seems fine, and people will be doing plenty of it
17:19:02 <kaeso> ajeddeloh: there is also "etcd2 vs etcd3"
17:19:18 * ajeddeloh hopes people will be using 3 on fcos
17:19:49 <bgilbert> CL originally shipped etcd in the OS, and the wrapper was a way to move it out
17:20:10 <ajeddeloh> other question is: regardless of *-wrapper, do we want to add fcct sugar for etcd, flannel, etc
17:20:30 <bgilbert> and I don't know that we want to couple as tightly to etcd anymore
17:20:59 <bgilbert> ajeddeloh: heh.  if we don't, CL migration gets harder
17:21:03 <kaeso> ajeddeloh: that seems a saner thing to do (generating the whole unit file)
17:21:28 <ajeddeloh> +1 I'm down for "not shipping wrappers but generating unit files with fcct"
17:21:50 <bgilbert> which does mean we can't update them later
17:22:08 <bgilbert> in sum, I guess I'm on the fence
17:22:30 <lorbus> do we need a vote on that? We can probably just info it..
17:23:00 <mrguitar> updating a unit file doesn't sound too painful ...though I guess we don't have the mco here
17:23:29 <kaeso> I see them as "containerize user apps", thus the user owns it lifecycle (which includes updates and migrations)
17:23:38 <kaeso> *containerized
17:23:41 <kaeso> *its
17:23:57 <ajeddeloh> mrguitar: in this case it would mean reprovisioning
17:24:23 <bgilbert> more to the point, we couldn't do it on behalf of users
17:24:27 <bgilbert> because the user owns their config
17:25:12 <lorbus> noting the time. 5min left and one more topic to go!
17:25:16 <ajeddeloh> I think I'm ok with that
17:25:31 <ajeddeloh> I'll create a tracker issue nonethekess
17:25:38 <kaeso> ajeddeloh++
17:25:38 <zodbot> kaeso: Karma for ajeddeloh changed to 4 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:26:07 <geoff-> maybe think about what the use cases are, then what's needed to support those use cases
17:26:20 <lorbus> #action ajeddeloh to create a tracker issue for the *-wrapper topic
17:26:42 <lorbus> #chair geoff-
17:26:42 <zodbot> Current chairs: ajeddeloh bgilbert dkliban dustymabe geoff- jlebon ksinny lorbus lucab mnguyen mrguitar redbeard slowrie
17:27:11 <lorbus> moving on...
17:27:14 <lorbus> #topic Conditionally disabling SMT by default
17:27:33 <lorbus> bgilbert is that you?
17:27:44 <bgilbert> yep
17:28:01 <bgilbert> with L1TF and MDS, enabling SMT by default doesn't seem like an entirely safe choice
17:28:42 <bgilbert> SMT is unsafe on certain processors (a great many of them) and certain workloads.  for entirely trusted workloads, it's fine, and provides some performance advantage
17:29:34 <bgilbert> for existing systems (e.g. the Container Linux installed base) it's difficult to just disable SMT via an update, since a) performance hit and b) only needed for certain workloads.
17:29:39 <mrguitar> we went back and forth on this w/ rhel, and ultimately decided to leave it on and warn users of the pros & cons
17:29:47 <bgilbert> but we're building a new OS and we have the opportunity to make different choices.
17:29:59 <bgilbert> the kernel supports two approaches:
17:30:02 <mrguitar> right
17:30:03 <bgilbert> `nosmt` disables unconditionally
17:30:28 <bgilbert> `mitigations=auto,nosmt` disables when needed to mitigate something
17:30:49 <slowrie> I agree with conditionally disabling it provided we provide sufficient documentation about the potential ramifications (e.x.: SMT might be shut off post-updates if newer kernels determine you're vulnerable), and how to enable/disable unconditionally
17:30:54 <bgilbert> that performs better (e.g., at the moment it leaves SMT enabled on AMT CPUs)
17:31:10 <bgilbert> but it also allows future updates to _change_ whether SMT is enabled.
17:31:15 <ajeddeloh> this of course is made difficult by the fact that we can't change it on first boot, need to reboot to take effect or disable later in boot unconditionally (there is no disable-if-affected option later in boot)
17:31:37 <bgilbert> so yeah, we'd want to document it, plus the force-on and force-off options
17:31:57 <bgilbert> but I think "we will keep you secure even if that means disabling HT" is a reasonable policy to set from day 1
17:32:30 <bgilbert> ajeddeloh: we might want to explore adding a mechanism to reboot, post-Ignition, from inside the initramfs
17:32:38 <slowrie> ajeddeloh: could we potentially add a hook in the initramfs we could jerry-rig into rebooting there that wouldn't break ConditionalFirstBoot?
17:32:51 <ajeddeloh> right
17:33:22 <bgilbert> (for context: rebooting halfway into the real root can cause later ConditionFirstBoot units never to run at all)
17:34:00 <lorbus> anybody want to info how we'll continue on this? time's up I'm afraid..
17:34:39 <ajeddeloh> create a tracker issue?
17:34:39 <bgilbert> #action bgilbert will file a ticket on conditionally disabling SMT, with links to PRs
17:34:44 <ajeddeloh> +1
17:34:53 <lorbus> thank you!
17:34:54 <lorbus> #endmeeting