16:30:11 #startmeeting fedora_coreos_meeting 16:30:11 Meeting started Wed May 15 16:30:11 2019 UTC. 16:30:11 This meeting is logged and archived in a public location. 16:30:11 The chair is lorbus. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:30:11 Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:30:11 The meeting name has been set to 'fedora_coreos_meeting' 16:30:18 #topic roll call 16:30:21 .hello2 16:30:21 .hello lucab 16:30:21 slowrie_: Sorry, but you don't exist 16:30:24 kaeso: lucab 'Luca Bruno' 16:30:25 .hello2 16:30:27 jlebon: jlebon 'None' 16:30:28 .hello2 16:30:30 ajeddeloh: ajeddeloh 'Andrew Jeddeloh' 16:30:31 .hello2 16:30:33 slowrie: slowrie 'Stephen Lowrie' 16:30:54 .hello sinnykumari 16:30:55 ksinny: sinnykumari 'Sinny Kumari' 16:31:17 #chair slowrie lucab jlebon ajeddeloh ksinny 16:31:17 Current chairs: ajeddeloh jlebon ksinny lorbus lucab slowrie 16:31:52 #info Please add items to the meeting agenda at https://public.etherpad-mozilla.org/p/20190515-FCOS-Meeting 16:32:07 .hello2 16:32:08 bgilbert: bgilbert 'Benjamin Gilbert' 16:32:27 #chair bgilbert 16:32:27 Current chairs: ajeddeloh bgilbert jlebon ksinny lorbus lucab slowrie 16:32:35 .hello mnguyen 16:32:39 mnguyen_: mnguyen 'Michael Nguyen' 16:32:47 .hello2 16:32:48 dustymabe: dustymabe 'Dusty Mabe' 16:33:03 .hello2 16:33:05 dkliban: dkliban 'Dennis Kliban' 16:33:12 #chair mnguyen dustymabe 16:33:12 Current chairs: ajeddeloh bgilbert dustymabe jlebon ksinny lorbus lucab mnguyen slowrie 16:33:22 #chair dkliban 16:33:22 Current chairs: ajeddeloh bgilbert dkliban dustymabe jlebon ksinny lorbus lucab mnguyen slowrie 16:33:43 .hello redbeard 16:33:44 red_beard: redbeard 'Brian 'redbeard' Harrington' 16:34:02 i am not sure if i used the correct command. this is my first time attneding this meeting 16:34:11 #chair redbeard 16:34:11 Current chairs: ajeddeloh bgilbert dkliban dustymabe jlebon ksinny lorbus lucab mnguyen redbeard slowrie 16:34:22 dkliban: I think you're good :) 16:34:24 .hello mrguitar 16:34:25 mrguitar: mrguitar 'Ben Breard' 16:34:41 #chair mrguitar 16:34:41 Current chairs: ajeddeloh bgilbert dkliban dustymabe jlebon ksinny lorbus lucab mnguyen mrguitar redbeard slowrie 16:34:43 geoff-: Geoff Levand 16:36:06 say hello to zod, geoff- :) 16:36:29 #topic Action items from last meeting 16:37:23 #undo 16:37:23 Removing item from minutes: 16:37:28 there are none :) 16:38:01 there is nothing on the tracker either, unless somebody wishes to revisit 16:38:07 "Root on raid and other complex devices" 16:38:26 https://github.com/coreos/fedora-coreos-tracker/issues/94 ^ 16:38:32 We talked about that last week, I forgot to remove the meeting label 16:38:39 ack 16:38:52 if someone has something ne we can talk about it again, but I don't have anything new 16:39:14 should we move to Open Floor straight away then? 16:39:23 we did followup and closed the FCCT casing 16:39:42 #link https://github.com/coreos/fedora-coreos-tracker/issues/176#issuecomment-491986978 16:39:57 lorbus: +1 16:40:06 tl;dr: snake_case 16:40:11 #topic Open Floor 16:40:14 alrighty, thanks kaeso! 16:40:33 we have to things on the agenda: 16:40:46 #topic No Python in FCOS now! 16:40:58 \o/ 16:40:58 #link https://github.com/coreos/fedora-coreos-tracker/issues/92 16:41:15 very nice! 16:41:24 +1 16:41:31 whoohee! 16:41:41 woot 16:41:43 we were discussing in private a followup, we should add relevant 'exclude's in the treefile 16:41:46 awesome work ksinny! cookie part is open 16:41:53 ksinny++ 16:41:54 lorbus: Karma for sinnykumari changed to 1 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 16:41:59 Added it just for info, don't have much to discuss. Basically all Python deps packges which were in FCOS has been taken care of :) 16:42:19 * mrguitar wipes tear from eye. thank you 16:42:20 kaeso: right 16:42:47 ksinny: do you want to self-action that yourself for both py2 and py3? 16:42:51 ksinny++ 16:43:01 kaeso: sure 16:43:19 ksinny++ 16:43:20 kaeso: Karma for sinnykumari changed to 2 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 16:43:35 I wonder how much smaller our images are now 16:43:46 ksinny: see also https://github.com/projectatomic/rpm-ostree/issues/1768 16:43:48 #action add relevant 'exclude's in the treefile to make sure we don't accidently pull in Python dependent packages in futture 16:43:54 #undo 16:43:54 Removing item from minutes: ACTION by ksinny at 16:43:48 : add relevant 'exclude's in the treefile to make sure we don't accidently pull in Python dependent packages in futture 16:44:00 also ksinny++ 16:44:01 jlebon: Karma for sinnykumari changed to 3 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 16:44:04 #action ksinny to add relevant 'exclude's in the treefile to make sure we don't accidently pull in Python dependent packages in futture 16:44:07 really great we've finally achieved this :) 16:44:19 ksinny: the functionality doesn't exist yet :) 16:44:33 though we *could* just spam exclude= in all the repo files for now 16:45:19 jlebon: ah, I read yesterday suggestion as if the treefile feature was already there 16:45:23 basically `exclude=python2 python3` should do it 16:45:39 jlebon: ah, thanks for the info. Will it be easier to implement or nees lots of changes? 16:45:49 needs* 16:45:54 misc thought: could we add tests in cosa to basically grep through the full package list and fail if it finds python? 16:46:11 ksinny: it shouldn't be very hard to implement 16:46:14 * ksinny is willing to try to implement that feature in rpm-ostree 16:46:29 wouldn't exclude succeed and silently remove python 16:46:52 jlebon: awesome, will create an issue and then we can discuss the implementation detail there 16:46:58 I'd think we'd want to fail hard rather than subtly suceed 16:47:36 ajeddeloh: exclude= essentially removes python from the pool of available pkgs. adding exclude= to the manifest though could have more powerful "assert-like" semantics 16:48:11 i think colin said it would fail with a depsolver error if some package required it but it couldn't be installed 16:48:28 oh, ok, great! nvm then 16:48:47 we'd have to verify that is true once we have the feature in place 16:49:11 is server-sive and client-side symmetrical on this? 16:49:15 *side 16:49:35 meaning could someone `rpm-ostree install python3` ? 16:49:47 that is, if we fill in "exclude"s, do we also want to stop client-side installs? 16:49:51 dustymabe: that, yes 16:50:02 no 16:50:05 hmm it won't no 16:50:23 that shouldn't impact client side since we will be defining exclude in manifest file while running rpm-ostree compose 16:50:48 (the server and client side stuff uses separate config formats for...historical reasons; really the only "data" passed between them is the rpmdb) 16:51:18 another good example of this is that disabling docs server side doesn't disable them client side 16:51:35 kaeso: would we actually want that anyway though? once you go pkg-layering it's up to you what's acceptable 16:52:22 jlebon: I was reading the GH ticket you posted and got confused. I don't think we want that, no. 16:52:33 do we have a follow-up ticket for this already that we can link to now? 16:53:17 otherwise, let's revisit next time. all ok to move on? 16:53:29 jlebon: I missed https://github.com/projectatomic/rpm-ostree/issues/1768 link earlier, just saw while looking at scrollback :) 16:53:44 #link https://github.com/projectatomic/rpm-ostree/issues/1768 16:53:48 #link https://github.com/projectatomic/rpm-ostree/issues/1768 16:53:54 #undo 16:53:54 Removing item from minutes: 16:54:37 #topic Pulp and OSTree support 16:54:39 #info possibly related to https://github.com/coreos/fedora-coreos-tracker/issues/169 16:54:55 I put this on here ... 16:55:11 I work on Pulp and we have a plugin for managing OSTree repositories 16:55:21 (sorry everyone, I seem to experience a bit of a Matrix-lag) 16:55:31 dkliban: go! :) 16:56:03 This OSTree plugin for PUlp 2 currently lets users mirror OSTree repos and copy branches of those repos from one repo to another in Pulp 16:57:05 We are now starting work on porting the OSTree plugin to Pulp 3. As we start this work, I was hoping that we could work close with FCOS to make sure that the features we are delivering are actually valuable 16:57:29 so as I was looking through the issue tracker I notied that there is some discussion about managing build artifacts 16:57:50 i thought that Pulp 3 could be a solution for managing OSTree build artifacts 16:58:32 Pulp 2 does not support any kind of upload use case for OSTreee, but I would like to chnage that for Pulp 3 16:59:11 i discussed this with jlebon yesterday and he suggested that i bring this up here today 16:59:31 dkliban: note there's Silverblue and IoT which also use vanilla OSTree 16:59:34 dkliban++ 100% thanks for bringing this up! 16:59:35 lorbus: Karma for dkliban changed to 1 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 17:00:08 flatpaks can also be distributed that way 17:00:15 easy OSTree mirroring i think is super useful 17:00:36 jlebon: but is there a need for content producers to be able to upload also? 17:01:01 does fedora-infra already hosts a Pulp 3 instance somewhere? 17:01:08 *host 17:01:10 kaeso: not yet 17:01:49 dkliban: yeah "uploading" OSTrees has always been tricky. if you can control the target host on which it is hosted, you can at least `ostree pull` from there. but otherwise, we've mostly relied on rsync 17:02:12 jlebon: is this due to the many number of files? 17:02:15 for fedora infra we've got a master ostree repo set up today 17:02:50 dkliban: mostly due to the lack of a "smart server". i.e. it's just files hosted over HTTP 17:02:59 dustymabe: do you rsync to that repo? 17:03:28 dkliban: we use NFS mounts and an `ostree pull-local` 17:03:54 to sync between ostree repos (we have a compose repo and a prod repo) 17:04:44 gotcha 17:05:30 so far it seems to me that we should continue to focus on mirroring 17:05:46 and in that case, how does cincinati come into play? 17:05:47 #info work on porting the Pulp 2 OSTree plugin to Pulp 3 has begun 17:06:44 dkliban: yeah, that's a good question. i think we're still figuring out where Cincinnati gets its data from (right kaeso?) for FCOS 17:07:51 dkliban: is anyone else asking you for the feature work you are proposing ? 17:08:08 (aside: how did mirroring work for CL? could people point the update agent at a local mirror?) 17:08:17 cincinnati only cares about ostree commit-hash, so where the repo lives should not be relevant to it 17:08:31 dustymabe: this is for Foreman (Satellite 6) 17:09:32 jlebon: for update payloads, there wasn't really a good way to do mirroring 17:09:49 dustymabe: we initially thought that since atomic host was going away that we would drop support for OSTree, but we are not realizing that we will want to let users manage OSTree content so they can have local copies of Fedora Siliverblue repo and Fedora CoreOS repo 17:09:55 jlebon: might be possible if you work at it, but we never tried to support it 17:10:25 dkliban: ok - yeah it should be pretty easy to do the mirror work (as you mentioned already done in pulp2) 17:10:34 kaeso: it seems like there are two use cases: one where clients keep contacting the upstream cincinnati server, but download from a local mirror, and another where you'd want to set up your own cincinnati server for your own cadence/rollout strategies? 17:10:44 I think FCOS is anyway different than CL one 17:10:48 I wouldn't spend time adding new features unless someone requests it 17:11:08 on FCOS there a matrix of custom-cincinnati and custom-ostree-repo 17:11:14 dustymabe: i agree ... that's why i wanted to talk to you all to see if you are interested in these features 17:11:23 and all the 4 combinations are client-side configs 17:11:27 kaeso: right, agreed 17:12:31 looking at the time, lets move on to the next topic 17:12:36 alright ... i think i got the answers i wanted out of this 17:12:44 i ageee ... let's go on to the next topic 17:12:58 thanks again for brining this up dkliban! 17:12:59 dkliban: thanks again for bringing up the subject! 17:13:05 #topic etcd-wrapper, flannel-wrapper equivilents? 17:13:07 thanks dkliban 17:13:14 jlebon: if there is some more details which seems unclear, let's discuss that out of band 17:13:27 basvdlei brought this up in IRC the other day 17:13:31 ajeddeloh that's yours I guess? 17:13:44 do we want to ship these wrappers in fcos? 17:14:11 on CL they're used as sort of a "happy path" for making setting up etcd and flannel 17:14:39 * ajeddeloh wonders if they can literally be copied over and used as is 17:15:09 likely not 17:15:14 does it make it super easy to join a cluster ? 17:15:16 they all use rkt inside 17:15:21 links? 17:15:25 https://github.com/coreos/coreos-overlay/tree/master/app-admin/etcd-wrapper 17:15:34 was going to say, don't those use stage 1 flys? 17:15:37 https://github.com/coreos/coreos-overlay/tree/master/app-admin/flannel-wrapper 17:15:40 #link https://github.com/coreos/coreos-overlay/tree/master/app-admin/etcd-wrapper 17:15:48 but even if we switch to podman, I don't think we have to 17:16:05 #link https://github.com/coreos/coreos-overlay/tree/master/app-admin/flannel-wrapper 17:16:53 can't speak to the flannel part 17:16:56 yeah - could easily add them in the future if we find it necessary 17:16:56 etcd-wrapper sort of encourages an etcd on every node, which isn't really the right thing 17:17:25 on CL, they were part of bigger story of "cluster orchestration" 17:17:39 together with kubelet-wrapper (and fleet?) 17:18:27 we don't have a really coherent story there yet, but it'll look different than on CL 17:18:32 bgilbert: would you prefer to not have an etcd-wrapper? 17:18:39 I think so? 17:18:49 or just document "hey don't do this on all your nodes" 17:18:53 "run this container" seems fine, and people will be doing plenty of it 17:19:02 ajeddeloh: there is also "etcd2 vs etcd3" 17:19:18 * ajeddeloh hopes people will be using 3 on fcos 17:19:49 CL originally shipped etcd in the OS, and the wrapper was a way to move it out 17:20:10 other question is: regardless of *-wrapper, do we want to add fcct sugar for etcd, flannel, etc 17:20:30 and I don't know that we want to couple as tightly to etcd anymore 17:20:59 ajeddeloh: heh. if we don't, CL migration gets harder 17:21:03 ajeddeloh: that seems a saner thing to do (generating the whole unit file) 17:21:28 +1 I'm down for "not shipping wrappers but generating unit files with fcct" 17:21:50 which does mean we can't update them later 17:22:08 in sum, I guess I'm on the fence 17:22:30 do we need a vote on that? We can probably just info it.. 17:23:00 updating a unit file doesn't sound too painful ...though I guess we don't have the mco here 17:23:29 I see them as "containerize user apps", thus the user owns it lifecycle (which includes updates and migrations) 17:23:38 *containerized 17:23:41 *its 17:23:57 mrguitar: in this case it would mean reprovisioning 17:24:23 more to the point, we couldn't do it on behalf of users 17:24:27 because the user owns their config 17:25:12 noting the time. 5min left and one more topic to go! 17:25:16 I think I'm ok with that 17:25:31 I'll create a tracker issue nonethekess 17:25:38 ajeddeloh++ 17:25:38 kaeso: Karma for ajeddeloh changed to 4 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 17:26:07 maybe think about what the use cases are, then what's needed to support those use cases 17:26:20 #action ajeddeloh to create a tracker issue for the *-wrapper topic 17:26:42 #chair geoff- 17:26:42 Current chairs: ajeddeloh bgilbert dkliban dustymabe geoff- jlebon ksinny lorbus lucab mnguyen mrguitar redbeard slowrie 17:27:11 moving on... 17:27:14 #topic Conditionally disabling SMT by default 17:27:33 bgilbert is that you? 17:27:44 yep 17:28:01 with L1TF and MDS, enabling SMT by default doesn't seem like an entirely safe choice 17:28:42 SMT is unsafe on certain processors (a great many of them) and certain workloads. for entirely trusted workloads, it's fine, and provides some performance advantage 17:29:34 for existing systems (e.g. the Container Linux installed base) it's difficult to just disable SMT via an update, since a) performance hit and b) only needed for certain workloads. 17:29:39 we went back and forth on this w/ rhel, and ultimately decided to leave it on and warn users of the pros & cons 17:29:47 but we're building a new OS and we have the opportunity to make different choices. 17:29:59 the kernel supports two approaches: 17:30:02 right 17:30:03 `nosmt` disables unconditionally 17:30:28 `mitigations=auto,nosmt` disables when needed to mitigate something 17:30:49 I agree with conditionally disabling it provided we provide sufficient documentation about the potential ramifications (e.x.: SMT might be shut off post-updates if newer kernels determine you're vulnerable), and how to enable/disable unconditionally 17:30:54 that performs better (e.g., at the moment it leaves SMT enabled on AMT CPUs) 17:31:10 but it also allows future updates to _change_ whether SMT is enabled. 17:31:15 this of course is made difficult by the fact that we can't change it on first boot, need to reboot to take effect or disable later in boot unconditionally (there is no disable-if-affected option later in boot) 17:31:37 so yeah, we'd want to document it, plus the force-on and force-off options 17:31:57 but I think "we will keep you secure even if that means disabling HT" is a reasonable policy to set from day 1 17:32:30 ajeddeloh: we might want to explore adding a mechanism to reboot, post-Ignition, from inside the initramfs 17:32:38 ajeddeloh: could we potentially add a hook in the initramfs we could jerry-rig into rebooting there that wouldn't break ConditionalFirstBoot? 17:32:51 right 17:33:22 (for context: rebooting halfway into the real root can cause later ConditionFirstBoot units never to run at all) 17:34:00 anybody want to info how we'll continue on this? time's up I'm afraid.. 17:34:39 create a tracker issue? 17:34:39 #action bgilbert will file a ticket on conditionally disabling SMT, with links to PRs 17:34:44 +1 17:34:53 thank you! 17:34:54 #endmeeting