16:32:07 #startmeeting fedora_coreos_meeting 16:32:07 Meeting started Wed Jan 16 16:32:07 2019 UTC. 16:32:07 This meeting is logged and archived in a public location. 16:32:07 The chair is bgilbert. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:32:07 Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:32:07 The meeting name has been set to 'fedora_coreos_meeting' 16:32:20 #topic roll call 16:32:20 .hello lorbus 16:32:21 lorbus[m]: lorbus 'Christian Glombek' 16:32:23 .hello2 16:32:24 bgilbert: bgilbert 'Benjamin Gilbert' 16:32:34 .hello sinnykumari 16:32:35 ksinny: sinnykumari 'Sinny Kumari' 16:32:36 .hello rfairley 16:32:38 rfairley: rfairley 'None' 16:32:40 .fas jasonbrooks 16:32:41 jbrooks: jasonbrooks 'Jason Brooks' 16:32:45 .hello2 16:32:46 ajeddeloh: ajeddeloh 'Andrew Jeddeloh' 16:32:57 .hello2 16:32:58 dustymabe: dustymabe 'Dusty Mabe' 16:33:30 #chair lorbus[m] ksinny rfairley jbrooks ajeddeloh dustymabe 16:33:30 Current chairs: ajeddeloh bgilbert dustymabe jbrooks ksinny lorbus[m] rfairley 16:33:33 .hello2 16:33:34 miabbott: miabbott 'Micah Abbott' 16:33:40 .hello mnguyen 16:33:41 mnguyen_: mnguyen 'Michael Nguyen' 16:34:29 #chair miabbott mnguyen_ 16:34:29 Current chairs: ajeddeloh bgilbert dustymabe jbrooks ksinny lorbus[m] miabbott mnguyen_ rfairley 16:34:59 #topic Action items from last meeting 16:35:30 * dustymabe to open design doc PR to close out openstack cloud agent issue #68 16:35:37 * ajeddeloh to add igntion 3.0.0 spec to roadmap 16:35:42 * bgilbert to coordinate with mattdm about FCOS metrics 16:35:45 .hello lucab 16:35:46 kaeso: lucab 'Luca Bruno' 16:35:46 .hello sayanchowdhury 16:35:48 #info bgilbert is coordinating with mattdm 16:35:49 sayan: sayanchowdhury 'Sayan Chowdhury' 16:35:58 #chair kaeso sayan 16:35:58 Current chairs: ajeddeloh bgilbert dustymabe jbrooks kaeso ksinny lorbus[m] miabbott mnguyen_ rfairley sayan 16:35:59 .hello flexo3001 16:36:00 flexo3001: flexo3001 'Marco Kundt' 16:36:10 #info dustymabe opened #117 for design doc PR for openstack (#68) 16:36:11 #chair flexo3001 16:36:11 Current chairs: ajeddeloh bgilbert dustymabe flexo3001 jbrooks kaeso ksinny lorbus[m] miabbott mnguyen_ rfairley sayan 16:36:18 #link https://github.com/coreos/fedora-coreos-tracker/pull/117 16:36:27 roadmap is PRd as of literally right now 16:36:31 https://github.com/coreos/fedora-coreos-tracker/pull/119 16:36:46 #info ajeddeloh PR'd roadmap for Ignition 3.0.0 spec 16:36:53 #link https://github.com/coreos/fedora-coreos-tracker/pull/119 16:37:01 bgilbert: oh hi i'm actually here :) 16:37:10 sorry for being awol on this 16:37:24 mattdm: hey, no problem! and I owe you another email :-) 16:37:27 #chair mattdm 16:37:27 Current chairs: ajeddeloh bgilbert dustymabe flexo3001 jbrooks kaeso ksinny lorbus[m] mattdm miabbott mnguyen_ rfairley sayan 16:37:35 are you in brno next week? 16:37:49 .hello2 16:37:50 jlebon: jlebon 'None' 16:38:15 mattdm: nope, unfortunately 16:38:27 #chair jlebon 16:38:27 Current chairs: ajeddeloh bgilbert dustymabe flexo3001 jbrooks jlebon kaeso ksinny lorbus[m] mattdm miabbott mnguyen_ rfairley sayan 16:38:45 ah well. I promise to respond to your email as soon as I get it :) 16:38:52 mattdm: :-) 16:39:16 #topic Autologin policy for cloud platform consoles 16:39:21 #link https://github.com/coreos/fedora-coreos-tracker/issues/114 16:39:33 I just wanted to bounce this around a bit 16:39:50 several clouds provide authenticated access to the serial or VGA console of a machine 16:39:57 which is nice for debugging, if you can actually log in. 16:40:02 FCOS machines will generally not have passwords. 16:40:14 we could _technically_ autologin on the console, since console access is authenticated 16:40:21 (by the cloud) 16:40:41 which provides better UX and worse security. 16:41:04 in the sense that a) we're very obviously trusting the cloud not to do things on the VM 16:41:21 and b) users may be surprised that the cloud is granting console access to some set of users 16:41:35 my concern is that users may have different policies for who has access to cloud things and who has access to contents of running machines 16:41:58 bgilbert: didn't we explicitly drop the "autologin" knob few months ago? I remember some previous discussion about that 16:42:10 kaeso: did we? I thought it was an oversight 16:42:29 (and thus filed https://github.com/coreos/fedora-coreos-tracker/issues/112) 16:42:57 bgilbert: let me try to verify my memory on that (and sorry didn't see the ticket this morning) 16:43:18 kaeso: np 16:43:42 while he's looking: 16:44:08 I think we should probably not enable autologin 16:44:13 of course we'd have docs about how to change that 16:44:23 but interested in views / use cases 16:44:38 re: 'Most Fedora CoreOS machines will not enable password-based login' 16:44:42 ajeddeloh: are there clouds that differentiate between console access and e.g. nuking the VM? 16:44:59 a user could set a password if they want? 16:45:12 also could make the pw login only valid on serial console (i.e. not via SSH) 16:45:27 I don't know, although GCEs controls are fine tuned enough I wouldn't be surprised 16:46:05 dustymabe: yes and yes 16:46:11 but my concern is being able to nuke VMs does not necesarily mean you have access to their contents 16:46:35 you could have permissions to spin up/down VMs but not be allowed to look at the data stored on it 16:47:52 disabled by default with instructions on how to turn it on SGTM 16:48:07 +1 16:48:37 kaeso: any luck? 16:48:51 so is this what we want? 16:49:01 +1 16:49:02 1. implement the feature, disabled by default 16:49:15 2. people can set a user pw and disable ssh PW if they want 16:49:22 3. people can enable autologin if they want 16:49:24 bgilbert: neither google nor my inbox have anything, so let's just say we didn't do/discuss that on purpose before 16:49:39 kaeso: sgtm 16:49:59 ok, just read the actual issue. so this is about making it easy on the kernel cmdline vs just documenting something like https://github.com/coreos/coreos-assembler/blob/16763204e836f8682e3ea0034794d8ab3206d3f6/src/cmd-run#L138 16:50:11 discussion on whether we _should_ have an autologin kernel command line parameter can go in https://github.com/coreos/fedora-coreos-tracker/issues/112, then 16:50:48 jlebon: https://github.com/coreos/init/blob/85a35815c7ffb14d6e398da950502dbabc8f4291/systemd/system-generators/coreos-autologin-generator 16:50:49 jlebon: #112 is about the kcmdline, #114 is about defaults on the cloud 16:51:06 we can talk about #112 explicitly if we'd like 16:51:28 bgilbert: +1 .. didn't realize you had also opened #112 16:51:36 and it's good to break apart those two topics 16:51:39 gotcha, splitting the discussion makes sense 16:51:47 dustymabe: +1 to talking about it here? 16:52:15 bgilbert: only speaking about #112, I think it would be valuable to have that mechanism file-based and documented 16:52:46 having a kernel param would be nice. definitely had times when I'd just nuke and reprovision a node with pw auth just to log in on the serial 16:52:49 bgilbert: (even if we don't enable it by default on any cloud) 16:53:01 #topic Consider adding kernel command-line argument for automatically logging in on console 16:53:08 #link https://github.com/coreos/fedora-coreos-tracker/issues/112 16:53:09 :-D 16:53:42 bgilbert: except for historical precedent, any reason for having it on the cmdline instead of a file touched/removed by Ignition? 16:54:04 kaeso: well 16:54:10 (assuming both are technically feasible, I haven't looked into that) 16:54:21 kaeso: all it's really doing is creating a dropin, which is technically a "file written by Ignition" 16:54:25 and we could document that 16:54:47 (in the usual Container Linux way of documenting little snippets to be written by an Ignition config) 16:54:58 you may want to be able to do it from the kernel command line anyway 16:55:11 we could provide CT sugar for it, even 16:55:20 but yes, also, it's convenient for breaking into machines 16:55:27 to have it on the kcmdline 16:55:28 like if something breaks and its the only way in 16:55:48 otherwise it's "rd.break" + "touch /sysroot/etc/let_me_in" 16:55:59 for bare metal you may have cases where physical access means you should be able to log in 16:56:08 "single" doesn't work because there's no root pw 16:56:18 hmm I guess with the rd.break there are other ways in 16:56:29 ajeddeloh: that is scoped to "if something breaks after we switched out of initramfs" no? 16:56:48 yeah 16:56:56 rd.break solves my concern 16:57:19 ajeddeloh: I was arguing against that, actually 16:57:21 it's clunky 16:57:47 (as a general trend, I'd like to see less kernel parameters instead of more) 16:58:42 another approach is to make "single" usable 16:58:46 by not requiring a pw 16:58:54 I may be wrong, bu my gut feeling is that the coreos-autologin-generator predates ignition 16:59:02 it has different semantics but is arguably better for emergency repairs 16:59:10 kaeso: I assume so 16:59:36 if that is true, the generator was needed because cloud-init was running too late for that 16:59:45 oh 16:59:56 now ignition runs before that generator 17:00:20 i added some questions to the ticket 17:00:20 and if the generator doesn't run, it means ignition failed 17:00:22 using Ignition has the problem that it only works on first boot, so emergency repairs are clunky 17:01:02 ajeddeloh: fair 17:01:26 do we agree that the kcmdline version is best for emergency repairs? 17:01:37 i.e., if someone wants autologin every time, they should use Ignition to configure it 17:01:47 ("best for" = "mainly useful for") 17:02:04 because if so, I think there's a good argument for fixing "single" rather than adding a new argument 17:02:10 I'm actualy not against kcmdline args as long as they're namespaces (i.e. ignition.X, coreos.X, etc) 17:03:29 bgilbert: if the main reason for the kcmdline is emergency, then yes it's basically "single" I guess 17:04:49 "single working" + "config-doc snippet to enable autologin" would work for me 17:05:05 I think historically autologin has also been used on platforms that haven't had functioning cloudinit or Ignition 17:05:50 but at this point I think it makes sense to call a platform without Ignition support "not actually an FCOS platform" 17:06:04 * bgilbert hides ISO boot under the carpet 17:06:04 i could see it being useful for spinning up an image in qemu and not wanting to figure out how to provide an ignition config 17:06:42 we often provide images to (X developer, where X is systemd or selinux or other) and ask them to try to investigate something 17:06:45 dustymabe: on CL we have the wrapper script, coreos_production_qemu.sh, which wraps that for you 17:06:59 finds your local SSH key and injects it 17:07:18 bgilbert: yeah.. typically people already have their own scripts for spinning up VMs and sometimes they balk at things like that 17:07:23 right 17:07:43 * dustymabe bees quiet now :) 17:08:20 so it seems like there's interest in dropping the karg but we're not sure if we can? 17:08:56 we can start without and add it later as soon as we find a situation where we really need it? 17:09:02 kaeso: was just about to say that 17:09:10 we can add it at any time, but we can ~never remove it 17:09:47 related feature we had in AH: https://www.projectatomic.io/blog/2016/03/introducing-atomic-devmode/ 17:10:35 it was directed at folks who just wanted to give it a spin without the cloud-init dance 17:10:37 jlebon: oh, interesting 17:10:58 okay, I'm thinking we should move on for time reasons 17:11:19 proposal: I'll write up the discussion in the bugs and we can continue discussion there? 17:11:29 ack 17:11:43 +1 17:11:45 (and thanks for summarizing) 17:12:57 #action bgilbert to summarize discussion in #112 and #114 17:13:06 #topic roadmap 17:13:10 #link https://github.com/coreos/fedora-coreos-tracker/blob/master/ROADMAP.md 17:13:18 dustymabe: want to take this? 17:13:55 yep 17:14:22 ok I sent in an update earlier today 17:14:31 let's review this week and then we'll talk about next week 17:14:59 2019-01-14 17:15:02 H - finalize strategy,collaborate Network Management #24 17:15:03 gaps identified feature work requested 17:15:06 H - finalize strategy Kubernetes/OKD strategy #93 17:15:08 M - finalize strategy Collect metrics from Fedora CoreOS machines design #86 17:15:10 M - complete bare metal installer: POC #91 17:15:12 Proof of concept complete 17:15:38 there is one that I marked as already complete with was H - investigate no cloud agents for digital ocean 17:15:44 thanks bgilbert ^^ 17:16:06 so let's go through them one by one 17:16:20 minitopic: H - finalize strategy,collaborate Network Management #24, gaps identified feature work requested 17:16:38 kaeso: did we do any followup with NM team to see where we stand? 17:17:10 not yet 17:17:20 worth noting that this is starting to block cloud support on some platforms :-( 17:17:49 ok sorry, will do that tomorrow morning 17:18:04 bgilbert: can you add the 'blocked' label and a comment to any issues that might be blocked? 17:18:15 kaeso: no need to apologize :) 17:18:23 dustymabe: sure 17:18:32 #action bgilbert to add "blocked" label to issues blocked on #24 17:18:53 minitopic: H - finalize strategy Kubernetes/OKD strategy #93 17:19:09 I think jbrooks was doing some investigation with systemd portable services 17:19:14 #action kaeso to follow up with NM team 17:19:56 That's right, I commented over in the system containers replacement thread https://github.com/coreos/fedora-coreos-tracker/issues/37#issuecomment-454550323 17:20:13 But the tldr is that I don't think portable services are quite ready for this 17:20:31 kaseo, dustymabe, jbrooks: we should get Lennart to have a quick look at the things that are missing to make it work 17:20:38 jbrooks: one thing I wonder is if the newer versions of systemd that just came out make any progress there ? 17:20:46 selinux support is one big thing 17:21:15 But then there's the bunches of bind mounts and stuff that need to be correct -- this is the sort of thing that made the system container a pain 17:21:28 It's just a really non-standard way to run the kubelet 17:21:53 jbrooks: for some reason I thought some of that pain would go away with portable services 17:22:00 I thought they weren't really "containers" 17:22:10 Nope, they are still contained 17:22:26 It's a lot like a more convenient, but less mature system containers 17:22:27 well "contained" 17:22:41 yeah 17:22:50 it's mostly just a chroot, plus a little extra stuff 17:23:29 ajeddeloh: portables services do all relevant namespacing dances 17:24:02 huh 17:24:22 yeah i haven't had a chance to play with them myself I guess I should 17:24:36 jbrooks: can we try to make sure there are issues opened for at least the selinux thing 17:24:36 I got the kubelet running, but not working 17:24:50 also maybe try newer systemd 17:24:55 There's an issue, and lennart was like, I need help w/ that 17:24:58 i think 240 or 241 just hit rawhide 17:25:07 IIRC, the kubelet "--containerized" flag was going to be deprecated as not really working 17:25:09 And I tried 240 17:25:10 so there is an open issue? 17:25:35 looking for the issue 17:25:53 thanks.. when you find it please add the link to the comment you alrady made 17:26:06 ok moving on.. to the next minitopic 17:26:19 minitopic: M - finalize strategy Collect metrics from Fedora CoreOS machines design #86 17:26:31 looks like mattdm and bgilbert have some circling to do on this 17:26:40 * bgilbert circles 17:26:49 which they mentioned in action items topic earlier 17:26:54 so.. next topic? 17:27:08 minitopic: M - complete bare metal installer: POC #91, Proof of concept complete 17:27:11 heh, actually not 17:27:19 #action bgilbert to email mattdm again 17:27:34 #undo 17:27:34 Removing item from minutes: ACTION by bgilbert at 17:27:19 : bgilbert to email mattdm again 17:27:39 #action bgilbert to reply to mattdm's email 17:27:43 better phrasing is better 17:28:05 dusty got sidetracked with a coreos-assembler issue last week, so will try to make progress again on #91 this week 17:28:22 ok so that's all for this week.. some of the topics will move forward (already discussed) 17:28:32 let's look at next week (which also happens to be devconf) 17:28:50 2019-01-21 17:28:51 Week of Devconf.cz 17:28:54 H - collaborate fedora releng integration #44 17:28:56 H - investigate no cloud agents #95 17:28:57 vmware #70, open new tickets for work items 17:29:00 M - finalize strategy burndown python dependencies #92 17:29:01 L - complete merge of fedora-toolbox and coreos-toolbox efforts #90 17:29:04 H - investigate no cloud agents #95 17:29:06 gce #67, open new tickets for work items 17:29:07 H - finalize strategy ostree mirroring for better UX #54 17:29:15 I moved ostree to next week (sinny will talk to fedora infra to solidify our approach there) 17:29:23 ksinny: good with that ? 17:29:24 +1 17:29:32 yup 17:29:39 and ksinny is already working on #92 17:29:50 I'm going to try to get together with releng for #44 17:30:23 ajeddeloh: has volunteered to look at #67, but if he gets bogged down we can have someone else that frees up to look at it 17:30:37 bgilbert: volunteered for the pain that is #70 17:30:52 hooray 17:31:04 bgilbert and I have a meeting with the gcloud folks today 17:31:05 i think yzhang was going to look at #90 but he is AFK today 17:31:13 that is low priority so we can move it if we need to 17:31:27 we can talk about gcloud support for fcos there 17:31:40 cool cool 17:31:58 ok any more comments on roadmap? 17:32:10 sorry bgilbert for running long :( 17:32:20 * dustymabe gives the reins back to bgilbert 17:32:30 dustymabe: I didn't give you much time :-( 17:32:31 #topic Open Floor Super-Quick 17:33:09 closing meeting in 60s 17:33:17 #info sayan opened a package review for mantle in fedora: https://bugzilla.redhat.com/show_bug.cgi?id=1665480 17:33:28 i'm working through the review with him 17:33:37 awesome 17:34:13 #endmeeting