16:30:41 #startmeeting fedora_coreos_meeting 16:30:41 Meeting started Wed Oct 20 16:30:41 2021 UTC. 16:30:41 This meeting is logged and archived in a public location. 16:30:41 The chair is dustymabe. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions. 16:30:41 Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:30:41 The meeting name has been set to 'fedora_coreos_meeting' 16:30:45 #topic roll call 16:30:48 .hi 16:30:49 dustymabe: dustymabe 'Dusty Mabe' 16:31:05 .hello sohank2602 16:31:06 skunkerk: sohank2602 'Sohan Kunkerkar' 16:31:10 .hello2 16:31:11 jlebon: jlebon 'None' 16:31:20 .hello2 16:31:21 fifofonix: fifofonix 'Fifo Phonics' 16:31:30 .hi 16:31:33 ueno: ueno 'Daiki Ueno' 16:31:48 .hi 16:31:49 scorreia: scorreia 'Sergio Correia' 16:31:51 .hello2 16:31:52 jaimelm: jaimelm 'Jaime Magiera' 16:31:59 .hello siosm 16:32:00 travier: siosm 'Timothée Ravier' 16:32:06 .hi 16:32:07 ravanelli: ravanelli 'Renata Andrade Matos Ravanelii' 16:32:13 .hi 16:32:14 walters: walters 'Colin Walters' 16:32:40 #chair fifofonix ueno scorreia jaimelm travier ravanelli walters 16:33:03 .hi 16:33:04 lucab: lucab 'Luca BRUNO' 16:33:05 .hi 16:33:06 .ping 16:33:08 bgilbert: bgilbert 'Benjamin Gilbert' 16:33:11 pong 16:33:22 #chair lucab bgilbert 16:33:23 .hello miabbott 16:33:24 miabbott: miabbott 'Micah Abbott' 16:33:41 hmm, weird not sure why it's not responding to the #chair's 16:33:57 #chair miabbott 16:34:12 ohhh. dustymabe do you have to #chair me first? 16:34:15 #chair dustymabe 16:34:15 Current chairs: dustymabe 16:34:34 .chair jlebon 16:34:35 jlebon is seated in a chair with a nice view of a placid lake, unsuspecting that another chair is about to be slammed into them. 16:34:48 #chair jlebon 16:34:48 Current chairs: dustymabe jlebon 16:34:53 #chair fifofonix ueno scorreia jaimelm travier ravanelli walters 16:34:53 Current chairs: dustymabe fifofonix jaimelm jlebon ravanelli scorreia travier ueno walters 16:34:55 #chair lucab bgilbert 16:34:55 Current chairs: bgilbert dustymabe fifofonix jaimelm jlebon lucab ravanelli scorreia travier ueno walters 16:34:58 #chair miabbott 16:34:58 Current chairs: bgilbert dustymabe fifofonix jaimelm jlebon lucab miabbott ravanelli scorreia travier ueno walters 16:35:14 🎉 16:35:20 .hello mnguyen 16:35:20 mnguyen: mnguyen 'Michael Nguyen' 16:35:30 #chair mnguyen 16:35:30 Current chairs: bgilbert dustymabe fifofonix jaimelm jlebon lucab miabbott mnguyen ravanelli scorreia travier ueno walters 16:35:35 Are these stackable, metal chairs? 16:35:44 nice turnout 16:35:47 * jmarrero watches in suspense 16:36:01 jaimelm: i'm imagining the metal chairs from WWF 16:36:13 right, exactly 16:36:25 last meeting we ran out of time before getting to https://github.com/coreos/fedora-coreos-tracker/issues/982, so let's try to have it near the top of the stack today 16:36:28 .hi 16:36:29 saqali: saqali 'Saqib Ali' 16:36:39 lucab: yep - it's at the top of the agenda 16:36:55 danke :) 16:37:02 #topic Action items from last meeting 16:37:23 * dustymabe to write docs for rpi4 including updating eeprom 16:37:24 * bgilbert to write docs for switching to FAT32 16:37:47 re-action for me :) 16:37:56 #action dustymabe to write docs for rpi4 including updating eeprom 16:38:07 .hello2 16:38:08 davdunc: davdunc 'David Duncan' 16:38:17 luckily they released a new version of the firmware so I can finish and publish the docs now 16:38:36 #info bgilbert filed https://github.com/coreos/fedora-coreos-docs/issues/326 for switching to FAT32 16:38:41 #chair davdunc 16:38:41 Current chairs: bgilbert davdunc dustymabe fifofonix jaimelm jlebon lucab miabbott mnguyen ravanelli scorreia travier ueno walters 16:38:54 +1 lucab, let's start with the Keylime discussion 16:39:20 bgilbert: were you planning to write up the docs too? 16:39:27 yup, it's on my list 16:39:38 ok cool. since there's an issue I won't re-action it 16:39:43 ok, let's move on! 16:39:44 +1 16:39:55 #topic New Package Request: rust-keylime_agent 16:39:59 #link https://github.com/coreos/fedora-coreos-tracker/issues/982 16:40:25 who wants to introduce this? 16:40:55 I can: it's an attester program of the keylime stack, which verifies that the running system in a good state, against remote verifier 16:42:07 using TPM and IMA - I guess it would be particularly useful to monitor IoT devices 16:44:18 i forgot a bit the context around keylime+OSTree, is the idea that only files in /usr will be measured? 16:45:39 It's a bit complex because there are two use cases at play here 16:46:11 The first one is remote attestation at boot time. The second one is integrity monitoring happening later 16:46:14 https://keylime-docs.readthedocs.io/en/latest/user_guide.html 16:46:47 AFAIK, there is nothing preventing keylime from running from a privileged container with a lot of access to the filesystem and devices, etc. 16:46:56 so reasonably this works only after 1) the system is normally booted (i.e. outside of initramfs), and 2) with the network fully up, correct? 16:47:48 the main question is more about how early do we want to be able to perform an attestation against a remote server. If the answer is before we pull any container image, then we need this agent on the node, otherwise we can use a container image 16:48:17 AFAIK yes, this is post initramfs and network up 16:49:38 worth noting things might've already been fetched over the network already (notably Ignition config and remote resources within the config) 16:50:47 Let me rephrase/clarify that as it's not about fetching but potential authorization: a setup could require all container images to be fetched from a given registry that would only allow attested node to fetch images 16:51:21 So you could have a node be able to fetch images from the cluster registry only after it has been attested as good 16:52:04 Of course the direct workaround here is to host this specific container in a public registry 16:52:51 The ignition config hash could also be included in the attestation too 16:53:19 travier: good point 16:53:38 ueno, scorreia I am making sense or is this not how keylime works right now? 16:54:06 travier, that makes sense to me 16:55:30 i don't quite know why someone would want to gate something as basic as fetching container images on attestation though 16:56:11 * jlebon has to go to another meeting, but bgilbert will take over 16:56:47 Well, we could have a feature that would prevents a node from doing anything in a cluster before it attest that it is in a good state 16:57:20 as it doesn't seem to be involved in bootstrapping the OS, it sounds like it could be easily layered in possibly? 16:57:23 This was a feature in CoreOS Tectonic 16:57:25 interesting 16:59:08 lucab: yeah that's what I'm wondering 16:59:28 well but it wasn't gating pulling all containers 16:59:28 right, Tectonic replaced the kubelet node join with attestation, but at least OpenShift today hard requires pulling container images before kubelet runs today, so any attempt to replicate that would require the ability to pull at least a subset of images 17:00:17 anyways I think my bottom line here is I'm not hard opposed but I haven't seen any attempt to try to containerize, and I think that should be the default first path 17:00:44 also because each container image has different authZ policies, so the base infra/bootstrapping ones are more freely pull-able 17:02:17 true, we should try container first and include it only if we find a case where this does not work 17:02:38 anyone want to do a #proposed? 17:03:04 and yes, the only case I can come up where we need this on the host is easily worked around 17:04:00 I might be missing some use cases) 17:04:02 (* 17:06:13 #proposed: We do not include the keylime agent in the host for now and will investigate if a container based deployment could satisfy the use cases. 17:06:40 who is going to investigate? 17:06:44 dustymabe: +1 17:07:01 we know where those unassigned action items go 17:07:02 travier: ack 17:07:23 in particular, are we planning to handle that in the FCOS WG? 17:07:36 Well, it's on my todo list to work on Keylime but I have not been able to get to it for now 17:08:13 * jdoss waves 17:08:50 is it something we could ask the requestors to investigate? 17:09:00 maybe in collaboration with travier 17:09:03 sorry ueno and scorreia as I had implied that we could include it before and we are going forward with another option now 17:09:24 dustymabe: yeah, we could investigate in collaboration with travier 17:09:24 travier, np, ok, it seems worth trying :-) 17:09:26 personally, I'd say "we believe this to fit better in a container. If that path does not get explored upstream, users can still layer in the RPM" 17:09:42 +1 lucab 17:10:01 want to do a new #proposed ? 17:10:10 *by upstream/community 17:10:38 #proposed we believe this to fit better in a container. If that path does not get explored by upstream/community, users can still layer in the RPM 17:11:53 ack - would be nice to understand this use case a bit more (maybe even add some docs?) 17:11:59 (to be clear, we may be wrong and this may be impossible to containerize) 17:12:00 ack 17:12:03 +1 17:12:39 ueno: scorreia: would you be able to investigate containerization as an option (along with travier) - package layering as a fallback - and maybe add some docs 17:12:53 would be nice to have our users use it, but they won't if they don't know it exists or how 17:13:22 dustymabe, sure 17:13:53 walters, +/-1? 17:14:00 +1 17:14:03 +1 17:14:29 We can revisit if it turns out some things can not work from inside a container 17:14:45 #agreed We believe this to fit better in a container. If that path does not get explored by upstream/community, users can still layer in the RPM. 17:14:57 (sorry I got context switched) but skimming back +1 17:15:07 looking for a topic that'll fit in 10 minutes 17:15:15 dustymabe, NIC renaming udev rules maybe? 17:15:51 WFM 17:16:01 might be good to get an update on the final F35 change 17:16:02 #topic forwarding NIC renaming udev rules from the initramfs 17:16:07 #undo 17:16:07 Removing item from minutes: 17:16:14 #topic F35: CHANGE: More flexible use of SSSD fast cache for local users 17:16:20 #link https://github.com/coreos/fedora-coreos-tracker/issues/875 17:16:45 travier, looks like you're the assignee 17:17:27 Still working on that one. Hopefully will close it this week. Nothing blocking us so far 17:17:30 +1 17:18:09 #topic forwarding NIC renaming udev rules from the initramfs 17:18:14 #link https://github.com/coreos/fedora-coreos-tracker/issues/553 17:18:26 dustymabe? 17:18:38 yep 17:18:45 OK for this one, we had a partner engineer working on OpenShift that hit this. They are seeing very inconsistent NIC naming with some of the new 5G ethernet cards they are using so they want to rename NICs based on MAC address and they are hitting on this issue. 17:19:27 right now they are going to have to create an ignition config per node to get the NIC renaming to work, but they think it's worth it because of how bad the NIC naming consistency is for them 17:20:05 the one trip up is if doing an install via PXE using the kargs then they won't have anything in that install boot that puts in place the NIC renaming in the real root 17:20:22 so currently: 17:20:24 1. set kargs 17:20:39 2. install boot ignition config (does this mean they can't use kargs to do the install too)? 17:20:54 3. first boot igition config 17:21:05 it sounds like the root cause is that stable NIC naming is broken for their case? 17:21:30 bgilbert: yes, it's the cause for them wanting to use NIC renaming 17:21:43 but you could have other reasons for wanting NIC renaming 17:22:34 well, sure, but do we? 17:22:42 it looks like the last decision was "don't pursue for now" 17:22:43 #link https://github.com/coreos/fedora-coreos-tracker/issues/553#issuecomment-658949868 17:23:08 "do we"? I think that's up to the user if they want to rename their NICs or not. 17:23:47 asking because, before this case came up, the bug was closed with "we'll reopen if we get a lot of user input" 17:23:48 https://github.com/coreos/fedora-coreos-tracker/issues/553#issuecomment-662554147 17:24:11 and absent that, I'm wondering whether we should just fix the bug 17:24:36 right, here's user input - and also one more data point I hadn't considered before, which is the PXE install extra ignition config needed 17:24:52 it sounds like a systemd/udev RFE where there should be some generator inspecting the cmdline and generating the udev rules? 17:24:55 bgilbert: yeah, I don't doubt we should fix the bug, not ure it's well defined, we could ask 17:25:32 lucab: it's kind of a lost feature in the transition from legacy network stack to NM IIRC 17:26:20 let me see if I can find the link 17:26:53 I think it was https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/issues/479 17:27:17 but also that's approximate, maybe not quite the exact same thing 17:27:45 so let me summarize 17:28:05 I think the new user input does change things slightly, mostly because the PXE install case was not considered before 17:28:29 and I also believe that one could have reason to rename their NICs outside of unreliable NIC naming 17:28:33 EOM 17:29:43 dustymabe, what do you think we should do? 17:29:45 also time check 17:30:16 bgilbert: it would be really nice if there was something at another level taking care of the renaming, but absent that we could copy forward the generated udev rule from the initramfs into the real root 17:30:33 so it would be part of the "propagate initramfs networking" logic 17:30:58 right 17:31:07 which is fragile, but it's the best we've got 17:31:26 should we try to drive toward a conclusion in the next few minutes, or defer? 17:32:09 dustymabe ^ 17:32:13 probably not much else to discuss I don't think, unless people have questions 17:32:18 +1 17:32:25 discussion? 17:32:49 I'm wondering if we could have a persist naming karg or option in Ignition 17:33:16 i could put my proposal in the ticket and allow for silent acceptance over the next week? 17:33:26 and invite more discussion there 17:33:27 dustymabe, wfm 17:33:46 travier: preferrably not in Ignition itself.. these people are trying to drive networking via kargs 17:33:53 https://www.freedesktop.org/software/systemd/man/systemd-network-generator.service.html#Kernel%20command%20line%20options 17:34:02 if we had what you propose we'd still need the install boot ignition config 17:34:30 lucab: interesting 17:34:53 wonder if we could add a feature to tell that network generator pieces to ignore 17:34:56 just found right now, I've no idea how link units play with NM 17:35:16 link units work great - it's an alternative that's documented in our docs 17:35:23 ok i'll add a summary to the ticket 17:35:30 thanks bgilbert 17:35:31 +1 17:35:36 dustymabe: thanks 17:35:38 #topic open floor 17:35:43 #info devconf.cz CFP closes this week 17:35:48 #link https://github.com/coreos/fedora-coreos-tracker/issues/988 17:36:01 #info we really need some volunteers for CI flakes. One example: https://github.com/coreos/fedora-coreos-tracker/issues/942 17:36:10 please submit a talk if you're interested to devconf.cz 17:36:13 unless it errors out on $something, I think it will just produce some unused stuff in /run 17:37:05 will close in 30s 17:37:36 #endmeeting