16:30:25 #startmeeting fedora_coreos_meeting 16:30:25 Meeting started Wed Mar 16 16:30:25 2022 UTC. 16:30:25 This meeting is logged and archived in a public location. 16:30:25 The chair is dustymabe. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions. 16:30:25 Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:30:25 The meeting name has been set to 'fedora_coreos_meeting' 16:30:32 #topic roll call 16:30:35 .hi 16:30:36 dustymabe: dustymabe 'Dusty Mabe' 16:30:41 .hello miabbott 16:30:42 miabbott: miabbott 'Micah Abbott' 16:30:50 .hello sohank2602 16:30:51 skunkerk: sohank2602 'Sohan Kunkerkar' 16:31:12 .hi 16:31:13 jmarrero: jmarrero 'Joseph Marrero' 16:31:16 .hello2 16:31:17 jlebon: jlebon 'None' 16:31:18 .hi 16:31:20 fifofonix: fifofonix 'Fifo Phonics' 16:31:57 .hello jasonbrooks 16:31:58 jbrooks: jasonbrooks 'Jason Brooks' 16:32:18 .hello marmijo 16:32:19 marmijo: marmijo 'Michael Armijo' 16:32:30 .heelo siosm 16:32:33 .hello siosm 16:32:34 travier: siosm 'Timothée Ravier' 16:32:37 #chair miabbott skunkerk jmarrero fifofonix jbrooks marmijo travier 16:32:37 Current chairs: dustymabe fifofonix jbrooks jmarrero marmijo miabbott skunkerk travier 16:32:55 .hello 16:32:57 davdunc: (hello ) -- Alias for "hellomynameis $1". 16:33:02 .hell2 16:33:05 .hello2 16:33:06 davdunc: davdunc 'David Duncan' 16:33:08 finally. 16:33:25 #chair davdunc 16:33:25 Current chairs: davdunc dustymabe fifofonix jbrooks jmarrero marmijo miabbott skunkerk travier 16:33:44 will get started here in a minute 16:33:52 .hi 16:33:53 ravanelli: ravanelli 'Renata Renata Andrade Matos Ravanelii' 16:34:23 #chair 16:34:23 Current chairs: davdunc dustymabe fifofonix jbrooks jmarrero marmijo miabbott skunkerk travier 16:34:23 #chair ravanelli 16:34:23 Current chairs: davdunc dustymabe fifofonix jbrooks jmarrero marmijo miabbott ravanelli skunkerk travier 16:34:40 #topic Action items from last meeting 16:34:53 There was an action item from last meeting but ravanelli showed up later in the meeting and we opened https://github.com/coreos/fedora-coreos-tracker/issues/1120 as a result of the new information she brought. 16:35:36 o/ 16:35:45 #topic news 16:35:56 .hi 16:35:59 lorbus: lorbus 'Christian Glombek' 16:36:08 #chair cverna lorbus 16:36:08 Current chairs: cverna davdunc dustymabe fifofonix jbrooks jmarrero lorbus marmijo miabbott ravanelli skunkerk travier 16:36:25 #info the Fedora 36 beta is fast approaching and we will be rebasing our `next` stream on top of Fedora 36 (most likely to be released next week) 16:36:45 \o/ 16:36:57 #info podman v4 is coming to the next stream along with Fedora 36 https://discussion.fedoraproject.org/t/fedora-coreos-moving-to-podman-v4/37303/2 16:37:26 #info iptables nft by default is coming to the `next` stream along with Fedora 36 https://discussion.fedoraproject.org/t/fedora-coreos-moving-to-iptables-nft/37302/2 16:38:19 #info The "dirty pipe" CVE-2022-0847 has now been fixed in all FCOS streams (as of today) 16:38:57 #info there are some regressions for some NFS servers brought in by the new kernel: https://discussion.fedoraproject.org/t/latest-kernel-update-can-cause-nfs-mount-failures/37555/2 16:39:07 a lot going on this week :) 16:39:30 sweet! 16:39:40 great! 16:39:44 .hi 16:39:45 bgilbert: bgilbert 'Benjamin Gilbert' 16:40:06 #info the Februrary update on happenings in Fedora CoreOS was posted by cverna: https://discussion.fedoraproject.org/t/this-month-in-fedora-coreos-february-2022/37477/2 16:40:09 hi bgilbert :) 16:40:12 cverna++ 16:40:29 * dustymabe moves to meeting topics soon 16:40:33 #chair bgilbert nemric 16:40:33 Current chairs: bgilbert cverna davdunc dustymabe fifofonix jbrooks jmarrero lorbus marmijo miabbott nemric ravanelli skunkerk travier 16:41:22 #topic Update the VMware metadata to new, modern defaults 16:41:27 #link https://github.com/coreos/fedora-coreos-tracker/issues/1119 16:41:55 .hi 16:41:57 lucab: lucab 'Luca BRUNO' 16:42:02 miabbott: - want to give us an update on this ticket? 16:42:06 sure thing 16:42:08 #chair lucab 16:42:08 Current chairs: bgilbert cverna davdunc dustymabe fifofonix jbrooks jmarrero lorbus lucab marmijo miabbott nemric ravanelli skunkerk travier 16:42:50 generally, I think making the proposed changes in that ticket are fine to do. we will likely have to produce some docs around the change and how users can modify the OVF metadata if they are on an older version of vSphere 16:43:49 one thing that users will need to be aware of is the use of SecureBoot during the initial boot of the node; they'll run into https://github.com/coreos/ignition/issues/1092 -> https://github.com/vmware/vmw-guestinfo/issues/20 16:44:21 so the workaround would be to disable SecureBoot for the initial boot of the node (when Ignition runs), then enable it for subsequent boots 16:44:45 thanks to fifofonix for testing the changes in his environment! 16:44:59 miabbott: is that problem only introduced because we are proposing switching to UEFI? 16:44:59 miabbott: the ticket implies that if we take no action, the image will break on newer versions of VMware. but I don't think that's actually the case? VMware has generally been quite good about back-compat. 16:45:01 no problem 16:45:32 I think we cherry-picked a patch in RHEL and were hoping to see the library fixed for Fedora and upstream 16:45:41 Should we really enable SecureBoot by default if it requires manual action?. 16:45:42 but that never happened 16:45:55 we should probably carry the same patch in Fedora too 16:46:22 dustymabe: the SecureBoot issue could be encountered by smart users who are configuring the UEFI firmware for their VM and enabling SecureBoot. we currently default to BIOS firmware in the OVF, but it is possible to change that setting when you provision the VM from the vSphere console 16:46:24 .hello mnguyen 16:46:25 mnguyen: mnguyen 'Michael Nguyen' 16:47:19 bgilbert: if we take no action, nothing breaks afaik. it's more of optics around what we consider the oldest version of vSphere we are supporting. 16:47:25 right 16:47:26 miabbott: but users would only get secureboot if they chose it? i.e. it's not the default 16:47:28 .hello copperi 16:47:29 copperi[m]: copperi 'Jan Kuparinen' 16:47:38 travier: the proposal is not to enable SecureBoot by default. only to use UEFI firware by default. 16:47:43 +1 16:47:59 #chair mnguyen copperi[m] 16:47:59 Current chairs: bgilbert copperi[m] cverna davdunc dustymabe fifofonix jbrooks jmarrero lorbus lucab marmijo miabbott mnguyen nemric ravanelli skunkerk travier 16:48:07 users would have to opt in/configure using SecureBoot when provisioning their VMs 16:48:11 +1 16:49:23 bgilbert: the other implication of not changing anything is that we would be producing RHCOS images that would technically not be supported by newer versions of OCP 16:49:53 miabbott: that's assuming we match RHCOS/FCOS, but yes 16:50:14 right, we could change `cosa` to special case RHCOS if we decide not to make the change broadly 16:50:26 lucab: i was unaware of any patches, do you have link? 16:51:25 miabbott: https://github.com/vmware/vmw-guestinfo/pull/21 16:51:29 here's how we left things last time: 16:51:32 oh that link :) 16:51:35 We aren't opposed to updating the OVA to contain more modern defaults, but would like to know how users can revert the behavior if they need to since the platforms aren't EOL yet. We'd either need some documentation to let the users know how or to know that the users can ignore warnings and still proceed. 16:52:22 So if that's still our current stance.. We'd need some documentation created? 16:52:51 yeah, it should be straight-forward to create those docs 16:53:00 +1 16:53:19 you can action me to do that...if we agree to the changes to the OVA :) 16:53:28 any opposed to moving forward with the changes *after* docs exist for reverting back if needed? 16:53:43 or docs first :P 16:54:01 +1 documenting how to make it work on older release and then doing it 16:54:37 Help from someone that has access to those environment would be appreciated :) 16:54:43 #proposed we will update the VMWare OVA to use hardware version 17 and also default to UEFI by default. We will also create documentation showing users how to revert to the old settings if needed. 16:54:53 did I miss anything ^^ 16:55:25 miabbott: if you need any more help running anything let me know. 16:55:27 how hard is the revert procedure? is it just decompress, edit file, recompress? 16:55:30 the other part of the proposal is updating the osType to something besides rhel7_64Guest 16:55:46 jlebon: correct 16:56:11 k, just confirming it's something reasonable to ask users to do 16:56:13 interesting observation is that open-vm-tools which i'm guessing a bunch of people will be using...updates the os name... 16:56:35 #proposed we will update the VMWare OVA to use hardware version 17 and an OS type newer than rhel7_64Guest. We'll also default to UEFI by default. We will also create documentation showing users how to revert to the old settings if needed. 16:56:42 maybe this has been answered already; why 17 and not 16? 16:56:45 #chair ryanjenkins 16:56:45 Current chairs: bgilbert copperi[m] cverna davdunc dustymabe fifofonix jbrooks jmarrero lorbus lucab marmijo miabbott mnguyen nemric ravanelli ryanjenkins skunkerk travier 16:56:59 we'd be forcing users to the latest Fusion/Workstation 16:57:14 good question 16:57:20 (because it's a lucky number) 16:57:32 bgilbert: 17 is the version required for vSphere 7.0 16:57:46 which is what OCP/RHCOS is targeting in the future 16:57:53 that's not how hw versions work 16:57:59 TIL 16:58:07 17 is the max version supported by vsphere 7.0 16:58:51 unless there are other constraints I don't know about 16:59:12 is the real goal to require vSphere >= 7.0 ? 16:59:26 that's the goal of the downstream 16:59:37 in which case I assume the max HW version for VSphere 6.X is less than 17 16:59:48 https://kb.vmware.com/s/article/1003746 16:59:54 ^ support table 17:00:10 so i guess i interpreted that table incorrectly 17:00:11 16 seems reasonable too 17:01:02 I guess I'm still confused about what problem we're trying to solve here 17:01:07 yeah, so upon further review...16 seems like a more reasonable minimum 17:01:28 if we're supporting X and all hypervisors that max out at X are EOL, then yeah, we should bump 17:01:36 if we need a feature that isn't supported by X, we could consider bumping 17:01:51 since we have taken up 30 minutes already, should we take some of this discussion to the ticket and we can revisit any proposals next week? 17:02:12 does it need much more discussion? 17:02:45 bgilbert: is there a HW version that you suggest (based on the criteria you just laid out)? 17:02:45 i'd like to make sure bgilbert has his questions answered and i don't want to take up additional time from the meeting 17:03:09 AFAICT the discussion has focused on the consequences of bumping, but I'm unclear on the benefits of bumping 17:03:28 this decision doesn't have to be agreed upon this week, which is why i am suggesting moving on to other topics 17:03:29 presumably something prompted this discussion though :-) 17:03:35 I'm okay with moving on 17:03:40 moving on... 17:03:49 #topic AWS: Add predictable symlinks for secondary disks 17:03:54 #link https://github.com/coreos/fedora-coreos-tracker/issues/1122 17:04:01 i'll reach out to bgilbert separately and record outcomes in the ticket 17:04:02 cc jlebon 17:04:05 +1 17:04:42 so we now have a stable symlink for the boot disk in FCOS, which is nice 17:05:24 another thing I think that would be nice to stabilize are cloud-specific nonboot block devices 17:05:55 e.g. on AWS, you can have instance storage which could or could not be NVMe 17:06:11 depending on the instance type 17:06:13 yeah all. I will need to leave earlier, I have a physiotherapy now. 17:06:30 so the same issue of writing a generic Ignition config shows up there 17:06:34 ravanelli: see you! :) 17:06:40 usually AWS calls it out as best practice to use the letters to sdf for the symlinks to ephemeral. 17:06:53 sd[b-f] 17:07:29 davdunc: i think it'd be nice to have something more explicit here 17:07:42 100% agreement jlebon 17:07:50 and I'm sure similar things show up on other clouds too 17:08:18 this is something I would like to determine how to get back into the ec2-utils 17:08:20 https://github.com/aws/amazon-ec2-utils 17:08:39 +1 for having this live upstream and not something we own 17:09:13 my understanding is it's also possible to attach additional EBS volumes, but I'm not sure what the current symlinks produced are 17:09:51 jlebon: in the ec2-utils, they are read from the comment region of the nvme detail. 17:10:12 the setting occurs in the instance configuration. 17:10:28 I'm not sure that coupling an udev rule to the metadata endpoint is a good thing 17:10:58 if it's in ec2-utils, how does that make it back to FCOS? would we have to add it as a package? 17:11:05 lucab: what's the concern? 17:11:05 lucab: it can be read out of the nvme comments 17:11:12 nvm 17:11:51 overall +1 to having it live somewhere more generic if there's a good place for it 17:12:16 davdunc: that's the disk name though I think (i.e. xvd*) 17:12:45 jlebon: I'd like to add it as a package and I can have that pushed out pretty quickly. 17:12:59 this seems like a nice to have 17:13:02 I've done a lot of cleanup on the license and whatnot already. 17:13:11 lucab: it's the disk name, yes. 17:13:42 identifying it as local might be complicated as they are all presenting as pseudo pci devices. 17:14:15 davdunc: we'd have to check if we want everything in there, which may not be the case :) 17:14:35 jlebon: that makes perfect sense. 17:15:03 davdunc: it is my understanding that this ephemeral-mapping is a different thing, and it lives in the IMDS 17:15:09 and we can not depend on Python :) 17:15:38 if it actually lives in the NVMe metadata, I'm much happier 17:15:55 yea. lucab it's a little more complicated in that it is tied tot he original instance from deployment, 17:15:58 * dustymabe wonders how this issue intersects with https://github.com/coreos/fedora-coreos-tracker/issues/601 17:16:24 so if you stop it and start on a different instance type that does not include the ephemeral disks, the metadata doesn't change. 17:16:52 dustymabe: I think they are pretty close. 17:17:26 it's better to rely on what can be discovered from the nvme data than the imds anyway. 17:17:31 IMO 17:17:32 bgilbert: going back to your question, I'm concerned that it's udev relying on network, and that the disk attachment is racing with the metadata content 17:18:03 I share that concern ^^ 17:18:13 lucab: +1 17:18:29 any conclusions we can draw from this conversation so far? 17:18:32 I don't want to trust the metadata as it may not be directly related to the current instance type. 17:18:34 hmm, we have pretty strict ordering already in the initramfs 17:18:45 to me too it looks like they are two different things, and that the xvd* one is saner (but perhaps it has a worse UX?) 17:18:46 i don't htink we'd want the udev rule to literally reach out to the network 17:19:55 the NVMe metadata would work for NVMe, but not other instance types 17:20:38 this can be worked around for now so I don't think there's any urgency here 17:20:48 we can let it simmer for now and get back to it 17:20:56 want to make a #proposed or an #info ? 17:21:19 * dustymabe always likes to write down some outcomes of a discussion, even if it's to keep discussing :) 17:22:09 #info having stable symlinks would be useful, though ideally it would live somewhere more generic and not be owned by us. there is also some apprehension about mixing networking and udev rules. 17:22:13 ? 17:22:28 LGTM - any particular actions for us to take at this point? 17:22:46 also should we re-kindle the efforts for https://github.com/coreos/fedora-coreos-tracker/issues/601 cc davdunc 17:22:56 i'll put in a package review for the ec2-net-utils and we can hopefully look at that for an upstream. 17:23:14 let's keep chatting in the ticket for now I think 17:23:29 kk 17:23:31 #topic Unable to disable zincati.service using Ignition 17:23:37 #link https://github.com/coreos/fedora-coreos-tracker/issues/392 17:24:53 jlebon: I think you tagged this one for the meeting 17:25:02 yup 17:25:14 #link https://github.com/systemd/systemd/pull/15205 17:25:16 so, this is something that keeps coming up again 17:25:23 is the underlying systemd issue I believe ^ 17:25:34 the TL;DR is that `enabled: false` in Ignition doesn't actually work 17:25:47 and making it work as is would require systemd changes 17:26:03 to which it doesn't seem like upstream is open 17:26:08 same problem with systemd-journald-flush 17:26:25 * dustymabe emailed lennart and zbyszek earlier this week to see if they could have a look 17:26:39 dustymabe++ 17:26:39 lorbus: Karma for dustymabe changed to 8 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 17:27:01 i realized this morning that there's actually a simpler way to solve this 17:27:20 jlebon: oh? 17:27:34 which is to have Ignition look for and delete any enablement symlinks when handling `enabled: false` 17:27:51 https://github.com/coreos/fedora-coreos-config/pull/363#issuecomment-1069323399 17:28:13 that said, I still think long-term we should figure out a way to have enablement symlinks be local state and not baked into the OSTree 17:28:19 won't that make ignition do the job of systemd ? 17:28:21 but... that's a much riskier change 17:28:56 nemric: systemd decided not to make it its job :) 17:29:11 yeah ... 17:29:24 can we do a full-clean of all of them before Ignition runs, not only the `enabled: false` one? 17:29:27 jlebon: did someone actually fully reject the PR? It just seemed like discussion lingered 17:29:44 same as reading ~/.config/systemd/user-preset ... 17:30:17 dustymabe: it's been open for almost 2 years now, so I'm not expecting any changes 17:30:36 lucab: that's what my current PR in https://github.com/coreos/fedora-coreos-config/pull/363 does 17:30:58 which is a valid approach too, but more blunt 17:31:39 * dustymabe notes we're running out of time 17:32:05 anyway, I think the Ignition approach is cleaner and is in scope 17:32:33 jlebon: ah sorry, I forgot about that. Yes I think that it isn't too scary, as it it first-boot only. 17:32:44 obviously ideally we have systemd do the right thing, but barring that.. 17:33:32 dustymabe: we can wait until you get an answer from your email before proceeding 17:33:38 +1 17:33:50 i'm not opposed to a tactical fix either, hopefully I get something back 17:33:53 even if it's having that PR closed and we move on with $something 17:34:00 #topic open floor 17:34:08 davdunc: from earlier I forgot to do this 17:34:48 #action davdunc to put a package review in for ec2-net-utils and brainstorm on how we can use that for #601 17:34:54 :) 17:35:16 yea. I am working on getting that systemd integration supported as a way forward. 17:35:19 also to everyone.. we need to schedule our test day for F36 https://github.com/coreos/fedora-coreos-tracker/issues/1123 17:35:25 #info coreos.live.rootfs_url now supports tftp:// 17:35:30 but I can do the ec2-net-utils now. 17:35:51 I'd like to not own coordinating the test day if possible.. Please do reach out or comment in the ticket if you'd like to help organize it 17:36:15 bgilbert: nice - maybe we can add that to https://github.com/coreos/fedora-coreos-tracker/issues/1117 17:36:24 I wanted to ask about the current state of plans for potentially unifying the ways how Silverblue/Kinoite, Fedora IoT and FCOS are built. I know there were some discussions with the osbuild folks in the past, but I never followed up on what the results were (if any) 17:36:35 anything else we should add to the "this month in FCOS" ticket? https://github.com/coreos/fedora-coreos-tracker/issues/1117 17:36:37 dustymabe: https://github.com/coreos/fedora-coreos-tracker/issues/1107#issuecomment-1043242480 17:36:54 it just went stable though 17:36:59 ahh 17:37:02 +1 17:37:13 (I can take that topic to the coreos channel though, looking at the time :) ) 17:37:25 lorbus: I don't know of any concrete plans to consolidate build tooling 17:37:31 there's an OpenSSL DoS CVE: https://lwn.net/Articles/887971/ 17:37:42 bgilbert: yay 17:37:51 caused by crafted server cert, and triggers _before_ cert validation 17:38:07 Fedora is... not efficiently putting out fixes 17:38:11 on this one 17:38:14 from what I read, it's a low-impact on RHEL/Fedora 17:38:29 closest context seems to be https://bugzilla.redhat.com/show_bug.cgi?id=2064453 17:38:31 (due to how openssl is built/patched there) 17:38:36 lucab: oh? 17:39:17 bgilbert: sec, I don't have the link at hand 17:39:20 coreos.inst.rootfs_url would be affected 17:40:03 thanks bgilbert for bringing that up - carry on conversation in #fedora-coreos ? 17:40:07 +1 17:40:20 * dustymabe waits for link from lucab (for the record) before closing the meeting 17:40:46 let's close, I'll post on -coreos once I find it again 17:40:57 #endmeeting