16:30:16 <dustymabe> #startmeeting fedora_coreos_meeting 16:30:16 <zodbot> Meeting started Wed Mar 9 16:30:16 2022 UTC. 16:30:16 <zodbot> This meeting is logged and archived in a public location. 16:30:16 <zodbot> The chair is dustymabe. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions. 16:30:16 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:30:16 <zodbot> The meeting name has been set to 'fedora_coreos_meeting' 16:30:22 <dustymabe> #topic roll call 16:30:26 <dustymabe> .hi 16:30:27 <zodbot> dustymabe: Something blew up, please try again 16:30:30 <zodbot> dustymabe: An error has occurred and has been logged. Please contact this bot's administrator for more information. 16:30:36 <dustymabe> oh zodbot, not this again 16:31:13 <jlebon> .hello2 16:31:14 <zodbot> jlebon: Something blew up, please try again 16:31:16 <travier> .hello siosm 16:31:17 <zodbot> jlebon: An error has occurred and has been logged. Please contact this bot's administrator for more information. 16:31:19 <aaradhak> .hello 16:31:20 <zodbot> travier: Something blew up, please try again 16:31:20 <skunkerk> .hello sohank2602 16:31:22 <zodbot> travier: An error has occurred and has been logged. Please contact this bot's administrator for more information. 16:31:25 <zodbot> aaradhak: (hello <an alias, 1 argument>) -- Alias for "hellomynameis $1". 16:31:28 <zodbot> skunkerk: Something blew up, please try again 16:31:31 <zodbot> skunkerk: An error has occurred and has been logged. Please contact this bot's administrator for more information. 16:32:17 <miabbott> .hello miabbott 16:32:18 <zodbot> miabbott: Something blew up, please try again 16:32:21 <zodbot> miabbott: An error has occurred and has been logged. Please contact this bot's administrator for more information. 16:32:51 <dustymabe> #chair jlebon travier aaradhak skunkerk miabbott 16:32:51 <zodbot> Current chairs: aaradhak dustymabe jlebon miabbott skunkerk travier 16:33:01 <davdunc[m> .hello2 16:33:01 <zodbot> davdunc[m: Error: Missing "]". You may want to quote your arguments with double quotes in order to prevent extra brackets from being evaluated as nested commands. 16:33:07 <fifofonix> .hello 16:33:07 <zodbot> fifofonix: (hello <an alias, 1 argument>) -- Alias for "hellomynameis $1". 16:33:09 <dustymabe> #chair davdunc[m 16:33:09 <zodbot> Current chairs: aaradhak davdunc[m dustymabe jlebon miabbott skunkerk travier 16:33:17 <dustymabe> #chair fifofonix 16:33:17 <zodbot> Current chairs: aaradhak davdunc[m dustymabe fifofonix jlebon miabbott skunkerk travier 16:33:34 <dustymabe> davdunc we should just at an e to your name 16:33:39 <dustymabe> davedunce 16:33:45 <dustymabe> davdunce* 16:34:25 <davdunc[m> everybody has the "e" in their moniker. I am the one without. ;) 16:34:58 <dustymabe> ok let's get started 16:35:07 <dustymabe> #topic Action items from last meeting 16:35:15 <dustymabe> * ravanelli to look into the F36 changes 127 to see if this change affects us on ppc64le 16:35:18 <dustymabe> * davdunc to see if we can get the revert applied to the latest fedora kernels again 16:36:02 <dustymabe> is ravanelli here today? 16:37:39 <miabbott> i've seen renata active on slack today 16:37:39 <davdunc> and i believe that jforbes is reverting the msi-x commit for the short term, but we are looking at removing the limitation as quickly as the AWS Xen team can move to the 4.4+ revisions 16:37:42 <dustymabe> davdunc: indeed he did revert the msi-x commit 16:37:51 <jforbes> It is removed in the 5.16.13 kernel in updates-testing right now 16:38:09 <miabbott> did we make the upstream committer aware of the problem we found re: msi-x? 16:38:30 <davdunc> but that is a short term fix and I am assured that the AWS service team is working on adding the requisite updates. 16:38:31 <dustymabe> miabbott: yes, we looped them in and gave them some information 16:38:34 <miabbott> :+1 16:38:47 <dustymabe> jforbes++ 16:38:47 <zodbot> dustymabe: Karma for jforbes changed to 1 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 16:38:57 <davdunc> indeed dustymabe ! 16:39:51 <dustymabe> #info the msi-x commit was reverted temporarily (again) in the Fedora kernel while AWS works to upgrade instances so they will no longer be affected https://github.com/coreos/fedora-coreos-tracker/issues/1066#issuecomment-1061789661 16:40:19 <davdunc> we are expecting to be able to let this revert go in a couple of months or so. 16:40:40 <dustymabe> davdunc: there was more information that Stefan Roese was asking for but we can't get without a serial console (see email thread) 16:40:50 <davdunc> ah. 16:40:53 <dustymabe> maybe you can get access to an instance serial console and give that information 16:41:19 <davdunc> yea. That's another problem that needs to be fixed in AWS so that we can dynamically allocate the serial console correctly for debug. 16:41:26 * dustymabe re-actions ravanelli for her item 16:41:33 <dustymabe> #action ravanelli to look into the F36 changes 127 to see if this change affects us on ppc64le 16:42:19 <dustymabe> ok let's move on 16:42:27 <dustymabe> #topic CVE-2022-0847 (The Dirty Pipe Vulnerability) 16:42:32 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1118 16:43:12 <dustymabe> #info there is a `testing` and `next` stream release going out today with the 5.16.13 kernel that should address the CVE 16:43:29 <travier> dustymabe++ 16:43:39 <dustymabe> luckily we had already started the conversation about the MSI-x revert last week. Otherwise this would have been a real fire 16:44:49 <dustymabe> and to state the obvious -- the `stable` release next week will have the updated kernel and fix for the CVE 16:45:05 <dustymabe> #topic Update the VMware metadata to new, modern defaults 16:45:10 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1119 16:45:12 <jforbes> miabbott: the upstream maintainer is aware, but from what I can gather, the bug isn't in the kernel, it is in the xen hypervisor's bad implementation of the hardware emulation, which they fixed long before the upstream code changed 16:45:23 <miabbott> jforbes: +1 16:45:31 <dustymabe> jforbes: thanks for the context 16:46:07 <dustymabe> miabbott: want to intro this one? 16:46:13 <miabbott> i'll give it a shot 16:46:25 <miabbott> i'm not a vmware expert, but here's what i understand 16:47:36 <miabbott> vsphere/esxi versions 6.5/6.7 are going EOL in oct 2022. we are currently setting the "hardware version" in the vmware OVF metadata for our artifacts that indicate the oldest version of vsphere/esxi on which you can run an FCOS guest is 6.5 16:49:05 <miabbott> it would make sense to update the OVF metadata to indicate a newer hardware version, so that we both 1) unlock features available in newer versions of vsphere/esxi and 2) aren't indicating to our users that they should try running fcos guests on older versions of vpshere/esxi 16:49:39 <miabbott> additionally, we can do things like defaulting to UEFI firmware and update the "osType" in the metadata to be more reasonable 16:50:14 <miabbott> ultimately this a change we need in downstream RHCOS, as OCP will be dropping support for vsphere/esxi 6.7 in the next release 16:50:34 <miabbott> so if this isn't something we decide to do for fcos + rhcos, we'll need a solution for rhcos only 16:50:58 <dustymabe> hmm 16:51:55 <dustymabe> I think I'm OK with updating to more recent "defaults" but could we publish some docs or something on how someone could crack open the OVA and adjust those if they needed to? 16:52:28 <dustymabe> i.e. if something isn't EOL yet then it feels weird to explicitly deny it 16:52:52 <dustymabe> makes sense for RHCOS (support, contracts, etc), but not so much for FCOS where everything is best effort anyway 16:53:19 <miabbott> i think so. the OVA is just tarball of the OVF + VMDK, so it should be straightforward for someone to crack it open, change things, and reassemble the OVA 16:53:50 <dustymabe> I think if we could add a page for that in our docs somewhere then I'd be happy with making the recommended changes 16:54:09 <dustymabe> we'd probably also want to send a mail as an FYI about it 16:54:27 <miabbott> one thing we can/should investigate is what happens if we update the hw version in the OVF and then try to run it on ESXi 6.5/6.7 16:54:38 <miabbott> i'm not certain about the repercussions in that situation 16:54:56 <jlebon> we could also make it a `cosa buildextend-vmware` knob which RHCOS can use now and we can default to once it's EOL proper 16:54:58 <dustymabe> yeah, not sure - if it's just a warning that the user can Ignore then maybe we don't need a docs page? 16:55:47 <dustymabe> jlebon: ehh - yeah, but not super happy with that either 16:55:58 <dustymabe> i'd prefer not to diverge to much if we don't have to 16:56:06 <dustymabe> and RHCOS leading FCOS feels awkward 16:56:35 <jlebon> don't disagree, but it depends how much we care about users that haven't migrated yet 16:56:39 <miabbott> let me gather more data with our vmware folks in OCP about the hw version and any warnings/errors that might happen if we change it 16:57:12 <miabbott> independent of the hw version change, we could change the firmware being used 16:57:19 <miabbott> i.e. moving from BIOS to UEFI 16:57:29 <miabbott> that unlocks the ability to use secureboot 16:58:17 <jlebon> seems reasonable to me 16:58:31 <dustymabe> #proposed We aren't opposed to updating the OVA to contain more modern defaults, but would like to know how users can revert the behavior if they need to since the platforms aren't EOL yet. We'd either need some documentation to let the users know how or to know that the users can ignore warnings and still proceed. 16:58:32 <ravanelli> .hi 16:58:33 <zodbot> ravanelli: ravanelli 'Renata Renata Andrade Matos Ravanelii' <renata.ravanelli@gmail.com> 16:58:36 <dustymabe> how about something like ^^ for now? 16:58:39 <dustymabe> #chair ravanelli 16:58:39 <zodbot> Current chairs: aaradhak davdunc[m dustymabe fifofonix jlebon miabbott ravanelli skunkerk travier 16:58:57 <dustymabe> ack/nack? 16:59:26 <travier> ack 16:59:31 <fifofonix> +1 (i'm on vsphere 6.7 with terraform deploying next ovf daily - infra team can't go 7.0 yet due to a blocker even if EOL) 17:00:07 <miabbott> oh good to know fifofonix. i had no idea how many people were using FCOS on vmware 17:00:23 <miabbott> dustymabe: could we do a separate proposal for the firmware change? 17:00:35 <miabbott> that one isn't tied to any hw version change 17:00:50 <jlebon> +0.5 17:01:05 <dustymabe> miabbott: depends.. I'm kind of a mind to bundle the changes together 17:01:26 <miabbott> ack; i'll take the action to finish gathering data then 17:01:47 <jlebon> i think this is probably fine, but also it's hard to evaluate this without knowing how many people will be affected 17:01:57 <dustymabe> miabbott: that's mostly so that we're not constantly giving users PSA 17:02:08 * miabbott nods 17:02:28 <ravanelli> dustymabe: Sorry, I'm late for today. Here is the action update #127. The gcc ppc64le team was able build fcos using F36 as host in a P9, P8 failed completely in the boot part for F36, they are looking at it, looks the issue is related to the bits change. I will keep an eye on how it is going 17:02:28 <dustymabe> ok I've got one ack one +1 and one .5 17:02:43 <miabbott> +1 to proposed 17:02:52 <fifofonix> (also happy to trial a next-modified version with new defaults for you in our 6.7 env if helpful to check no adverse messages) 17:03:06 <dustymabe> fifofonix: that's awesome 17:03:11 <jlebon> fifofonix++ 17:03:11 <zodbot> jlebon: Karma for fifofonix changed to 2 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 17:03:17 <travier> fifofonix: thanks! that would be really useful 17:03:22 <miabbott> fifofonix: thanks for the offer; we can make a test OVA with the changes and give it to you 17:03:43 <dustymabe> #agreed We aren't opposed to updating the OVA to contain more modern defaults, but would like to know how users can revert the behavior if they need to since the platforms aren't EOL yet. We'd either need some documentation to let the users know how or to know that the users can ignore warnings and still proceed. 17:03:46 <fifofonix> no problem. always eager to participate in next stuff if needs be. don't hesitate to ask. 17:03:53 <miabbott> fifofonix++ 17:03:53 <zodbot> miabbott: Karma for fifofonix changed to 3 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 17:04:01 <dustymabe> miabbott: want me to leave the `meeting` label on the ticket and we'll see if there is more information next week? 17:04:08 <miabbott> yup, works for me 17:04:36 <dustymabe> #topic open floor 17:04:58 <dustymabe> #info Here is the action update #127. The gcc ppc64le team was able build fcos using F36 as host in a P9, P8 failed completely in the boot part for F36, they are looking at it, looks the issue is related to the bits change. I will keep an eye on how it is going 17:05:17 <dustymabe> ravanelli: want me to open a ticket for that F36 change so we can collect notes there? 17:06:09 <ravanelli> dustymabe: sure, where where exactly should I open this? 17:06:17 <dustymabe> i'll open it and link you back to it 17:06:49 <dustymabe> any other topics for open floor? 17:07:11 <jlebon> neat, lots of time today :) 17:07:21 <dustymabe> anything we should add to our running "this month in FCOS" ticket? https://github.com/coreos/fedora-coreos-tracker/issues/1117 17:07:51 <fifofonix> no open items but just to note i' 17:08:04 <jlebon> we'll be adding the next-devel rebase there soon 17:08:12 <fifofonix> m happy with stability of terraform aws gpu nodes running tensorflow across next/testing/stable. :o) 17:08:22 <dustymabe> fifofonix: awesome! 17:08:34 <davdunc[m> fifofonix: that's newsworthy! 17:09:05 <dustymabe> fifofonix: are you able to manually specify an AMI to your setup? soon we'll have an AMI based on F36 that might be good to have you run through your setup 17:09:27 <jlebon> cool! 17:09:51 <fifofonix> can do. i was waiting for next to have f36 and on tenter hooks. i think the comms on podman is good but want to see it. 17:10:07 <travier> fifofonix++ 17:10:20 <dustymabe> fifofonix: if I gave you an AMI today could you use it? I'll kick off a manual build if so 17:10:51 <fifofonix> yes. not today but in the next few days I can use it. 17:10:56 <dustymabe> nice 17:11:03 <dustymabe> fifofonix++ 17:11:09 <dustymabe> any other topics for open floor? 17:11:21 * dustymabe looks to see when time changes happen 17:12:47 <ravanelli> dustymabe: Sounds good, thanks ;) 17:12:48 <dustymabe> #info Your time may shift in your locale in the coming weeks, but FCOS community meetings will stay at 16:30 UTC. 17:13:07 <dustymabe> FYI everyone ^^ 17:13:17 <dustymabe> the US changes March 13th 17:14:22 <dustymabe> anything else for open floor before I close out? 17:15:44 <dustymabe> #endmeeting