16:30:16 <dustymabe> #startmeeting fedora_coreos_meeting
16:30:16 <zodbot> Meeting started Wed Mar  9 16:30:16 2022 UTC.
16:30:16 <zodbot> This meeting is logged and archived in a public location.
16:30:16 <zodbot> The chair is dustymabe. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:30:16 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:30:16 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:30:22 <dustymabe> #topic roll call
16:30:26 <dustymabe> .hi
16:30:27 <zodbot> dustymabe: Something blew up, please try again
16:30:30 <zodbot> dustymabe: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:30:36 <dustymabe> oh zodbot, not this again
16:31:13 <jlebon> .hello2
16:31:14 <zodbot> jlebon: Something blew up, please try again
16:31:16 <travier> .hello siosm
16:31:17 <zodbot> jlebon: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:31:19 <aaradhak> .hello
16:31:20 <zodbot> travier: Something blew up, please try again
16:31:20 <skunkerk> .hello sohank2602
16:31:22 <zodbot> travier: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:31:25 <zodbot> aaradhak: (hello <an alias, 1 argument>) -- Alias for "hellomynameis $1".
16:31:28 <zodbot> skunkerk: Something blew up, please try again
16:31:31 <zodbot> skunkerk: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:32:17 <miabbott> .hello miabbott
16:32:18 <zodbot> miabbott: Something blew up, please try again
16:32:21 <zodbot> miabbott: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:32:51 <dustymabe> #chair jlebon travier aaradhak skunkerk miabbott
16:32:51 <zodbot> Current chairs: aaradhak dustymabe jlebon miabbott skunkerk travier
16:33:01 <davdunc[m> .hello2
16:33:01 <zodbot> davdunc[m: Error: Missing "]".  You may want to quote your arguments with double quotes in order to prevent extra brackets from being evaluated as nested commands.
16:33:07 <fifofonix> .hello
16:33:07 <zodbot> fifofonix: (hello <an alias, 1 argument>) -- Alias for "hellomynameis $1".
16:33:09 <dustymabe> #chair davdunc[m
16:33:09 <zodbot> Current chairs: aaradhak davdunc[m dustymabe jlebon miabbott skunkerk travier
16:33:17 <dustymabe> #chair fifofonix
16:33:17 <zodbot> Current chairs: aaradhak davdunc[m dustymabe fifofonix jlebon miabbott skunkerk travier
16:33:34 <dustymabe> davdunc we should just at an e to your name
16:33:39 <dustymabe> davedunce
16:33:45 <dustymabe> davdunce*
16:34:25 <davdunc[m> everybody has the "e" in their moniker. I am the one without. ;)
16:34:58 <dustymabe> ok let's get started
16:35:07 <dustymabe> #topic Action items from last meeting
16:35:15 <dustymabe> * ravanelli to look into the F36 changes 127 to see if this change affects us on ppc64le
16:35:18 <dustymabe> * davdunc to see if we can get the revert applied to the latest fedora kernels again
16:36:02 <dustymabe> is ravanelli here today?
16:37:39 <miabbott> i've seen renata active on slack today
16:37:39 <davdunc> and i believe that jforbes is reverting the msi-x commit for the short term, but we are looking at removing the limitation as quickly as the AWS Xen team can move to the 4.4+ revisions
16:37:42 <dustymabe> davdunc: indeed he did revert the msi-x commit
16:37:51 <jforbes> It is removed in the 5.16.13 kernel in updates-testing right now
16:38:09 <miabbott> did we make the upstream committer aware of the problem we found re: msi-x?
16:38:30 <davdunc> but that is a short term fix and I am assured that the AWS service team is working on adding the requisite updates.
16:38:31 <dustymabe> miabbott: yes, we looped them in and gave them some information
16:38:34 <miabbott> :+1
16:38:47 <dustymabe> jforbes++
16:38:47 <zodbot> dustymabe: Karma for jforbes changed to 1 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
16:38:57 <davdunc> indeed dustymabe !
16:39:51 <dustymabe> #info the msi-x commit was reverted temporarily (again) in the Fedora kernel while AWS works to upgrade instances so they will no longer be affected https://github.com/coreos/fedora-coreos-tracker/issues/1066#issuecomment-1061789661
16:40:19 <davdunc> we are expecting to be able to let this revert go in a couple of months or so.
16:40:40 <dustymabe> davdunc: there was more information that Stefan Roese was asking for but we can't get without a serial console (see email thread)
16:40:50 <davdunc> ah.
16:40:53 <dustymabe> maybe you can get access to an instance serial console and give that information
16:41:19 <davdunc> yea. That's another problem that needs to be fixed in AWS so that we can dynamically allocate the serial console correctly for debug.
16:41:26 * dustymabe re-actions ravanelli for her item
16:41:33 <dustymabe> #action ravanelli to look into the F36 changes 127 to see if this change affects us on ppc64le
16:42:19 <dustymabe> ok let's move on
16:42:27 <dustymabe> #topic CVE-2022-0847 (The Dirty Pipe Vulnerability)
16:42:32 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1118
16:43:12 <dustymabe> #info there is a `testing` and `next` stream release going out today with the 5.16.13 kernel that should address the CVE
16:43:29 <travier> dustymabe++
16:43:39 <dustymabe> luckily we had already started the conversation about the MSI-x revert last week. Otherwise this would have been a real fire
16:44:49 <dustymabe> and to state the obvious -- the `stable` release next week will have the updated kernel and fix for the CVE
16:45:05 <dustymabe> #topic Update the VMware metadata to new, modern defaults
16:45:10 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/1119
16:45:12 <jforbes> miabbott: the upstream maintainer is aware, but from what I can gather, the bug isn't in the kernel, it is in the xen hypervisor's bad implementation of the hardware emulation, which they fixed long before the upstream code changed
16:45:23 <miabbott> jforbes: +1
16:45:31 <dustymabe> jforbes: thanks for the context
16:46:07 <dustymabe> miabbott: want to intro this one?
16:46:13 <miabbott> i'll give it a shot
16:46:25 <miabbott> i'm not a vmware expert, but here's what i understand
16:47:36 <miabbott> vsphere/esxi versions 6.5/6.7 are going EOL in oct 2022.  we are currently setting the "hardware version" in the vmware OVF metadata for our artifacts that indicate the oldest version of vsphere/esxi on which you can run an FCOS guest is 6.5
16:49:05 <miabbott> it would make sense to update the OVF metadata to indicate a newer hardware version, so that we both 1) unlock features available in newer versions of vsphere/esxi and 2) aren't indicating to our users that they should try running fcos guests on older versions of vpshere/esxi
16:49:39 <miabbott> additionally, we can do things like defaulting to UEFI firmware and update the "osType" in the metadata to be more reasonable
16:50:14 <miabbott> ultimately this a change we need in downstream RHCOS, as OCP will be dropping support for vsphere/esxi 6.7 in the next release
16:50:34 <miabbott> so if this isn't something we decide to do for fcos + rhcos, we'll need a solution for rhcos only
16:50:58 <dustymabe> hmm
16:51:55 <dustymabe> I think I'm OK with updating to more recent "defaults" but could we publish some docs or something on how someone could crack open the OVA and adjust those if they needed to?
16:52:28 <dustymabe> i.e. if something isn't EOL yet then it feels weird to explicitly deny it
16:52:52 <dustymabe> makes sense for RHCOS (support, contracts, etc), but not so much for FCOS where everything is best effort anyway
16:53:19 <miabbott> i think so.  the OVA is just tarball of the OVF + VMDK, so it should be straightforward for someone to crack it open, change things, and reassemble the OVA
16:53:50 <dustymabe> I think if we could add a page for that in our docs somewhere then I'd be happy with making the recommended changes
16:54:09 <dustymabe> we'd probably also want to send a mail as an FYI about it
16:54:27 <miabbott> one thing we can/should investigate is what happens if we update the hw version in the OVF and then try to run it on ESXi 6.5/6.7
16:54:38 <miabbott> i'm not certain about the repercussions in that situation
16:54:56 <jlebon> we could also make it a `cosa buildextend-vmware` knob which RHCOS can use now and we can default to once it's EOL proper
16:54:58 <dustymabe> yeah, not sure - if it's just a warning that the user can Ignore then maybe we don't need a docs page?
16:55:47 <dustymabe> jlebon: ehh - yeah, but not super happy with that either
16:55:58 <dustymabe> i'd prefer not to diverge to much if we don't have to
16:56:06 <dustymabe> and RHCOS leading FCOS feels awkward
16:56:35 <jlebon> don't disagree, but it depends how much we care about users that haven't migrated yet
16:56:39 <miabbott> let me gather more data with our vmware folks in OCP about the hw version and any warnings/errors that might happen if we change it
16:57:12 <miabbott> independent of the hw version change, we could change the firmware being used
16:57:19 <miabbott> i.e. moving from BIOS to UEFI
16:57:29 <miabbott> that unlocks the ability to use secureboot
16:58:17 <jlebon> seems reasonable to me
16:58:31 <dustymabe> #proposed We aren't opposed to updating the OVA to contain more modern defaults, but would like to know how users can revert the behavior if they need to since the platforms aren't EOL yet. We'd either need some documentation to let the users know how or to know that the users can ignore warnings and still proceed.
16:58:32 <ravanelli> .hi
16:58:33 <zodbot> ravanelli: ravanelli 'Renata Renata Andrade Matos Ravanelii' <renata.ravanelli@gmail.com>
16:58:36 <dustymabe> how about something like ^^ for now?
16:58:39 <dustymabe> #chair ravanelli
16:58:39 <zodbot> Current chairs: aaradhak davdunc[m dustymabe fifofonix jlebon miabbott ravanelli skunkerk travier
16:58:57 <dustymabe> ack/nack?
16:59:26 <travier> ack
16:59:31 <fifofonix> +1 (i'm on vsphere 6.7 with terraform deploying next ovf daily - infra team can't go 7.0 yet due to a blocker even if EOL)
17:00:07 <miabbott> oh good to know fifofonix.  i had no idea how many people were using FCOS on vmware
17:00:23 <miabbott> dustymabe: could we do a separate proposal for the firmware change?
17:00:35 <miabbott> that one isn't tied to any hw version change
17:00:50 <jlebon> +0.5
17:01:05 <dustymabe> miabbott: depends.. I'm kind of a mind to bundle the changes together
17:01:26 <miabbott> ack; i'll take the action to finish gathering data then
17:01:47 <jlebon> i think this is probably fine, but also it's hard to evaluate this without knowing how many people will be affected
17:01:57 <dustymabe> miabbott: that's mostly so that we're not constantly giving users PSA
17:02:08 * miabbott nods
17:02:28 <ravanelli> dustymabe: Sorry, I'm late for today. Here is the action update #127. The gcc ppc64le team was able build fcos using F36 as host in a P9, P8 failed completely in the boot part for F36, they are looking at it, looks the issue is related to the bits change. I will keep an eye on how it is going
17:02:28 <dustymabe> ok I've got one ack one +1 and one .5
17:02:43 <miabbott> +1 to proposed
17:02:52 <fifofonix> (also happy to trial a next-modified version with new defaults for you in our 6.7 env if helpful to check no adverse messages)
17:03:06 <dustymabe> fifofonix: that's awesome
17:03:11 <jlebon> fifofonix++
17:03:11 <zodbot> jlebon: Karma for fifofonix changed to 2 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:03:17 <travier> fifofonix: thanks! that would be really useful
17:03:22 <miabbott> fifofonix: thanks for the offer; we can make a test OVA with the changes and give it to you
17:03:43 <dustymabe> #agreed We aren't opposed to updating the OVA to contain more modern defaults, but would like to know how users can revert the behavior if they need to since the platforms aren't EOL yet. We'd either need some documentation to let the users know how or to know that the users can ignore warnings and still proceed.
17:03:46 <fifofonix> no problem.   always eager to participate in next stuff if needs be.  don't hesitate to ask.
17:03:53 <miabbott> fifofonix++
17:03:53 <zodbot> miabbott: Karma for fifofonix changed to 3 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:04:01 <dustymabe> miabbott: want me to leave the `meeting` label on the ticket and we'll see if there is more information next week?
17:04:08 <miabbott> yup, works for me
17:04:36 <dustymabe> #topic open floor
17:04:58 <dustymabe> #info Here is the action update #127. The gcc ppc64le team was able build fcos using F36 as host in a P9, P8 failed completely in the boot part for F36, they are looking at it, looks the issue is related to the bits change. I will keep an eye on how it is going
17:05:17 <dustymabe> ravanelli: want me to open a ticket for that F36 change so we can collect notes there?
17:06:09 <ravanelli> dustymabe: sure, where where exactly should I open this?
17:06:17 <dustymabe> i'll open it and link you back to it
17:06:49 <dustymabe> any other topics for open floor?
17:07:11 <jlebon> neat, lots of time today :)
17:07:21 <dustymabe> anything we should add to our running "this month in FCOS" ticket? https://github.com/coreos/fedora-coreos-tracker/issues/1117
17:07:51 <fifofonix> no open items but just to note i'
17:08:04 <jlebon> we'll be adding the next-devel rebase there soon
17:08:12 <fifofonix> m happy with stability of terraform aws gpu nodes running tensorflow across next/testing/stable. :o)
17:08:22 <dustymabe> fifofonix: awesome!
17:08:34 <davdunc[m> fifofonix: that's newsworthy!
17:09:05 <dustymabe> fifofonix: are you able to manually specify an AMI to your setup? soon we'll have an AMI based on F36 that might be good to have you run through your setup
17:09:27 <jlebon> cool!
17:09:51 <fifofonix> can do.  i was waiting for next to have f36 and on tenter hooks. i think the comms on podman is good but want to see it.
17:10:07 <travier> fifofonix++
17:10:20 <dustymabe> fifofonix: if I gave you an AMI today could you use it? I'll kick off a manual build if so
17:10:51 <fifofonix> yes.  not today but in the next few days I can use it.
17:10:56 <dustymabe> nice
17:11:03 <dustymabe> fifofonix++
17:11:09 <dustymabe> any other topics for open floor?
17:11:21 * dustymabe looks to see when time changes happen
17:12:47 <ravanelli> dustymabe: Sounds good, thanks ;)
17:12:48 <dustymabe> #info Your time may shift in your locale in the coming weeks, but FCOS community meetings will stay at 16:30 UTC.
17:13:07 <dustymabe> FYI everyone ^^
17:13:17 <dustymabe> the US changes March 13th
17:14:22 <dustymabe> anything else for open floor before I close out?
17:15:44 <dustymabe> #endmeeting