2024-06-26 16:30:39 <@jbtrystram:matrix.org> !startmeeting fedora_coreos_meeting
2024-06-26 16:30:41 <@meetbot:fedora.im> Meeting started at 2024-06-26 16:30:39 UTC
2024-06-26 16:30:41 <@meetbot:fedora.im> The Meeting name is 'fedora_coreos_meeting'
2024-06-26 16:30:44 <@jbtrystram:matrix.org> !topic roll call
2024-06-26 16:30:46 <@jbtrystram:matrix.org> !hi
2024-06-26 16:30:48 <@zodbot:fedora.im> Jean-Baptiste Trystram (jbtrystram) - he / him / his
2024-06-26 16:30:52 <@aaradhak:matrix.org> !hi aaradhak
2024-06-26 16:30:53 <@zodbot:fedora.im> Aashish Radhakrishnan (aaradhak)
2024-06-26 16:30:56 <@hricky:fedora.im> !hi
2024-06-26 16:30:57 <@zodbot:fedora.im> Hristo Marinov (hricky) - he / him / his
2024-06-26 16:32:30 <@gurssing:matrix.org> !hi gursewak
2024-06-26 16:32:31 <@zodbot:fedora.im> Gursewak Singh (gursewak)
2024-06-26 16:32:31 <@mnguyen:fedora.im>  !hi
2024-06-26 16:32:55 <@mnguyen:fedora.im> !hi mnguyen
2024-06-26 16:32:56 <@zodbot:fedora.im> Michael Nguyen (mnguyen)
2024-06-26 16:32:59 <@marmijo:fedora.im> !hi
2024-06-26 16:33:01 <@zodbot:fedora.im> Michael Armijo (marmijo)
2024-06-26 16:33:09 <@jlebon:fedora.im> !hi
2024-06-26 16:33:10 <@zodbot:fedora.im> None (jlebon)
2024-06-26 16:34:17 <@jbtrystram:matrix.org> okay let's start ! 
2024-06-26 16:34:53 <@jbtrystram:matrix.org> The meeting last week was aborted due to zodbot not working and i could not find any action items from the week before 
2024-06-26 16:35:16 <@jbtrystram:matrix.org> so let's jump right into our topics for today as we have a few lined up
2024-06-26 16:35:28 <@jbtrystram:matrix.org> !topic Trigger a bootupd update before landing latest 6.9 kernel update in Fedora CoreOS
2024-06-26 16:35:37 <@jbtrystram:matrix.org> !link https://github.com/coreos/fedora-coreos-tracker/issues/1752
2024-06-26 16:36:39 <@jbtrystram:matrix.org> I'll intro this one since travier is not around today.  The 6.9 kernel won't boot with a shim bootloader that dates from F39
2024-06-26 16:38:09 <@jbtrystram:matrix.org> since we don't update bootloader by default in fedora coreOS, users having secureboot enabled will fail to boot once the kernel 6.9 lands
2024-06-26 16:39:06 <@jbtrystram:matrix.org> we did the same process not too long ago but that was limited to aarch64 
2024-06-26 16:39:51 <@jlebon:fedora.im> i'm confused though. shim hasn't changed in a while, not even in f40. is it the shim EFI executables or the grub ones that are stale?
2024-06-26 16:40:21 <@dustymabe:matrix.org> !hi
2024-06-26 16:40:23 <@zodbot:fedora.im> Dusty Mabe (dustymabe) - he / him / his
2024-06-26 16:40:24 <@jlebon:fedora.im> ok, https://bodhi.fedoraproject.org/updates/?packages=shim shows an f38 update 3 months ago
2024-06-26 16:40:31 <@jbtrystram:matrix.org> (not too long means a bit more than a year ago for kernel 6.2 in https://github.com/coreos/fedora-coreos-tracker/issues/1441)
2024-06-26 16:40:43 <@jlebon:fedora.im> presumably that one is what's in f38+ currently
2024-06-26 16:41:36 <@jbtrystram:matrix.org> why was is it marked only for f38 in bodhi ? 
2024-06-26 16:41:51 <@jlebon:fedora.im> it entered FCOS around that time too via https://github.com/coreos/fedora-coreos-config/commit/88e72b4af8ed8379430630c446ff2425caca23b6
2024-06-26 16:42:36 <@jlebon:fedora.im> when it was still on f39
2024-06-26 16:42:36 <@jlebon:fedora.im> so _assuming_ this is the fixed version, anything installed before that FCOS 39 release is what's broken
2024-06-26 16:42:53 <@jlebon:fedora.im> jbtrystram: shim is kinda special, it's not versioned per Fedora release and is shared across multiple
2024-06-26 16:43:28 <@dustymabe:matrix.org> fun - i guess we kinda knew we'd hid something like this one day 
2024-06-26 16:43:51 <@jbtrystram:matrix.org> Jonathan Lebon: you did the test today, did you check what shim version was that ?
2024-06-26 16:44:11 <@jbtrystram:matrix.org> Jonathan Lebon: you did the test today, did you check what shim version was in the intial f38 ?
2024-06-26 16:44:19 <@jlebon:fedora.im> what's unfortunate is that this was reported in FSB for a while now, but somehow we didn't connect the dots.
2024-06-26 16:44:31 <@jlebon:fedora.im> jbtrystram: that test was from f38 to f40
2024-06-26 16:44:49 <@jlebon:fedora.im> so definitely older shim
2024-06-26 16:46:02 <@dustymabe:matrix.org> we do have upgrade tests that run with secureboot enabled, right?
2024-06-26 16:46:15 <@jbtrystram:matrix.org> will we be able to enable auto-updates once https://github.com/coreos/bootupd/pull/669 lands ?
2024-06-26 16:46:15 <@dustymabe:matrix.org> we do have extended upgrade tests that run with secureboot enabled, right?
2024-06-26 16:47:14 <@jlebon:fedora.im> jbtrystram: i think that's the key last bit, yeah
2024-06-26 16:47:54 <@jbtrystram:matrix.org> cool 
2024-06-26 16:48:15 <@jlebon:fedora.im> 6.9 is already in testing-devel, so shipping a barrier release which updates the ESP would require ad-hoc releases
2024-06-26 16:48:22 <@dustymabe:matrix.org> am I muted?
2024-06-26 16:48:42 <@jlebon:fedora.im> dustymabe: i see you
2024-06-26 16:48:49 <@jlebon:fedora.im> see https://github.com/coreos/fedora-coreos-tracker/issues/1752#issuecomment-2192031828
2024-06-26 16:49:27 <@jlebon:fedora.im> though i guess we could pin to 6.8 in testing-devel for one cycle too
2024-06-26 16:49:37 <@dustymabe:matrix.org> ahh. wonder why we wouldn't have seen this earlier in rawhide though ?
2024-06-26 16:51:33 <@jbtrystram:matrix.org> not in any production stream though? what do you mean by `ad-hoc` release ?  a release outside of the usual cadence ?
2024-06-26 16:51:42 <@jlebon:fedora.im> ahhh, for the last successful rawhide build that triggered upgrade tests, those kola runs have already been GC'ed
2024-06-26 16:52:45 <@jbtrystram:matrix.org> Do we care if streams outside of the production ones are broken ? (i.e. should we do the shim updates on all streams or just production)  
2024-06-26 16:52:57 <@dustymabe:matrix.org> production streams are all that matter
2024-06-26 16:53:16 <@jlebon:fedora.im> jbtrystram: yeah, exactly
2024-06-26 16:53:31 <@dustymabe:matrix.org> i think what jlebon was saying is we either need to do an ad-hoc release OR (another option) is we pin the kernel in testing-devel 
2024-06-26 16:53:32 <@jlebon:fedora.im> (re. "a release outside of the usual cadence")
2024-06-26 16:54:25 <@jbtrystram:matrix.org> ad-hoc or not we need a barrier and a migration systemd unit
2024-06-26 16:54:47 <@dustymabe:matrix.org> agree - we can probably use some of the same code/logic from the last time we encountered the issue
2024-06-26 16:55:13 <@dustymabe:matrix.org> it wasn't the same issue, but it was the same fix (bootloader needed updating)
2024-06-26 16:55:44 <@jbtrystram:matrix.org> yeah, thanks for doing that the first time :) 
2024-06-26 16:56:04 <@jlebon:fedora.im> https://github.com/coreos/fedora-coreos-tracker/issues/1752#issuecomment-2192203204
2024-06-26 16:56:07 <@jbtrystram:matrix.org> should we vote (between ad-hoc release or pin linux-6.8) ?
2024-06-26 16:56:38 <@jbtrystram:matrix.org> Nice find Jonathan Lebon 
2024-06-26 16:57:18 <@dustymabe:matrix.org> one thing we should consider doing at the same time is shipping a fix for https://github.com/coreos/fedora-coreos-tracker/issues/1724 
2024-06-26 16:57:31 <@jlebon:fedora.im> i think my inclination is: unless someone is ready to own the ad-hoc releases, I'd prefer pinning to 6.8 for now and adding/removing the barrier during regular cycles
2024-06-26 16:58:06 <@dustymabe:matrix.org> +1 for pin to 6.8 and normal release cadence 
2024-06-26 16:58:31 <@dustymabe:matrix.org> 
2024-06-26 16:58:31 <@dustymabe:matrix.org> i am wondering though, should we just not conditionalize this 
2024-06-26 16:58:51 <@marmijo:fedora.im> I dont mind doing the ad-hoc releases if needed, but +1 from me on pinning also
2024-06-26 17:00:14 <@jlebon:fedora.im> dustymabe: until we have safer auto-updates (which hopefully should be soon), I'd rather be more conservative
2024-06-26 17:00:45 <@jlebon:fedora.im> hmm, i wonder also how/whether this affects aarch64. something to dig into
2024-06-26 17:01:06 <@jlebon:fedora.im> does SB work on aarch64? /me tries
2024-06-26 17:01:15 <@dustymabe:matrix.org> IIUC secureboot isn't supported on aarch64 in Fedora
2024-06-26 17:01:35 <@dustymabe:matrix.org> but we should confirm that
2024-06-26 17:01:47 <@jlebon:fedora.im> i have a vague memory of that as well, yeah
2024-06-26 17:01:48 <@jbtrystram:matrix.org> +1 to pin and regular cadence, because it's less work and delaying a kernel update for a few weeks isn't too bad
2024-06-26 17:01:52 <@dustymabe:matrix.org> i just don't think we have the signing infra for it
2024-06-26 17:03:14 <@jlebon:fedora.im> ok, jbtrystram want to do a proposed? :)
2024-06-26 17:03:45 <@dustymabe:matrix.org> Jonathan Lebon: can we agree to try to fix https://github.com/coreos/fedora-coreos-tracker/issues/1724 at the same time? 
2024-06-26 17:04:10 <@nhanlon:beeper.com> it's not supported -- fedora would need to get a a64 shim signed
2024-06-26 17:05:02 <@jbtrystram:matrix.org> !info proposed : pin kernel 6.8 and add a barrier release updating the bootloader on secureboot enabled system for the next release  
2024-06-26 17:05:13 <@jlebon:fedora.im> dustymabe: let me refresh my memory on this a bit
2024-06-26 17:05:42 <@jbtrystram:matrix.org> should we decide now how long we pin the kernel  ? i guess unpinning right after is fine since it's a barrier anyway ?
2024-06-26 17:06:22 <@jlebon:fedora.im> dustymabe: ok right, yeah. i think we basically _have_ to fix that too since it can prevent bootupd from working, which we need here
2024-06-26 17:07:00 <@dustymabe:matrix.org> Jonathan Lebon: right. it's a limited few starting alephs that could be affected, but would be good to fix those while we are here
2024-06-26 17:07:30 <@jlebon:fedora.im> - do regular release
2024-06-26 17:07:30 <@jlebon:fedora.im> - remove migration code
2024-06-26 17:07:30 <@jlebon:fedora.im> - unpin 6.8
2024-06-26 17:07:30 <@jlebon:fedora.im> - pin to 6.8
2024-06-26 17:07:30 <@jlebon:fedora.im> - add migration code for shim and aleph bugs
2024-06-26 17:07:30 <@jlebon:fedora.im> - mark release as barrier
2024-06-26 17:08:13 <@jbtrystram:matrix.org> dustymabe: you mention "while we are here" : it will be separate code, right, if we want to limit the update to secureboot enabled nodes. So still two "risky"operation (one barrier though)
2024-06-26 17:08:56 <@jlebon:fedora.im> it should be two separate systemd units, but the bootupd one would be conditionalized on uefi-secureboot and run After= the first one
2024-06-26 17:10:03 <@jbtrystram:matrix.org> agreed, i just wanted to clear out that it's not a "two birds one stone" situation :)
2024-06-26 17:10:20 <@jbtrystram:matrix.org> okay, so i'll update proposed
2024-06-26 17:10:41 <@jbtrystram:matrix.org> unless we can agree directly, as no-one says they were against so far
2024-06-26 17:11:43 <@jbtrystram:matrix.org>  - pin to 6.8
2024-06-26 17:11:43 <@jbtrystram:matrix.org> !info proposed: 
2024-06-26 17:11:43 <@jbtrystram:matrix.org>   -remove migration code
2024-06-26 17:11:43 <@jbtrystram:matrix.org>  - unpin 6.8
2024-06-26 17:11:43 <@jbtrystram:matrix.org>  - mark release as barrier
2024-06-26 17:11:43 <@jbtrystram:matrix.org>  - do regular release
2024-06-26 17:11:43 <@jbtrystram:matrix.org>  - add migration code for shim and aleph bugs
2024-06-26 17:14:24 <@jlebon:fedora.im> +1 from me
2024-06-26 17:14:24 <@jlebon:fedora.im> i'm used to proposals being prose. but i guess that works too. :)
2024-06-26 17:14:24 <@jlebon:fedora.im> maybe worth specifying though that the ESP update will be confined to systems running secureboot, while the aleph fix will apply to all systems
2024-06-26 17:14:46 <@jbtrystram:matrix.org> +1 for me
2024-06-26 17:14:54 <@dustymabe:matrix.org> +1
2024-06-26 17:15:04 <@jbtrystram:matrix.org> I'll do a nicer sentence for agreed :)
2024-06-26 17:15:13 <@marmijo:fedora.im> +1
2024-06-26 17:15:18 <@aaradhak:matrix.org> +1
2024-06-26 17:16:46 <@jbtrystram:matrix.org> !agreed we will pin kernel 6.8 in a barrier release that will carry code to fix aleph on all impacted systems then update the bootloader only on secureboot enabled systems. Then we will unpin kernel 6.8 and remove the migration code for the next release.  
2024-06-26 17:17:29 <@jbtrystram:matrix.org> let's move to our next topic
2024-06-26 17:17:44 <@jbtrystram:matrix.org> !topic New Package Request: firewalld
2024-06-26 17:17:51 <@jbtrystram:matrix.org> !link https://github.com/coreos/fedora-coreos-tracker/issues/1747
2024-06-26 17:18:13 <@jbtrystram:matrix.org> Hristo Marinov: want to intro that one ?
2024-06-26 17:18:23 <@dustymabe:matrix.org> I feel like last time we skipped this because travier wasn't present - is that accurate?
2024-06-26 17:19:14 <@jbtrystram:matrix.org> ok sorry about that, i'll keep it for next week
2024-06-26 17:19:31 <@jbtrystram:matrix.org> qed firmware missing or f41 changes ?
2024-06-26 17:20:16 <@jbtrystram:matrix.org> !topic qed firmware missing from 40.20240519.3.0
2024-06-26 17:20:22 <@jbtrystram:matrix.org> !link https://github.com/coreos/fedora-coreos-tracker/issues/1746
2024-06-26 17:20:43 <@jlebon:fedora.im> i can intro this
2024-06-26 17:21:16 <@jlebon:fedora.im> so, this slipped by, but it's essentially another instance of linux-firmware splitting off more subpackages that no longer get pulled in
2024-06-26 17:21:41 <@jlebon:fedora.im> this time, firmware for an enterprise-y network adapter
2024-06-26 17:22:02 <@dustymabe:matrix.org> hmm. usually the splitting doesn't happen mid major release 
2024-06-26 17:22:20 <@dustymabe:matrix.org> i wish they would only split on the rawhide branch so we can lump all of those together 
2024-06-26 17:22:37 <@jbtrystram:matrix.org> could we always pull subpackages only for certains packages ?
2024-06-26 17:22:57 <@jbtrystram:matrix.org> or do we want to discuss each firmware we include or not ?
2024-06-26 17:23:02 <@jlebon:fedora.im> we definitely don't want all the subpackages of linux-firmware
2024-06-26 17:23:52 <@jlebon:fedora.im> dustymabe: yeah, not sure what happened there. possibly something we can bring up with the maintainers. ISTM like this would break traditional systems too
2024-06-26 17:25:15 <@jlebon:fedora.im> anyway, i think short-term we should add it back. longer-term we can discuss it; it's not clear to me if the adapter needs the blob to function at all, or only for peak performance/additional tuning
2024-06-26 17:25:23 <@dustymabe:matrix.org> i feel like at least for f40 we should add it back 
2024-06-26 17:25:41 <@dustymabe:matrix.org> but if the subpackage is small enough, even long term we should just keep it? 
2024-06-26 17:26:11 <@jbtrystram:matrix.org> it's 9.5M FYI
2024-06-26 17:26:26 <@jlebon:fedora.im> that's the thing, yeah :) noticed that earlier too when looking at this
2024-06-26 17:26:57 <@jlebon:fedora.im> so assuming it's not required for bootstrapping, we could possibly make an argument for making it day-2 instead (or eventually, part of your container build)
2024-06-26 17:27:49 <@jbtrystram:matrix.org> the users reports having no network at all without the firmware
2024-06-26 17:30:16 <@jlebon:fedora.im> indeed. just want to prod them a bit more on that aspect (e.g. if it can be configured in a way that doesn't require the firmware file)
2024-06-26 17:30:52 <@dustymabe:matrix.org> yeah, but if it's too asinine then maybe let's just add it back (since it was there before) and move on to other fish
2024-06-26 17:31:41 <@jlebon:fedora.im> agreed
2024-06-26 17:33:08 <@jbtrystram:matrix.org> i'll ask in the issue and if setting up network isn't possible without the firmware i'll add it back
2024-06-26 17:33:30 <@jbtrystram:matrix.org> !action jbtrystram to follow-up on `qed-firmware` removal 
2024-06-26 17:33:52 <@jbtrystram:matrix.org> we are out of time for today, let's move to open floot
2024-06-26 17:33:55 <@jbtrystram:matrix.org> floor*
2024-06-26 17:34:34 <@jbtrystram:matrix.org> !topic Open Floor
2024-06-26 17:34:35 <@jlebon:fedora.im> jbtrystram: ahh, just replied in the ticket. i think let's add it back for now at the very least. and let's see what they reply to the question for the longer term
2024-06-26 17:34:56 <@jbtrystram:matrix.org> Jonathan Lebon: thanks ! 
2024-06-26 17:35:10 <@dustymabe:matrix.org> thanks for running the meeting! 
2024-06-26 17:38:01 <@jlebon:fedora.im> otherwise have nothing for open floor. anyone wants to bring something up?
2024-06-26 17:38:01 <@jlebon:fedora.im> we should discuss ownership on that barrier release, but we can do that in the coreos channel.
2024-06-26 17:39:30 <@jlebon:fedora.im> sounds like we can end this meeting :)
2024-06-26 17:40:00 <@jbtrystram:matrix.org> !endmeeting