<@jbtrystram:matrix.org>
16:30:39
!startmeeting fedora_coreos_meeting
<@meetbot:fedora.im>
16:30:41
Meeting started at 2024-06-26 16:30:39 UTC
<@meetbot:fedora.im>
16:30:41
The Meeting name is 'fedora_coreos_meeting'
<@jbtrystram:matrix.org>
16:30:44
!topic roll call
<@jbtrystram:matrix.org>
16:30:46
!hi
<@zodbot:fedora.im>
16:30:48
Jean-Baptiste Trystram (jbtrystram) - he / him / his
<@aaradhak:matrix.org>
16:30:52
!hi aaradhak
<@zodbot:fedora.im>
16:30:53
Aashish Radhakrishnan (aaradhak)
<@hricky:fedora.im>
16:30:56
!hi
<@zodbot:fedora.im>
16:30:57
Hristo Marinov (hricky) - he / him / his
<@gurssing:matrix.org>
16:32:30
!hi gursewak
<@zodbot:fedora.im>
16:32:31
Gursewak Singh (gursewak)
<@mnguyen:fedora.im>
16:32:31
!hi
<@mnguyen:fedora.im>
16:32:55
!hi mnguyen
<@zodbot:fedora.im>
16:32:56
Michael Nguyen (mnguyen)
<@marmijo:fedora.im>
16:32:59
!hi
<@zodbot:fedora.im>
16:33:01
Michael Armijo (marmijo)
<@jlebon:fedora.im>
16:33:09
!hi
<@zodbot:fedora.im>
16:33:10
None (jlebon)
<@jbtrystram:matrix.org>
16:34:17
okay let's start !
<@jbtrystram:matrix.org>
16:34:53
The meeting last week was aborted due to zodbot not working and i could not find any action items from the week before
<@jbtrystram:matrix.org>
16:35:16
so let's jump right into our topics for today as we have a few lined up
<@jbtrystram:matrix.org>
16:35:28
!topic Trigger a bootupd update before landing latest 6.9 kernel update in Fedora CoreOS
<@jbtrystram:matrix.org>
16:35:37
!link https://github.com/coreos/fedora-coreos-tracker/issues/1752
<@jbtrystram:matrix.org>
16:36:39
I'll intro this one since travier is not around today. The 6.9 kernel won't boot with a shim bootloader that dates from F39
<@jbtrystram:matrix.org>
16:38:09
since we don't update bootloader by default in fedora coreOS, users having secureboot enabled will fail to boot once the kernel 6.9 lands
<@jbtrystram:matrix.org>
16:39:06
we did the same process not too long ago but that was limited to aarch64
<@jlebon:fedora.im>
16:39:51
i'm confused though. shim hasn't changed in a while, not even in f40. is it the shim EFI executables or the grub ones that are stale?
<@dustymabe:matrix.org>
16:40:21
!hi
<@zodbot:fedora.im>
16:40:23
Dusty Mabe (dustymabe) - he / him / his
<@jlebon:fedora.im>
16:40:24
ok, https://bodhi.fedoraproject.org/updates/?packages=shim shows an f38 update 3 months ago
<@jbtrystram:matrix.org>
16:40:31
(not too long means a bit more than a year ago for kernel 6.2 in https://github.com/coreos/fedora-coreos-tracker/issues/1441)
<@jlebon:fedora.im>
16:40:43
presumably that one is what's in f38+ currently
<@jbtrystram:matrix.org>
16:41:36
why was is it marked only for f38 in bodhi ?
<@jlebon:fedora.im>
16:41:51
it entered FCOS around that time too via https://github.com/coreos/fedora-coreos-config/commit/88e72b4af8ed8379430630c446ff2425caca23b6
<@jlebon:fedora.im>
16:42:36
when it was still on f39
<@jlebon:fedora.im>
16:42:36
so _assuming_ this is the fixed version, anything installed before that FCOS 39 release is what's broken
<@jlebon:fedora.im>
16:42:53
jbtrystram: shim is kinda special, it's not versioned per Fedora release and is shared across multiple
<@dustymabe:matrix.org>
16:43:28
fun - i guess we kinda knew we'd hid something like this one day
<@jbtrystram:matrix.org>
16:43:51
Jonathan Lebon: you did the test today, did you check what shim version was that ?
<@jbtrystram:matrix.org>
16:44:11
Jonathan Lebon: you did the test today, did you check what shim version was in the intial f38 ?
<@jlebon:fedora.im>
16:44:19
what's unfortunate is that this was reported in FSB for a while now, but somehow we didn't connect the dots.
<@jlebon:fedora.im>
16:44:31
jbtrystram: that test was from f38 to f40
<@jlebon:fedora.im>
16:44:49
so definitely older shim
<@dustymabe:matrix.org>
16:46:02
we do have upgrade tests that run with secureboot enabled, right?
<@jbtrystram:matrix.org>
16:46:15
will we be able to enable auto-updates once https://github.com/coreos/bootupd/pull/669 lands ?
<@dustymabe:matrix.org>
16:46:15
we do have extended upgrade tests that run with secureboot enabled, right?
<@jlebon:fedora.im>
16:47:14
jbtrystram: i think that's the key last bit, yeah
<@jbtrystram:matrix.org>
16:47:54
cool
<@jlebon:fedora.im>
16:48:15
6.9 is already in testing-devel, so shipping a barrier release which updates the ESP would require ad-hoc releases
<@dustymabe:matrix.org>
16:48:22
am I muted?
<@jlebon:fedora.im>
16:48:42
dustymabe: i see you
<@jlebon:fedora.im>
16:48:49
see https://github.com/coreos/fedora-coreos-tracker/issues/1752#issuecomment-2192031828
<@jlebon:fedora.im>
16:49:27
though i guess we could pin to 6.8 in testing-devel for one cycle too
<@dustymabe:matrix.org>
16:49:37
ahh. wonder why we wouldn't have seen this earlier in rawhide though ?
<@jbtrystram:matrix.org>
16:51:33
not in any production stream though? what do you mean by `ad-hoc` release ? a release outside of the usual cadence ?
<@jlebon:fedora.im>
16:51:42
ahhh, for the last successful rawhide build that triggered upgrade tests, those kola runs have already been GC'ed
<@jbtrystram:matrix.org>
16:52:45
Do we care if streams outside of the production ones are broken ? (i.e. should we do the shim updates on all streams or just production)
<@dustymabe:matrix.org>
16:52:57
production streams are all that matter
<@jlebon:fedora.im>
16:53:16
jbtrystram: yeah, exactly
<@dustymabe:matrix.org>
16:53:31
i think what jlebon was saying is we either need to do an ad-hoc release OR (another option) is we pin the kernel in testing-devel
<@jlebon:fedora.im>
16:53:32
(re. "a release outside of the usual cadence")
<@jbtrystram:matrix.org>
16:54:25
ad-hoc or not we need a barrier and a migration systemd unit
<@dustymabe:matrix.org>
16:54:47
agree - we can probably use some of the same code/logic from the last time we encountered the issue
<@dustymabe:matrix.org>
16:55:13
it wasn't the same issue, but it was the same fix (bootloader needed updating)
<@jbtrystram:matrix.org>
16:55:44
yeah, thanks for doing that the first time :)
<@jlebon:fedora.im>
16:56:04
https://github.com/coreos/fedora-coreos-tracker/issues/1752#issuecomment-2192203204
<@jbtrystram:matrix.org>
16:56:07
should we vote (between ad-hoc release or pin linux-6.8) ?
<@jbtrystram:matrix.org>
16:56:38
Nice find Jonathan Lebon
<@dustymabe:matrix.org>
16:57:18
one thing we should consider doing at the same time is shipping a fix for https://github.com/coreos/fedora-coreos-tracker/issues/1724
<@jlebon:fedora.im>
16:57:31
i think my inclination is: unless someone is ready to own the ad-hoc releases, I'd prefer pinning to 6.8 for now and adding/removing the barrier during regular cycles
<@dustymabe:matrix.org>
16:58:06
+1 for pin to 6.8 and normal release cadence
<@dustymabe:matrix.org>
16:58:31
<@dustymabe:matrix.org>
16:58:31
i am wondering though, should we just not conditionalize this
<@marmijo:fedora.im>
16:58:51
I dont mind doing the ad-hoc releases if needed, but +1 from me on pinning also
<@jlebon:fedora.im>
17:00:14
dustymabe: until we have safer auto-updates (which hopefully should be soon), I'd rather be more conservative
<@jlebon:fedora.im>
17:00:45
hmm, i wonder also how/whether this affects aarch64. something to dig into
<@jlebon:fedora.im>
17:01:06
does SB work on aarch64? /me tries
<@dustymabe:matrix.org>
17:01:15
IIUC secureboot isn't supported on aarch64 in Fedora
<@dustymabe:matrix.org>
17:01:35
but we should confirm that
<@jlebon:fedora.im>
17:01:47
i have a vague memory of that as well, yeah
<@jbtrystram:matrix.org>
17:01:48
+1 to pin and regular cadence, because it's less work and delaying a kernel update for a few weeks isn't too bad
<@dustymabe:matrix.org>
17:01:52
i just don't think we have the signing infra for it
<@jlebon:fedora.im>
17:03:14
ok, jbtrystram want to do a proposed? :)
<@dustymabe:matrix.org>
17:03:45
Jonathan Lebon: can we agree to try to fix https://github.com/coreos/fedora-coreos-tracker/issues/1724 at the same time?
<@nhanlon:beeper.com>
17:04:10
it's not supported -- fedora would need to get a a64 shim signed
<@jbtrystram:matrix.org>
17:05:02
!info proposed : pin kernel 6.8 and add a barrier release updating the bootloader on secureboot enabled system for the next release
<@jlebon:fedora.im>
17:05:13
dustymabe: let me refresh my memory on this a bit
<@jbtrystram:matrix.org>
17:05:42
should we decide now how long we pin the kernel ? i guess unpinning right after is fine since it's a barrier anyway ?
<@jlebon:fedora.im>
17:06:22
dustymabe: ok right, yeah. i think we basically _have_ to fix that too since it can prevent bootupd from working, which we need here
<@dustymabe:matrix.org>
17:07:00
Jonathan Lebon: right. it's a limited few starting alephs that could be affected, but would be good to fix those while we are here
<@jlebon:fedora.im>
17:07:30
- do regular release
<@jlebon:fedora.im>
17:07:30
- remove migration code
<@jlebon:fedora.im>
17:07:30
- unpin 6.8
<@jlebon:fedora.im>
17:07:30
- pin to 6.8
<@jlebon:fedora.im>
17:07:30
- add migration code for shim and aleph bugs
<@jlebon:fedora.im>
17:07:30
- mark release as barrier
<@jbtrystram:matrix.org>
17:08:13
dustymabe: you mention "while we are here" : it will be separate code, right, if we want to limit the update to secureboot enabled nodes. So still two "risky"operation (one barrier though)
<@jlebon:fedora.im>
17:08:56
it should be two separate systemd units, but the bootupd one would be conditionalized on uefi-secureboot and run After= the first one
<@jbtrystram:matrix.org>
17:10:03
agreed, i just wanted to clear out that it's not a "two birds one stone" situation :)
<@jbtrystram:matrix.org>
17:10:20
okay, so i'll update proposed
<@jbtrystram:matrix.org>
17:10:41
unless we can agree directly, as no-one says they were against so far
<@jbtrystram:matrix.org>
17:11:43
- pin to 6.8
<@jbtrystram:matrix.org>
17:11:43
!info proposed:
<@jbtrystram:matrix.org>
17:11:43
-remove migration code
<@jbtrystram:matrix.org>
17:11:43
- unpin 6.8
<@jbtrystram:matrix.org>
17:11:43
- mark release as barrier
<@jbtrystram:matrix.org>
17:11:43
- do regular release
<@jbtrystram:matrix.org>
17:11:43
- add migration code for shim and aleph bugs
<@jlebon:fedora.im>
17:14:24
+1 from me
<@jlebon:fedora.im>
17:14:24
i'm used to proposals being prose. but i guess that works too. :)
<@jlebon:fedora.im>
17:14:24
maybe worth specifying though that the ESP update will be confined to systems running secureboot, while the aleph fix will apply to all systems
<@jbtrystram:matrix.org>
17:14:46
+1 for me
<@dustymabe:matrix.org>
17:14:54
+1
<@jbtrystram:matrix.org>
17:15:04
I'll do a nicer sentence for agreed :)
<@marmijo:fedora.im>
17:15:13
+1
<@aaradhak:matrix.org>
17:15:18
+1
<@jbtrystram:matrix.org>
17:16:46
!agreed we will pin kernel 6.8 in a barrier release that will carry code to fix aleph on all impacted systems then update the bootloader only on secureboot enabled systems. Then we will unpin kernel 6.8 and remove the migration code for the next release.
<@jbtrystram:matrix.org>
17:17:29
let's move to our next topic
<@jbtrystram:matrix.org>
17:17:44
!topic New Package Request: firewalld
<@jbtrystram:matrix.org>
17:17:51
!link https://github.com/coreos/fedora-coreos-tracker/issues/1747
<@jbtrystram:matrix.org>
17:18:13
Hristo Marinov: want to intro that one ?
<@dustymabe:matrix.org>
17:18:23
I feel like last time we skipped this because travier wasn't present - is that accurate?
<@jbtrystram:matrix.org>
17:19:14
ok sorry about that, i'll keep it for next week
<@jbtrystram:matrix.org>
17:19:31
qed firmware missing or f41 changes ?
<@jbtrystram:matrix.org>
17:20:16
!topic qed firmware missing from 40.20240519.3.0
<@jbtrystram:matrix.org>
17:20:22
!link https://github.com/coreos/fedora-coreos-tracker/issues/1746
<@jlebon:fedora.im>
17:20:43
i can intro this
<@jlebon:fedora.im>
17:21:16
so, this slipped by, but it's essentially another instance of linux-firmware splitting off more subpackages that no longer get pulled in
<@jlebon:fedora.im>
17:21:41
this time, firmware for an enterprise-y network adapter
<@dustymabe:matrix.org>
17:22:02
hmm. usually the splitting doesn't happen mid major release
<@dustymabe:matrix.org>
17:22:20
i wish they would only split on the rawhide branch so we can lump all of those together
<@jbtrystram:matrix.org>
17:22:37
could we always pull subpackages only for certains packages ?
<@jbtrystram:matrix.org>
17:22:57
or do we want to discuss each firmware we include or not ?
<@jlebon:fedora.im>
17:23:02
we definitely don't want all the subpackages of linux-firmware
<@jlebon:fedora.im>
17:23:52
dustymabe: yeah, not sure what happened there. possibly something we can bring up with the maintainers. ISTM like this would break traditional systems too
<@jlebon:fedora.im>
17:25:15
anyway, i think short-term we should add it back. longer-term we can discuss it; it's not clear to me if the adapter needs the blob to function at all, or only for peak performance/additional tuning
<@dustymabe:matrix.org>
17:25:23
i feel like at least for f40 we should add it back
<@dustymabe:matrix.org>
17:25:41
but if the subpackage is small enough, even long term we should just keep it?
<@jbtrystram:matrix.org>
17:26:11
it's 9.5M FYI
<@jlebon:fedora.im>
17:26:26
that's the thing, yeah :) noticed that earlier too when looking at this
<@jlebon:fedora.im>
17:26:57
so assuming it's not required for bootstrapping, we could possibly make an argument for making it day-2 instead (or eventually, part of your container build)
<@jbtrystram:matrix.org>
17:27:49
the users reports having no network at all without the firmware
<@jlebon:fedora.im>
17:30:16
indeed. just want to prod them a bit more on that aspect (e.g. if it can be configured in a way that doesn't require the firmware file)
<@dustymabe:matrix.org>
17:30:52
yeah, but if it's too asinine then maybe let's just add it back (since it was there before) and move on to other fish
<@jlebon:fedora.im>
17:31:41
agreed
<@jbtrystram:matrix.org>
17:33:08
i'll ask in the issue and if setting up network isn't possible without the firmware i'll add it back
<@jbtrystram:matrix.org>
17:33:30
!action jbtrystram to follow-up on `qed-firmware` removal
<@jbtrystram:matrix.org>
17:33:52
we are out of time for today, let's move to open floot
<@jbtrystram:matrix.org>
17:33:55
floor*
<@jbtrystram:matrix.org>
17:34:34
!topic Open Floor
<@jlebon:fedora.im>
17:34:35
jbtrystram: ahh, just replied in the ticket. i think let's add it back for now at the very least. and let's see what they reply to the question for the longer term
<@jbtrystram:matrix.org>
17:34:56
Jonathan Lebon: thanks !
<@dustymabe:matrix.org>
17:35:10
thanks for running the meeting!
<@jlebon:fedora.im>
17:38:01
otherwise have nothing for open floor. anyone wants to bring something up?
<@jlebon:fedora.im>
17:38:01
we should discuss ownership on that barrier release, but we can do that in the coreos channel.
<@jlebon:fedora.im>
17:39:30
sounds like we can end this meeting :)
<@jbtrystram:matrix.org>
17:40:00
!endmeeting