<@coremodule:fedora.im>
16:00:37
!startmeeting Fedora ARM Working Group Meeting
<@meetbot:fedora.im>
16:00:38
Meeting started at 2024-02-27 16:00:37 UTC
<@meetbot:fedora.im>
16:00:38
The Meeting name is 'Fedora ARM Working Group Meeting'
<@jforbes:fedora.im>
16:00:45
morning
<@coremodule:fedora.im>
16:00:46
!chair pwhalen pbrobinson jlinton jforbes coremodule
<@coremodule:fedora.im>
16:00:52
!topic roll call
<@coremodule:fedora.im>
16:00:57
morning jforbes
<@coremodule:fedora.im>
16:01:09
!topic 1) ==== Stable Release ====
<@linuxfriend01:fedora.im>
16:01:12
Good afternoon
<@coremodule:fedora.im>
16:01:46
!info Fedora-39-20240227.0
<@coremodule:fedora.im>
16:01:52
!link https://openqa.fedoraproject.org/tests/overview?distri=fedora&version=39&build=Fedora-Cloud-39-20240227.0&groupid=5
<@coremodule:fedora.im>
16:01:57
Good afternoon Andreas
<@coremodule:fedora.im>
16:02:11
Anything for F39? Nothing from me.
<@jforbes:fedora.im>
16:02:27
Nothing of particular interest from me
<@coremodule:fedora.im>
16:02:53
!topic 2) ==== Candidate Release ====
<@coremodule:fedora.im>
16:02:58
!info Fedora-40-20240227.n.0
<@coremodule:fedora.im>
16:03:03
!link https://openqa.fedoraproject.org/tests/overview?distri=fedora&version=40&build=Fedora-40-20240227.n.0&groupid=5
<@coremodule:fedora.im>
16:03:28
!info Raspberry Pi 4 and 400 won't boot into graphical environment
<@coremodule:fedora.im>
16:03:32
!link https://bugzilla.redhat.com/show_bug.cgi?id=2264415
<@coremodule:fedora.im>
16:03:36
!info Fedora 40 aarch64 Workstation disk image is oversize
<@coremodule:fedora.im>
16:03:40
!link https://bugzilla.redhat.com/show_bug.cgi?id=2247611
<@coremodule:fedora.im>
16:03:48
These are the major bugs I see for aarch64.
<@coremodule:fedora.im>
16:03:56
Let me look at the RPi bug quick...
<@jlinton:fedora.im>
16:05:08
One of these years we are going to have to come up with a better way to handle DTs.
<@coremodule:fedora.im>
16:05:21
Hi pwhalen, jlinton
<@pwhalen:fedora.im>
16:06:21
coremodule howdy! Are you looking for the lvm bug?
<@pwhalen:fedora.im>
16:07:30
For the display/dtb issues I believe Peter has found the issue (but could be wrong), and will be working on it when he returns from PTO (tomorrow)
<@coremodule:fedora.im>
16:08:03
sorry, multitasking. No, I was just looking at the display bug
<@jlinton:fedora.im>
16:09:22
!link https://bugzilla.redhat.com/show_bug.cgi?id=2259264
<@jlinton:fedora.im>
16:09:27
is still a problem too.
<@coremodule:fedora.im>
16:09:53
jlinton, right, I keep missing that one.
<@jlinton:fedora.im>
16:10:27
You need some newer EDK2 machines, and it would be hard to forget.
<@coremodule:fedora.im>
16:10:41
😂 dont curse me with that
<@coremodule:fedora.im>
16:11:11
It looks like we're waiting for pjones, who is in turn waiting on Microsoft at this point?
<@jlinton:fedora.im>
16:11:45
Well pbrobinsons comment seems appropriate, since its shim-unsigned which is the problem, and that shouldn't require microsoft.
<@jforbes:fedora.im>
16:12:02
shim-unsigned should have a recent build
<@jlinton:fedora.im>
16:12:34
shim-unsigned-aarch64, which doesn't unless its not appearing on koji.
<@jforbes:fedora.im>
16:13:33
Oh, I bet he builds other arches when he does the signed package
<@jlinton:fedora.im>
16:14:47
Makes sense because if there are issues during the signing, respinning all the unsigned ones might create some churn, but we also want a signed one.
<@jforbes:fedora.im>
16:15:12
Once the singed binary comes back, the easiest way to get it into an update is to build shim with everything and file it as an update
<@jlinton:fedora.im>
16:15:24
Same goes for systemd-boot-unsigned.
<@jforbes:fedora.im>
16:15:45
signed even, though singed might be appropriate too.
<@jlinton:fedora.im>
16:16:02
which by itself might be nice if it were just signed with the fedora keys, and then people could flush their secure boot setup and manually enroll the fedora ones.
<@jlinton:fedora.im>
16:16:11
which is long term better than asking MS.
<@jlinton:fedora.im>
16:17:47
Which will break multiboot, but isn't worse than having to go in/out of the firmware to enable/disable secure boot for windows+fedora.
<@jforbes:fedora.im>
16:18:01
Perhaps, though we do try to keep the barrier to entry lower than "enroll this key"
<@jlinton:fedora.im>
16:19:13
It could be largely transparent, with a giant red stop/warning sign to inform people of the side effects (nothing but fedora's going to boot).
<@coremodule:fedora.im>
16:21:12
Well, that's all I had for 40, let's move on to the kernel if there's nothing else
<@coremodule:fedora.im>
16:21:32
!topic 3) ==== Kernel Status ====
<@coremodule:fedora.im>
16:21:39
!info F39 kernel-6.7.6-200.fc39
<@coremodule:fedora.im>
16:21:44
!link https://koji.fedoraproject.org/koji/buildinfo?buildID=2408601
<@coremodule:fedora.im>
16:21:48
!info F40 kernel-6.8.0-0.rc6.49.fc40
<@coremodule:fedora.im>
16:21:51
!link https://koji.fedoraproject.org/koji/buildinfo?buildID=2410823
<@coremodule:fedora.im>
16:21:55
!info Rawhide kernel-6.8.0-0.rc6.49.fc41
<@coremodule:fedora.im>
16:21:59
!link https://koji.fedoraproject.org/koji/buildinfo?buildID=2410583
<@jforbes:fedora.im>
16:22:06
Nothing particularly exciting for the kernel. Rawhide and F30 are on rc6, configs are set, so please test and let us know if anything needs to change
<@jforbes:fedora.im>
16:22:12
err F40
<@coremodule:fedora.im>
16:22:35
Cool, will do jforbes. Thanks for the update. I haven't booted a recent kernel on hardware since last week.
<@pwhalen:fedora.im>
16:23:45
I still see issues with no kernel output on kernels with the git suffix, is that known? Might just be in virt
<@jforbes:fedora.im>
16:24:00
Of note, in just over a week, I will be out for an extended period of time for medical reasons. acaringi will be handling stable Fedora, and I think Patrick will do rawhide builds. kernel-ark should continue as usual.
<@jforbes:fedora.im>
16:24:27
oh? There was an issue with no output on virt, but that was a mutter problem
<@nirik:matrix.scrye.com>
16:24:37
jforbes: best of luck... hope everything goes super smoothly.
<@jforbes:fedora.im>
16:24:46
Thanks, me too
<@jlinton:fedora.im>
16:25:09
Yes, I hope everything goes well for you.
<@coremodule:fedora.im>
16:25:25
jforbes: Yes, agreed. Hope you're OK jforbes.
<@jforbes:fedora.im>
16:25:35
pwhalen: Oh, is this the thing with git suffix kernels only, like rc5 was fine, git snapshots in between were not, rc6 is fine?
<@pwhalen:fedora.im>
16:25:48
jforbesexactly
<@jforbes:fedora.im>
16:26:32
I thought that was specific to coreos, but there is no reason that should be the case. The only difference is the name. We do not build git snapshots any differently
<@jlinton:fedora.im>
16:26:37
Its not clear if its a dracut issue, but the Renesas RZ doesn't sound like its booting the installers cleanly yet, so there might be more config options (networking?) that need tweaking.
<@jforbes:fedora.im>
16:26:54
perhaps something has issue with the uname length?
<@pwhalen:fedora.im>
16:27:06
jforbesspecifically on iot, so ostree as well
<@jforbes:fedora.im>
16:28:09
Right, so I would guess something in ostree has an issue with the filename length for git snapshots or similar?
<@dustymabe:matrix.org>
16:28:15
jforbes: yeah not sure what the issue is. The systems just don't boot
<@dustymabe:matrix.org>
16:28:39
https://github.com/coreos/fedora-coreos-tracker/issues/1647
<@dustymabe:matrix.org>
16:28:43
it's just on aarch64 for us
<@pwhalen:fedora.im>
16:28:53
Same for iot
<@jforbes:fedora.im>
16:29:13
but they used to... What else changed when this trend started?
<@dustymabe:matrix.org>
16:29:14
if it was an issue with filename length I would expect it not to be aarch64 specific
<@pwhalen:fedora.im>
16:29:33
dustymabedo they boot ok on actual hardware?
<@dustymabe:matrix.org>
16:29:34
jforbes: for us it was 6.8 IIUC
<@jforbes:fedora.im>
16:29:59
dustymabe: aarch64 is the longest arch name, so it could still be
<@dustymabe:matrix.org>
16:30:53
pwhalen: I haven't really had time to dig into this. We asked someone on our team to look into it, but I think they've got other fires going on too
<@pwhalen:fedora.im>
16:31:38
dustymabeok, will be doing some more testing on hw today too, will let you know how it goes
<@jforbes:fedora.im>
16:32:24
might be worth manually editing the filename, shave 1 char off of it and see what that does. Would need to be both vmlinuz and the initramfs
<@dustymabe:matrix.org>
16:33:06
> dustymabe: aarch64 is the longest arch name, so it could still be
I guess an easy way to test this theory is to get a build with a few characters of the snapshot chopped off? could you do that?
<@pwhalen:fedora.im>
16:33:14
will do
<@pwhalen:fedora.im>
16:33:49
(that was to jforbes :))
<@dustymabe:matrix.org>
16:34:28
yeah. I wasn't sure if it would be that easy to test by just hand editting files or if a more complete test would be to get a kernel build with fewer chars across the board
<@jforbes:fedora.im>
16:34:31
Yeah, though still interested in trying just renaming the file too. That tells us whether it is filename or uname
<@coremodule:fedora.im>
16:36:29
!topic 4) ==== Bootloader Status ====
<@coremodule:fedora.im>
16:36:34
!info uboot-tools-2024.04-0.2.rc2.fc40
<@coremodule:fedora.im>
16:36:37
!link https://koji.fedoraproject.org/koji/buildinfo?buildID=2403298
<@coremodule:fedora.im>
16:36:50
I don't think anything has changed since last week here. Test of course is welcome
<@coremodule:fedora.im>
16:37:39
!topic 5) ==== Fedora Rawhide status ===
<@coremodule:fedora.im>
16:37:43
!info OpenQA Fedora-Rawhide-20240227.n.0
<@coremodule:fedora.im>
16:37:50
!info https://openqa.fedoraproject.org/tests/overview?distri=fedora&version=Rawhide&build=Fedora-Rawhide-20240227.n.0&groupid=5
<@coremodule:fedora.im>
16:38:26
Rawhide looks OK, but of course not a big focus right now. Major testing efforts should be on F40 at this stage.
<@coremodule:fedora.im>
16:39:08
!topic 6) ==== Open Floor ====
<@coremodule:fedora.im>
16:39:16
And that's all I had! Anything for open floor time?
<@pwhalen:fedora.im>
16:39:36
Testing and karma appreciated - https://bodhi.fedoraproject.org/updates/FEDORA-2024-5b91f30bca
<@pwhalen:fedora.im>
16:40:06
that should fix problems running on a server install
<@coremodule:fedora.im>
16:40:46
Cool, I will test later today!
<@jforbes:fedora.im>
16:41:06
pwhalen dustymabe https://koji.fedoraproject.org/koji/taskinfo?taskID=114139309
<@jforbes:fedora.im>
16:41:29
Will take a bit, but I shaved some chars off of that one
<@pwhalen:fedora.im>
16:41:40
perfect, thanks. I will try the rename too
<@jforbes:fedora.im>
16:42:05
Yeah, if this works, I would still like a plain rename without the rebuild so that uname is longer
<@nirik:matrix.scrye.com>
16:43:09
just out of curiosity, I noticed that the aarch64 kernel builds take almost twice the time the rest do? anyone know why that might be the case?
<@jforbes:fedora.im>
16:43:51
Slow hardware. It used to be neck and neck between x86_64 and aarch64 until the recent x86 hardware upgrades
<@jlinton:fedora.im>
16:44:22
Is there still a 8 core parallelization limit?
<@nirik:matrix.scrye.com>
16:44:59
the hw is... .not slow usually, unless I did something wrong with the vm's
<@jlinton:fedora.im>
16:45:09
Those ampere's are more than capable of burning most recent x86 hardware due to the shear number of cores they have, but this has been a problem for a while now.
<@nirik:matrix.scrye.com>
16:45:22
the virthosts are new mt snow boxes with nvme drives... builders are vm's on them.
<@jforbes:fedora.im>
16:46:51
nirik: aarch64 and x86_64 had been taking the same amount of time for a long while now, basically since we upgraded the aarch64 hardware (it used to be much longer). For a long time, I never knew which arch would finish last per build... It is only since the hardware upgrade for x86 that it is consistently building much faster
<@nirik:matrix.scrye.com>
16:46:56
anyhow, just noticed it...
<@nirik:matrix.scrye.com>
16:47:30
ah... so they both used to take around 2 hours and now x86 finishes in an hour is?
<@jforbes:fedora.im>
16:47:43
That said, I am all in favor of anything we could do to speed thing up
<@jforbes:fedora.im>
16:47:55
Yes, they both used to take 2 hoursish
<@jforbes:fedora.im>
16:48:06
aarch64 before the last upgrade was ~3hrs
<@nirik:matrix.scrye.com>
16:48:28
ok. yeah, I am not sure what I can do, but if anyone has ideas let me know.
<@nirik:matrix.scrye.com>
16:49:22
the buildvm's have 12 cpus each now... they used to have 8 before this last move
<@jforbes:fedora.im>
16:49:23
scratch builds. x86 is slower still I think
<@nirik:matrix.scrye.com>
16:49:46
yeah, scratch builds wouldn't get the dedicated builders...
<@jforbes:fedora.im>
16:49:57
Well, I am always in favor of anything that speeds up builds, but I think 2 hours is acceptable
<@nirik:matrix.scrye.com>
16:50:24
its been a lot worse in the past. ;)
<@jforbes:fedora.im>
16:50:42
We have some stuff pending for koji, hopefully this summer which will split us into "task per variant" not just task per arch. That should speed us up too. debug will build on separate hardware from standard
<@jforbes:fedora.im>
16:51:21
And allows us to add things like 16k for aarch, and rt without massively slowing the builds
<@jforbes:fedora.im>
16:52:54
For Fedora, we only build debug kernels for x86 and aarch too, so ppc and s390x are building 1 kernel each, x86 and aarch are building 2
<@coremodule:fedora.im>
16:59:04
!endmeeting