2024-02-27 16:00:37 <@coremodule:fedora.im> !startmeeting Fedora ARM Working Group Meeting
2024-02-27 16:00:38 <@meetbot:fedora.im> Meeting started at 2024-02-27 16:00:37 UTC
2024-02-27 16:00:38 <@meetbot:fedora.im> The Meeting name is 'Fedora ARM Working Group Meeting'
2024-02-27 16:00:45 <@jforbes:fedora.im> morning
2024-02-27 16:00:46 <@coremodule:fedora.im> !chair pwhalen pbrobinson jlinton jforbes coremodule
2024-02-27 16:00:52 <@coremodule:fedora.im> !topic roll call
2024-02-27 16:00:57 <@coremodule:fedora.im> morning jforbes
2024-02-27 16:01:09 <@coremodule:fedora.im> !topic 1) ==== Stable Release ====
2024-02-27 16:01:12 <@linuxfriend01:fedora.im> Good afternoon
2024-02-27 16:01:46 <@coremodule:fedora.im> !info Fedora-39-20240227.0  
2024-02-27 16:01:52 <@coremodule:fedora.im> !link https://openqa.fedoraproject.org/tests/overview?distri=fedora&version=39&build=Fedora-Cloud-39-20240227.0&groupid=5
2024-02-27 16:01:57 <@coremodule:fedora.im> Good afternoon Andreas
2024-02-27 16:02:11 <@coremodule:fedora.im> Anything for F39? Nothing from me.
2024-02-27 16:02:27 <@jforbes:fedora.im> Nothing of particular interest from me
2024-02-27 16:02:53 <@coremodule:fedora.im> !topic 2) ==== Candidate Release ====
2024-02-27 16:02:58 <@coremodule:fedora.im> !info Fedora-40-20240227.n.0   
2024-02-27 16:03:03 <@coremodule:fedora.im> !link https://openqa.fedoraproject.org/tests/overview?distri=fedora&version=40&build=Fedora-40-20240227.n.0&groupid=5
2024-02-27 16:03:28 <@coremodule:fedora.im> !info Raspberry Pi 4 and 400 won't boot into graphical environment
2024-02-27 16:03:32 <@coremodule:fedora.im> !link https://bugzilla.redhat.com/show_bug.cgi?id=2264415
2024-02-27 16:03:36 <@coremodule:fedora.im> !info Fedora 40 aarch64 Workstation disk image is oversize
2024-02-27 16:03:40 <@coremodule:fedora.im> !link https://bugzilla.redhat.com/show_bug.cgi?id=2247611
2024-02-27 16:03:48 <@coremodule:fedora.im> These are the major bugs I see for aarch64.
2024-02-27 16:03:56 <@coremodule:fedora.im> Let me look at the RPi bug quick...
2024-02-27 16:05:08 <@jlinton:fedora.im> One of these years we are going to have to come up with a better way to handle DTs.
2024-02-27 16:05:21 <@coremodule:fedora.im> Hi pwhalen, jlinton
2024-02-27 16:06:21 <@pwhalen:fedora.im> coremodule howdy! Are you looking for the lvm bug? 
2024-02-27 16:07:30 <@pwhalen:fedora.im> For the display/dtb issues I believe Peter has found the issue (but could be wrong), and will be working on it when he returns from PTO (tomorrow)
2024-02-27 16:08:03 <@coremodule:fedora.im> sorry, multitasking. No, I was just looking at the display bug
2024-02-27 16:09:22 <@jlinton:fedora.im> !link https://bugzilla.redhat.com/show_bug.cgi?id=2259264
2024-02-27 16:09:27 <@jlinton:fedora.im> is still a problem too.
2024-02-27 16:09:53 <@coremodule:fedora.im> jlinton, right, I keep missing that one.
2024-02-27 16:10:27 <@jlinton:fedora.im> You need some newer EDK2 machines, and it would be hard to forget.
2024-02-27 16:10:41 <@coremodule:fedora.im> 😂 dont curse me with that
2024-02-27 16:11:11 <@coremodule:fedora.im> It looks like we're waiting for pjones, who is in turn waiting on Microsoft at this point?
2024-02-27 16:11:45 <@jlinton:fedora.im> Well pbrobinsons comment seems appropriate, since its shim-unsigned which is the problem, and that shouldn't require microsoft.
2024-02-27 16:12:02 <@jforbes:fedora.im> shim-unsigned should have a recent build
2024-02-27 16:12:34 <@jlinton:fedora.im> shim-unsigned-aarch64, which doesn't unless its not appearing on koji.
2024-02-27 16:13:33 <@jforbes:fedora.im> Oh, I bet he builds other arches when he does the signed package
2024-02-27 16:14:47 <@jlinton:fedora.im> Makes sense because if there are issues during the signing, respinning all the unsigned ones might create some churn, but we also want a signed one.
2024-02-27 16:15:12 <@jforbes:fedora.im> Once the singed binary comes back, the easiest way to get it into an update is to build shim with everything and file it as an update
2024-02-27 16:15:24 <@jlinton:fedora.im> Same goes for systemd-boot-unsigned.
2024-02-27 16:15:45 <@jforbes:fedora.im> signed even, though singed might be appropriate too.
2024-02-27 16:16:02 <@jlinton:fedora.im> which by itself might be nice if it were just signed with the fedora keys, and then people could flush their secure boot setup and manually enroll the fedora ones.
2024-02-27 16:16:11 <@jlinton:fedora.im> which is long term better than asking MS.
2024-02-27 16:17:47 <@jlinton:fedora.im> Which will break multiboot, but isn't worse than having to go in/out of the firmware to enable/disable secure boot for windows+fedora.
2024-02-27 16:18:01 <@jforbes:fedora.im> Perhaps, though we do try to keep the barrier to entry lower than "enroll this key"
2024-02-27 16:19:13 <@jlinton:fedora.im> It could be largely transparent, with a giant red stop/warning sign to inform people of the side effects (nothing but fedora's going to boot).
2024-02-27 16:21:12 <@coremodule:fedora.im> Well, that's all I had for 40, let's move on to the kernel if there's nothing else
2024-02-27 16:21:32 <@coremodule:fedora.im> !topic 3) ==== Kernel Status ====
2024-02-27 16:21:39 <@coremodule:fedora.im> !info F39 kernel-6.7.6-200.fc39
2024-02-27 16:21:44 <@coremodule:fedora.im> !link https://koji.fedoraproject.org/koji/buildinfo?buildID=2408601
2024-02-27 16:21:48 <@coremodule:fedora.im> !info F40 kernel-6.8.0-0.rc6.49.fc40
2024-02-27 16:21:51 <@coremodule:fedora.im> !link https://koji.fedoraproject.org/koji/buildinfo?buildID=2410823
2024-02-27 16:21:55 <@coremodule:fedora.im> !info Rawhide kernel-6.8.0-0.rc6.49.fc41
2024-02-27 16:21:59 <@coremodule:fedora.im> !link https://koji.fedoraproject.org/koji/buildinfo?buildID=2410583
2024-02-27 16:22:06 <@jforbes:fedora.im> Nothing particularly exciting for the kernel. Rawhide and F30 are on rc6, configs are set, so please test and let us know if anything needs to change
2024-02-27 16:22:12 <@jforbes:fedora.im> err F40
2024-02-27 16:22:35 <@coremodule:fedora.im> Cool, will do jforbes. Thanks for the update. I haven't booted a recent kernel on hardware since last week.
2024-02-27 16:23:45 <@pwhalen:fedora.im> I still see issues with no kernel output on kernels with the git suffix, is that known? Might just be in virt 
2024-02-27 16:24:00 <@jforbes:fedora.im> Of note, in just over a week, I will be out for an extended period of time for medical reasons. acaringi will be handling stable Fedora, and I think Patrick will do rawhide builds. kernel-ark should continue as usual.
2024-02-27 16:24:27 <@jforbes:fedora.im> oh? There was an issue with no output on virt, but that was a mutter problem
2024-02-27 16:24:37 <@nirik:matrix.scrye.com> jforbes: best of luck... hope everything goes super smoothly.
2024-02-27 16:24:46 <@jforbes:fedora.im> Thanks, me too
2024-02-27 16:25:09 <@jlinton:fedora.im> Yes, I hope everything goes well for you.
2024-02-27 16:25:25 <@coremodule:fedora.im> jforbes: Yes, agreed. Hope you're OK jforbes.
2024-02-27 16:25:35 <@jforbes:fedora.im> pwhalen: Oh, is this the thing with git suffix kernels only, like rc5 was fine, git snapshots in between were not, rc6 is fine?
2024-02-27 16:25:48 <@pwhalen:fedora.im> jforbesexactly 
2024-02-27 16:26:32 <@jforbes:fedora.im> I thought that was specific to coreos, but there is no reason that should be the case. The only difference is the name. We do not build git snapshots any differently
2024-02-27 16:26:37 <@jlinton:fedora.im> Its not clear if its a dracut issue, but the Renesas RZ doesn't sound like its booting the installers cleanly yet, so there might be more config options (networking?) that need tweaking. 
2024-02-27 16:26:54 <@jforbes:fedora.im> perhaps something has issue with the uname length?
2024-02-27 16:27:06 <@pwhalen:fedora.im> jforbesspecifically on iot, so ostree as well 
2024-02-27 16:28:09 <@jforbes:fedora.im> Right, so I would guess something in ostree has an issue with the filename length for git snapshots or similar?
2024-02-27 16:28:15 <@dustymabe:matrix.org> jforbes: yeah not sure what the issue is. The systems just don't boot
2024-02-27 16:28:39 <@dustymabe:matrix.org> https://github.com/coreos/fedora-coreos-tracker/issues/1647
2024-02-27 16:28:43 <@dustymabe:matrix.org> it's just on aarch64 for us
2024-02-27 16:28:53 <@pwhalen:fedora.im> Same for iot 
2024-02-27 16:29:13 <@jforbes:fedora.im> but they used to... What else changed when this trend started?
2024-02-27 16:29:14 <@dustymabe:matrix.org> if it was an issue with filename length I would expect it not to be aarch64 specific 
2024-02-27 16:29:33 <@pwhalen:fedora.im> dustymabedo they boot ok on actual hardware?
2024-02-27 16:29:34 <@dustymabe:matrix.org> jforbes: for us it was 6.8 IIUC
2024-02-27 16:29:59 <@jforbes:fedora.im> dustymabe: aarch64 is the longest arch name, so it could still be
2024-02-27 16:30:53 <@dustymabe:matrix.org> pwhalen: I haven't really had time to dig into this. We asked someone on our team to look into it, but I think they've got other fires going on too 
2024-02-27 16:31:38 <@pwhalen:fedora.im> dustymabeok, will be doing some more testing on hw today too, will let you know how it goes 
2024-02-27 16:32:24 <@jforbes:fedora.im> might be worth manually editing the filename, shave 1 char off of it and see what that does. Would need to be both vmlinuz and the initramfs
2024-02-27 16:33:06 <@dustymabe:matrix.org> > dustymabe: aarch64 is the longest arch name, so it could still be

I guess an easy way to test this theory is to get a build with a few characters of the snapshot chopped off? could you do that?
2024-02-27 16:33:14 <@pwhalen:fedora.im> will do 
2024-02-27 16:33:49 <@pwhalen:fedora.im> (that was to jforbes :))
2024-02-27 16:34:28 <@dustymabe:matrix.org> yeah. I wasn't sure if it would be that easy to test by just hand editting files or if a more complete test would be to get a kernel build with fewer chars across the board 
2024-02-27 16:34:31 <@jforbes:fedora.im> Yeah, though still interested in trying just renaming the file too.  That tells us whether it is filename or uname
2024-02-27 16:36:29 <@coremodule:fedora.im> !topic 4) ==== Bootloader Status ====
2024-02-27 16:36:34 <@coremodule:fedora.im> !info uboot-tools-2024.04-0.2.rc2.fc40
2024-02-27 16:36:37 <@coremodule:fedora.im> !link https://koji.fedoraproject.org/koji/buildinfo?buildID=2403298
2024-02-27 16:36:50 <@coremodule:fedora.im> I don't think anything has changed since last week here. Test of course is welcome
2024-02-27 16:37:39 <@coremodule:fedora.im> !topic 5) ==== Fedora Rawhide status ===
2024-02-27 16:37:43 <@coremodule:fedora.im> !info OpenQA Fedora-Rawhide-20240227.n.0     
2024-02-27 16:37:50 <@coremodule:fedora.im> !info https://openqa.fedoraproject.org/tests/overview?distri=fedora&version=Rawhide&build=Fedora-Rawhide-20240227.n.0&groupid=5
2024-02-27 16:38:26 <@coremodule:fedora.im> Rawhide looks OK, but of course not a big focus right now. Major testing efforts should be on F40 at this stage.
2024-02-27 16:39:08 <@coremodule:fedora.im> !topic 6) ==== Open Floor ====
2024-02-27 16:39:16 <@coremodule:fedora.im> And that's all I had! Anything for open floor time?
2024-02-27 16:39:36 <@pwhalen:fedora.im> Testing and karma appreciated - https://bodhi.fedoraproject.org/updates/FEDORA-2024-5b91f30bca
2024-02-27 16:40:06 <@pwhalen:fedora.im> that should fix problems running on a server install 
2024-02-27 16:40:46 <@coremodule:fedora.im> Cool, I will test later today!
2024-02-27 16:41:06 <@jforbes:fedora.im> pwhalen dustymabe  https://koji.fedoraproject.org/koji/taskinfo?taskID=114139309
2024-02-27 16:41:29 <@jforbes:fedora.im> Will take a bit, but I shaved some chars off of that one
2024-02-27 16:41:40 <@pwhalen:fedora.im> perfect, thanks. I will try the rename too 
2024-02-27 16:42:05 <@jforbes:fedora.im> Yeah, if this works, I would still like a plain rename without the rebuild so that uname is longer
2024-02-27 16:43:09 <@nirik:matrix.scrye.com> just out of curiosity, I noticed that the aarch64 kernel builds take almost twice the time the rest do? anyone know why that might be the case?
2024-02-27 16:43:51 <@jforbes:fedora.im> Slow hardware. It used to be neck and neck between x86_64 and aarch64 until the recent x86 hardware upgrades
2024-02-27 16:44:22 <@jlinton:fedora.im> Is there still a 8 core parallelization limit? 
2024-02-27 16:44:59 <@nirik:matrix.scrye.com> the hw is... .not slow usually, unless I did something wrong with the vm's
2024-02-27 16:45:09 <@jlinton:fedora.im> Those ampere's are more than capable of burning most recent x86 hardware due to the shear number of cores they have, but this has been a problem for a while now.
2024-02-27 16:45:22 <@nirik:matrix.scrye.com> the virthosts are new mt snow boxes with nvme drives... builders are vm's on them.
2024-02-27 16:46:51 <@jforbes:fedora.im> nirik: aarch64 and x86_64 had been taking the same amount of time for a long while now, basically since we upgraded the aarch64 hardware (it used to be much longer).  For a long time, I never knew which arch would finish last per build... It is only since the hardware upgrade for x86 that it is consistently building much faster
2024-02-27 16:46:56 <@nirik:matrix.scrye.com> anyhow, just noticed it...
2024-02-27 16:47:30 <@nirik:matrix.scrye.com> ah... so they both used to take around 2 hours and now x86 finishes in an hour is?
2024-02-27 16:47:43 <@jforbes:fedora.im> That said, I am all in favor of anything we could do to speed thing up
2024-02-27 16:47:55 <@jforbes:fedora.im> Yes, they both used to take 2 hoursish
2024-02-27 16:48:06 <@jforbes:fedora.im> aarch64 before the last upgrade was ~3hrs
2024-02-27 16:48:28 <@nirik:matrix.scrye.com> ok. yeah, I am not sure what I can do, but if anyone has ideas let me know.
2024-02-27 16:49:22 <@nirik:matrix.scrye.com> the buildvm's have 12 cpus each now... they used to have 8 before this last move
2024-02-27 16:49:23 <@jforbes:fedora.im> scratch builds. x86 is slower still I think
2024-02-27 16:49:46 <@nirik:matrix.scrye.com> yeah, scratch builds wouldn't get the dedicated builders...
2024-02-27 16:49:57 <@jforbes:fedora.im> Well, I am always in favor of anything that speeds up builds, but I think 2 hours is acceptable
2024-02-27 16:50:24 <@nirik:matrix.scrye.com> its been a lot worse in the past. ;)
2024-02-27 16:50:42 <@jforbes:fedora.im> We have some stuff pending for koji, hopefully this summer which will split us into "task per variant" not just task per arch. That should speed us up too.  debug will build on separate hardware from standard
2024-02-27 16:51:21 <@jforbes:fedora.im> And allows us to add things like 16k for aarch, and rt without massively slowing the builds
2024-02-27 16:52:54 <@jforbes:fedora.im> For Fedora, we only build debug kernels for x86 and aarch too, so ppc and s390x are building 1 kernel each, x86 and aarch are building 2
2024-02-27 16:59:04 <@coremodule:fedora.im> !endmeeting