15:06:32 <pwhalen> #startmeeting Fedora ARM Working Group Meeting
15:06:32 <zodbot> Meeting started Tue Aug  9 15:06:32 2022 UTC.
15:06:32 <zodbot> This meeting is logged and archived in a public location.
15:06:32 <zodbot> The chair is pwhalen. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
15:06:32 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:06:32 <zodbot> The meeting name has been set to 'fedora_arm_working_group_meeting'
15:06:41 <pwhalen> #chair pwhalen pbrobinson jlinton coremodule
15:06:41 <pwhalen> #topic roll call
15:06:41 <zodbot> Current chairs: coremodule jlinton pbrobinson pwhalen
15:06:48 <pwhalen> Sorry for the late start, who's here today?
15:06:56 * jlinton wavews
15:07:16 <jlinton> I was about to conclude it was canceled.
15:07:28 <pwhalen> howdy jlinton, how're you doing?
15:07:40 <pwhalen> no, sorry. Just distracted :)
15:07:50 <jlinton> Good, how about yourself?
15:08:09 <jforbes> morning
15:09:15 <pwhalen> Not too bad, thanks. It's cooling off here, so outside is once again bearable
15:09:21 <pwhalen> morning jforbes
15:09:56 <jlinton> Yah, the weather has been a bit cooler than I expected, its only 90 something rather than 100 something.
15:10:24 <jlinton> But that seems to happen here when the pacific northwest gets really hot
15:10:41 * coremodule is here
15:10:55 <jforbes> I am very much looking forward to spending some time in Ireland with much lower temps.
15:11:02 <pwhalen> heh, any incremental improvement is welcome
15:11:22 <jlinton> Plumbers is virtual for me.
15:11:41 <pwhalen> ok, lets get started
15:11:43 <pwhalen> #topic 1) ==== Stable Releases ====
15:12:05 <pwhalen> good morning, coremodule!
15:12:10 <jlinton> I opened that bug about virt-manager/libvirt getting broken in F36/rawhide
15:12:44 <pwhalen> have a link handy?
15:12:51 <jlinton> That seems like it could be sorta a big deal. https://bugzilla.redhat.com/show_bug.cgi?id=2116534
15:12:58 * nirik is lurking in the back with a new 32bit arm problem (sadly)
15:14:06 <pwhalen> #info Bug 2116534 - Text console appears garbled following recent edk2 updates when using virt-manager serial consoles
15:15:56 <pwhalen> sounds like we should flag it as a blocker for beta
15:16:26 <pwhalen> nirik: thanks for joining, whats the issue?
15:16:30 <jlinton> I'm surprised it didn't break CI somewhere
15:16:30 <jforbes> sure, but if it is in F36 as well...
15:17:19 <nirik> I reinstalled the 32bit arm builders with f36 and they have been locking up... just dead in the water. ;( I've been trying to gather more info, but it's not been easy.
15:17:33 <jlinton> TX2's?
15:18:04 <pwhalen> nirik: it never is :)
15:18:39 <jlinton> sorry, those were the older ah, ampere machines.
15:18:53 <nirik> anyhow, this is f36 on emags with lpae kernel, 40g ram...
15:18:59 <pwhalen> coremodule: would you be able to try to reproduce and file a bug?
15:19:46 <pbrobinson> nirik: is there any crash or output either on the guest or hypervisor?
15:19:56 <nirik> https://paste.centos.org/view/64758644 was a dmesg I managed to get from one... but it wasn't completely locked up so I am not 100% sure it was the same issue
15:20:18 <nirik> [Mon Aug  8 08:47:34 2022] usercopy: Kernel memory overwrite attempt detected to SLUB object 'task_struct' (offset 96, size 4)!
15:20:27 <pwhalen> #info 32bit arm builders with F36 have been locking up, needs further investigation.
15:21:24 <pbrobinson> yes, if we could get that into a bug that would be fab
15:21:43 <coremodule> pwhalen, yes, I can attempt to reproduce
15:21:52 <coremodule> will report in-channel later today
15:22:01 <pwhalen> thanks
15:22:05 <jforbes> We are running btrfs on the builders?
15:22:08 <pwhalen> coremodule++
15:23:53 <pwhalen> nirik: is that btrfs?
15:23:59 <nirik> yeah... amusingly it was the way to get them stable in the past.
15:24:11 <nirik> I can try back on xfs if it looks btrfs related.
15:24:34 <pwhalen> might be worth a shot.
15:24:57 <pwhalen> coremodule: I can assist getting some hw to try to reproduce with
15:25:18 <pwhalen> Any other issues?
15:25:24 <nirik> yeah, would just about figure. They did seem fine on f35 with btrfs. Perhaps thats worth a try too... take them back to f35.
15:26:59 <pwhalen> Right, the end is nigh
15:27:02 <pwhalen> #topic 2) ==== Kernel Status ====
15:27:05 <jforbes> 5.19.0 would be interesting to try as well.
15:27:18 <pwhalen> #info kernel-5.19.0-65.fc37
15:27:18 <pwhalen> #link https://koji.fedoraproject.org/koji/buildinfo?buildID=2040274
15:27:21 <cmurf> i'd sooner try going up or down a major kernel version, either 5.19 from f37 or down, there's a metric f ton of mm changes in 5.19
15:27:43 <cmurf> >kernel BUG at mm/usercopy.c:100!
15:27:54 <nirik> jforbes: good idea.
15:28:03 <jforbes> Speaking of, 5.19 is going to be the release kernel for f37 and the branch is already created in kernel-ark
15:28:27 <jforbes> nirik: I can do a build for you against F36 since we don't have a 32bit f36 build
15:28:37 <jforbes> err f37 build
15:29:01 <pwhalen> #info Kernel 5.19 will be the release kernel for Fedora 37. Please test and report any issues to #fedora-arm or the mailing list.
15:29:07 <nirik> oh yeah, right. sure, that would be great if you can...
15:30:01 <nirik> oh, another thing I can mention to you all... we have 5 new arm boxes in (but not yet racked/etc). They should replace a bunch of emags...
15:30:24 <pbrobinson> so I did a bunch of testing on the weekend and basically moved everything to 5.19 and it looks OK
15:30:43 <pbrobinson> nirik: they're the new Ampere Altras?
15:31:06 <nirik> mt snow I think it is...
15:31:20 <pwhalen> I've also done some testing with 5.19, no issues so far
15:31:56 <nirik> 80 core altra yeah...
15:33:35 <pwhalen> Nice.
15:33:45 <pwhalen> Any other kernel news/issues?
15:34:04 <pwhalen> #topic 3) ==== Bootloader Status ====
15:34:06 <jlinton> 80 fairly reasonable cores in 1 socket.. so yah, that should be a nice upgrade from the emag.
15:34:22 <pwhalen> #info uboot-tools-2022.10-0.1.rc1.fc37
15:34:22 <pwhalen> #link https://koji.fedoraproject.org/koji/buildinfo?buildID=2038295
15:34:52 <jforbes> I want 80 reasonable cores in 1 socket
15:35:08 <pbrobinson> there's at least an issue on the rpi4 here that I need to bisect
15:35:59 <pwhalen> pbrobinson: whats the issue? Seems ok running a recent iot rawhide
15:36:16 <pbrobinson> using the latest U-Boot?
15:36:27 <pbrobinson> it's resetting in the USB stack
15:36:41 <pbrobinson> maybe it works if you have a keyboars/storage plugged in
15:36:44 <pwhalen> DMI: Unknown Unknown Product/Unknown Product, BIOS 2022.10-rc1 10/01/2022
15:36:56 <pbrobinson> fun!
15:37:13 * pbrobinson notes he's at times cursed with HW
15:37:40 <pwhalen> heh, I have nothing plugged into it, running headless in the laundry room
15:38:36 <pwhalen> Anything else for uboot?
15:39:14 <pwhalen> #topic 4) ==== Fedora 37 status ===
15:39:24 <pwhalen> #info OpenQA Testing
15:39:24 <pwhalen> #info https://openqa.fedoraproject.org/tests/overview?distri=fedora&version=Rawhide&build=Fedora-Rawhide-20220808.n.0&groupid=5
15:40:23 <pwhalen> Looks pretty good, does anything stand out?
15:40:57 <jlinton> I moved a few of my machines to it last week, and other than that virt bug, everything seems to be ok at the moment.
15:42:22 <pwhalen> I'm moving stuff over as well, no issues so far. Will do some more this week
15:42:23 <jlinton> The honeycomb still needs restool reverted back to an older version to get the nic's working, so at some point I need to track down what exactly changed.
15:42:32 <pbrobinson> I'll be moving most of mine when the first branched compose compleetes but been testing kernel etc
15:43:03 <pbrobinson> jlinton: yes, it's on my list to test that, I'm due another firmware which updates some of the vendor firmwares too
15:43:21 <jlinton> Although "working" means that a 10G port, does about 100Mbits, so the throughput is still broken somewhere.
15:46:40 <pwhalen> #topic 5) ==== Open Floor ====
15:46:48 <pwhalen> Anything else for this week?
15:47:41 <jlinton> I'm still hacking anaconda/systemd-boot, although I said a couple weeks back I would post something, its getting to the point where it might actually work without me having to drop to the command prompt to fix things.
15:48:02 <jlinton> so, RSN.
15:48:34 <pwhalen> nice, thanks for the update
15:49:36 <jlinton> its also aarch64 only at the moment too, so that might be fun.
15:52:16 <pwhalen> heh, never a dull moment :)
15:52:32 <pwhalen> ok, thanks for joining!
15:52:34 <pwhalen> #endmeeting