15:01:27 <pwhalen> #startmeeting Fedora ARM and AArch64 Status Meeting
15:01:27 <pwhalen> #chair pwhalen pbrobinson jlinton coremodule
15:01:27 <pwhalen> #topic roll call
15:01:27 <zodbot> Meeting started Tue Jan 26 15:01:27 2021 UTC.
15:01:27 <zodbot> This meeting is logged and archived in a public location.
15:01:27 <zodbot> The chair is pwhalen. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:27 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:01:27 <zodbot> The meeting name has been set to 'fedora_arm_and_aarch64_status_meeting'
15:01:27 <zodbot> Current chairs: coremodule jlinton pbrobinson pwhalen
15:01:50 <coremodule> Good morning!
15:01:51 <pwhalen> good morning folks, who's here today?
15:01:53 * coremodule is here.
15:01:54 * jlinton waves
15:01:56 <pwhalen> howdy coremodule
15:02:47 <rcbaus> hello
15:02:47 * mahmoudajawad is here
15:02:51 * pbrobinson is sort of here
15:03:20 <pwhalen> ok, lets get started
15:03:28 <pwhalen> #topic 1) ==== Userspace Status  ====
15:03:46 <pwhalen> Any new user space issues?
15:04:24 <pwhalen> nothing coming to mind
15:04:42 <pwhalen> #info No new issues reported.
15:04:59 <jlinton> Its getting "boring" as it should be.
15:05:11 <pwhalen> :)
15:05:15 <pwhalen> #topic 2) ==== Kernel Status ====
15:05:19 <pwhalen> kernel isnt as boring
15:05:30 <pwhalen> #info kernel-5.11.0-0.rc5.134.fc34
15:05:30 <pwhalen> #link https://koji.fedoraproject.org/koji/buildinfo?buildID=1673663
15:05:37 * mahmoudajawad agrees xD
15:06:35 <jlinton> Vaguly off topic, but the ark kernel configs don't build standalone anymore, they require the gcc plugins from the package.
15:06:40 <pwhalen> armhfp still has issues booting. On non-lpae we get no output at all, on the lpae kernel, it boots but then fails to unpack the initramfs (cpio error) and then kernel panics
15:07:01 <pwhalen> ah :( yea, seeing other complain about that too
15:08:05 <pwhalen> the kernel has gotten larger, and the lpae is a little smaller than non-lpae.. so perhaps there is an overlap in memory?
15:08:50 <pwhalen> interestingly, uefi boot does work on some platforms when booting the lpae kernel only
15:09:06 <pwhalen> rpi2/3 work
15:09:30 <pwhalen> but we arent getting any uefi disk images, because it doesnt boot in qemu to run imagefactory
15:09:41 <pwhalen> qemu uefi boot doesnt work
15:10:04 <jlinton> So that broke again, is the screen capture working?
15:10:32 <pwhalen> I'm gonna say no, but I havent looked
15:11:06 <mahmoudajawad> pwhalen: For last point, you are building and testing to run fc34 with the latest rc for 5.11 all emulated?
15:12:11 <jlinton> Its kvm
15:12:15 <pwhalen> mahmoudajawad: the disk images are created with imagefactory which uses qemu.
15:12:21 <pbrobinson> I've not had time to look at the armv7, I'm going to try to get to it this weekend but if anyone has cycles it would be very useful
15:12:50 <mahmoudajawad> jlinton: Yes, wrong term I should've said virtualised.
15:13:32 <mahmoudajawad> pwhalen: If uefi boot is important for checking, I can do on one of the RPi 3's I have. Unless, as to my knowledge, it's out of scoop for fedora?
15:14:37 <pbrobinson> mahmoudajawad: UEFI is the primary way we boot aarch64 and the intention is to move armv7 to that for f34+
15:14:38 <pwhalen> mahmoudajawad: we're moving to uefi disk images in F34 for armhfp
15:15:54 <mahmoudajawad> Aha! Excuse my ignorance, my info is outdated.
15:16:17 <pwhalen> so, yes please. Anyone that has cycles to take a look and discuss in #fedora-arm
15:16:52 <pwhalen> otherwise, aarch64 is looking ok. Need to test RC5
15:17:08 <pwhalen> Anyone else aware of any kernel issues?
15:18:55 <pwhalen> #info Please test and report any issues to the list or #fedora-arm.
15:19:03 <pwhalen> #topic 3) ==== Bootloader Status ====
15:19:21 <pwhalen> #info uboot-tools-2021.01-1.fc34
15:19:21 <pwhalen> #link https://koji.fedoraproject.org/koji/buildinfo?buildID=1670854
15:19:38 <pbrobinson> so the fixes I mentioned previously are there
15:19:40 <pwhalen> Anyone seeing any issues?
15:19:47 <pwhalen> yep, working well for me
15:19:49 <pbrobinson> overall it seems pretty reasonable from my testing
15:19:59 <coremodule> i haven't looked in a month or so; were the rpi400 bits added in from upstream?
15:20:37 <pbrobinson> coremodule: yes, with the upstream u-boot and the latest firmware the rpi400 boots for me
15:20:41 <pwhalen> rpi4-400 boots, I dont have usb though
15:21:01 <coremodule> pbrobinson, awesome, I'll dedicate some cycles this week to testing it
15:21:03 <pbrobinson> there's some quirks around USB and there's more work to do but it should boot just fine
15:21:21 <pbrobinson> the wifi doesn't work, trying to get the vendor to release the firmware for that
15:21:27 <pwhalen> yep, works great headless, the display came up but then an update killed gnome
15:21:42 <pbrobinson> the 5.10.x kernel works with wifi
15:21:46 <pbrobinson> bah
15:21:48 <pbrobinson> wrong
15:21:54 <pbrobinson> the 5.10.x kernel works for USB
15:22:27 <pbrobinson> but looking at a complete upstream solution for 5.11+
15:22:46 <pbrobinson> but wired ethernet should work
15:23:37 <coremodule> nice, okay. do you want me to add it as "unofficially supported, but works" on the rpi arm page?
15:23:44 <pwhalen> #info No known issues, should resolve previous bugs booting UEFI enabled images. Please test and report any issues to the list or #fedora-arm.
15:24:50 <pwhalen> coremodule: maybe hold off until it works a little more?
15:24:55 <coremodule> okay, sure
15:25:01 <pwhalen> imo, pbrobinson?
15:25:19 <pbrobinson> coremodule: not really because people assume that means a stable F-33 release
15:25:30 <pbrobinson> coremodule: I tend to just add it when releases are out
15:25:43 <coremodule> sure, makes sense. i wont touch the page then
15:25:48 <pbrobinson> and I've not tested anything other than serial console
15:26:03 <pwhalen> #topic 4) ==== Fedora 34 ====
15:26:26 <pwhalen> so, mass rebuild was delayed for various reasons
15:27:46 <pwhalen> one of which was the armhfp builders, upgraded to f33 but failing to build some packages.. build works ok but kojid OOM, build gets reassigned and its stuck in a loop
15:28:06 <pwhalen> I tired to reproduce, but even running other tasks the build is fine.
15:29:03 <pwhalen> F32 works ok. When I installed f32 I noticed we got a swap partition by default, wheras f33 uses zram, so that may be the difference.
15:29:32 <smooge> let me look at the f33 kickstart
15:29:44 <pwhalen> But! I am pretty sure nirik tried f32 userspace with the f33 kernel and hit the same issue
15:29:50 <pwhalen> smooge: thanks
15:30:41 <smooge> not sure he did that. for f33 part swap --fstype swap --size=32768
15:31:07 <smooge> so i think we had a real swap partition on the system. not sure if zswap is added onto that though
15:33:18 <smooge> they both have a swap partition and both have ext4. I think nirik did try newer kernels on f32 which also failed.. but he didn't ahve enough time to walk through 5.6.9 to 5.7.10 to see where it happened
15:33:21 <smooge> eol
15:33:34 <pwhalen> thanks smooge. I had some hope. Perhaps if we killed the zram there may be a difference. Runnning test builds it barely touches swap
15:34:22 <pwhalen> sadly, many of those kernels arent usable because of the "pause" issue
15:34:24 <pbrobinson> yea, I wouldn't imagine huge amounts of swap due to being more single use
15:34:53 <pbrobinson> easy enough for me to do scratch builds for the latest of each of the major releases if that's helpful
15:35:35 <pwhalen> I would also need a reproducer, so not worthwhile yet
15:37:45 <pwhalen> #info Latest nominated compose
15:37:45 <pwhalen> #link https://fedoraproject.org/wiki/Test_Results:Current_Installation_Test
15:38:19 <pwhalen> #info Initial-setup bug fixed in Rawhide.
15:39:09 <pwhalen> Has anyone hit any issues not yet reported?
15:40:19 <nirik> I tried with and without zram0... it didn't seem to matter
15:40:39 <pwhalen> :(
15:41:31 <pwhalen> nirik: I havent been able to reproduce, builds work ok with kojid running. Not sure how to reproduce it without
15:42:00 <pwhalen> s/with/without
15:42:23 <pwhalen> if anyone has any suggestions, please let me know
15:43:06 <pwhalen> #topic 5)  == Open Floor ==
15:43:12 <nirik> pwhalen: try a python3.8 build? that was my test..
15:43:40 <pwhalen> nirik: its been running in a loop without issue
15:44:25 <pwhalen> 24 builds so far
15:44:26 <nirik> huh, and this is f33 + 5.10.10 + lpae kernel + 24gb mem + 5cpus ?
15:45:36 <pwhalen> 5.10.9-201.fc33.armv7hl+lpae, 24GB ram... cpus, checking
15:46:01 <nirik> wonder if it could be firmware differences somehow?
15:46:10 <pwhalen> only two cpus
15:46:28 <nirik> 5.10.9 should have the pause issue... but that might be only for specific builds to trigger
15:46:29 <pwhalen> I'll try with 5, using 2 completes the build in about 45 minutes-ish
15:46:49 <nirik> yeah, oddly, 5 also completes in about 45min. ;) or crashes after about 40
15:46:51 <pwhalen> ah, I thought 5.10.9 had the fix, not hit the pause
15:47:14 <pwhalen> will try the newer kernel, add more cpus
15:47:17 <nirik> oh, you are right.
15:47:24 <nirik> 5.10.9-201 should be ok
15:47:25 <pwhalen> phew
15:47:36 <nirik> this last week has been a blur. ;)
15:47:50 <pwhalen> ok, thanks. No doubt. Really sorry nirik
15:49:07 <pwhalen> I'll add more cpus, continue to try to find a reproducer without kojid running
15:49:33 <pwhalen> coremodule, does this time for the meeting work for you?
15:49:58 <coremodule> Yes, this time is a-okay, its 8am my time
15:50:03 <nirik> I can also redo stg now with f33 and have a handy reproducing box
15:50:30 <coremodule> the iot meeting is a little early for me, but that's a different story
15:50:51 <pwhalen> nirik: would I be able to get access run through the kernels? pbrobinson has offered to do some scratch kernels to see if we can track down when this started
15:51:15 <nirik> sure, we can see about setting that up...
15:51:17 <pwhalen> coremodule: right, ok. So lets discuss rescheduling iot  tomorrow
15:51:36 <pwhalen> ok, thanks nirik
15:52:36 <pwhalen> Anything else for open floor?
15:53:11 <pwhalen> #endmeeting