15:02:39 <pwhalen> #startmeeting Fedora ARM and AArch64 Status Meeting
15:02:39 <zodbot> Meeting started Tue Oct 27 15:02:39 2020 UTC.
15:02:39 <zodbot> This meeting is logged and archived in a public location.
15:02:39 <zodbot> The chair is pwhalen. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:02:39 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:02:39 <zodbot> The meeting name has been set to 'fedora_arm_and_aarch64_status_meeting'
15:02:53 <pwhalen> Sorry for the late start, my last meeting ran over
15:03:00 <pwhalen> Who's here today?
15:03:12 <mattdm> I am!
15:03:13 <mboddu> And sorry for starting releng meeting here :)
15:03:25 <mattdm> bcotton said he would also guest-star
15:03:30 * jlinton waves
15:03:44 <pwhalen> #chair pwhalen pbrobinson jlinton
15:03:44 <zodbot> Current chairs: jlinton pbrobinson pwhalen
15:03:57 <pwhalen> #chair pwhalen pbrobinson jlinton mattdm bcotton
15:03:57 <zodbot> Current chairs: bcotton jlinton mattdm pbrobinson pwhalen
15:04:07 * pbrobinson o/
15:04:13 <bcotton> .hello2
15:04:14 <zodbot> bcotton: bcotton 'Ben Cotton' <bcotton@redhat.com>
15:04:29 <pwhalen> #topic 0) Fedora 33
15:05:16 <pwhalen> #info Fedora 33 GA released today! https://fedoramagazine.org/announcing-fedora-33/
15:05:34 <pwhalen> unfortunately we have some issues on arm that I missed.
15:05:36 <mattdm> yay! and everything went flawlessly
15:05:44 <mattdm> oh wait :)
15:06:00 <mattdm> Let's not make that "that I missed"
15:06:09 <pwhalen> #link https://fedoraproject.org/wiki/Common_F33_bugs#ARM_.26_AArch64
15:06:10 * coremodule is here
15:06:17 <coremodule> .hello2
15:06:18 <zodbot> coremodule: coremodule 'Geoffrey Marr' <gmarr@redhat.com>
15:06:27 <pwhalen> I added a note for the affected hardware
15:06:44 <mattdm> let's figure out what we need to do so we get the hardware we want tested tested every release
15:06:51 <pbrobinson> yes, the plan is to debug/fix/respin the affected images
15:07:00 <dustymabe> pwhalen: it might be good to link to the beta image in those entries in the common bugs page
15:07:16 <pwhalen> dustymabe: will do.
15:08:17 <pbrobinson> we should be able to do a minimal run against the GA bits with just some overrides in place
15:08:35 <nb> .hello2
15:08:36 <zodbot> nb: nb 'Nick Bebout' <nick@bebout.net>
15:08:40 <mattdm> pbrobinson++
15:08:42 <zodbot> mattdm: Karma for pbrobinson changed to 1 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
15:08:45 <pwhalen> For the nano, do we even know why it doesnt boot the GA image?
15:08:48 <pbrobinson> most people in the arm meeting in a long time.....
15:09:06 <pbrobinson> pwhalen: not yet, I'm digging
15:09:19 <pwhalen> For the Pine64* devices, it was a change in the kernel, but the nano is a little mystery right now
15:09:22 <pbrobinson> it will run a GA kernel if you update from an earlier image
15:09:39 <mattdm> pbrobinson: I basically just run from fire to fire, so where there's smoke....
15:09:40 <pwhalen> right, it was tested and expected to work
15:10:04 <pbrobinson> which is why it was missed because I just did boot testing of the new kernel expecting that would be enough
15:10:37 <mattdm> We need an explicit test case for that, I guess
15:10:45 <pbrobinson> mattdm: I don't even have time to run, the burning dumsters just roll on past
15:11:23 <coremodule> pbrobinson, when you say boot testing, what exactly do you mean?
15:11:37 <mattdm> And we should have specific test cases for the hardware we care about the most
15:11:44 <pwhalen> coremodule: just make sure the kernel boots
15:11:52 <mattdm> like, N devices, where N is > 1 and < 6
15:11:54 <coremodule> ahhh.
15:11:55 <pbrobinson> coremodule: update the kernel on a running system and reboot as opposed to flashing out a full new workstation image
15:12:14 <coremodule> ah okay.
15:12:33 <coremodule> mattdm, we do have testcases that cover that, a new fresh install.
15:12:46 <mattdm> coremodule -- but they weren't done?
15:13:00 <pbrobinson> so basically my running theory ATM is there's a dep that changed in the kernel update and a core module/firmware isn't pulled into the generic initrd that was previously
15:13:13 <pbrobinson> s/dep/mod dep/
15:13:41 <pbrobinson> mattdm: coremodule: they were likely done, just not on that particular HW
15:14:07 <pwhalen> the specific hardware testing for the pine64 is where I discovered the problem
15:14:24 <jlinton> Yah, I've been running/installing the images on honeycomb+rpi4, so everything was working a week or so ago.
15:14:26 <pwhalen> Nano was tested after sign off.
15:14:28 <coremodule> mattdm, thats true. i don't think anyone on the qa team has this specific hardware: jetson nano, pine 64. I know I personally only have an assortment of rpi and a few rock64 devices.
15:14:51 <mattdm> Yeah, so: that's part two of my suggestion
15:15:00 <pwhalen> mattdm: I think coremodule would be a great recipient if you're buying hardware
15:15:01 <mattdm> We have some Fedora budget this year that's not being spent on travel
15:15:36 <mattdm> So let's identify the hardware we really care the most about
15:15:42 <mattdm> and make sure everyone has it
15:15:55 <coremodule> mattdm, i think the arm/iot teams goal has been specifically to *not* have a list of hardware that they care about most...
15:16:10 <coremodule> going by some conversations that we had in the last month or so...
15:16:16 <coremodule> im trying to find the ticket
15:16:17 <mattdm> coremodule: but it turns out that secretly we do?
15:16:30 <coremodule> *shrugs* yeah, heh
15:16:39 <coremodule> https://pagure.io/fedora-qa/issue/615
15:16:41 <coremodule> ^^
15:16:43 <mattdm> and also, while that's an admirable goal, I don't think the ARM world is anywhere close to making that possible
15:16:46 <nb> I am willing to help test hardware
15:16:48 <mattdm> x86 is barely there really
15:16:51 <pbrobinson> coremodule: because we've got bent over and fucked up QA in the past over lists of hardware
15:17:16 <pbrobinson> so I try and keep it generic so people cause me less stress
15:17:30 <pwhalen> RIght.
15:17:31 <mattdm> I think the generic is causing you stress too :)
15:17:43 <coremodule> pbrobinson, I understand that, and I agree with it. it's a pain to herd each individual device.
15:18:32 <mattdm> yeah, but if we want to make sure some specific devices work, we need to actually list those devices
15:18:51 <mattdm> And I can't buy generic devices to send to people. They have to actually be specific devices  :)
15:18:58 <coremodule> I think the end goal here is to avoid this in the future... Which could *possibly* be solved by having a list of "officially supported hardware"
15:19:08 <pbrobinson> mattdm: SBSA is generic ;-)
15:19:14 <mattdm> "officially supported" is too strong
15:19:21 <coremodule> but I do agree that the end goal should be to have all devices "supported"
15:19:24 <jlinton> Right, and none of those devices are SBSA
15:19:24 <mattdm> but something like "recommended platforms" or something
15:19:42 <coremodule> maybe "officially tested"? "tested and known working"?
15:19:43 <pwhalen> Our postion has been to evaluate the hardware issues as they happen, as they do on x86, rather than create a blocking list
15:19:58 <jlinton> really unless you _REALLY_ care about a given device its better to know one of them might be broken, and only block if it turns out there is a common problem.
15:20:16 <coremodule> jlinton, makes total sense to me
15:20:32 <pbrobinson> right, and try and get as much of a runway to test them as possible rather than trying and testing them all in a tiny window from when the RC lands to a no-doze meeting
15:22:04 <mattdm> So, is it helpful then to try to get as much *different* hardware to lots of people?
15:22:38 <pbrobinson> mattdm: I think we should come up with a small list of devices that people likely want and get a selection to people
15:22:51 <mattdm> pbrobinson Okay, cool.
15:23:02 <mattdm> That's what I am saying; I thought people were disagreeing
15:23:07 <pbrobinson> the problem we've had with HW in the past is that you send it to people and they still don't test/use it and it's been a waste of time
15:23:29 <mattdm> Yeah, we need actually committed people. We don't have money to throw away.
15:23:39 <bcotton> that's where i think explicitly adding hardware to the requirements helps
15:23:39 <mattdm> But I'm willing to take some chances too.
15:23:41 <pbrobinson> but I'm happy to assist with the HW list if other people can deal with the list of people that should have ti
15:24:00 <pwhalen> RIght, and test the boards within a small window before we release. Sometimes less than 24 hours.
15:24:05 <coremodule> pbrobinson, yes, if you can assist there, you have the brain we need to assemble the list
15:24:05 <bcotton> it's not so much a "we're blocking on this device" as a "having it on the list will help motivate testers to test"
15:24:34 <mattdm> right, if the test case says "does the Jetson Nano boot" it's easy for someone to whom we have sent a Jetson Nano to take responsibility
15:24:42 <nb> bcotton mattdm good points
15:24:48 <coremodule> I can deal with changing the docs to reflect your list
15:25:12 <bcotton> i think in a lot of cases, we can test earlier than the RC and be fine. but if a kernel gets an FE right before the compose, that's where it becomes troublesome
15:25:42 <mattdm> yeah, that's unfortunate -- but late-breaking kernel updates that fix big security flaws are a fact of life
15:26:22 <pbrobinson> mattdm: yes, but there needs to be run way to deal with the rest of the fall out that causes
15:26:38 <pwhalen> bcotton: in the case of the nano, we dont know why the GA images doesnt boot. Kernel is fine.
15:27:04 <bcotton> one thing we could do to minimize the risk is to have a "no FEs more than X time before th Go/No-Go. blocker fixes only". I don't think it'd help in this case, but it at least narrows the set of possible changes
15:27:13 <bcotton> pwhalen: sure, i'm speaking more generally
15:27:33 <mattdm> bcotton++
15:27:33 <zodbot> mattdm: Karma for bcotton changed to 4 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
15:27:43 <coremodule> bcotton, that's a good idea. you should propose that at the next qa MEETING
15:27:48 <coremodule> whoops, caps lock
15:28:31 <bcotton> coremodule: ack
15:31:12 <mattdm> excellent all sorted then :)
15:31:18 <pwhalen> heh.
15:31:20 <coremodule> lol
15:32:00 <mattdm> #action mattdm to work with mnordin to make process for getting "grant" hardware for ARM QA
15:32:34 <pbrobinson> if only it was so easy, it's a start :)
15:32:43 <pwhalen> :)
15:33:21 <bcotton> hey, we've only released fedora 33 times. give us another 33 and we'll have everything just right :-)
15:33:45 <mattdm> action $SOMEONE_ELSE to come up with the "prioritized hardware" list?
15:33:50 <coremodule> pbrobinson, if you assembly a list and can send it to me, ill update the docs if you dont want to
15:33:52 <kushal> bcotton, we can do that in another 30times itself :)
15:34:04 * pbrobinson will be in some home sucking his thumb by then bcotton
15:34:27 <mattdm> let's make sure the AWS ARM systems (both virt and bare metal) are on the list too
15:34:39 <pbrobinson> coremodule: I'll work with pwhalen on this list and then he can work with you to get it updated?
15:34:46 <pwhalen> mattdm: this would be a big list. We also need to cover enterprise hardware.. and right, aws
15:34:47 <coremodule> pbrobinson, perfect
15:35:07 <mattdm> I think we need a list no bigger than 5
15:35:15 <pingou> pwhalen: no worries, mattdm said he has budget ;-)
15:35:19 <mattdm> and then there can be a second list of other important hardware
15:35:42 <mattdm> We've got limited resources, limited people, limited time, limited budget
15:35:50 <mattdm> There's no option but to narrow down
15:36:14 <pbrobinson> mattdm: yep, I was going to probably do 4-5.... else I run out of fingers
15:36:19 <mattdm> pbrobinson++
15:36:30 <pbrobinson> mattdm: cover key things like certain GPUs etc
15:36:31 <mattdm> and you're sucking on your thumb, so four it is
15:36:34 <pwhalen> The enterprise stuff has to work or internal people come hunting for us. (and it all works in f33)
15:36:56 <pbrobinson> pwhalen: sure, and of course internal people could actually test that too ;-)
15:37:03 <pwhalen> HAHAHA
15:37:20 <mattdm> no really though. they should!
15:37:36 <pbrobinson> mattdm: welcome to try to encourage them ;-)
15:38:08 <mattdm> I will. Tell me who to bug :)
15:38:16 <pbrobinson> jfeeney
15:38:35 <pwhalen> That'll just come back to me.
15:38:50 <pwhalen> Dont worry about enterprise, I got it.
15:39:44 <pwhalen> For SBC's - Raspberry Pis (2,3,4), Jetson Nano, Pine64
15:40:18 <pbrobinson> RockPro64
15:40:22 <pbrobinson> Pinebook Pro
15:40:34 <pbrobinson> we'll do a list and circle back
15:40:41 <coremodule> cool
15:41:21 <jlinton> There is the "edge" hardware too mcbin/honeycomb.
15:41:52 <pbrobinson> jlinton: yes, I was going to review a bunch of all of those
15:42:21 <pbrobinson> the later isn't cheap though in the scheme of things so not something likely to be on the list and we have people testing it already
15:42:49 <pwhalen> Agreed, I dont have those either so I can't test.
15:43:03 <coremodule> is that the macchiatobin?
15:43:19 <jlinton> Right, I guess its more a question of making sure someone is testing it, I dropped much of my mcbin testing, because I can't use that many platforms
15:43:20 <pbrobinson> coremodule: yes
15:45:19 <pwhalen> ok, shall we revisit the list next week? Anything else we want to talk about?
15:45:25 <pbrobinson> nothing from me
15:45:53 <coremodule> here's one: has this affected the iot spin at all? do we know?
15:45:58 <coremodule> *on that hardware?
15:46:30 <pwhalen> yes, Pine64
15:47:26 <coremodule> okay, good to know
15:47:32 <pwhalen> ok, moving on
15:47:40 <pwhalen> #topic 1) ==== Userspace Status  ====
15:47:51 <pwhalen> Any new user space issues?
15:47:56 <pbrobinson> coremodule: possibly, but we respin all of that regularly anyway
15:48:42 <pwhalen> I'm not aware of any user space issues
15:48:56 <pwhalen> #info No issues reported.
15:49:07 <pwhalen> #topic 2) ==== Kernel Status ====
15:49:47 <pwhalen> #info Kernel Test Week! https://fedoraproject.org/wiki/Test_Day:2020-10-26_Kernel_5.9_Test_Week
15:50:05 <pwhalen> If people could take a few moments and test on the hardware you care about.
15:50:25 <pwhalen> #info 5.9 still needs the xgene PMU patch added, will panic on boot.
15:51:08 <pwhalen> HPE Apollo only prints EFI messages, I'll try to get a bug filed on that today.
15:52:32 <pwhalen> Do we know when 5.9 will be released in f33, or is it just pending karma?
15:54:31 <pwhalen> We're almost out of time, and it seems I've lost folks..
15:54:45 <pwhalen> #topic 3)  == Open Floor ==
15:54:50 <pwhalen> Anything else for today?
15:56:36 <coremodule> nothing from me
15:56:46 <pwhalen> #endmeeting