15:02:39 <pwhalen> #startmeeting Fedora ARM and AArch64 Status Meeting 15:02:39 <zodbot> Meeting started Tue Oct 27 15:02:39 2020 UTC. 15:02:39 <zodbot> This meeting is logged and archived in a public location. 15:02:39 <zodbot> The chair is pwhalen. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:02:39 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 15:02:39 <zodbot> The meeting name has been set to 'fedora_arm_and_aarch64_status_meeting' 15:02:53 <pwhalen> Sorry for the late start, my last meeting ran over 15:03:00 <pwhalen> Who's here today? 15:03:12 <mattdm> I am! 15:03:13 <mboddu> And sorry for starting releng meeting here :) 15:03:25 <mattdm> bcotton said he would also guest-star 15:03:30 * jlinton waves 15:03:44 <pwhalen> #chair pwhalen pbrobinson jlinton 15:03:44 <zodbot> Current chairs: jlinton pbrobinson pwhalen 15:03:57 <pwhalen> #chair pwhalen pbrobinson jlinton mattdm bcotton 15:03:57 <zodbot> Current chairs: bcotton jlinton mattdm pbrobinson pwhalen 15:04:07 * pbrobinson o/ 15:04:13 <bcotton> .hello2 15:04:14 <zodbot> bcotton: bcotton 'Ben Cotton' <bcotton@redhat.com> 15:04:29 <pwhalen> #topic 0) Fedora 33 15:05:16 <pwhalen> #info Fedora 33 GA released today! https://fedoramagazine.org/announcing-fedora-33/ 15:05:34 <pwhalen> unfortunately we have some issues on arm that I missed. 15:05:36 <mattdm> yay! and everything went flawlessly 15:05:44 <mattdm> oh wait :) 15:06:00 <mattdm> Let's not make that "that I missed" 15:06:09 <pwhalen> #link https://fedoraproject.org/wiki/Common_F33_bugs#ARM_.26_AArch64 15:06:10 * coremodule is here 15:06:17 <coremodule> .hello2 15:06:18 <zodbot> coremodule: coremodule 'Geoffrey Marr' <gmarr@redhat.com> 15:06:27 <pwhalen> I added a note for the affected hardware 15:06:44 <mattdm> let's figure out what we need to do so we get the hardware we want tested tested every release 15:06:51 <pbrobinson> yes, the plan is to debug/fix/respin the affected images 15:07:00 <dustymabe> pwhalen: it might be good to link to the beta image in those entries in the common bugs page 15:07:16 <pwhalen> dustymabe: will do. 15:08:17 <pbrobinson> we should be able to do a minimal run against the GA bits with just some overrides in place 15:08:35 <nb> .hello2 15:08:36 <zodbot> nb: nb 'Nick Bebout' <nick@bebout.net> 15:08:40 <mattdm> pbrobinson++ 15:08:42 <zodbot> mattdm: Karma for pbrobinson changed to 1 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 15:08:45 <pwhalen> For the nano, do we even know why it doesnt boot the GA image? 15:08:48 <pbrobinson> most people in the arm meeting in a long time..... 15:09:06 <pbrobinson> pwhalen: not yet, I'm digging 15:09:19 <pwhalen> For the Pine64* devices, it was a change in the kernel, but the nano is a little mystery right now 15:09:22 <pbrobinson> it will run a GA kernel if you update from an earlier image 15:09:39 <mattdm> pbrobinson: I basically just run from fire to fire, so where there's smoke.... 15:09:40 <pwhalen> right, it was tested and expected to work 15:10:04 <pbrobinson> which is why it was missed because I just did boot testing of the new kernel expecting that would be enough 15:10:37 <mattdm> We need an explicit test case for that, I guess 15:10:45 <pbrobinson> mattdm: I don't even have time to run, the burning dumsters just roll on past 15:11:23 <coremodule> pbrobinson, when you say boot testing, what exactly do you mean? 15:11:37 <mattdm> And we should have specific test cases for the hardware we care about the most 15:11:44 <pwhalen> coremodule: just make sure the kernel boots 15:11:52 <mattdm> like, N devices, where N is > 1 and < 6 15:11:54 <coremodule> ahhh. 15:11:55 <pbrobinson> coremodule: update the kernel on a running system and reboot as opposed to flashing out a full new workstation image 15:12:14 <coremodule> ah okay. 15:12:33 <coremodule> mattdm, we do have testcases that cover that, a new fresh install. 15:12:46 <mattdm> coremodule -- but they weren't done? 15:13:00 <pbrobinson> so basically my running theory ATM is there's a dep that changed in the kernel update and a core module/firmware isn't pulled into the generic initrd that was previously 15:13:13 <pbrobinson> s/dep/mod dep/ 15:13:41 <pbrobinson> mattdm: coremodule: they were likely done, just not on that particular HW 15:14:07 <pwhalen> the specific hardware testing for the pine64 is where I discovered the problem 15:14:24 <jlinton> Yah, I've been running/installing the images on honeycomb+rpi4, so everything was working a week or so ago. 15:14:26 <pwhalen> Nano was tested after sign off. 15:14:28 <coremodule> mattdm, thats true. i don't think anyone on the qa team has this specific hardware: jetson nano, pine 64. I know I personally only have an assortment of rpi and a few rock64 devices. 15:14:51 <mattdm> Yeah, so: that's part two of my suggestion 15:15:00 <pwhalen> mattdm: I think coremodule would be a great recipient if you're buying hardware 15:15:01 <mattdm> We have some Fedora budget this year that's not being spent on travel 15:15:36 <mattdm> So let's identify the hardware we really care the most about 15:15:42 <mattdm> and make sure everyone has it 15:15:55 <coremodule> mattdm, i think the arm/iot teams goal has been specifically to *not* have a list of hardware that they care about most... 15:16:10 <coremodule> going by some conversations that we had in the last month or so... 15:16:16 <coremodule> im trying to find the ticket 15:16:17 <mattdm> coremodule: but it turns out that secretly we do? 15:16:30 <coremodule> *shrugs* yeah, heh 15:16:39 <coremodule> https://pagure.io/fedora-qa/issue/615 15:16:41 <coremodule> ^^ 15:16:43 <mattdm> and also, while that's an admirable goal, I don't think the ARM world is anywhere close to making that possible 15:16:46 <nb> I am willing to help test hardware 15:16:48 <mattdm> x86 is barely there really 15:16:51 <pbrobinson> coremodule: because we've got bent over and fucked up QA in the past over lists of hardware 15:17:16 <pbrobinson> so I try and keep it generic so people cause me less stress 15:17:30 <pwhalen> RIght. 15:17:31 <mattdm> I think the generic is causing you stress too :) 15:17:43 <coremodule> pbrobinson, I understand that, and I agree with it. it's a pain to herd each individual device. 15:18:32 <mattdm> yeah, but if we want to make sure some specific devices work, we need to actually list those devices 15:18:51 <mattdm> And I can't buy generic devices to send to people. They have to actually be specific devices :) 15:18:58 <coremodule> I think the end goal here is to avoid this in the future... Which could *possibly* be solved by having a list of "officially supported hardware" 15:19:08 <pbrobinson> mattdm: SBSA is generic ;-) 15:19:14 <mattdm> "officially supported" is too strong 15:19:21 <coremodule> but I do agree that the end goal should be to have all devices "supported" 15:19:24 <jlinton> Right, and none of those devices are SBSA 15:19:24 <mattdm> but something like "recommended platforms" or something 15:19:42 <coremodule> maybe "officially tested"? "tested and known working"? 15:19:43 <pwhalen> Our postion has been to evaluate the hardware issues as they happen, as they do on x86, rather than create a blocking list 15:19:58 <jlinton> really unless you _REALLY_ care about a given device its better to know one of them might be broken, and only block if it turns out there is a common problem. 15:20:16 <coremodule> jlinton, makes total sense to me 15:20:32 <pbrobinson> right, and try and get as much of a runway to test them as possible rather than trying and testing them all in a tiny window from when the RC lands to a no-doze meeting 15:22:04 <mattdm> So, is it helpful then to try to get as much *different* hardware to lots of people? 15:22:38 <pbrobinson> mattdm: I think we should come up with a small list of devices that people likely want and get a selection to people 15:22:51 <mattdm> pbrobinson Okay, cool. 15:23:02 <mattdm> That's what I am saying; I thought people were disagreeing 15:23:07 <pbrobinson> the problem we've had with HW in the past is that you send it to people and they still don't test/use it and it's been a waste of time 15:23:29 <mattdm> Yeah, we need actually committed people. We don't have money to throw away. 15:23:39 <bcotton> that's where i think explicitly adding hardware to the requirements helps 15:23:39 <mattdm> But I'm willing to take some chances too. 15:23:41 <pbrobinson> but I'm happy to assist with the HW list if other people can deal with the list of people that should have ti 15:24:00 <pwhalen> RIght, and test the boards within a small window before we release. Sometimes less than 24 hours. 15:24:05 <coremodule> pbrobinson, yes, if you can assist there, you have the brain we need to assemble the list 15:24:05 <bcotton> it's not so much a "we're blocking on this device" as a "having it on the list will help motivate testers to test" 15:24:34 <mattdm> right, if the test case says "does the Jetson Nano boot" it's easy for someone to whom we have sent a Jetson Nano to take responsibility 15:24:42 <nb> bcotton mattdm good points 15:24:48 <coremodule> I can deal with changing the docs to reflect your list 15:25:12 <bcotton> i think in a lot of cases, we can test earlier than the RC and be fine. but if a kernel gets an FE right before the compose, that's where it becomes troublesome 15:25:42 <mattdm> yeah, that's unfortunate -- but late-breaking kernel updates that fix big security flaws are a fact of life 15:26:22 <pbrobinson> mattdm: yes, but there needs to be run way to deal with the rest of the fall out that causes 15:26:38 <pwhalen> bcotton: in the case of the nano, we dont know why the GA images doesnt boot. Kernel is fine. 15:27:04 <bcotton> one thing we could do to minimize the risk is to have a "no FEs more than X time before th Go/No-Go. blocker fixes only". I don't think it'd help in this case, but it at least narrows the set of possible changes 15:27:13 <bcotton> pwhalen: sure, i'm speaking more generally 15:27:33 <mattdm> bcotton++ 15:27:33 <zodbot> mattdm: Karma for bcotton changed to 4 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 15:27:43 <coremodule> bcotton, that's a good idea. you should propose that at the next qa MEETING 15:27:48 <coremodule> whoops, caps lock 15:28:31 <bcotton> coremodule: ack 15:31:12 <mattdm> excellent all sorted then :) 15:31:18 <pwhalen> heh. 15:31:20 <coremodule> lol 15:32:00 <mattdm> #action mattdm to work with mnordin to make process for getting "grant" hardware for ARM QA 15:32:34 <pbrobinson> if only it was so easy, it's a start :) 15:32:43 <pwhalen> :) 15:33:21 <bcotton> hey, we've only released fedora 33 times. give us another 33 and we'll have everything just right :-) 15:33:45 <mattdm> action $SOMEONE_ELSE to come up with the "prioritized hardware" list? 15:33:50 <coremodule> pbrobinson, if you assembly a list and can send it to me, ill update the docs if you dont want to 15:33:52 <kushal> bcotton, we can do that in another 30times itself :) 15:34:04 * pbrobinson will be in some home sucking his thumb by then bcotton 15:34:27 <mattdm> let's make sure the AWS ARM systems (both virt and bare metal) are on the list too 15:34:39 <pbrobinson> coremodule: I'll work with pwhalen on this list and then he can work with you to get it updated? 15:34:46 <pwhalen> mattdm: this would be a big list. We also need to cover enterprise hardware.. and right, aws 15:34:47 <coremodule> pbrobinson, perfect 15:35:07 <mattdm> I think we need a list no bigger than 5 15:35:15 <pingou> pwhalen: no worries, mattdm said he has budget ;-) 15:35:19 <mattdm> and then there can be a second list of other important hardware 15:35:42 <mattdm> We've got limited resources, limited people, limited time, limited budget 15:35:50 <mattdm> There's no option but to narrow down 15:36:14 <pbrobinson> mattdm: yep, I was going to probably do 4-5.... else I run out of fingers 15:36:19 <mattdm> pbrobinson++ 15:36:30 <pbrobinson> mattdm: cover key things like certain GPUs etc 15:36:31 <mattdm> and you're sucking on your thumb, so four it is 15:36:34 <pwhalen> The enterprise stuff has to work or internal people come hunting for us. (and it all works in f33) 15:36:56 <pbrobinson> pwhalen: sure, and of course internal people could actually test that too ;-) 15:37:03 <pwhalen> HAHAHA 15:37:20 <mattdm> no really though. they should! 15:37:36 <pbrobinson> mattdm: welcome to try to encourage them ;-) 15:38:08 <mattdm> I will. Tell me who to bug :) 15:38:16 <pbrobinson> jfeeney 15:38:35 <pwhalen> That'll just come back to me. 15:38:50 <pwhalen> Dont worry about enterprise, I got it. 15:39:44 <pwhalen> For SBC's - Raspberry Pis (2,3,4), Jetson Nano, Pine64 15:40:18 <pbrobinson> RockPro64 15:40:22 <pbrobinson> Pinebook Pro 15:40:34 <pbrobinson> we'll do a list and circle back 15:40:41 <coremodule> cool 15:41:21 <jlinton> There is the "edge" hardware too mcbin/honeycomb. 15:41:52 <pbrobinson> jlinton: yes, I was going to review a bunch of all of those 15:42:21 <pbrobinson> the later isn't cheap though in the scheme of things so not something likely to be on the list and we have people testing it already 15:42:49 <pwhalen> Agreed, I dont have those either so I can't test. 15:43:03 <coremodule> is that the macchiatobin? 15:43:19 <jlinton> Right, I guess its more a question of making sure someone is testing it, I dropped much of my mcbin testing, because I can't use that many platforms 15:43:20 <pbrobinson> coremodule: yes 15:45:19 <pwhalen> ok, shall we revisit the list next week? Anything else we want to talk about? 15:45:25 <pbrobinson> nothing from me 15:45:53 <coremodule> here's one: has this affected the iot spin at all? do we know? 15:45:58 <coremodule> *on that hardware? 15:46:30 <pwhalen> yes, Pine64 15:47:26 <coremodule> okay, good to know 15:47:32 <pwhalen> ok, moving on 15:47:40 <pwhalen> #topic 1) ==== Userspace Status ==== 15:47:51 <pwhalen> Any new user space issues? 15:47:56 <pbrobinson> coremodule: possibly, but we respin all of that regularly anyway 15:48:42 <pwhalen> I'm not aware of any user space issues 15:48:56 <pwhalen> #info No issues reported. 15:49:07 <pwhalen> #topic 2) ==== Kernel Status ==== 15:49:47 <pwhalen> #info Kernel Test Week! https://fedoraproject.org/wiki/Test_Day:2020-10-26_Kernel_5.9_Test_Week 15:50:05 <pwhalen> If people could take a few moments and test on the hardware you care about. 15:50:25 <pwhalen> #info 5.9 still needs the xgene PMU patch added, will panic on boot. 15:51:08 <pwhalen> HPE Apollo only prints EFI messages, I'll try to get a bug filed on that today. 15:52:32 <pwhalen> Do we know when 5.9 will be released in f33, or is it just pending karma? 15:54:31 <pwhalen> We're almost out of time, and it seems I've lost folks.. 15:54:45 <pwhalen> #topic 3) == Open Floor == 15:54:50 <pwhalen> Anything else for today? 15:56:36 <coremodule> nothing from me 15:56:46 <pwhalen> #endmeeting