13:41:11 <mattdm> #startmeeting emergency-brakes-on-f33-release
13:41:11 <zodbot> Meeting started Tue Oct 27 13:41:11 2020 UTC.
13:41:11 <zodbot> This meeting is logged and archived in a public location.
13:41:11 <zodbot> The chair is mattdm. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:41:11 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
13:41:11 <zodbot> The meeting name has been set to 'emergency-brakes-on-f33-release'
13:41:11 <decathorpe> that sounds ominous
13:41:23 <x3mboy> .hello2
13:41:24 <zodbot> x3mboy: x3mboy 'Eduard Lucena' <eduardlucena@gmail.com>
13:41:28 * nirik waves
13:41:39 <bcotton_> .hello bcotton
13:41:40 <zodbot> bcotton_: bcotton 'Ben Cotton' <bcotton@redhat.com>
13:41:43 <mattdm> #info Apparently some key ARM devices never got tested with the kernel updated that landed in the last RC
13:41:46 <kalev> .hello kalev
13:41:47 <zodbot> kalev: kalev 'Kalev Lember' <klember@redhat.com>
13:41:49 <mattdm> #info and turns out they don't boot
13:41:55 <nb> .hi
13:41:59 <zodbot> nb: nb 'Nick Bebout' <nick@bebout.net>
13:42:06 <mattdm> #info specifically, the Rocks64 devices and the Jetson Nano
13:42:43 <nirik> bummer. ;(
13:42:56 <mattdm> #topic quick roll call
13:42:59 * pwhalen is here
13:43:01 <mattdm> #chair bcotton_
13:43:01 <zodbot> Current chairs: bcotton_ mattdm
13:43:04 <relrod> .hello codeblock
13:43:05 <mattdm> #chair nirik
13:43:05 <zodbot> Current chairs: bcotton_ mattdm nirik
13:43:05 <zodbot> relrod: codeblock 'Rick Elrod' <codeblock@elrod.me>
13:43:10 <mattdm> #chair relrod
13:43:10 <zodbot> Current chairs: bcotton_ mattdm nirik relrod
13:43:10 <x3mboy> .hello2
13:43:12 <zodbot> x3mboy: x3mboy 'Eduard Lucena' <eduardlucena@gmail.com>
13:43:16 <mattdm> #chair x3mboy
13:43:16 <zodbot> Current chairs: bcotton_ mattdm nirik relrod x3mboy
13:43:20 <nb> .hello nb
13:43:20 <zodbot> nb: nb 'Nick Bebout' <nick@bebout.net>
13:43:25 <mattdm> do we have people from qa here?
13:43:31 <frantisekz> myself, lruzicka
13:43:33 <mattdm> pwhalen: you around?
13:43:36 <lruzicka> yes
13:43:37 <mattdm> #chair frantisekz
13:43:37 <zodbot> Current chairs: bcotton_ frantisekz mattdm nirik relrod x3mboy
13:43:43 <mattdm> #chair lruzicka
13:43:43 <zodbot> Current chairs: bcotton_ frantisekz lruzicka mattdm nirik relrod x3mboy
13:43:44 <x3mboy> There are tons of people already downloading the ISO
13:43:53 <mattdm> x3mboy: yeah. well, it's before the release :)
13:43:55 <nirik> yeah, IMHO it's too late to stop release today.
13:43:56 <pwhalen> mattdm: I am
13:43:59 <mattdm> and if they're not on ARM it's okay
13:44:07 <x3mboy> Ok
13:44:19 <x3mboy> I know, they are refreshing getfedora, and the announcement is not on
13:44:26 <mattdm> #topic it's definitely too late in 5 minutes. so let's talk fast
13:44:30 <x3mboy> But, well, we need to publish with a note
13:44:48 <pbrobinson> mattdm: pine64 devices, not rock64
13:44:54 <mattdm> I've already added a vague note to the magazine article
13:45:06 <x3mboy> Let me check if I can read
13:45:20 <nirik> I think a common bugs/add to announcement + we look at perhaps doing a release with that fixed for arm later?
13:45:21 <lruzicka> is there a link to that article?
13:45:38 <decathorpe> oh, that's unfortunate. I can imagine that a lot of people will want to try fedora 33 on Pine* devices :(
13:45:42 <mattdm> https://fedoramagazine.org/?p=32102&preview=true&preview_id=32102&_thumbnail_id=32103
13:45:55 <mattdm> yeah and peter worked really hard to get extra support working
13:46:00 <mattdm> just something is broken with the new kernel
13:46:02 <pbrobinson> nirik: we should be able to run the compose with the bits needed tagged into an override correct? And then just cherry pick the bits needed
13:46:04 <mattdm> upgrades apparently work fine
13:46:12 <frantisekz> so, is there any way to not publish arm images anywhere before a fix comes? I mean not listing 33 for arm on getfedora and maybe changing permissions on iso files so people can't download them?
13:46:27 <pbrobinson> likely do it with the new shim
13:46:28 <lruzicka> mattdm, the article does not exist, 404
13:46:32 <nirik> pbrobinson: we could make a special pungi config to make just those I hope... would have to play with it, but should be possible
13:46:47 <nb> We could add a warning on the download page that the arm images might not boot on some devices
13:46:48 <mattdm> try https://fedoramagazine.org/?p=32102&preview=1&_ppp=161d56f052
13:46:54 <nirik> pbrobinson: we didn't get the new shim. ;)
13:46:57 <frantisekz> if upgrades work fine, it's one problem less and great news
13:46:58 <pbrobinson> nirik: yes, actually just like IoT does, consume the rest of the GA compose
13:47:06 * nirik nods
13:47:57 <nirik> frantisekz: the images work on many other devices tho...
13:48:01 <mattdm> So, I guess it's too scattered for a vote, so I'm going to make an FPL call. Give me your opinions, y'all
13:48:32 <mboddu> Hello
13:48:32 <mattdm> FWIW I guess my inclination is to release including the arm images, but note that there's a known problem on some devices and that we'll have updated images ... soonish
13:48:41 <nb> Too late to hold the release, make a note on the announcement, commonbugs, and on download page
13:48:41 <nirik> It's too late to stop release, we release with a note in announcemnt/common bug and promise to look at a updated image, we look at that and announce it when we have it.
13:48:45 <pwhalen> The Jetson Nano works when upgraded, but for some reason the GA image doesnt. The Pine64* also works, but networking is broken in kernels 5.15+, some devices work slowly, others do not even get an IP.
13:48:53 <nb> Yeah, try to make an updated image when we can
13:48:59 <frantisekz> nirik: yeah, I know, but (I am optimistic) and assume we'll have some fix in matter of days? users on working devices can wait a bit? don't know, just how I am feeling it
13:49:09 <mattdm> pwhalen: what does "doesn't work" look like for the Nano?
13:49:12 <Southern_Gentlem> +1 nirik, mattdm,nb
13:49:29 <pwhalen> mattdm: no kernel messages during boot, just the initial efi messages..
13:49:34 <bcotton_> it's going to be hard to stop the train at this point. i think we have to let it go and folks can install with 32 media and upgrade when a fix is available :-(
13:49:37 <mattdm> so _really_ doesn't work.
13:49:37 <lruzicka> So, could they install F32 and upgrade from there to get a working stuff?
13:49:43 <pwhalen> Not sure why, the kernel was tested and works when upgraded from say beta
13:49:47 <mattdm> It's ... exciting ... that they fail to work in different ways.
13:49:55 <mattdm> lruzicka: yes, apparently.
13:50:12 <pbrobinson> mattdm: I could do with less of that sort of exciting TBH
13:50:21 <frantisekz> :D
13:50:22 <mattdm> pbrobinson: no kidding.
13:50:23 <mboddu> Too late to hold the release
13:50:24 <x3mboy> Publish and make a note for new installations, since upgrade works
13:50:32 <mattdm> okay, seems like we've got consensus.
13:50:43 <pbrobinson> lruzicka: could even install beta and upgrade, but it's problematic because it causes support issues and a lot of queries
13:50:50 <mattdm> in the future, let's make explicit test cases for this hardware, and make sure more people have it
13:50:54 <pwhalen> My plan was to document those issues in common bugs.
13:51:05 <mattdm> #info if you are willing to test and need hardware, LET ME KNOW WE CAN MAKE THIS HAPPEN
13:51:10 <mboddu> Maybe if and when we run the point release, we can verify them
13:51:17 <lruzicka> pbrobinson, but the end users will not have access to beta images, will they? we need a workaround for CommonBugs
13:51:22 <pbrobinson> pwhalen: the nano is the big issue because it doesn't boot and we've done a bunch of work with nvidia on this
13:51:22 * nirik has some old pine64 stuff, but not any of the new stuff.
13:51:23 <mattdm> pwhalen: can you get something in there in the next ten minutes? even just a placeholder?
13:51:42 <pwhalen> mattdm: yes
13:51:48 <mattdm> pwhalen thanks
13:51:52 <pbrobinson> lruzicka: how do you work around something that doesn't boot? And we didn't officially support nano on F-32 even thought it worked
13:52:15 * cverna waves
13:52:22 <x3mboy> mattdm: I'm interested!
13:52:23 <mattdm> pbrobinson: workaround is install from f32 and upgrade, i guess?
13:52:29 <nb> mattdm I am interested in testing
13:52:50 <relrod> me too, happy to play with hardware and test in the future
13:52:54 <bcotton_> let's focus on the now and take care of improving our number of people with test devices later, eh?
13:52:58 * mboddu is happy to help as well
13:53:02 <pbrobinson> mattdm: the display didn't work on Workstation for F-32.... hence why it wasn't fully supported
13:53:48 <pbrobinson> mattdm: so we get into the "do this work around to work around the work around of the work around" ..... terrible way of supporting and just causes issues with newbies, hence why we don't yet fully support the rpi4
13:53:52 <nb> relrod can we add something to the download page that it doesn't work on some devices?
13:54:01 <mattdm> pbrobinson: yes, agree that it's terrible.
13:54:19 <mattdm> keep the download page vague and link to commonbugs
13:55:03 <mattdm> pwhalen, pbrobinson is getting more people testing Pinebook 64 helpful? because that's the easiest to get to people from my point of view
13:55:07 <pbrobinson> we should have a link to common bugs on the download page anyway shouldn't we?
13:55:17 <relrod> Feel free to suggest wording and I can work on getting a banner on gf.o download pages and arm.fp.o
13:55:24 <relrod> pbrobinson: we have one at the bottom in the footer
13:55:28 <pwhalen> I would definitely appreciate the help testing these devices. There are so many to test for arm and aarch64 from these small SBC's to the enterprise hardware we care about.
13:56:01 <pbrobinson> mattdm: 1) do you mean pinebook pro? 2) I need to finish documenting the setup of that 3) if the PBP it's not an affected device
13:56:14 <mattdm> pbrobinson: yes whatever the latest one is.
13:56:39 <mattdm> ok, so: let's talk about this in the ARM meeting. what time is that?
13:56:39 <pbrobinson> mattdm: it's not affected by this issue (as far as I'm aware)
13:56:54 <mattdm> meanwhile, I'm sending the release announcement now :-/
13:56:58 <mattdm> #info thanks all!
13:57:00 <lruzicka> sorry, had some network issies
13:57:02 <pbrobinson> pwhalen would know, the EU TZ has moved stuff so I'm lost
13:57:04 <decathorpe> mattdm: If both f32 and f33 GA images don't work, would pointing people with affected devices to the f33 RC 1.1 image be better, if that still had a working kernel?
13:57:19 <pwhalen> mattdm: in one hour
13:57:24 <lruzicka> mattdm, so what is the result:
13:57:25 <lruzicka> ?
13:57:29 <mattdm> pwhalen: which channel?
13:57:30 <pbrobinson> I think we just keep it vague and I'll work to get fixed images
13:57:31 <mattdm> lruzicka: shippit
13:57:37 <lruzicka> mattdm, ok
13:57:56 <pbrobinson> we've not widely announced the NV stuff so we can just do a follow up magazine article once the new images are in place
13:58:01 <pwhalen> mattdm: this one
13:58:01 <lruzicka> mattdm, so how CommonBugs should be updated? workaround with F32?
13:58:13 <mattdm> pbrobinson: yeah we can make some jam out of these lemons
13:58:23 <mattdm> pwhalen: convenient!
13:58:26 <mattdm> #endmeeting