15:00:38 <rbergeron> #startmeeting Go/No Go Continuation, Round Two, RC3, F17
15:00:38 <zodbot> Meeting started Thu Apr  5 15:00:38 2012 UTC.  The chair is rbergeron. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:38 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:00:53 <rbergeron> #meetingname Go/No-Go Continuation F17 beta rc3 part two or three
15:00:53 <zodbot> The meeting name has been set to 'go/no-go_continuation_f17_beta_rc3_part_two_or_three'
15:00:59 <ianweller> round two: electric boogaloo
15:01:03 <rbergeron> #topic Who's up
15:01:13 <rbergeron> wwoods, tflink, adamw, nirik
15:01:15 <ianweller> oh god, i can forsee an interesting meetbot bug
15:01:16 <ianweller> hold on
15:01:26 <rbergeron> ianweller: yeah, i have fudged that one up a few times
15:01:27 * nirik is around.
15:01:29 <rbergeron> i forget when i'm sleepy
15:01:41 <ianweller> never mind, it handles it okay :)
15:01:48 * pschindl_ is here
15:02:01 <adamw> yo
15:02:28 * tflink is around
15:02:39 <adamw> both bcl and hughsie are active, just trying to get them both in here
15:03:14 <rbergeron> okay
15:03:16 <adamw> we're gonna need some intellectual firepower on preupgrade
15:03:18 <rbergeron> #chair nirik adamw tflink
15:03:18 <zodbot> Current chairs: adamw nirik rbergeron tflink
15:03:42 <rbergeron> Do we want to look at test list, new proposed, or preupgrade drama first
15:03:45 <adamw> <hughsie> adamw, i'm got a RH conf call now, sorry
15:04:31 <adamw> the last two are the same...
15:04:37 <adamw> preupgrade drama is one of the (two) new proposed
15:05:06 <adamw> or three. or what the hell ever.
15:07:03 <adamw> we're gonna need _someone_ who can take a stab at the preupgrade bug, though.
15:08:25 <rbergeron> three, yeah
15:08:29 <adamw> we can nibble around the edges
15:09:19 <rbergeron> okay, want to look at the validation list?
15:11:11 * rbergeron looks for anyone to agree with her
15:11:22 <tflink> works for me
15:11:26 <rbergeron> just to get any administrative bits out of the way :)
15:11:29 <tflink> we have to do it at some point :)
15:11:31 <rbergeron> #topic Test Results
15:11:37 <rbergeron> #link https://fedoraproject.org/wiki/Test_Results:Fedora_17_Beta_RC3_Install
15:12:11 <rbergeron> https://fedoraproject.org/wiki/Test_Results:Fedora_17_Beta_RC3_Base
15:12:22 <rbergeron> https://fedoraproject.org/wiki/Test_Results:Fedora_17_Beta_RC3_Desktop
15:12:40 <adamw> i'm happy with the coverage at this point
15:12:57 <tflink> yeah, other than preupgrade, I'm not seeing anything that is a huge red flag
15:12:57 <adamw> we can pull the missing tests from rc2, especially for kde
15:13:16 <adamw> installation and base are full except for scsi, which we have no hardware for.
15:13:30 <adamw> er, 'full' wrt beta.
15:13:46 <adamw> all fails that ought to be proposed as blockers are, i'd say
15:14:00 <adamw> i'm not horribly worried about any of the efi fails, as enough methods work
15:14:52 <adamw> they're all marked 'alpha' because we want in the big picture EFI to work at alpha, but we don't really need every single permutation to pass.
15:15:43 <rbergeron> okay
15:16:08 <rbergeron> so ... agreed we are okay with the level of test coverage as of now on RC3
15:16:14 <rbergeron> ?
15:16:26 <adamw> ack
15:16:31 <rbergeron> #agreed we are okay with the level of test coverage as of now on RC3
15:16:41 <rbergeron> #topic Blockers
15:17:07 <rbergeron> adamw: I leave it to your discretion about the tackling of preupgrade vs. other bugs
15:17:11 <rbergeron> or tflink
15:17:17 * rbergeron suspects y'all are tyring to catch someone who can give feedback
15:17:24 <adamw> yes.
15:17:29 <adamw> oh, hey bcl, thanks
15:17:37 <rbergeron> #chair bcl
15:17:37 <zodbot> Current chairs: adamw bcl nirik rbergeron tflink
15:17:56 <bcl> so, where are we at?
15:18:02 <adamw> about to start looking at blockers
15:18:17 <adamw> let's take the easy ones first
15:18:19 <adamw> #topic https://bugzilla.redhat.com/show_bug.cgi?id=741594
15:18:39 <adamw> my feelings on this can be summarized by the words BIG RED NO, accompanied by several other words not fit for print.
15:18:44 <Martix> adamw: easy?
15:18:48 * nirik clicks
15:18:54 <Martix> https://bugzilla.redhat.com/show_bug.cgi?id=741594#c14
15:19:14 <Martix> please refresh page
15:19:35 <adamw> the answer to that is to have your shit together for a test day more than one day in advance.
15:19:51 <nirik> this was a yum update on a running live media? not a install from live media?
15:19:53 <adamw> yes,
15:20:04 <Martix> adamw: I did, but dev is doing what he thinks
15:20:14 <nirik> -1 blocker.
15:20:18 <adamw> bottom line, there has never been the slightest expectation that yum update on running live would work, not in the critera, not anywhere else. it's absolutely not a beta blocker.
15:20:22 <tflink> I'm pretty much with adam on this one, but a little less extreme :)
15:20:26 <adamw> Martix: i didn't mean personally, i meant in general.
15:20:30 <bcl> yum update on live really isn't a good idea.
15:20:34 <tflink> -1 blocker
15:20:35 <Martix> adamw: ok
15:20:41 <adamw> if you're relying on your users being able to yum update a live image, You Have Already Lost. :)
15:20:46 <nirik> it might be nice to capture the attempt and warn or something.
15:20:58 <rbergeron> -1 blocker :)
15:20:59 <Martix> yum update is often workflow during Test Days
15:21:08 <nirik> unless you have a LOT of memory. ;)
15:21:16 <bcl> Martix: a full update or specific packages?
15:21:17 <adamw> Martix: it wasn't when i was running them ;)
15:21:33 <Martix> nirik: 4GB isnt lot of memory?
15:21:38 <nirik> no.
15:21:48 <Martix> adamw: Gnome Shell SW rendering?
15:22:03 <Martix> nirik: what about persistent overlay?
15:22:10 <bcl> a yum update of a small set (1) of packages *may* be ok. but I wouldn't want to support it in general.
15:22:13 <nirik> every file added, _and_ every file removed has to be kept in the dm overlay. Aside from isolated small packges being installed, yum upgrade from a running live media is doomed.
15:22:15 <adamw> Martix: yeah, i didn't run that. i did tell desktop team several times to make sure they had an updated live image ready.
15:22:26 <Martix> bcl: what is small?
15:22:29 <adamw> (1)
15:22:38 <adamw> persistent overlay is /home only, isn't it?
15:22:50 <nirik> sure, if you have usb and persistent overlay you are in much better shape.
15:23:03 <nirik> but it still goes away a lot faster than you think.
15:23:18 <nirik> it's not just keeping the current change... but everything in between.
15:23:35 <Martix> btw AFAIR yum update always worked on F14 or F15 on Live media
15:23:50 <bcl> luck
15:23:53 <tflink> yeah, I remember seeing goofy errors when I attempted to debug a kernel issue  by updating the kernel in usb+overlay live
15:24:28 <adamw> i've certainly seen it explode in 14/15
15:24:32 * satellit_laptop liveinst to 8 GB USB is OK
15:24:39 <adamw> it's obviously going to depend on how many updates happen to be available at the time
15:24:43 <bcl> anyway, not a blocker. -1
15:24:44 <tflink> it turned out to be easier to just build livecds or do an install to a USB HD
15:25:06 <Martix> adamw: it was updates after release
15:25:28 <Martix> but 100 or 200 MB of update shouldnt kill system with 4 GB RAM or overlay
15:25:44 <nirik> I suspect it will.
15:25:50 <adamw> anyhow
15:25:51 <tflink> that sounds like a really bad idea to me
15:25:52 <adamw> we have solid - votes
15:25:58 <tflink> yep
15:26:06 <nirik> 100-200MB of updates is xz compressed payload... unpacking is... larger.
15:26:13 <Martix> free space is never exhausted when I run df -h
15:26:22 <adamw> propose #agreed 741594 does not hit any of the criteria and the cited scenarios are not significant enough to worry about delaying the beta
15:26:27 <tflink> ack
15:26:48 <nirik> Martix: it's not in a df... it's in a overlay.
15:27:43 <adamw> acks welcome
15:27:46 <rbergeron> ack
15:27:54 <nirik> Martix: 'dmsetup status live-rw'
15:28:05 <bcl> ack
15:28:30 <nirik> ack
15:29:18 <adamw> #agreed 741594 does not hit any of the criteria and the cited scenarios are not significant enough to worry about delaying the beta
15:29:26 <rbergeron> moving on then
15:29:56 <rbergeron> wait, there were three, now there are five
15:30:48 <rbergeron> which one's next :)
15:31:10 <adamw> let's go for a kde party
15:31:12 <adamw> #topic https://bugzilla.redhat.com/show_bug.cgi?id=810226
15:31:46 <adamw> "Cursor invisible in KDE"
15:32:05 <adamw> i can reproduce this, but only in a VM set up with qxl as the driver but VNC as the viewer, which has never been a default config
15:32:19 <adamw> the old default was cirrus/vnc (which hits another bug we'll come to), the new default is qxl/spice
15:32:29 <adamw> you can only get cirrus/spice by manual config
15:32:33 <adamw> has anyone else poked this one?
15:32:49 <tflink> adamw: how long as qxl/spice been the default?
15:33:06 <bcl> oh, hey, I hit that.
15:33:22 <tflink> a vm I just created on F16 x64 is using cirrus/vnc by default
15:33:39 <bcl> same conditions as adamw, I was working around the shell screen corruption bug.
15:34:00 <adamw> tflink: a new machine definition?
15:34:20 <adamw> then i really give up on figuring out those frickin' defaults. how did you create it?
15:34:24 <tflink> adamw: yeah, new disk, new VM
15:34:26 <adamw> bcl: so, qxl/vnc ?
15:34:29 <tflink> created w/ VMM
15:34:30 <bcl> right
15:34:49 <adamw> if no-one can reproduce with cirrus/vnc or qxl/spice i'm inclined towards -1 on this, though it's close
15:35:03 <tflink> the only non-default thing I do is lvm backed storage instead of flatfile
15:35:28 * tflink hasn't tried, to be honest
15:36:33 <bcl> adamw: oh, and this isn't KDE specific. I see it in anaconda which uses metacity and gtk
15:36:40 <adamw> ah, that's interesting
15:37:14 <Kevin_Kofler> Looks like an issue deeper in the stack than the KDE Workspaces for sure.
15:38:05 <adamw> seeing if i can confirm with desktop
15:38:19 <adamw> yup
15:38:28 <adamw> so basically any qxl/vnc is invisible cursor.
15:39:07 <tflink> not sure if it's a beta blocker, though
15:39:19 <tflink> since it doesn't hit what we think the 2 most common configs are
15:39:24 <adamw> yeah, i'm kinda on the fence
15:39:28 <tflink> qxl/spice and cirrus/vnc
15:39:40 <adamw> asking in #virt if they have a feel for whether qxl/vnc is popular and/or supported
15:40:22 <adamw> anyone else have a feeling on this one?
15:40:47 <rdieter_work> adamw: I'd tend to agree with you, depends on how common that config is.
15:40:53 <bcl> I'd lean towards -1 -- I guess I need to learn how to use spice with qemu
15:40:55 <adamw> oh, hey, look what the first google result for 'qxl vnc' is.
15:40:55 <adamw> http://patchwork.freedesktop.org/patch/9198/
15:41:07 <rdieter_work> lolz
15:41:36 <rbergeron> folks: I have to depart now for dentist but I will be back on shortly to keep going in meeting
15:41:38 * nirik leans toward -1 as well... especially if it can be fixed in an update.
15:41:42 <rbergeron> #chair rdieter_work
15:41:42 <zodbot> Current chairs: adamw bcl nirik rbergeron rdieter_work tflink
15:41:58 <tflink> yeah, this can't be fixed on the live images - it's a host problem
15:42:01 * rbergeron would lean towards -1
15:42:04 <adamw> bcl: http://www.linux-kvm.org/page/SPICE
15:42:05 <tflink> unless I'm missing something
15:42:15 <rbergeron> oh, ew
15:42:47 <nirik> well, if you are using the live image in virt instead of installing in virt. Not sure how common either is off hand.
15:42:48 <bcl> adamw: thanks
15:43:11 <adamw> nirik: both v. common.
15:43:24 <Kevin_Kofler> If that's really the problem, it's indeed on the host end as tflink says, so not a bug in the live images at all.
15:43:30 <bcl> so if that bug is correct this has nothing to do with the media. It is a qemu bug and can be fixed later.
15:43:41 <tflink> but if this is a driver problem on the host, how can we fix it in beta?
15:43:48 <adamw> the patch is in qemu?
15:43:49 <nirik> well, I meant which is more common... but yeah.
15:44:38 <adamw> tflink: you can update the host?
15:44:39 <tflink> adamw: if it isn't, won't this affect F16, too?
15:44:45 <adamw> tflink: presumably. i can test that.
15:45:05 * tflink has a F16 VM sitting in front of him, will try
15:45:08 <adamw> if the bug's in the host, then definitely -1.
15:46:04 <adamw> yup, reproduced with a 16 desktop live.
15:46:23 <adamw> so given that, i'm a definite -1. i'll try and hack up a fixed qemu to confirm it can be fixed with a host update, but sounds convincing.
15:46:32 <bcl> same here with my qemu-kvm and f16 netboot
15:46:36 <bcl> -1 blocker
15:47:13 <tflink> -1 blocker
15:48:04 <adamw> propose #agreed 810226 only hits a non-default configuration (qxl as the graphics driver, VNC as the display agent) and the bug is on the host side, not the guest side, so it can safely be fixed with an update: we do not need to re-spin images to fix it. therefore, rejected as a blocker
15:48:15 <tflink> ack
15:48:20 <bcl> ackack
15:48:29 <nirik> ack
15:48:37 <tflink> adamw: we probably want to commonugs it, though
15:48:40 <adamw> point
15:48:44 <tflink> commonbugs, even
15:48:45 * nirik nods.
15:48:57 <adamw> #topic https://bugzilla.redhat.com/show_bug.cgi?id=810161
15:49:04 <adamw> so, this one's likely to be a bit more sticky. =)
15:49:32 <adamw> background: we had a blocker bug, https://bugzilla.redhat.com/show_bug.cgi?id=809052 , which was really bad display corruption in shell on a cirrus/VNC VM
15:50:08 <Kevin_Kofler> So now we have KDE Platform applications being corrupted instead.
15:50:16 <adamw> the update https://admin.fedoraproject.org/updates/xorg-x11-drv-cirrus-1.3.2-19.fc17,mesa-8.0.2-2.fc17 fixes it and was pulled into rc3, but it introduces a (MHO) less serious display corruption regression in KDE on cirrus/VNC
15:50:19 <Kevin_Kofler> Plus, the "fix" degrades overall display quality by reducing color depth.
15:50:46 <Kevin_Kofler> IMHO, this kind of change is landing way too late for its impact and should be pulled out.
15:50:46 <adamw> Kevin_Kofler: i don't believe that's actually true, i'm pretty sure i've heard ajax explain before that there's actually no practical difference between 16 and 24. the details escape me, though.
15:51:24 <adamw> ajax's rationale for doing it this way is that the 24bpp path is used by almost no drivers and not well tested; he thinks all drivers should be on 32 or 16
15:51:33 * adamw trying to find records
15:51:56 <Kevin_Kofler> 24 bpp is what cirrus used for ages, and in fact we went out of the way to fix Qt to work with it back in the day.
15:52:05 <bcl> adamw: so this only hits KDE in VM and only with cirrus/vnc?
15:52:12 <Kevin_Kofler> And now we get force-switched to 16 between Beta RC2 and Beta RC3. :-/
15:52:21 <adamw> so, important that we understand the issue fully: has anyone seen anything besides the pink scrollbars?
15:52:46 <adamw> i only tested briefly, enough to see the pink stuff, i didn't see anything further than that immediately but i might've missed something
15:52:47 <adamw> bcl: yes.
15:52:48 <nirik> is this a cirrus bug? or a general 16bpp issue with kwin/etc?
15:53:03 <Kevin_Kofler> Those pink scrollbars are so blatant they immediately strike a user as "broken".
15:53:04 <adamw> bcl: desktop is fine with cirrus/vnc, kde is fine with qxl/spice.
15:53:05 <nirik> I guess cirrus
15:53:08 <bcl> adamw: so does it work ok with spice/qxl?
15:53:13 <adamw> Kevin_Kofler: sure, i just want to make sure we catch if there's anything worse
15:53:26 <bcl> I'd say -1 blocker then.
15:53:30 <adamw> Kevin_Kofler: i don't want to be discussing it on the basis of 'pink scrollbars' if the bug is actually 'pink scrollbars and my baby explodes'
15:53:42 <rdieter_work> hard to say, do any other drivers use 16bpp by default?
15:53:56 <adamw> rdieter: ajax says all the other 'fallback drivers' use 16bpp
15:53:59 <Kevin_Kofler> vesa does, I think.
15:54:02 <Martix> adamw: when I was discussing next TD with jreznik he told me, there are more artefacts than just scrollbars
15:54:08 * rdieter_work only has old i945 i965 to test
15:54:11 <adamw> i'm not entirely clear what he means by 'fallback drivers' but i believe we're talking vesa, fbdev, that kinda stuff.
15:54:13 <Kevin_Kofler> I don't think we've ever seen that particular issue with vesa though.
15:54:19 <adamw> Martix: is he around?
15:54:24 <Kevin_Kofler> So I suspect a driver bug.
15:54:31 <Martix> adamw: unfortunately not now
15:54:48 <adamw> Kevin_Kofler: i tried to ping ajax/airlied to discuss it
15:55:19 <adamw> hi ajax
15:55:23 <adamw> so we're arguing about https://bugzilla.redhat.com/show_bug.cgi?id=810161
15:55:36 <adamw> the kde regression introduced by the fix for https://bugzilla.redhat.com/show_bug.cgi?id=809052
15:55:54 <ajax> oh good, choice.
15:55:59 <adamw> =)
15:56:15 <ajax> i actually found something similar in gnome while trying to fix 16bpp gears to not be translucent
15:56:15 <adamw> kev was questioning switching to 16bpp so late, any background on that? specifically the question of what drivers use what depths has come up
15:56:35 <ajax> normally we try to use 32bpp if at all possible
15:56:40 <Kevin_Kofler> Looks like cirrus is badly broken in 16 bpp then.
15:56:44 <ajax> Kevin_Kofler: no.
15:56:56 <ajax> cirrus is correct.  the desktop is broken.
15:57:07 <Kevin_Kofler> I'm not aware of ever having seen this on vesa though.
15:57:12 <Kevin_Kofler> And vesa is 16 bpp, isn't it?
15:57:17 <ajax> also no.
15:57:23 <Kevin_Kofler> Oh?
15:57:37 <ajax> if you'd let me talk...
15:57:56 <adamw> you're not both using the same keyboard, are you? :)
15:58:02 <Kevin_Kofler> ^^
15:58:23 <ajax> vesa tries for 32 then 16 then 24.  because 24bpp is easily the worst-tested path in the world.
15:58:35 <ajax> the cirrus card that qemu emulates can't do 24bpp.
15:58:36 <adamw> ah, checkboxes are half purple, there's purple spots on menu entries (File, Window etc)
15:58:40 <ajax> excuse me. can't do 32bpp.
15:58:56 * rbergeron peeks in
15:59:05 <ajax> it turns out that gnome is a little broken for 16bpp too, and it's _weird_
15:59:09 <adamw> slider widgets have purple backgrounds.
16:00:06 <adamw> let's say there's a definite lilac theme to the desktop. =) i haven't found anything that's actually unusable yet though.
16:00:07 <ajax> what happens is when cogl goes to create a glx pixmap for texture_from_pixmap, it guesses whether to do xrgb or argb based on whether the number of bits in the window's visual matches the pixmap's bit depth.
16:00:26 <ajax> but it turns out that's the wrong thing to do
16:00:42 <ajax> because the backing pixmap can often be 32bpp, due to the way composite works
16:00:58 <ajax> so you have a 16-bit visual but 32 bpp real storage
16:01:30 <ajax> so in cogl this manifests as the alpha channel being 0 a lot of the time, and gears is accidentally translucent
16:01:44 <ajax> but i'm reasonably sure you'd find something similar in the kde code
16:01:53 <Kevin_Kofler> But KDE does not use OpenGL at all on cirrus.
16:02:01 <ajax> did i say opengl?
16:02:06 <adamw> is anyone else testing this, btw? it's easy to poke
16:02:13 <Kevin_Kofler> glx, texture_from_pixmap etc.
16:02:15 <adamw> just boot up the kde live in a cirrus/vnc vm
16:02:16 <ajax> it's not a property of opengl.  it's a property of Composite.
16:02:37 <ajax> redirected windows can end up with a backing pixmap that doesn't match the X visual.
16:02:49 <ajax> if kde is still compositing even on cirrus, then you'd hit this.
16:03:03 <Kevin_Kofler> It's compositing using XRender.
16:03:07 <ajax> well then there you go.
16:03:22 <ajax> it might manifest differently but i suspect it's the same underlying conceptual bug.
16:03:50 <Kevin_Kofler> I guess we need to test by disabling effects/compositing in KWin entirely.
16:04:05 <adamw> how do i do that?
16:04:15 <rdieter_work> so, we have possible workaround 3, disable desktop effects
16:04:18 <Kevin_Kofler> KDE System Settings, there's a Desktop Effects entry there.
16:04:22 <adamw> ah, got it.
16:04:25 <rdieter_work> adamw: ALT-SHIFT-F12
16:04:31 <rdieter_work> should toggle effects on/off
16:04:53 <Kevin_Kofler> Upstream falls back to XRender when there's no suitable OpenGL (and llvmpipe is not "suitable" under KWin's current definition, due to both corruption issues and performance issues).
16:05:39 <ajax> corruption you'd probably fix if you did the moral equivalent of http://pkgs.fedoraproject.org/gitweb/?p=cogl.git;a=blob;f=cogl-1.8.2-lp-no-framebuffer-blit.patch;h=c6564772d3c45abc5f3db6c8188a801c290c8427;hb=HEAD
16:06:04 <ajax> wouldn't fix the performance bit, of course
16:06:13 <adamw> turning effects off doesn't seem to fix things.
16:06:16 <adamw> anyhow
16:06:23 <ajax> well hell.
16:06:27 <adamw> we don't really need to know how to fix it in this meeting...question is, is it serious enough to be a blocker
16:06:52 <tflink> which gets us back to the question of how common the verious VM settings are
16:07:07 <ajax> qemu defaults to cirrus because they're idiots.
16:07:12 <adamw> well since you seem to bleedin' well always get cirrus/vnc as your default...=)
16:07:51 <adamw> ajax: i've long thought they switched to qxl/spice as default with f16, but tflink says he gets cirrus/vnc as default even for a new 64-bit VM on his f16 machine. so i'm really not sure what the story is there.
16:08:02 <adamw> tflink: also the question of just how bad we think the bug is, to be honest.
16:08:04 <bcl> adamw: I'm -1 blocker. Seems like you can use qxl/spice to get around it.
16:08:20 <Kevin_Kofler> And doesn't the switch to 16 bpp also mean that graphics will look worse than before (even if not corrupted)?
16:08:21 <bcl> I'd be +1 final blocker though.
16:08:21 <rdieter_work> imo, slight corruption no real impact , workarounds available, -1 beta blocker, can NTH later perhaps
16:08:25 <adamw> looks silly? sure. but i've yet to find a symptom which renders the desktop unusable.
16:08:26 <rbergeron> maybe jforbes knows offhand, but....
16:08:31 <ajax> i could probably fix it by getting cirrus to expose 32bpp visuals and papering it over in the shadowfb upload hook
16:08:32 * nirik is -1 blocker, +1 common bugs for sure.
16:08:51 <adamw> oh, of course if someone's on an old enough host, spice/qxl may be unavailable or really buggy.
16:09:01 <adamw> i'm not sure how good the spice/qxl in f15 is. anyone used that combo lately?
16:09:02 <ajax> which is probably not a bad idea anyway
16:09:08 <rbergeron> i would be -1 for beta, but commonbugs... final though maybe
16:09:10 <tflink> adamn: just because it always happens to me, doesn't mean that it always happens for everyone:)
16:09:42 <tflink> I'm probably -1 blocker, +1 commonbugs, +1 nth
16:09:44 <adamw> i'm a slight -1 mainly because it's just not a terrible bug, it looks silly but everything works okay
16:09:51 <adamw> +1 nth +1 commonbugs obviously
16:10:17 <Kevin_Kofler> Well, we have a fix available, revert the change. So let's exercise the NTH with that fix.
16:10:32 <adamw> not really an option, as that would re-open a blocker.
16:10:46 <Kevin_Kofler> Then that blocker needs to be reconsidered, or fixed differently.
16:10:58 <Martix> Kevin_Kofler: +1 for different fix
16:11:11 <adamw> Kevin_Kofler: sure, if we wind up slipping, we have several days to come up with something which fixes both
16:11:17 <bcl> Kevin_Kofler: Remember this is BETA, there's still time to fix it.
16:11:32 <adamw> if we don't slip, though, i'm okay with shipping this, and commonbugsing it.
16:11:47 <nirik> adamw: +1
16:11:51 <tflink> same here - the workaround is pretty easy
16:12:02 <adamw> for the record, not because this is KDE. if we had a KDE bug as bad as 809052, i'd call it blocker.
16:12:10 <bcl> yep.
16:12:19 <Kevin_Kofler> tflink: The same workaround would also fix the other bug which was accepted as a blocker and whose "fix" caused this late regression.
16:12:22 <tflink> yeah, 809052 made shell completely un-usable
16:12:35 <ajax> Kevin_Kofler: i look forward to your patch.
16:13:13 <adamw> propose #agreed 810161 is rejected as a blocker because the impact is minor (it's essentially a cosmetic bug, desktop is still usable), also it only hits one specific virt configuration so it can at least be worked around. accepted as NTH and will be commonbugs'ed if we ship with it
16:13:21 <Martix> ajax: adamw for the record, we have KDE 4.8 Test Day on Tuesday, so #810161 fix would be nice
16:13:28 <rbergeron> ack
16:13:30 <nirik> ack
16:13:30 <bcl> ack
16:13:31 <adamw> Martix: thanks
16:13:38 <tflink> ack
16:13:43 <ajax> Martix: i'll see what i can do.
16:13:52 <Martix> ajax: thank you
16:13:56 <adamw> #agreed 810161 is rejected as a blocker because the impact is minor (it's essentially a cosmetic bug, desktop is still usable), also it only hits one specific virt configuration so it can at least be worked around. accepted as NTH and will be commonbugs'ed if we ship with it
16:14:24 <adamw> #topic https://bugzilla.redhat.com/show_bug.cgi?id=810289
16:14:36 * ajax afk for lunch
16:14:37 <adamw> so, this is annoying - it's the one case bcl and i didn't think of in the free space for upgrade check discussion
16:15:08 <adamw> what happens if you don't have enough space for upgrade but you do have enough space for /usr move? answer: /usr move happens, and so even though anaconda catches that you don't have enough space for upgrade, you're stuck with a basically busted system
16:15:24 <adamw> you can't just go back to 16, as it's been /usrmoved. you have to free up space and do the upgrade.
16:15:38 <adamw> ajax: thanks, i think that's all we needed you for
16:15:59 <Kevin_Kofler> Just yet another example of how UsrMove is broken… :-/
16:16:02 <adamw> bcl: wdyt on this one?
16:16:15 <bcl> adamw: I don't think we check for space on upgrade.
16:16:27 <adamw> "5. usr-move is executed, anaconda will not upgrade your machine, bc. of missing
16:16:27 <adamw> space"
16:16:44 <bcl> And really, we can't know how much space will be needed.
16:17:32 <bcl> oh, sorry, I hadn't noticed he'd filed a bug with details.
16:17:35 <adamw> i guess that could just mean 'it fails', not 'anaconda refuses to try'.
16:17:45 <adamw> it's a bit unclear.
16:18:47 <cpuobsessed> if root and usr are separate partition then if usr free > usr + root?
16:19:37 <Kevin_Kofler> AFAIK, UsrMove actually operates on a copy of the directories, so it does need some additional space.
16:19:39 * wwoods is here if still needed
16:19:39 <adamw> bcl: so we can't really do anything better than hardwire an ugly arbitrary free space check into dracut or something here?
16:19:50 <adamw> wwoods: you will be, we'll be coming to the preupgrade bug later
16:19:54 <bcl> I'd guess that it fails during package install.
16:20:06 <adamw> Kevin_Kofler: apparently it uses hardlinks
16:20:07 <Kevin_Kofler> It could be done with basically no space requirements if it'd MOVE files from one directory into the other without using a copy, but that'd be more risky.
16:20:10 <pjones> Kevin_Kofler: they sell 3TB drives now, you know.
16:20:10 <adamw> Kevin_Kofler: so it doesn't actually need much.
16:20:18 <bcl> adamw: yeah, I don't know what else to do.
16:20:23 <pjones> the hardlink tree isn't exactly what you'd call large.
16:20:29 <adamw> i can't see that there's really much we can _do_ here
16:20:39 <adamw> since usrmove happens in dracut, the free space check needs to be before it
16:20:50 <adamw> and we can hardly go wiring 'check the installed package set and estimate size needed for upgrade' code into dracut...
16:20:58 <nirik> perhaps dracut could grow a 'unupgrade' ?
16:21:00 <Kevin_Kofler> pjones: I know. :-)
16:21:06 <bcl> Kevin_Kofler: I talked to harald and convertfs fails gracefully if it runs out of space.
16:21:10 <nirik> so if the install fails, anaconda could tell it to revert on boot?
16:21:18 <adamw> and it's hard to just set an arbitrary minimum because the disk space needed for a minimal install upgrade is going to be a lot less than for a full install
16:22:16 <bcl> right. that's why I didn't see a problem when I tried it with 1G free.
16:22:27 <Kevin_Kofler> nirik: Only really possible if you save a list of what files were where.
16:22:35 <Kevin_Kofler> Undoing a merge is harder than doing it. ;-)
16:22:36 <nirik> yeah.
16:22:44 <bcl> this isn't a new problem either, it's just that usrmove makes it slightly worse.
16:22:57 <adamw> true. and i guess it is 'slightly'
16:22:58 <pjones> yeah, you're down in the territory where %post space estimation is a problem.
16:23:02 <pjones> i.e. the weeds.
16:23:11 <pjones> %postin rather
16:23:14 <adamw> because if anaconda has no check, when it gets halfway through an upgrade and fails, you likely have a busted system anyhow.
16:23:16 <bcl> right.
16:23:34 <bcl> At best we could test for (1G?) free and prompt to continue.
16:23:48 * adamw is trying to reproduce the reporter's scenario to see exactly how anaconda fails
16:23:55 <adamw> bcl: yeah, it's a bit microsoft-y but can't see what else
16:24:57 <bcl> wwoods: do you have any feeling for metadata overhead on package installs?
16:25:21 <bcl> maybe we could look at the # of installed packages and make a guess.
16:25:39 <adamw> so if we've never had a space check in anaconda anyway, i guess i'm -1 to this
16:26:18 <adamw> because that would mean we've shipped a dozen releases which can explode on low space upgrade and everyone's just said 'boy, that fedora sure is a lil scamp'
16:26:21 <bcl> I'm -1 blocker +1 NTH but we should probably come up with something for final.
16:26:26 <tflink> would you have ended up with a busted system if this happened with earlier releases?
16:26:39 <bcl> yes, depending on where in the package list it failed.
16:26:45 <adamw> tflink: i'm pretty sure 'yes', if the only failure mode anaconda has is 'try the upgrade anyway then explode at some point'.
16:26:55 <adamw> usr move is not making things a whole lot worse, in practical terms.
16:26:58 <tflink> then -1 blocker, +1 NTH
16:28:09 <adamw> i wish my test would finish so i can confirm we don't hit some kind of anaconda check bcl has forgotten about =)
16:29:02 <adamw> well, we can circle back if my test finds something new
16:29:14 <rbergeron> -1 blocker, +1 nth....... unless adam finds something
16:29:21 <rbergeron> bad something
16:29:22 <rbergeron> :)
16:30:04 <adamw> propose #agreed #810289 is rejected as a blocker on the basis that we've never actually checked for low space on upgrade (and hence have implictly never considered it a blocking issue). anaconda will just try to upgrade and explode. so this isn't in fact a regression compared to earlier releases, and we don't have a specific requirement for a low space check in criteria or anywhere else
16:30:46 <bcl> ack
16:31:27 <tflink> ack
16:31:27 <nirik> ack
16:31:38 <adamw> #agreed #810289 is rejected as a blocker on the basis that we've never actually checked for low space on upgrade (and hence have implictly never considered it a blocking issue). anaconda will just try to upgrade and explode. so this isn't in fact a regression compared to earlier releases, and we don't have a specific requirement for a low space check in criteria or anywhere else. accepted as NTH
16:31:57 <adamw> okay, fun time :/
16:31:59 <adamw> #topic https://bugzilla.redhat.com/show_bug.cgi?id=810136
16:32:07 <adamw> so, as we feared, preupgrade is still borked.
16:32:23 <adamw> this is one of those where we've squelched exactly one bug in each tc/rc, and we're not at the end of the chain yet.
16:32:25 * tflink wants to figure out a better way to test this earlier
16:32:31 <adamw> wwoods: floor show
16:32:40 <tflink> we always seem to have problems with preupgrade at the last minute
16:32:45 <adamw> tflink: i at least figured out how you can force preupgrade to pull from some specific location, so it should be possible.
16:33:12 <tflink> and there is a TODO in the preupgrade code to make the location configurable
16:33:13 <adamw> obviously, it's important to know whether this bug will be fixed preupgrade side or anaconda side
16:33:41 <adamw> but even if it's preupgrade side, there's the possibility we'll find ANOTHER bug behind it which is in anaconda
16:34:16 <bcl> repo=hd::/var/ is missing a UUID
16:34:42 <bcl> so that points to preupgrade, not anaconda/dracut
16:35:01 <adamw> so, that would be preupgrade?
16:35:05 <bcl> yes.
16:35:20 <adamw> okay, so i still have that vm running - i can reboot it and hack a UUID in there
16:35:37 <bcl> NO guarantee that it would actually work, but you could try to edit the cmdline and point it to the same drive as the others.
16:35:49 <wwoods> it's not missing
16:35:51 <wwoods> it's not necessary
16:35:56 <bcl> oh?
16:36:11 <wwoods> "stage2=" is supposed to override repo=, so we'd ignore that
16:36:43 <wwoods> "repo=hd::.." is special preupgrade-only notation that means "use whatever device we're upgrading"
16:36:53 <adamw> okay. and changing it doesn't seem to help.
16:36:54 <wwoods> which anaconda understands
16:37:00 <adamw> next idea? :)
16:37:14 * adamw has the VM running and is taking requests
16:37:28 <wwoods> "no suitable images" is the confusing bit
16:38:02 <wwoods> add "rd.debug" to the boot args so you can examine /run/initramfs/init.log
16:38:16 <wwoods> adamw: does /tmp/ks.cfg.done exist?
16:38:43 <wwoods> uh
16:38:47 <wwoods> upgrade --root-device=UUID=None
16:39:00 <wwoods> that's definitely wrong, but outside the scope of the problem you're seeing
16:39:11 <adamw> there's only export.orig in /tmp
16:40:06 <adamw> we're not hitting another case of the parameter split bug here, are we?
16:40:07 <wwoods> adamw: that's.. confusing. I'm gonna set up my own VM so I can swear^Wplay along
16:40:25 <adamw> wwoods: it'll probably take you a while :/
16:40:36 <adamw> oh hey!
16:40:42 <wwoods> adamw: not really, since I just need a squashfs.img in the right place
16:40:48 <adamw> ah k.
16:40:52 <wwoods> adamw: very likely yes, now that you mention it
16:41:02 <adamw> i could give you root login on my machine, then you could access my VM via virt-manager...=)
16:41:39 <adamw> is the parameter split bug fixable with an updates.img (so i could test)?
16:42:02 <bcl> adamw: is preupgrade beta criteria? Or can we insist they just use anaconda?
16:42:09 <adamw> bcl: we put preupgrade in beta
16:42:20 <adamw> in our young, foolish and optimistic phase, no doubt
16:42:35 <bcl> well, really, we need to test sooner and more often.
16:42:38 <adamw> ironically, it's in beta because it's our 'recommended method' for upgrading a running system (as opposed to yum)
16:42:48 <wwoods> adamw: no. although I keep thinking it might be useful to have a way to update initramfs inside initramfs
16:42:49 <adamw> yet it always seems to turn out that we have yum upgrades basically working before preupgrade does. :/
16:42:54 <bcl> right. and we'd like to make it the only one eventually.
16:42:59 <adamw> yo dawg...
16:43:03 <wwoods> but that requires being able to cleanly restart initramfs /init, which we're still working on
16:43:41 * cpuobsessed points out he noticed it was broke in alpha :D
16:43:50 <adamw> cpuobsessed: oh, we knew
16:43:57 <adamw> cpuobsessed: but it was broken for a completely *different* reason
16:44:08 <cpuobsessed> but broken nonetheless
16:44:09 <adamw> cpuobsessed: that's the problem with preupgrade - we can basically fix one bug in it per candidate
16:44:21 <adamw> so we find one bug, fix it, then with the next compose we find the next bug, fix it...etc etc
16:44:38 <cpuobsessed> trudge trudge trudge
16:45:47 <adamw> well, shit, i'm an idiot. for the free space on upgrade test, i carefully installed a system and tweaked it down to 700MB free disk space. an *f17* system. d'oh.
16:46:11 <rbergeron> doh
16:48:27 <adamw> wwoods: any eta?
16:48:37 <adamw> sorry to take up people's time, but it's pretty important we figure out the preupgrade story
16:49:48 <wwoods> adamw: patch is already sent, reviewed, and applied
16:49:53 <wwoods> oh wait blarg
16:50:26 <adamw> i meant, eta on you reproducing and investigatin'
16:51:48 * rbergeron peeks in
16:53:06 <wwoods> adamw: so here's what you can do - boot with 'rd.debug rd.break' and grep 'stage2=' /run/initramfs/init.log
16:53:42 <adamw> roger
16:54:38 <wwoods> if it's missing the path, it's the splitsep bug
16:54:49 <adamw> there's no /run/initramfs/init.log
16:55:01 <wwoods> ...
16:55:35 <adamw> huh, it's like my rd.debug got dropped.
16:56:38 <bcl> try it w/o rd.break, I had some problems with it truncating logs a while back.
16:56:49 <adamw> okay, worked this time
16:56:52 <adamw> i moved the parameters
16:56:53 <bcl> since you're already getting the shell you shouldn't need the break.
16:57:09 <bcl> oh? that's not good.
16:57:17 <adamw> um, i get three results
16:57:49 <adamw> 27parse-anaconda-repo.sh@8(source): getarg stage2=inst.stage2=
16:58:09 <adamw> 27parse-anaconda-repo.sh@8(source): stage2=hd:UUID=blahblah:/upgrades/squashfs.img
16:58:21 <adamw> 27parse-anaconda-repo.sh@12(source): arg=stage2
16:58:35 <adamw> so that's a fail?
17:00:09 <adamw> bcl: it's possible i stuck the parameters on the initrd line first try. i am pretty tired.
17:00:12 <cpuobsessed> adamw: you forgot to rub your tummy and pat your head
17:00:13 <adamw> wwoods: ^^^
17:00:34 <adamw> cpuobsessed: sorry, one hand is typing and the other is holding the breakfast whiskey.
17:00:34 <rbergeron> buddha!
17:00:51 <adamw> in case anyone's wondering, this is the last proposed blocker
17:00:56 <adamw> so we're not blocking discussion of any other bug
17:01:36 <adamw> bcl: should i try the root= style workaround that we've used for the other cases of the getarg breakage?
17:01:51 <bcl> may as well try.
17:02:08 <bcl> though that really isn't a good solution.
17:02:13 <adamw> root=live:hd or what?
17:02:27 <bcl> yeah, the live needs to be there.
17:02:31 <wwoods> no hd
17:02:37 <wwoods> hd: is anaconda-specific
17:02:42 <wwoods> only works with things that start with inst.
17:03:18 <adamw> ...i'm getting mixed messages =)
17:03:39 <wwoods> in fact I might drop it, since it's obvious you're talking about a disk device when you say "/dev/sdb" or "LABEL=funk" or whatever
17:03:49 <wwoods> root=live:DEVICE
17:03:58 <adamw> so, live:UUID=blah
17:04:03 <wwoods> yes. and live: doesn't take paths
17:04:06 <adamw> k
17:04:22 <adamw> i'm cloning the machine first, so i have a 'clean' copy of the problem state.
17:04:35 <adamw> even if this test explodes things.
17:04:41 <wwoods> so unless the image is at /LiveOS/squashfs.img it might not work
17:05:20 <adamw> wwoods: tflink said his workaround for 810005 is "root=live:http://path/to/repo/LiveOS/squashfs.img" ?
17:05:31 <wwoods> that'd work
17:05:51 <wwoods> "root=live:DEVICE rd.live.dir=upgrades" might work too
17:05:58 <wwoods> then you wouldn't need to http-fetch it
17:06:21 <adamw> well i'm just sayin, he passed a path.
17:06:33 <wwoods> that's a URL, not a path
17:06:44 <adamw> well played, sir
17:06:48 <adamw> you win THIS round
17:07:30 <wwoods> live:{http,https,ftp,nfs} get handled differently
17:07:33 <adamw> might i invite you to a round of golf in my space lair?
17:07:55 <adamw> still cloning...
17:07:58 <wwoods> the one with all the death lasers? sounds very un-traplike!
17:08:58 <adamw> oh, yes. it has a big sign that says DEFINITELY NOT A TRAP, and you can't say fairer than that.
17:09:16 <wwoods> is everything in the RC tagged and live in the repos, or do I need to grab some updates to make images?
17:09:40 <adamw> hey, that second one seems to get further.
17:09:48 <tflink> wwoods: everything has been pushed to stable AFAIK, may not be out to the mirrors yet, though
17:09:56 <adamw> now it's trying to get the kickstart from /dev/vda2:/upgrade/ks.cfg...and goodnight.
17:10:19 <adamw> i'm not sure _everything_ is pushed, but everything image-compose-relevant ought to be
17:10:31 <adamw> we pushed anaconda and lorax and forced a tree compose
17:10:57 <adamw> so, yeah: that gets to 'fetching kickstart from /dev/vda2:/upgrade/ks.cfg' and then fails, 'Unable to process initqueue' and shell.
17:11:56 <wwoods> adamw: any /tmp/ks.cfg* exist?
17:12:11 <adamw> yeah, /tmp/ks.cfg and /tmp/ks.cfg.done and /tmp/ks.info .
17:12:17 <adamw> so i guess it actually fails after kickstart.
17:12:29 <wwoods> okay then
17:13:06 <wwoods> try: cd /lib/dracut/hooks/initqueue/finished
17:13:17 <wwoods> . /lib/dracut-lib.sh
17:13:38 <wwoods> for f in *; do . $f; echo $f: $?; done
17:14:12 <wwoods> one or more of those scripts is returning non-zero - figure out which
17:14:57 <adamw> it doesn't like the filenames...
17:15:11 <adamw> the two 'devexists' ones with lots of slashes in them just give 'file not found'
17:16:08 <wwoods> okay. what's in those scripts?
17:16:48 <wwoods> also, "devexists" should be like, devexists-\xXXdev\xXXroot or something. if it's just a bunch of slashes, maybe the device name got lost
17:17:10 <adamw> yeah, it's like that
17:17:32 <adamw> all three seem to return 0, but i fiddled a bit...
17:17:36 <adamw> lemme reboot and retry
17:18:13 <adamw> actually, the x2fdev one gave 1 on that last try
17:18:25 <adamw> yeah, seems to be giving 1 every try now...
17:18:27 <wwoods> what are the contents of those files?
17:18:43 <adamw> the one that fails just has one line
17:18:47 <wwoods> should be something like "[ -e "DEVICE" ] || warn ..
17:18:52 <adamw> [ -e "/dev/mapper/live-rw" ]
17:19:10 <wwoods> and /dev/mapper/live-rw doesn't exist, I take it
17:19:21 <adamw> you would be correct, sir
17:19:30 <wwoods> so probably when you dropped into the shell it said  "/dev/mapper/live-rw does not exist"
17:19:39 <adamw> indeed it did, missed that.
17:19:40 <adamw> sorry.
17:21:21 <adamw> so, big picture, so far we're sure the parameter interpretation bug is the first thing breaking it, but we could theoretically patch preupgrade to work around it, and now we're hitting a different bug?
17:22:08 <wwoods> seems that way
17:22:20 * nirik sighs. a nasty Matryoshka of bugs.
17:23:41 <adamw> any idea what this second bug is?
17:23:55 <adamw> nirik: ooh, you should change preupgrade's Summary line to that. :P
17:23:56 * Kevin_Kofler thinks we should make direct yum the one supported upgrade method and drop all the others.
17:24:00 <Kevin_Kofler> Really it's what works best.
17:24:23 <Kevin_Kofler> We've now been discussing how to fix preupgrade for over an hour.
17:24:35 <Kevin_Kofler> And the DVD upgrades are known for their upgrade path issues.
17:25:01 <Kevin_Kofler> Why do we keep insisting that the best working upgrade method is "unsupported"?
17:25:14 <wwoods> because it has its own host of problems which you are conveniently ignoring.
17:25:29 <wwoods> but hey, support whichever you want
17:25:46 <wwoods> they've all got their own problems
17:26:13 * adamw would like to try and get to the smallest matroshka before we make a final vote/determination
17:26:25 <adamw> so we have some degree of confidence whether we can make preupgrade work with the rc3 images or not
17:28:30 <rbergeron> yesplz
17:31:17 <wwoods> adamw: does 'anaconda_live_root_dir' show up in init.log? if so, what are the args?
17:32:03 <adamw> whoops, didn't boot with rd.debug, just a sec
17:34:14 <adamw> wwoods: nope, no result from grep.
17:35:27 <wwoods> adamw: can you copy the init.log out of there and fpaste it or something?
17:35:37 <wwoods> mount the working disk and copy it in there or something
17:35:47 <adamw> i was already working on it =)
17:35:59 <wwoods> aces
17:37:09 <adamw> well shit. /dev/vda2 is already mounted ro as the install source and i can't mount the other disk as it's an LVM
17:37:09 <adamw> gr
17:38:51 <adamw> okay, a usb stick works. yay dracut.
17:39:29 <adamw> wwoods: http://fpaste.org/peao/
17:41:45 <wwoods> oh. this is when you're doing root=live:UUID...
17:42:25 <wwoods> well, this needs poking at too
17:43:44 <adamw> sorry, you mean it might just be caused by the 'workaround'?
17:45:38 <wwoods> adamw: is the file in /upgrades/squashfs.img on /dev/vda2?
17:46:02 <adamw> /dev/mapper/live-rw ?
17:46:09 <wwoods> no
17:46:30 <wwoods> is /upgrades/squashfs.img the file on /dev/vda2
17:46:34 <wwoods> does that file exist on that partition
17:46:38 <adamw> oh, i see.
17:46:55 <adamw> it's upgrade/squashfs.img
17:46:58 <adamw> not upgrades/squashfs.img
17:47:00 <adamw> but it's there
17:47:06 <wwoods> there's your problem then
17:47:09 <adamw> ah.
17:47:21 <wwoods> rd.live.dir=upgrade, not upgrades
17:47:28 <adamw> d'oh =)
17:47:31 <adamw> shoulda caught that.
17:47:33 <adamw> retrying
17:48:55 <adamw> well, we have stage2
17:48:58 <adamw> but:
17:49:47 <adamw> An error occurred mounting device /dev/vda2 as /boot: mount failed: (32, 'mount: /dev/vda2 is already mounted or /mnt/sys/image/boot busy\n   /dev/vda2 is already mounted on /run/initramfs/live'). This is a fatal error and the install cannot continue.
17:49:52 <adamw> sic
17:50:03 <adamw> it does have that weird cut-off message, that's not a typo
17:50:56 <wwoods> that's probably because of the upgrade line being wrong
17:51:11 <adamw> oh, you mentioned that earlier
17:51:20 <wwoods> change ks.cfg so the upgrade --root-device=DEVICE is correct
17:51:23 <wwoods> and see if that works
17:51:33 <wwoods> that's a preupgrade problem, not an anaconda one
17:51:48 <wwoods> but also: if you can get an init.log from trying to boot with the default args, that'd be helpful
17:51:56 <adamw> so it should be device=UUID=thesameUUIDasonthekernelcmdline ?
17:52:15 <adamw> uh, so an init.log from this state it's in now?
17:52:18 <wwoods> yeah. or whatever device you're trying to upgrade
17:53:14 <wwoods> no, you changed the boot args from what anaconda set - I'm looking for init.log after booting with "repo=hd::... stage2=hd:UUID:...:" etc.
17:53:30 <adamw> oh, without the root= workaround?
17:53:33 <wwoods> yes
17:53:42 <adamw> oh, okay. i thought we already knew the fix for that, but sure
17:53:47 <wwoods> no, we don't
17:53:51 <adamw> ok
17:53:56 <adamw> i thought it was the getparam bug
17:54:18 <wwoods> I don't know. that's why I want th elog.
17:54:34 <wwoods> what I got from you earlier was inconclusive
17:56:24 <adamw> ah ok
17:56:40 <adamw> wwoods: http://fpaste.org/Ayn6/
18:00:29 <adamw> wwoods: so just to double-check, for my Tottering Pile Of Workarounds path, we're up to 'upgrade --root-device=UUID=f9f8667a-5806-41be-a96d-93bbac0334e4' in the ks.cfg, right?
18:00:34 <adamw> that's the correct syntax?
18:00:50 <wwoods> should be
18:01:01 <adamw> k
18:01:52 <wwoods> adamw: root=anaconda-disk:UUID=f9...4e4
18:02:00 <wwoods> no path. ayup, that's the splitsep error
18:02:07 <adamw> ok
18:02:43 <adamw> still get the vda2 error after correcting the upgrade line
18:03:20 <adamw> i can see the 'corrected' line in /run/install/ks.cfg so it looks like it's being picked up
18:04:52 <rbergeron> tottering!
18:05:01 <adamw> i certainly am
18:05:47 <adamw> rbergeron: so, status is: we know what the first bug is, it's 810005, same parameter splitting bug we discussed yesterday
18:06:04 <adamw> once i work around that we can make it to stage2, which promptly explodes trying to mount something wot is already mounted
18:06:22 <adamw> wwoods: /dev/vda2 is mounted at /run/initramfs/live , of course.
18:07:02 <adamw> and anaconda is trying to mount it at /mnt/sysimage/boot .
18:07:15 <wwoods> well yeah, but anaconda is supposed to recognize this situation and not attempt to mount it again
18:07:24 <wwoods> is 'preupgrade' in the boot args?
18:07:37 <adamw> yup.
18:07:56 <adamw> BOOT_IMAGE=/upgrade/vmlinuz preupgrade repo=.......
18:08:04 <adamw> (that's cat /proc/cmdline)
18:10:15 <tflink> btw, let me know if you want something tried again - I have a resettable system ready for (pre)upgrade
18:12:08 <adamw> wwoods: any logs that would help?
18:17:28 * adamw going for a quick shower, brb
18:17:36 <wwoods> adamw: the traceback from that error would be helpful
18:18:14 <adamw> wwoods: it's an anaconda error dialog, it doesn't hit the traceback handler and there's no /tmp/anaconda-tb-*
18:18:20 <adamw> wwoods: is there somewhere else i might find it?
18:18:46 <wwoods> not really. can you tell me what the last step mentioned in anaconda.log is?
18:19:03 <adamw> movoing (1) to step upgrademount
18:19:09 <adamw> upgrademount is a direct step
18:19:30 <wwoods> 's what I figured
18:19:33 <adamw> then it mounts lv_root as /mnt/sysimage, and next tries to mount /dev/vda2 as /mnt/sysimage/boot
18:19:59 <adamw> program.log shows a resize2fs as the last command before the failed mount
18:24:42 <adamw> wwoods: where's the code that's supposed to recognize the preupgrade cmdline parameter? I don't see it
18:26:38 <adamw> feh. i'll go take that shower. =)
18:27:00 <wwoods> adamw: storage.log might be helpful, actually. but yeah I'm still tracing stuff here.
18:29:02 <adamw> i'll get it in a sec
18:36:29 <adamw> wwoods: http://fpaste.org/YVHo
18:36:40 <adamw> could be the use of root= is breaking this, i guess? i'll test with a network sourced root=
18:39:25 <wwoods> that's a pretty good theory actually
18:40:57 <adamw> yeah, with network sourced root it's got further
18:40:58 <rbergeron> theories are good, good theories aee awesome
18:41:06 <rbergeron> :)
18:41:08 <adamw> it's doing packages now
18:41:43 <adamw> kinda hard to tell if it's using the preupgrade-downloaded packages, but eh
18:41:45 <rbergeron> at some point we need to call it one way or the other so people know what to do this weekend :/
18:41:51 <wwoods> well, that's not really a solution though, since you're just dodging the mount
18:41:52 <adamw> rbergeron: we're still diagnosing
18:41:58 <adamw> wwoods: sure, just as a triage
18:42:03 <rbergeron> not pressuring just... saying so nobody can say it wasnt said
18:42:11 <rbergeron> adamw: i know :)
18:42:42 <adamw> well...so right now you actually can do a preupgrade with at least one hacky workaround (possibly two, not sure if the 'fix' to the kickstart is required)
18:43:00 <adamw> and the hacky workaround requires network access for the start of install at least
18:43:48 <adamw> well, maybe a bit early to declare that. it didn't get to bootloader install yet.
18:53:11 * adamw taps fingers
19:01:46 <adamw> wwoods: any new thoughts your end? does it sound like we won't be able to get a working local workaround for rc3?
19:01:53 <adamw> or is there some way we could finesse it?
19:11:13 <adamw> my upgrade is cleanup now
19:11:37 <wwoods> I don't know. I'm trying to figure out how we used to avoid the "/boot is already mounted" thing
19:12:53 <adamw> i can run a 15->16 preupgrade and try to watch what happens, but it might take a while :/
19:14:15 <wwoods> yeah I'm reading the source and trying to bug dlehman about it
19:15:08 * adamw starts cloning his 15 master
19:20:08 <adamw> okay, well, the upgrade with the remote root workaround succeeds
19:20:10 <adamw> so, yay, i guess
19:20:26 <adamw> i should test whether the kickstart fix was needed, though
19:21:16 <adamw> wwoods: are you likely to need anything from the machine that succeeded, with the ks fix and remote root workaround? or can i blow it away now?
19:30:40 <adamw> so, just passing root=live:http://path/to/squashfs.img seems to be good enough as a workaround - the 'error' in the kickstart doesn't seem to break it
19:31:11 <adamw> but that does require a working network connection.
19:31:54 <adamw> wwoods: i'm kicking off an f15 preupgrade now
19:39:02 <wwoods> isn't f15 where we first combined the images, so we had a big install.img?
19:39:40 <adamw> i really don't remember that far back
19:40:00 <adamw> so...you think we have a good enough understanding to make a call now?
19:40:35 * adamw updates bug
19:40:39 <adamw> ping rbergeron
19:40:41 <adamw> ping nirik
19:40:48 <nirik> yes?
19:40:53 * nirik reads up
19:41:14 <wwoods> ayup, F15 is where we combined the images
19:41:45 <rbergeron> hi
19:41:48 <nirik> so, work around, but it needs network... but really you should have net anyhow doing preupgrade, right?
19:41:48 <wwoods> so we gotta go back to F14 to figure this out, and yeah, we used to copy install.img into /tmp and unmount
19:41:51 <wwoods> which is why it used to work
19:41:56 * rbergeron looks for spot
19:42:04 <wwoods> nirik: not during the upgrade, usually, but sure
19:42:13 <nirik> yeah
19:42:20 <wwoods> it's not really workable
19:42:34 <wwoods> you'd need to bring up whatever crazy-ass network the person might be on inside initramfs
19:42:43 <wwoods> which means you have to plug into a wired network to run preupgrade
19:42:52 <wwoods> 'cuz we don't do wireless
19:42:52 <wwoods> etc
19:44:04 <nirik> ;(
19:44:22 * rbergeron reiterates the pouty face
19:44:47 <adamw> oh yeah that's a point
19:44:53 <adamw> you need a *wired* connection
19:45:37 <adamw> #agreed 810136 turns out to be a dupe of 810005, moving discussion
19:45:42 <adamw> #topic https://bugzilla.redhat.com/show_bug.cgi?id=810005
19:46:02 <adamw> okay, so i updated #810005 with a summary of the current state as I grok it: https://bugzilla.redhat.com/show_bug.cgi?id=810005#c6
19:46:20 <adamw> now we need to make a call. please god let's just make a call and i can get out of this room.
19:46:39 <tflink> I don't know, we could debate it a bit longer ...
19:46:43 <adamw> =)
19:47:11 <adamw> what's everyone's feeling on the acceptable-ness of the workaround?
19:47:27 <adamw> it does mean you can't really do preupgrade via wireless :/
19:47:41 <spot> yuck.
19:48:04 <nirik> do we have preupgrade critera? or just 'should work by beta' ?
19:48:14 <wwoods> or VPN, or with proxies, or basically anything except "wired network with DHCP"
19:48:15 <tflink> nirik: it's specifically required
19:48:32 <adamw> just in case everyone's thinking we should fudge it to make the whole validation sprint worth it, i'll stick my neck out and say +1 blocker. really, strictly going by the protocol, this workaround isn't good enough to consider the criterion not breached, imo.
19:48:44 <spot> +1 blocker from me, as much as i hate it
19:48:48 * nirik nods sadly. slip and +1 blocker here too.
19:48:52 <tflink> "The installer must be able to successfully complete an upgrade installation from a clean, fully updated default installation (from any official install medium) of the previous stable Fedora release, either via preupgrade or by booting to the installer manually. The upgraded system must meet all release criteria"
19:48:57 <adamw> yup.
19:49:02 <tflink> yeah, I'm +1 blocker
19:49:24 * nirik plays the sad trombone.
19:49:37 <adamw> spot: it's a pita to slip, but on the plus side, it means we can fix some nasty nths. we're not really at a point of diminishing returns.
19:49:51 <rbergeron> nasty easyfix nths?
19:49:52 <spot> adamw: thats very positive of you. ;)
19:50:10 <adamw> rbergeron: the other paths that hit 810005
19:50:11 <rbergeron> it's because his breakfast whiskey turned into lunch whiskey
19:50:14 <adamw> that NetworkManager crash thingy
19:50:35 <adamw> i think i had something else on my list, i'll have to look for it.
19:50:44 <rbergeron> at two weeks i want to be verrry careful to not introduce new awesome
19:50:48 <adamw> rbergeron: point.
19:50:48 <rbergeron> gnome 3.4?
19:50:53 <adamw> 3.4 was in rc3
19:50:58 <adamw> it worked out fine, shockingly enough. no problem.
19:51:03 <rbergeron> ohhhright
19:51:39 <adamw> propose #agreed 810005 is accepted as a blocker per criterion "The installer must be able to successfully complete an upgrade installation from a clean, fully updated default installation (from any official install medium) of the previous stable Fedora release, either via preupgrade or by booting to the installer manually. The upgraded system must meet all release criteria". the known workaround has too many limitations to be considered 'good enough'
19:51:52 <tflink> ack
19:52:06 <rbergeron> uck
19:52:07 <rbergeron> :)
19:52:12 <rbergeron> but ack
19:52:38 <nirik> ugh ack ugh. ;)
19:52:50 <tflink> we sounds like cavepeople
19:53:23 <adamw> #agreed 810005 is accepted as a blocker per criterion "The installer must be able to successfully complete an upgrade installation from a clean, fully updated default installation (from any official install medium) of the previous stable Fedora release, either via preupgrade or by booting to the installer manually. The upgraded system must meet all release criteria". the known workaround has too many limitations to be considered 'good enough'
19:53:33 <adamw> rberge you want to finish up the bureaucracy?
19:53:38 <adamw> we should make an official no-go call
19:53:50 <rbergeron> well
19:54:00 <rbergeron> blockers = no go but :)
19:54:12 <rbergeron> #topic Go or no-go?
19:54:27 <rbergeron> #info we have a blocker.
19:54:31 <nirik> sadly, no go
19:54:35 <adamw> per our policy, QA votes no-go as there is an open unresolved blocker.
19:55:00 <adamw> rbergeron: oh, on the nth front, it'd be nice to fix that kde corruption.
19:55:18 <rbergeron> #info qa votes no-go, ass do various others from engineering, fesco, pm
19:55:26 <rbergeron> adamw: agreed
19:55:32 <tflink> and pxe boot and the missing livecd fixes
19:55:49 <rbergeron> adamw, wwoods: we do have a fix for 810005?
19:55:52 <adamw> tflink: we'll get pxe boot with the parameter fix
19:56:00 <adamw> rbergeron: i believe there's one on the anaconda list, yeah
19:56:05 <adamw> we'll need a fix for teh second bug too though
19:56:07 <rbergeron> or when can we get to rc4, given holiday weekend and etc
19:56:10 <tflink> adamw: yep, but it'll still be fixed where it doesn't work w/ RC3
19:56:30 <adamw> i'm happy to do the qa crap for it over the weekend, holiday or no, if wwoods can get anaconda fixes in
19:56:40 <adamw> dunno where dgilmore stands
19:56:49 <tflink> oh yeah, there's a holiday this weekend
19:57:07 <nirik> dgilmore is out today I know for sure, not sure if he's around the rest of the weekend or not.
19:57:13 <rbergeron> i mean i'll be around all weekend but i dont know about  others
19:57:16 * nirik should be around mostly if he can help with anything.
19:57:29 <adamw> tflink: missing livecd fixes, you mean the efi thing?
19:57:30 <rbergeron> holiday means kids have candy and crap from grandparents and are fully occupied
19:57:34 <tflink> adamw: yeah
19:57:41 <adamw> yeah, that'd be nice.
19:58:23 <rbergeron> that said i'm not entirely helpful beyond making sad faces and meeting notes and delivering sad news
19:58:27 <adamw> #agreed we are no-go on F17 Beta RC3 sadly
19:58:36 <rbergeron> oh, thx
19:58:55 <rbergeron> #action rbergeron to update schedule.... again
19:59:32 <rbergeron> i can send out a slip mail in prob an hr
19:59:47 <rbergeron> blocker meeting tomorrow?
20:00:02 <tflink> yep, I'll send out the announcement as soon as I see your email re: slip
20:00:02 <rbergeron> or monday in qa meeting or ad hoc as we are hailed :)
20:00:09 <rbergeron> okay
20:00:19 <rbergeron> #action rbergero to send slip mail
20:00:22 * tflink has afeeling that we're in for some lively discussion around the kde 16bpp thing
20:00:38 <rbergeron> #action tflink to do blocker meeting mail following slip notice
20:02:47 <rbergeron> any other loose ends?
20:03:09 <rbergeron> #action qa heroes of awesome to get some rest plz
20:03:10 * adamw is going golfing
20:03:11 <rbergeron> :/
20:03:20 <kalev> nirik: if dgilmore is out today, could you do a stable push with the NTH fixes that were included in RC3?
20:03:23 <rbergeron> #action adamw to get a hole in one
20:03:42 <nirik> kalev: if they have enough karma.
20:03:58 <rbergeron> we may want to cherry pick those - adamw, tflink, spot, nirik, thoughts?
20:03:58 <nirik> kalev: I couldn't get it to push the gnome update (I can check with lmacken on why)
20:04:28 <adamw> cherry pick what? the rc3 packages? yeah, we've been doing that for this release, pushing the stuff that was pulled into the TCs/RCs stable
20:04:39 <tflink> I thought that we got most of them, though
20:04:40 <adamw> we were already working on it last night, just had a problem with the gnome update as nirik says
20:04:53 <kalev> nirik: ah, lmacken did some manual hackery on that update, I bet he knows what the issue might be
20:04:56 <nirik> there's a few others that didn't have enough karma.
20:05:01 <rbergeron> okay
20:05:04 <adamw> we'll get those karma'ed and pushed.
20:05:07 <nirik> kalev: it just ignored me trying to push it. ;(
20:05:13 <adamw> and i'll try and pull together an nth list later.
20:05:20 <adamw> things it's worth fixing.
20:05:47 <rbergeron> adamw: thanks, i assume in a ticket nirik can watch for (or dgilmore)
20:05:58 <adamw> well, just via nth process at first.
20:06:10 <rbergeron> okay
20:06:27 <rbergeron> i'll just be quiet and let you guys do your work and golfing :)
20:07:16 * kalev goes to file a new NTH bug.
20:07:35 <nirik> so, shall we close out this meeting?
20:07:38 <nirik> (finally) ;)
20:07:53 <adamw> yup
20:08:46 <adamw> thanks for the work everyone, esp anaconda team
20:08:56 <adamw> #info we will re-convene tomorrow or monday
20:08:59 <adamw> #endmeeting