16:02:29 <adamw> #startmeeting Fedora QA meeting
16:02:29 <zodbot> Meeting started Mon Feb 29 16:02:29 2016 UTC.  The chair is adamw. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:02:29 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:02:29 <zodbot> The meeting name has been set to 'fedora_qa_meeting'
16:02:32 <adamw> #meetingname fedora-qa
16:02:32 <zodbot> The meeting name has been set to 'fedora-qa'
16:02:36 <adamw> #topic Roll call
16:02:42 <adamw> ahoyhoy folks, who's around for QA meety fun?
16:02:51 * handsome_pirate waves
16:03:06 * pschindl is here
16:05:46 * kparal is here
16:06:05 <adamw> #chair kparal pschindl
16:06:05 <zodbot> Current chairs: adamw kparal pschindl
16:06:06 <adamw> anyone else?
16:07:47 * linuxmodder is  here
16:07:50 <adamw> hi hi
16:07:53 <linuxmodder> .fas corey84
16:07:53 <zodbot> linuxmodder: corey84 'Corey Sheldon' <sheldon.corey@gmail.com>
16:08:05 <handsome_pirate> .chair adamw
16:08:05 <zodbot> adamw is seated in a chair with a nice view of a placid lake, unsuspecting that another chair is about to be slammed into them.
16:09:07 <adamw> nice.
16:09:18 <adamw> i only *wish* there was a placid lake here.
16:09:22 <adamw> hell, i'd take Lake Placid.
16:09:49 <adamw> alrighty, let's get rolling, i guess
16:09:53 <adamw> #topic Previous meeting follow-up
16:10:01 <adamw> aka All The Things AdamW Didn't Do Last Week
16:11:01 * pwhalen looks in
16:11:04 <linuxmodder> slacker
16:11:44 <handsome_pirate> pwhalen:  you may want to run away before you get tasked
16:12:07 * adamw plays hold music while he does one of the things
16:13:06 <adamw> #info "tflink to merge D689 already, damnit flink" - as of thirty seconds ago, it's landed (reviewed by tflink and mkrizek)
16:13:12 <pwhalen> heh :)
16:13:22 <adamw> that was the commit that adds non-media blocker support to blockerbugs
16:13:31 <adamw> now we get to see what it breaks!
16:13:46 <handsome_pirate> all the things
16:14:11 <adamw> #info "adamw to complete matrix changes for i686 no longer blocking media" - good grief, i totally forgot about that. got sandbagged by Pungi 4. i'll put it back in the list
16:14:19 <adamw> #action adamw to complete matrix changes for i686 no longer blocking media
16:14:35 <adamw> "kparal to implement changes to the process pages to specify that fixes for non-media blockers must be pushed stable for the appropriate release(s) for QA to vote Go" - kparal?
16:14:50 <kparal> I sent the proposal to test list today
16:15:00 <adamw> rockin'
16:15:08 <linuxmodder> re:pungi4   adamw  can you update me on its status  post mtg
16:15:10 <adamw> #info "kparal to implement changes to the process pages to specify that fixes for non-media blockers must be pushed stable for the appropriate release(s) for QA to vote Go" - kparal sent the proposed changes out to test@ for review today
16:15:21 <adamw> linuxmodder: i can talk your leg off about it! we have blocker review next though
16:15:34 <adamw> in fact there's gonna be some pungi4 stuff in this meeting, whether you like it or not. ;)
16:15:50 <linuxmodder> adamw, i will be in that as much as possible  will likely  be  out for a mtg  during  BR
16:15:58 <adamw> alrighty
16:16:04 <adamw> so i think that's everything from the previous meeting
16:18:13 <adamw> moooving along
16:18:30 * adamw is all over the place this morning
16:18:53 <adamw> #topic Fedora 24 status and compose changes
16:19:07 <adamw> so...executive summary, last week everything changed!
16:19:24 <adamw> would it be useful for me to go through what the hell's going on with F24 at present, as far as I know?
16:20:10 <kparal> is it something not covered in the emails you sent out recently?
16:20:33 <handsome_pirate> adamw:  all the things are broken
16:21:06 <adamw> kparal: well, there might be a few things.
16:21:09 <adamw> i'll try and be brief
16:21:21 <adamw> #info Fedora 24 branched last week.
16:22:34 <adamw> #info At the same time, Rawhide and Fedora 24 nightly composes switched to using Pungi 4: the bits in development/24 and development/rawhide are now produced by Pungi 4, meaning we get a full set of images each day (when image compose works)
16:23:29 <adamw> #info recent 'nightly' composes are now stashed at https://kojipkgs.fedoraproject.org/compose/ rather than https://kojipkgs.fedoraproject.org/mash/ , and the directory names are slightly different.
16:24:02 <adamw> #info Pungi 4 produced images mostly seem to more or less work, except for the DVDs: https://bugzilla.redhat.com/show_bug.cgi?id=1311795
16:24:13 <dgilmore> adamw: that should be fixed
16:24:27 <adamw> #info later in the week, glibc landed a change splitting all locales into 'langpack' subpackages, which has basically broken the world
16:25:03 <adamw> #info anaconda no longer starts up, and generation of live media and cloud images is not happening because both require anaconda to work
16:25:44 <adamw> #info upgraded systems run into lots of bugs because all locales disappear on upgrade: https://bugzilla.redhat.com/show_bug.cgi?id=1312956
16:26:10 <linuxmodder> what was the logic of that  split  that  I never understood from the ml
16:26:13 <kparal> charming changes
16:26:17 <adamw> #info and finally, our old friend the systemd-selinux bug is still lying in wait for anyone who somehow manages to get past all the other bear traps
16:26:38 <adamw> linuxmodder: aiui it's to let us make cloud-y and container-y things smaller.
16:27:11 <linuxmodder> so we now have to hacky  make a subpackage that is  useless   ---lovely
16:27:28 <adamw> #info openQA scheduling has been converted to work with pungi 4 composes, so we get openQA jobs for the new nightlies.
16:27:50 <handsome_pirate> which fail because the nightlies are broken?
16:27:50 <adamw> #info the Rawhide/Branched report emails have changed format and are currently missing dependency check information, releng is aware and working on fixing that.
16:27:59 <adamw> handsome_pirate: exactly, you can go look at the sea of red, though. :)\
16:28:30 * handsome_pirate wonders how he managed a successful upgrade to F24
16:28:54 <adamw> #info the wiki release validation bits are not yet converted to a Pungi 4 world, so we can't really create validation events right now. my priority for this week is to at least get nightly validation events working again.
16:29:21 <adamw> handsome_pirate: upgrades will basically *work*, just you wind up with no locales until you install one manually, which causes various odd effects (e.g. gnome-terminal doesn't work, ssh broke for me)
16:29:42 <handsome_pirate> huh
16:29:51 <handsome_pirate> I didn't manually install any, and everything just worked
16:29:53 <jsmith> Yeah, it's really friendly :-p
16:29:59 <nirik> luckily dnf does still work. ;)
16:30:04 <handsome_pirate> including gnome-terminal and ssh
16:30:15 <adamw> #info I have check-compose code which will work for pungi4 composes, but ideally would like to move the image check stuff to the Rawhide/Branched report emails as it makes more sense there, and just have compose-check be an openQA status email.
16:30:25 <adamw> handsome_pirate: guess you're lucky!
16:30:28 <handsome_pirate> nirik:  actually, when I upgraded, dnf broke in another way.
16:30:34 <adamw> handsome_pirate: when did you upgrade?
16:30:48 <handsome_pirate> adamw:  Friday?
16:30:52 <adamw> huh.
16:31:00 <adamw> oh, maybe the dnf langpack stuff worked for you?
16:31:07 * nirik hasn't seen new dnf breakage, but then I am not on the same release most of you are. ;)
16:31:08 <adamw> can you run 'rpm -qa | grep glibc' and see what you've got?
16:31:09 <handsome_pirate> well, dnf itself broke
16:31:47 <handsome_pirate> adamw:  no glibc-langpack-en
16:32:09 <adamw> so...that's about where F24 is, so far as I know. the compose changeover basically worked but the bits are completely broken right now so we can't do more extensive testing to find gremlins in the livemedia-creator generated lives or anything until the glibc stuff at least is fixed.
16:32:20 <adamw> handsome_pirate: interesting
16:32:24 <handsome_pirate> adamw:  so, things /ought/ to be broken, but aren't?
16:32:42 * handsome_pirate must not be very good at QA as things have a tendency to not break
16:32:54 <handsome_pirate> at least, not in the ways they break for everyone else
16:33:04 * handsome_pirate gets weird breakage that no one else can reproduce
16:34:08 <adamw> welp, it's not super important right now
16:34:46 <handsome_pirate> adamw:  http://paste.fedoraproject.org/331214/45676367
16:34:59 <adamw> so, oh yeah - we're still going to need to have things that look a lot like "TCs" and "RCs" for F24, but pungi4 has a slightly different concept for what it calls 'production' composes, so we'll have to figure out with releng what we want to do there
16:36:11 <linuxmodder> handsome_pirate,  on what  again?
16:36:17 <adamw> how much does anyone care about exactly what the candidate names and wiki page names look like?
16:36:37 <adamw> basically i'm asking do we want to have a big irc and/or list discussion about it, or do you just want me to go away and figure it out with dennis
16:36:43 <tflink> not really, so long as they're consistent and make some sort of sense
16:36:57 <handsome_pirate> linuxmodder:  hrm?
16:37:05 <linuxmodder> the breakage you mentioned
16:37:23 <handsome_pirate> linuxmodder:  ah, dnf was blowing up
16:37:36 <linuxmodder> ah
16:37:45 <handsome_pirate> linuxmodder:  don't recall what I did to fix it.
16:37:51 <handsome_pirate> linuxmodder:  wasn't able to reproduce
16:37:58 <linuxmodder> post mtg  pm me  some  logs or  something I  am great at  repro'ing  weird  shit
16:38:16 <adamw> for the nightly events i'll probably just change the page names slightly to indicate the respin as well as the date, that's quite easy; deciding what we do with milestones is more open ended.
16:38:26 <handsome_pirate> adamw:  only thing, re tcs and rcs, is people are probably expecting tcs and rcs
16:38:29 <adamw> linuxmodder: you definitely seem to have some kind of bug with your spacebar. :)
16:38:47 <adamw> handsome_pirate: yeah, that's true.
16:39:10 <handsome_pirate> adamw, linuxmodder:  https://xkcd.com/1172/
16:39:33 <adamw> so to briefly explain: pungi4 builds in some assumptions based I think on the RHEL process. basically what pungi4 expects us to build is something like "Alpha 1.1", "Alpha 1.2", "Alpha 2.1" etc etc
16:40:09 <linuxmodder> lol handsome_pirate
16:40:11 <adamw> where the .1 / .2 part is a 'respin' and is not entirely part of the "public" version, so really it expects us to build an Alpha 1 then an Alpha 2 then a Beta 1 then a Beta 2 then an RC 1 then an RC 2 then (if I understand correctly) pick an RC to release
16:41:03 <linuxmodder> that makes  decent  sense  logic wise
16:41:05 <adamw> we can either try to sort of map our TC/RC process onto pungi4's expectations (and just build a series of releases that so far as Pungi is concerned are 'Alpha 1', 'Alpha 2', 'Alpha 3' etc and maybe label them differently in emails and directory names), or we can teach pungi about our process.
16:41:23 <adamw> linuxmodder: it's a perfectly OK process, but as handsome_pirate says it's not what fedora people expect
16:41:39 <adamw> we've never had numbered milestone releases, we've just had 'the' Alpha and 'the' Beta
16:41:42 <linuxmodder> we are  great at  adapting  tho
16:43:19 <adamw> we could also basically treat the respin as the TC/RC indicator, so we just build Alpha 1.1 then Alpha 1.2 then Alpha 1.3 and we never go to Alpha 2.anything, we just keep doing Alpha 1 'respins', then ship the last one we do and call it Alpha
16:43:39 <adamw> i think that's kinda the easiest way to translate between the systems.
16:43:53 <kparal> or Alpha 1.0, Alpha 2.0, etc?
16:44:23 <handsome_pirate> adamw:  One mystery solved; turns out I upgraded before the langpack split had landed, hence my lack of issues
16:44:28 <adamw> kparal: yeah, we can do it that way too, but i think the 'respin' concept maps a bit better to TC/RC than the 'milestone number' concept. i dunno for sure though.
16:44:49 <adamw> handsome_pirate: well partly, but you've updated since then, i think.
16:44:56 <handsome_pirate> hrm, true
16:44:59 * handsome_pirate doesn't know
16:45:09 <handsome_pirate> either way, I'm manually installing the langpack
16:45:10 <linuxmodder> Alpha 1.0-1.n   Alpha 2.x (beta )   Beta being  the  RC ?
16:45:24 <adamw> linuxmodder: uh, that sounds awful? :)
16:45:34 <adamw> linuxmodder: i was just using Alpha as an example
16:45:40 <handsome_pirate> adamw:  maybe, the 1.x respins are tcs, and 2.x respins are rcs ?
16:46:26 <adamw> so in my idea we'd have Alpha 1.1 (== TC1), Alpha 1.2 (== TC2), pretend that freeze point happens next so Alpha 1.3 (== RC1), Alpha 1.4 (== RC2), then we pass all tests so we ship Alpha 1.4 and call it 'Alpha'...then for Beta we'd start at Beta 1.1 and follow the same process.
16:46:45 <adamw> handsome_pirate: yeah, that's another option if we want to somehow indicate TCness vs. RCness.
16:46:54 <adamw> i'm not sure whether we really need to do that for any reason
16:47:50 <adamw> i.e. whether we actually have any practical use for the TC vs. RC designation
16:48:12 <handsome_pirate> true
16:50:04 <adamw> so anyone have other thoughts or strong preferences about what we ought to do?
16:50:18 <adamw> dgilmore: anything on your end?
16:51:16 <dgilmore> adamw: sorry in another meeting
16:51:45 <handsome_pirate> adamw:  Do we want to keep the concept of TCness versus RCness?
16:51:55 <adamw> handsome_pirate: that's kinda my question atm
16:51:58 * handsome_pirate can see it both ways
16:52:06 <adamw> off the top of my head i don't *think* we have any real need to indicate it
16:52:18 <handsome_pirate> on the one hand, all our process does center on it
16:52:27 <handsome_pirate> on the other, we don't really need it.
16:52:44 <adamw> handsome_pirate: actually i don't think it'd be very difficult to twiddle things to drop the TC vs. RC concept
16:52:52 <adamw> doing numbered milestone releases for e.g. would be a *much* more disruptive change
16:53:26 <handsome_pirate> hrm
16:53:46 <handsome_pirate> maybe drop tc/rc this round, do numbered milestones for f25?
16:53:55 <adamw> so how bout this: for now i'll talk to dgilmore about it when he has a couple of cycles, and send out a proposal to the ML, maybe with a couple of options
16:54:04 <adamw> handsome_pirate: i don't think I want to do numbered milestones, personally, that was just an example.
16:54:21 <adamw> handsome_pirate: if we're gonna get all long-term changey, the long-term change i would like to see is *no* milestone releases, but that's a whole other can of worms.
16:55:01 <handsome_pirate> oy
16:55:32 <adamw> #action adamw to mail out a proposal with a few options for adjusting the release validation process to pungi4's expectations
16:56:24 <adamw> alrighty, i wanted to check in on test days but we're kinda at time :/
16:56:48 <adamw> does anyone have a few cycles to check in on the test day tickets and make sure we're not ignoring anyone?
16:56:58 <adamw> if not i'll throw it on my teetering todo pile
16:57:29 <handsome_pirate> adamw:  I can look at it tomorrow
16:57:33 <adamw> thanks
16:57:48 <adamw> #action handsome_pirate to check in on the Test Day trac tickets and make sure we're keeping up our end on all proposals
16:58:04 <adamw> i believe from the qa-devel meeting earlier we have some folks working to get the testdays webapp working again
16:58:17 <adamw> so we have blocker review in ~2 mins in #fedora-blocker-review
16:58:37 <adamw> i'll leave roll call open for a few minutes to give folks time to grab coffee and answer the call of the wild ;)
16:58:51 <adamw> anything wildly urgent for open floor?
16:59:14 * adamw sets the fuse
16:59:57 <adamw> thanks for coming everyone!
17:00:01 <adamw> #endmeeting