16:00:20 <adamw> #startmeeting Fedora QA meeting
16:00:20 <zodbot> Meeting started Mon Feb  2 16:00:20 2015 UTC.  The chair is adamw. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:20 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:00:23 <adamw> #meetingname fedora-qa
16:00:23 <zodbot> The meeting name has been set to 'fedora-qa'
16:00:27 <adamw> #topic Roll call
16:00:27 * pschindl is here
16:00:28 <adamw> ahoyhoy
16:00:44 <pschindl> adamw: good morning
16:00:52 * mkrizek is here
16:00:53 * kparal is here
16:00:53 * roshi is here
16:00:57 * tflink is here
16:01:28 * satellit listening
16:01:46 <adamw> #chair satellit kparal
16:01:46 <zodbot> Current chairs: adamw kparal satellit
16:02:59 * smccann is here
16:04:15 * adamw was expecting a flaming-torch-and-pitchfork turnout today
16:04:40 <adamw> :P
16:04:47 <roshi> how man more people are needed for that?
16:04:52 * tflink knew he forgot something
16:05:05 * roshi is used to things like, dozen, couple, few...
16:05:05 <tflink> I can make it to home depot and back in 20 minutes, though
16:05:18 <adamw> well, not more people, just more flaming torches, pitchforks, and effigies of anaconda developers
16:05:57 <roshi> I have a shovel and a turkey frier, can that work as a standing for my torch and pitchfork?
16:06:09 <roshi> s/standing/stand in/
16:06:18 <adamw> it's the thought that counts
16:06:21 <adamw> allllrighty
16:06:25 <adamw> #topic Previous meeting follow-up
16:06:52 <adamw> "kparal to check in on Preupgrade Assistant feature, what testing is desired and how heavily it will be promoted" - kparal?
16:06:53 * tflink duct tapes his camp stove to the end of a shovel
16:07:16 <kparal> yeah, so I talked to... remembering name...
16:07:33 <kparal> Petr Hracek
16:08:03 <kparal> he would love to have this tool mention in official upgrade guidelines
16:08:04 * danofsatx is hear
16:08:18 <adamw> hi dan
16:08:19 <kparal> so if he succeeds, it will affect us
16:08:42 <adamw> have you asked fesco about their take on it?
16:09:33 <kparal> no I didn't, should I? I guess he should initiative such a ticket, it he wants to push it that way
16:10:38 <adamw> just figured that sort of thing should be reviewed as part of the Change...
16:10:47 <kparal> anyway, if I understood it correctly, it should be just a tool which collects some information about your current system and prints various helpful information about what's going to change in the upgraded system and how you'll be affected. so it should affect the system anyhow, change anything
16:10:55 <kparal> assuming I understood it correctly
16:11:11 <kparal> also it will need some content from "providers", which should be major system parts maintainers
16:11:18 <kparal> like systemd I assume
16:11:18 <adamw> i see
16:11:26 <kparal> he said there's no content there yet
16:11:54 <adamw> so, we should check it out, but it's not likely to actively break anything...and i guess worst case we can always just drop it from the docs/publicity
16:11:58 <kparal> so this tool should not add significant QA effort, I assume. it's just a helper
16:12:24 <adamw> thanks for that
16:12:28 <kparal> documentation helper
16:12:33 <kparal> sure
16:12:56 <adamw> #info "kparal to check in on Preupgrade Assistant feature, what testing is desired and how heavily it will be promoted" - the Change owner wants to tool to be promoted in official docs, but its impact is low (just an information provider) - we should check in on it later in the cycle
16:13:35 <adamw> "roshi and corey84 to try and work up some methodical testing plans for dnf" - roshi? nice small job like this should be done by now, right? :)
16:13:41 <roshi> haha
16:14:01 <roshi> oh yeah, and all the testing is done - I declare things Notionally Complete :p
16:14:44 <adamw> awesome, activate Project Colada stage 2
16:15:08 <roshi> I spent last week getting to know all the points yum is used throughout the whole project, narrowed it down to what we as QA will want to actually test
16:16:05 <adamw> cool
16:16:15 <roshi> the idea is to have a Project Wide overview, just so people from other teams can take a look and see if things are covered
16:16:20 <roshi> it's a far from trivial change
16:16:21 <adamw> are you writing any kind of notes as you go along that could go on a wiki page or anything?
16:16:54 <roshi> for us, ABRT, FedUp and packageKit/Apper will be the big things
16:17:20 <roshi> yeah, I have notes, but it's a scattered spreadsheet at the moment - not for human consumption
16:17:29 <roshi> I'll try to get something written up this week
16:18:04 <adamw> don't spend too much time on that, i just figured if they were in shareable form anyway it'd be nice
16:18:08 <adamw> but most important thing is to get the job done...
16:18:27 <roshi> I thought getting something written up was "done" ;P
16:18:43 <adamw> #info "roshi and corey84 to try and work up some methodical testing plans for dnf" - this is a long-term job but so far roshi has been identifying areas where yum is currently used and trying to stake out a test scope
16:20:43 <adamw> #info "adamw to revise the package set criterion proposal again and move forward with multiboot proposal" - I went ahead and put the package set proposal in place last week, multiboot is still pending but pjones signed off on the tl;dr summary in that last email i sent
16:20:52 <adamw> we have a topic for this later too
16:21:28 <adamw> #info "adamw to document 'efibootmgr -n' workaround for failure to boot windows from grub with SB enabled (#1170245)" - did that, https://fedoraproject.org/wiki/Common_F21_bugs#grub-secure-boot
16:23:48 <adamw> #topic Anaconda "test day" planning
16:23:56 * amita joining late
16:24:00 <adamw> hi amita
16:24:15 <amita> do you need article for Anaconda "test day"
16:24:19 <adamw> so, dcantrell (the anaconda team's manager) asked me last week about putting together a 'test day' relating to anaconda CI
16:24:28 <adamw> amita: probably not, but wait up while i explain :)
16:24:33 <amita> sure
16:24:40 * danofsatx thought anaconda was a bad word now
16:24:58 <adamw> this is what they asked for:
16:25:09 <adamw> "I'd like to see us work through as many of the test cases you guys use for Fedora QA as well as our own and:"
16:25:16 <adamw> "* Turn them in to Jenkins test cases we can use in our CI environment"
16:25:24 <adamw> "* File as many upstream DNF bugs as possible and tag with 'anaconda' to help them prioritize the work (requested by DNF)"
16:25:32 <adamw> "* Try to get a better handle on the general usability of DNF as the installer backend in rawhide at the moment."
16:25:57 <tflink> isn't their CI environment non-public?
16:25:57 <adamw> i'm not totally sure how points 2 and 3 relate to point 1 there, but in any case it seems like they'll want fairly experienced release testing folks
16:26:13 <adamw> let me see if we can grab someone from #anaconda to chat
16:26:56 <adamw> on condition that no-one yells at them about passwords. :P
16:27:24 <adamw> hi, dacntrell
16:27:49 <adamw> dcantrell: so far i just explained what you asked for (i pasted the bullets from your first email), and tflink asked "isn't their CI environment non-public?"
16:28:20 <dcantrell> yes, our CI environment is in the installer lab in Westford
16:28:33 <dcantrell> but that really shouldn't affect a test day effort
16:28:45 <kparal> what is the CI capable of, and how can we help to create those new test cases?
16:28:56 <dcantrell> what I want from the outcome of the test day is an enumerated set of tests which we can put in to the CI system for continued use
16:29:27 <dcantrell> kparal: right now it's most useful for automated install tests where we can define the test using kickstart
16:29:51 <adamw> so basically you're looking for kickstarts that cover specific validation testing cases, and expected outcomes?
16:29:53 <dcantrell> kparal: what I want from Fedora QA is what sort of tests you guys run through during testing.  perhaps some of those can be defined as tests for our CI system
16:30:00 <dcantrell> adamw: right
16:30:03 <adamw> hi bcl
16:30:10 <dcantrell> that's one part of what we're after
16:30:19 <kparal> sure, we can do that. how do we know what you have already?
16:30:20 <dcantrell> I would say the more important aspect of a test day right now is to hammer the DNF backend
16:30:36 <adamw> right, i wasn't totally sure how those two bits of the scope added up
16:31:12 <dcantrell> adamw: so think of the test day request more of "we want to hammer DNF and make sure it works, oh and a nice side effect of this is maybe we can collect some test cases to add to the CI environment"
16:31:22 <adamw> when you say 'hammer the DNF back end' you want us to all get together and just do exploratory manual testing on it, messing around with repos and package sets and stuff? or were you thinking of coming up with CI tests for that during the event too?
16:31:22 <nirik> is there any further thoughts/decision on python3 anaconda/dnf?
16:31:38 <kparal> is dnf enabled by default or do we need to force it somehow?
16:31:44 <dcantrell> kparal: I can collect what we have right now and post it somewhere, but there's not really much.  that's why I'm wanting to get a list or description of what Fedora QA tests
16:31:58 <dcantrell> kparal: DNF should be enabled by default right now, bcl can verify for me
16:32:09 <adamw> kparal: since ~20141210 it's in use by default
16:32:10 <bcl> yep, it has been on for a couple months now
16:32:25 <dcantrell> nirik: the main reason for this test day coordination request is so we can work through dnf and python3 bugs in anaconda
16:32:43 <adamw> dcantrell: basically our entire set of planned tests is https://fedoraproject.org/wiki/Test_Results:Current_Installation_Test .
16:32:46 <roshi> is there a specific rawhide build you want to use for testing, or are we going to need to get a special image for the testday?
16:32:47 <dcantrell> nirik: we're wanting Fedora QA's help before we hit the alpha freeze deadline
16:32:51 <dcantrell> adamw: thanks
16:32:56 <nirik> dcantrell: ok, so python3 will be landed this cycle?
16:33:06 <adamw> dcantrell: or https://fedoraproject.org/wiki/Template:Installation_test_matrix , same list really.
16:33:07 <nirik> or still TBD
16:33:31 <dcantrell> nirik: that will be determined based on the outcome of this coordinated testing
16:33:32 <kparal> dcantrell: do you have some specific date in mind for this test day?
16:33:54 <adamw> to me it seems like the earlier the better
16:33:55 <nirik> ok, so the testing will take place with python3 enabled anaconda / dnf then I assume?
16:33:55 <dcantrell> kparal: as soon as possible
16:34:08 * roshi can help figure out some of the details and help get the wiki pages and whatnot around
16:34:18 * nirik doesn't think it should land this cycle, but perhaps it will all go smoothly. ;)
16:34:47 <roshi> in an ideal world, we'd only change one thing at a time while testing :p
16:35:05 <bcl> nirik: we don't have a py3 anaconda/blivet yet.
16:35:06 <kparal> feb 5 - 10 is not optimal, conferences
16:35:15 <bcl> So I'm not sure how that'll work.
16:35:30 <roshi> Feb 12, perhaps?
16:35:43 <adamw> bcl: well, it was dcantrell who just said you wanted to "work through python3 bugs in anaconda", which seems difficult without a python3 build of anaconda :P
16:35:48 <nirik> yeah. also dnf-3 hasn't really gotten much testing... which makes all the years of testing dnf kinda not as useful
16:36:02 <dcantrell> wait, sorry, I am combining python3 and dnf in my discussions here
16:36:06 <dcantrell> the testing should be around DNF
16:36:10 <dcantrell> dnf is the focus here
16:36:13 <roshi> dcantrell: did you have an image in mind, or does it need to be created for the testday?
16:36:14 <adamw> ah, ok.
16:36:34 <adamw> roshi: assuming nightly generation is working again by then, we can probably use a nightly.
16:36:36 <dcantrell> if dnf is not usable for the install test cases that Fedora QA cares about, we need to file bugs upstream and then use that information to make a decision as to whether or not dnf is ready for f22
16:36:54 <adamw> so, does Feb 12 sound OK for you?
16:36:55 <dcantrell> so far all of the "let's move to dnf" discussion has been made without any real empirical evidence as to the functionality
16:37:01 <dcantrell> Feb 12 is fine with me
16:37:07 <adamw> ok
16:37:33 <kparal> sounds good
16:37:34 <dcantrell> roshi: let's just run with rawhide for now
16:37:43 <roshi> works for me :)
16:37:44 <kparal> pschindl: Feb 12 for you?
16:37:51 <dcantrell> creating a special build seems like too much extra work right now
16:37:52 <nirik> thats after branching, but otherwise...
16:37:53 <adamw> #info main focus of proposed 'test day' is to exercise DNF and check whether it's in good enough state to be used for F22, tests for anaconda's CI would be a useful by-product
16:38:09 <adamw> dcantrell: i think we'll be on Branched by then, but that's just a note, it shouldn't make a practical difference
16:38:19 <dcantrell> sure
16:38:25 <adamw> #info we will aim to set this up for 2015-02-12
16:38:32 <amita> I think we QEcamp at tht time, not sure if that matters
16:38:45 <tflink> amita: that's the 9th and 10th
16:38:59 <adamw> i guess i'll take the fall for this one...
16:39:13 <adamw> #action adamw to organize/publicize anaconda/DNF test day
16:39:15 <pschindl> kparal: Feb 12 is ok
16:39:27 <amita> ok
16:39:57 <kparal> adamw: if you're going to send an announcement, please put it into fedocal as well
16:39:58 <adamw> so as it seems we'll be aiming to do some broad manual testing, it probably will be useful to have publicity...amita, i'll get in touch once i've got a bit more concrete stuff in place so we can organize an article
16:40:04 <adamw> kparal: thanks, good point
16:40:20 <adamw> dcantrell: bcl: this all sound OK for you? anything else to add?
16:40:21 <amita> adamw, sure
16:40:46 <bcl> kickstart testing is good as well, and easier to translate into test cases.
16:41:03 <dcantrell> adamw: this is sounding good.  we'll get coordinated on our side.  problems we find we'll want to triage and get assigned to dnf or the right component.  I expect this to be bumpy, so testers should be prepared for that
16:41:15 <adamw> eh, i was kinda including that...writing a kickstart on the spot and firing it off is pretty much manual testing, for me. anyhow
16:41:21 <kparal> bcl: just to make sure, there's no point in e.g. gui testing which can't be translated into kickstart, right?
16:41:23 <roshi> we're always ready for bumby
16:41:30 <adamw> kparal: i think we need to do that too.
16:41:46 <kparal> in general sure, but for this CI effort...
16:42:04 <dcantrell> gui testing is fine, but since the focus is on dnf, whatever we can use to drive the dnf backend is key.  the %packages section in kickstart allows more customization than the gui, for instance
16:42:06 <adamw> kparal: as i understood it it's primarily a 'check out dnf' effort and secondarily a 'produce CI tests' effort.
16:42:07 <kparal> we should basically produce a handful of kickstarts?
16:42:20 <kparal> ok
16:42:38 <adamw> dcantrell: the GUI code itself hooks into the package manager library for a bunch of what it does, so we do need to test all the GUI interactions too.
16:43:06 <adamw> dcantrell: at least a couple of the bugs we hit with DNF so far are in the GUI - see https://bugzilla.redhat.com/show_bug.cgi?id=1177002 for e.g.
16:43:28 <dcantrell> adamw: fair enough, definitely should be inclulded
16:44:59 <adamw> ok, we can bash out other details as we go along, i'll set up a trac ticket and a skeleton wiki page asap
16:45:03 <adamw> thanks, bcl/dcantrell
16:45:08 <nirik> related: can folks test dnf-3 as much as they can. Last I heard dnf maintainers wanted to switch to that by default soon. (so yes, if anaconda stays python2, it would use python2 dnf and everyone else command line will be using python 3 dnf).
16:45:27 <adamw> good grief.
16:45:33 <amita> :)
16:45:59 <dcantrell> adamw: np
16:46:20 <adamw> #topic Release criteria / test case revisions
16:46:30 <dcantrell> also, on the python3 subject, we are still on python2 primarily because of the libuser python module.  it has not been updated to python3 yet
16:46:33 <dcantrell> just an fyi
16:46:39 <adamw> bcl: you might be interested in this too i guess, just to know what we have in scope for f22 blocking
16:47:19 <bcl> hey, I'm not the f22 lead I just do rawhide right now ;)
16:47:24 <adamw> so in case anyone had trouble following the 'package set' proposal as it went along, i'll give a quick summary of what i put in place so anyone can point out any problems
16:47:58 <adamw> for alpha, we basically just have 'install out of the box' for all release blocking images - whatever the default package set, it should work
16:48:04 <adamw> that's no different from f21 in practice
16:48:37 <adamw> (except we'll now have the couple of new media: workstation netinst and 'generic' netinst. so more testing. yay.)
16:48:48 <roshi> :D
16:49:27 <danofsatx> yippe. I'm having trouble containing my excitemnet.
16:49:34 <adamw> for Beta, we expect the default package set for each blocking image to be 'correct', and package set selection to work on the 'generic' image
16:49:37 <adamw> danofsatx: :P
16:50:20 <adamw> technically i think we never explicitly required those before, but it was kinda an oversight; if either had been wrong we'd probably have raised a blocker somehow or other
16:51:11 <adamw> for Final, we expect minimal install from the 'generic' image with no updates repository enabled to work
16:51:20 <jreznik> sorry for being late, catching up with stuff after I came back from FOSDEM - we talked with mkolman and he actually started planning test day for anaconda/dnf
16:51:33 <adamw> we *could* require more there, but since the Products have their own netinsts it seemed unnecessary
16:51:46 <adamw> jreznik: ah, ok - i'll co-ordinate with him then...thanks for the heads-up
16:52:08 <kparal> adamw: why no updates repo?
16:52:23 <adamw> kparal: because the updates repo can be...updated
16:52:46 <adamw> in practice i'm sure we'll also check that it works with the packages in 'updates' at release time and yell if it doesn't, but the thing that we can't fix post-release is the frozen package set
16:53:18 <kparal> I meant that it will be empty until release, so we don't really need to insist in needs to be disabled in the criteria
16:53:19 <adamw> and it *is* plausible that someone might set up some kind of large scale unattended install process that they really need to be sure will always work, and it'd make sense to use the frozen repo for that
16:53:49 <adamw> kparal: that could possibly change (as we discussed in relation to the upgrade testing thing) and there's the timing of when exactly the installer stops using updates-testing to consider as well...
16:54:00 <kparal> ok, fair enough
16:54:06 <Southern_Gentlem> adamw, so like the historic fedora folder
16:54:52 <adamw> so in practice i think that keeps the test coverage manageable...
16:55:26 <adamw> i added a requirement in https://fedoraproject.org/wiki/QA:Testcase_Boot_default_install that the default package set be 'correct' (to check that criterion)
16:55:40 <adamw> and i resurrected https://fedoraproject.org/wiki/QA:Testcase_Package_Sets_Minimal_Package_Install to cover the Final criterion
16:56:01 <adamw> and added the two new images to the "Default boot and install" table in https://fedoraproject.org/wiki/Template:Installation_test_matrix
16:56:08 <adamw> that's the extent of the changes so far as testing goes
16:56:42 <adamw> does anyone see any issues with that stuff?
16:57:21 * roshi doesn't
16:57:35 <roshi> but we can adjust as we go if we find something
16:57:37 <adamw> sure
16:57:56 <adamw> so other than that the multiboot revision is still on the go, i'll try and finish that this week
16:58:37 <adamw> and there's a trac ticket with a bunch of places we still need test cases to match criteria, if anyone wants something to work on - https://fedorahosted.org/fedora-qa/ticket/397
16:58:54 <adamw> but it's nearly top of the hour, so let's do a quick open floor then kick over to blocker review...
16:58:56 <adamw> #topic Open floor
16:59:03 <adamw> anyone have anything we didn't cover?
16:59:31 <kparal> we could talk about using those QA Contact fields in bugzilla for blocker bug meeting purposes, but we have 1 minute left
16:59:32 * roshi doesn't
16:59:54 <roshi> oh yeah, good call kparal
16:59:59 <roshi> +1 from me
17:00:31 <kparal> should we discuss this during blocker bug meeting instead?
17:02:06 <roshi> works for me, only have a couple blockers to go through
17:02:10 <roshi> should be a short meeting
17:02:23 <adamw> yeah, sounds good
17:02:31 <adamw> oh, i totally forgot the planned OpenQA topic
17:02:32 <adamw> go me!
17:02:56 <amita> really ,, short blocker bug meeting.. surprise :)
17:02:57 <kparal> jskladan is not here, we can discuss it next time
17:03:22 <amita> OpenQA ?
17:03:32 <adamw> #info some of the folks in RH's Brno office have been working on automating some of the release validation tests with OpenQA, and they've got quite a bit up and running already - code can be found at https://bitbucket.org/rajcze/openqa_fedora_tools/src/d48b884aa1244f5acbb9cad052f5b8a765cc08ad/tools/openqa_trigger/?at=master . we'll talk about it more next time!
17:04:00 <amita> nice
17:04:32 <adamw> amita: it's a test automation thingy that SUSE wrote, https://en.opensuse.org/openSUSE:OpenQA
17:04:47 <kparal> http://os-autoinst.github.io/openQA/
17:04:50 <amita> yeah, got it
17:04:57 <adamw> amita: one of the SUSE folks got some fedora tests running in it as a 'fun project' over the holidays, and some of the brno guys were inspired to try it out too
17:05:11 <adamw> note it's not the one at http://www.openqa.org/ that's dead :)
17:05:20 <adamw> ok folks, thanks for coming everyone!
17:05:30 <adamw> next up: blocker review in #fedora-blocker-review
17:05:50 <adamw> #endmeeting