16:01:14 <jlaska> #startmeeting Fedora QA Meeting
16:01:14 <zodbot> Meeting started Mon Jan  4 16:01:14 2010 UTC.  The chair is jlaska. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:14 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:01:21 <jlaska> #meetingname qa
16:01:21 <zodbot> The meeting name has been set to 'qa'
16:01:34 <jlaska> #topic gathering
16:02:03 <adamw> hellllo
16:02:04 <jlaska> Howdy folks ... let's do a show of hands
16:02:30 * wwoods shows off his hands
16:02:37 <jlaska> ooh, look at those hands
16:02:53 <wwoods> theyr'e the new '10 model
16:02:53 * kparal waves too
16:02:55 <adamw> who'd you cut them off? and where'd you get the jars?
16:02:57 * Viking-Ice raises his hand for a high five on the new year..
16:03:26 <wwoods> Viking-Ice: o/ \o
16:03:35 <jlaska> down low
16:03:47 <adamw> wwoods can give you a high twenty
16:04:01 <jlaska> alright ... let's get things started
16:04:19 <jlaska> we've got a light agenda today ... but always room for 'open discussion'
16:04:28 <jlaska> #topic Previous meeting follow-up
16:04:46 <jlaska> okay ... trying to figure out which meeting was the previous
16:05:04 <jlaska> the last set of tasks I'm tracking are listed at https://fedoraproject.org/wiki/QA/Meetings/20100104#Previous_meeting_follow-up
16:05:20 <jlaska> #info jlaska to post recommendations on F-12 QA retrospective
16:05:36 <jlaska> these are posted and available at https://fedoraproject.org/wiki/Fedora_12_QA_Retrospective#Recommendations
16:05:54 <jlaska> I've left most of the agenda for walking through this
16:06:08 <jlaska> next I've got ...
16:06:26 <jlaska> #info adamw to escalate security policy issue to fesco
16:07:03 <jlaska> I don't have any updates on that one ... did that already happen?
16:07:12 <adamw> err. i think i filed a ticket. just a sec
16:07:45 <adamw> yeah, I did
16:07:47 <adamw> https://fedorahosted.org/fesco/ticket/297
16:08:01 <jlaska> ah great
16:08:06 <adamw> no reply yet
16:08:12 * jlaska lost the linky
16:08:24 <jlaska> so that'll come up for discussion at the next FESCO meeting right?
16:08:53 <jlaska> Next up ...
16:08:57 <jlaska> #info jlaska to set up a meeting to co-ordinate test plan changes for F13
16:09:30 <jlaska> We had a quick sprint the week prior to Christmas with adamw, lili, rhe and myself walking through the list of F-13 test plan tickets
16:10:09 <jlaska> still lots of kinks to iron out in the sprint process, and there are still a few inprogress tickets
16:10:15 * jlaska searches for ticket query
16:11:22 <jlaska> https://fedorahosted.org/fedora-qa/query?status=new&status=assigned&status=reopened&component=Test+Review&milestone=Fedora+13&order=priority
16:11:44 <jlaska> I'll circle back and bump the status on a few of those later today
16:11:55 <jlaska> last item ...
16:11:58 <jlaska> #info adamw and jlaska to look at setting up f13 test day process
16:12:14 <jlaska> No updates from me on this ... I'd like to chat w/ adamw this week to iron out an approach
16:12:39 <maxamillion> is there something that is going to be greatly different from the last test day process(es)?
16:12:48 <jlaska> we've got initial ideas posted to the wiki already ... https://fedoraproject.org/wiki/Talk:QA/Fedora_13_test_days
16:13:05 <jlaska> maxamillion: I don't expect so ... just need to get on top of it and actively work the schedule
16:13:46 <jlaska> okay, anything not mentioned?
16:13:49 <maxamillion> ah, nice
16:13:56 <maxamillion> I'll be sure to read that later
16:14:04 <jlaska> if not ... we'll jump into the F-13 QA recommendations
16:14:19 <jlaska> maxamillion: cool, and of course, test day topic suggestions encouraged too!
16:14:29 <jlaska> #topic F-13 QA Recommendations
16:14:57 <jlaska> So ... I've taking all the QA retrospective feedback and tried to organize things into several groups
16:15:04 <maxamillion> jlaska: I'll be working with kevin and christoph on the xfce spin test day plans and try to add suggestions if I find any :)
16:15:32 <jlaska> maxamillion: awesome, that's just what I had in mind.  Any additional spin test days would be fun (much like the sugar event)
16:15:39 <Viking-Ice> I was thinking about going knee deep in LXDE so I should be able to write something..
16:16:19 <maxamillion> Viking-Ice: be sure to touch base with cwickert, i'm sure he'd welcome the interest in LXDE
16:16:24 <jlaska> so I've organized the feedback into the following themes: process, planning, communication and automation
16:16:47 <jlaska> these aren't hard themes, just what bubbled up from how I read the data
16:17:05 <jlaska> so there is overlap between these themes .. but in general, I was happy with the grouping
16:17:06 <Viking-Ice> maxamillion: I was planning to hear from him..
16:17:16 <maxamillion> Viking-Ice: cool cool :)
16:17:35 <jlaska> maxamillion: Viking-Ice should I add an action item for next week on this?  or too soon?
16:17:49 <jlaska> ^--> re: LXDE and XFCE spin test days?
16:18:23 <Viking-Ice> you can add action item for me
16:18:30 <jlaska> okay
16:18:37 <maxamillion> we could do next week, that would give me a bit of motivation to get on the ball about it
16:18:41 <maxamillion> errr I*
16:18:51 * Viking-Ice works best under pressure..
16:18:56 <jlaska> #action Viking-Ice investigating adding an LXDE test day to the F-13 test day schedule
16:19:29 <jlaska> #action maxamillion investigating scheduling a XFCE test day to F-13 test day schedule
16:20:15 <maxamillion> w00t! my first action item in the QA world :P
16:20:25 <jlaska> maxamillion: heh, you've arrived :D
16:20:42 <jlaska> so have folks had a chance to look through the recommendations?
16:20:49 <jlaska> https://fedoraproject.org/wiki/Fedora_12_QA_Retrospective#Recommendations
16:21:06 <jlaska> there shouldn't be surprises there ... I've basically reorganized the feedback provided
16:21:36 <jlaska> what I'd like to do is walk through the list and see if we can identify owners for some of the tasks
16:21:44 <maxamillion> jlaska: do you want to #link that so its in the minutes? (that might not even be the right tag, but you get the idea) :)
16:21:55 <jlaska> maxamillion: sure ...
16:22:05 <jlaska> #link https://fedoraproject.org/wiki/Fedora_12_QA_Retrospective#Recommendations
16:22:57 <adamw> just refreshing my memory
16:23:01 <jlaska> many of the listed recommendations are things several of you are already on top of
16:23:35 <adamw> i could take "Promote install testing as a QA activity"
16:24:18 <jlaska> adamw: thanks, Hurry has some ideas on this already ... would you two like to sync up?
16:24:41 <jlaska> perhaps her experience on the install side iwth your wiki mastery can come up with some good ideas
16:24:59 <adamw> also most of the 'communication' stuff
16:25:10 <adamw> well yeah i was just meaning i could do the wiki bits
16:25:21 <adamw> stick me in a #action
16:25:25 <Viking-Ice> I need a little explanation with this "Determine whether guides can be offered by Fedora QA as a result of a test day/feature process"
16:25:47 <jlaska> #action adamw and rhe to discuss ways to add install testing as a QA activity
16:26:12 <jlaska> adamw: kparal had some social networking suggestions already, perhaps we can divide and conquer there?
16:26:29 <jlaska> Viking-Ice: my thinking there was to follow your lead for dracut
16:26:37 <kparal> not really mine suggestions, but I can help
16:26:53 <jlaska> kparal: oh right ...these were from the lunch'n'learn
16:27:00 <kparal> correct
16:27:02 <Viking-Ice> Advertice on the Fedora Project network on FB
16:27:20 <jlaska> Viking-Ice: basically, that dracut debug page rocks ... I'd like to see we can follow that model again for maybe a few features?
16:27:23 <jlaska> what do you think?
16:27:43 <Viking-Ice> sounds raz..
16:28:19 <Viking-Ice> I'll run through the feature stuff and see what I can write..
16:28:44 <jlaska> Viking-Ice: sweet, I think that would be tremendously helpful.  I just don't know a good way to target specfiic features
16:28:58 <jlaska> dracut was a good choice since it was something *everyone* would need
16:29:09 <jlaska> perhaps python3 might be similar in that regard?  or others
16:30:22 <jlaska> #link https://fedoraproject.org/wiki/Releases/13/FeatureList
16:30:53 <jlaska> #topic F-13 QA Recommendations - test planning
16:31:02 <jlaska> on the test planning front ... much of this is already underway
16:31:32 <jlaska> #info Maintenance of the Fedora Install Test Plan - Liam and Hurry are doing a lot of heavy lifting here
16:31:55 <jlaska> #info Draft a more formal, but basic and lightweight, security checklist/test plan - discussed earlier, adamw has a fesco ticket filed ... stay tuned
16:32:14 <jlaska> #info Draft a test plan to cover the desktop verification from Fedora_Release_Criteria
16:32:19 <jlaska> #link https://fedoraproject.org/wiki/Fedora_Release_Criteria
16:32:26 <kparal> I noticed adamw is creating test plan for the desktop stuff: https://fedoraproject.org/wiki/Category:Desktop_Acceptance_Test_Cases
16:32:37 <adamw> yeah, i'm in teh middle of that
16:32:43 <adamw> got about halfway through before the xmas break
16:32:46 <adamw> will be back on it this week
16:32:47 <jlaska> kparal: perfect, thanks for the link
16:33:00 <kparal> I will gladly help with that if time permits
16:33:15 <adamw> sure, if anyone wants to chip in tests that'd be great
16:33:15 <Viking-Ice> I think we need to get in touch with the X team to come up with an formal Xorg test plan ( not just x11-drv-* )
16:33:26 <jlaska> Viking-Ice: good segway ... that was next
16:33:30 <adamw> well we talked about that quite a lot at the sprint
16:33:31 <jlaska> #info We have no formal Xorg-x11-drv test plan ... either create one, or develop proposal for gathering the same feedback through other means (smolt, bugzilla)
16:33:50 <adamw> i actually don't think a test plan for X would achieve much
16:33:58 <adamw> we started doing a test matrix then realized it was pretty pointless
16:34:15 <jlaska> adamw: can you talk more about how we came to that conclusion?
16:34:18 <adamw> the tricky thing with X is that the main problem with it is that we're dealing with seventy zillion different bits of hardware
16:34:31 <jlaska> yeah that ---^ :)
16:34:33 <adamw> and the level of functionality we currently care about is pretty basic, this isn't really deep level stuff
16:34:52 <maxamillion> I think there could be a matrix of hardware and "Works for Me" and maybe some break down of how much works (KMS, direct rendering, etc)
16:35:02 <adamw> the types of problems we get with X are not 'this one out of the millions of really complex levels of functionality suddenly regressed' but 'o shit this card doesn't work any more'
16:35:18 <adamw> maxamillion: that would rapidly get HORRIBLY unsustainable
16:35:20 <jlaska> maxamillion: if someone wanted to look into how to pull this out of smolt and present the information ... that would be a fun project
16:35:27 <adamw> i really can't emphasize the amount of hardware we're talking about here enough
16:35:35 <maxamillion> adamw: for a test day?
16:35:38 <Viking-Ice> what about input ( keyboards usb etc ) and for example multitouch ?
16:35:39 <maxamillion> or you mean long term
16:35:40 <adamw> we already do test days
16:35:41 <maxamillion> ?
16:35:47 <maxamillion> oh ok, i misunderstood
16:35:55 <adamw> we're talking about some kind of test plan
16:36:03 <maxamillion> sorry, I'm in a meeting at $dayjob and trying to keep up with this one ... don't mind me :P
16:36:14 <adamw> i just don't think that's an approach that fixes the X situation; it's not the paradigm that addresses the kinds of problems we actually get with X
16:36:29 <maxamillion> adamw: yeah, I agree
16:36:31 <jlaska> maxamillion: no it's a good suggestion ... it's just trying to figure out how to sustainably execute the plan and track the data
16:36:55 <adamw> Viking-Ice: um. in theory an input matrix is a useful idea, but in practice it kinda depends how much we care about anything besides basic mouse+keyboard stuff
16:37:02 <adamw> er, s/matrix/test plan/
16:37:13 <adamw> multitouch is way too basic at this point to worry about
16:37:20 <adamw> you can't really _use_ it in any meaningful way yet
16:37:58 <adamw> jlaska still reckons some kind of test plan for X is a good idea, but i still don't quite understand exactly what we can achieve with one that we're not achieving already; this is where we left it in the sprint :)
16:37:58 <jlaska> one way we can use the existing test cases is much how they've been used so far
16:38:51 <jlaska> for test days and for use to help isolate the true failure of reported bugs
16:39:05 <Viking-Ice> adamw: Ok Talking about meaningful did we receive any feed back from nirik and others on the front ( forums #fedora etc ) about what they get bombarded with ( perhaps something we should track and target for test days )
16:39:18 <jlaska> Viking-Ice: good suggestion
16:39:47 * nirik looks up.
16:39:56 <adamw> well, i'm mostly on teh front for X stuff, and yeah, we do co-ordinate a bit on that
16:40:00 <nirik> you mean specific stuff? or in general areas we see a lot of stuff?
16:40:21 <jlaska> nirik: other soft spots from the F-12 release that could modify the QA plan for F-13
16:40:29 <Viking-Ice> all proprietary aside the issues most end users are faced with
16:40:34 <adamw> i don't think we can add much to our process based on that feedback. i'd say broadly what you get splits into two camps: 'HELP MY X/COMPIZ DOESN'T WORK' and 'HELP MY PROPRIETARY DRIVER DOESN'T WORK'
16:40:40 <adamw> second is out of our scope
16:40:43 <adamw> first is already covered
16:40:43 * nirik nods.
16:40:48 <jlaska> true
16:41:16 <jlaska> okay ... last I have on the planning side is Test Days
16:41:27 <adamw> as i said, X is fundamentally a shallow area: the issues are due to hardware variety (and the developers rewriting everything every three months). there just isn't a complex tangle of functionality to test
16:42:13 <jlaska> adamw: if we had a pull of every hardware adapter available ... would there be some type of feedback to provide to X maintainers that would be valuable?
16:42:51 <nirik> having the list of workarounds especially for install has been great on the common bugs list.
16:42:55 <jlaska> oh god ... s/pull/pool/
16:43:00 <adamw> well, yeah. if we had a Big Test Lab with every graphics adapter in the world in it, sure. what they'd be interested in is knowing with every day's rawhide which systems basically worked. for bonus points, which could do enough 3D to run compiz.
16:43:10 <jlaska> heh
16:43:23 <adamw> i mean, that's really it. like i keep saying, it's not complicated =)
16:43:26 <jlaska> adamw: so instead, we have a highly distributed test lab
16:43:40 <adamw> there was that thread about 'stop working on rawhide and work on f12' or whatever on devel-list
16:43:44 <wwoods> (if you consider the set of Interested Fedora Users as a distributed test lab..)
16:44:00 <adamw> dave airlie (or ajax, I forget) posted the X team's List Of Things We Really Care About to that thread, which is basically what I just said
16:44:02 <jlaska> wwoods: true true
16:44:10 <jlaska> adamw: gotcha
16:44:21 <jlaska> alright ... so last I had on the planning topic was test days
16:44:37 <jlaska> anyone know if they are plans to revisit the 'fit and finish' events?
16:44:41 <Viking-Ice> adamw:  Big Test Lab the world not enough ;)
16:44:43 <jlaska> if not, I can follow-up w/ mclasen
16:44:55 <adamw> sure. how i look at that is that the blocking points in that way of looking at things all involve making the lab work efficiently. the test plan is not part of that. the test plan defines _what to test_, which as I keep saying is really simple and everyone knows it already
16:45:38 <adamw> all the problems with X testing, if we look at the 'the world is a big test lab' theory, lie in actually co-ordinating the distributed systems and people who make up the 'lab' to do testing and provide useful info. which a test plan can't really help with, i don' think. but hey. sorry, jlaska wants to move on :)
16:46:05 <jlaska> adamw, no thanks ... you explained it better than I could have hoped to
16:46:57 <Viking-Ice> Let's move along..
16:47:03 <jlaska> #action jlaska will reach out to mclasen for thoughts on F-13 fit'n'finish test days
16:47:14 <jlaska> okay ... any other suggestions on the test planning side for F-13?
16:47:33 <jlaska> for reference ... https://fedoraproject.org/wiki/Fedora_12_QA_Retrospective#Test_Planning_3
16:48:23 <jlaska> alright ... the next group is where wwoods and kparal rock
16:48:32 <jlaska> #topic F-13 QA Recommendations - test automation
16:48:43 <jlaska> #link https://fedoraproject.org/wiki/Fedora_12_QA_Retrospective#Test_Automation
16:48:53 <jlaska> The first obvious one is ...
16:49:01 <jlaska> #info Deploy AutoQA 'is rawhide broken?' front-end into fedora infrastructure
16:49:12 <jlaska> this is inprogress ... and going through package review at the moment
16:49:25 <jlaska> #action jlaska to reach out ot fedora-{test,devel}-list for trading reviews
16:49:36 <jlaska> #info Deploy build-time sanity checking mechanism for package maintainers (rpmguard, rpmlint etc...)
16:49:52 <jlaska> Kamil and Will are knee deep in this now
16:50:05 <jlaska> with both rpmlint and rpmguard working into the system
16:50:06 <kparal> and will be deeper :)
16:50:11 <jlaska> hehe
16:50:37 <jlaska> The last item I have here is
16:50:40 <jlaska> #ifo Develop proof-of-concept for automating install test matrix
16:50:44 <jlaska> #info Develop proof-of-concept for automating install test matrix
16:50:48 <wwoods> I'm pretty sure rpmguard will obsolete rpmlint eventually
16:51:14 <kparal> I don't think so, they have different goal
16:51:25 <jlaska> Liam and Hurry are working towards a automated DVD install test at the moment ... and then likely will need wwoods expertise for integrating the tests into AutoQA
16:51:26 <wwoods> kparal: hm, good point
16:51:42 <wwoods> jlaska: yeah, I think this will probably be my first big goal for the new year?
16:52:07 <jlaska> wwoods: cool, this would be huge to see traction on before F-13 kicks into high gear
16:52:33 <jlaska> Liam was looking into playing with install.py over the break and expanding it to do a basic DVD install ... I'll be checking in to see how he's making out soon
16:52:47 <jlaska> anything else under automation that wasn't covered ... that needs to be on the radar?
16:52:54 <wwoods> yeah, IIRC it involves designing a test that takes as input:
16:53:02 <wwoods> 1) a description of the test system hardware (disks, disk sizes),
16:53:04 <wwoods> 2) a kickstart (which describes partitioning etc),
16:53:09 <wwoods> 3) an installable tree/image
16:53:43 <jlaska> yeah, I should have cc'd you on the mail I sent him
16:53:49 <jlaska> basically the same list of inputs
16:53:51 <wwoods> pretty sure a test that would do that would cover the entire test matrix, but we'll see
16:54:57 <jlaska> okay ... we touched on each of the other sections already but here they are for the logs
16:55:11 <jlaska> #topic F-13 QA Recommendations - process
16:55:16 <jlaska> #link https://fedoraproject.org/wiki/Fedora_12_QA_Retrospective#Process_4
16:55:19 <Viking-Ice> One thing that came to my mind when I hit bug 549944 is this something we can automate an test for ( looks like version mismatch )
16:55:20 <buggbot> Bug https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=549944 medium, low, ---, jskala, NEW, Updating ppp to 2.4.5-x breaks NetworkManager-pptp vpn connection...
16:56:53 <jlaska> Viking-Ice: hrmm, yeah if there's something there that could have been detecting with rpmguard, we can get a ticket in for a future test case idea
16:57:02 <jlaska> so for process ... I had a few things
16:57:28 <jlaska> #info Move User:Liam/Draft_Install_Test_SOP out of draft status - to me this is part of making install testing easier
16:57:54 <jlaska> #info Move User:Poelstra/blocker_bug_meeting_sop out of draft status - adamw had some improvements during FUDCon, I think we just need to dust it off before the blocker reviews kick in again
16:58:07 <jlaska> #info Draft a SOP (or extract from existing BugsAndFeatureRequests page) that describes how to document bugs (e.g. common_bugs or release notes or install guide etc...)
16:58:32 <jlaska> ^^^ I'm a simple caveman ... so the docs might already be out there, but I could stand to be reminded how we advise folks on documenting issues
16:58:51 <adamw> we don't have anything really
16:59:00 <adamw> well, each document advises people how to contribute to it
16:59:08 <adamw> the common_bugs page advises how to add to the common_bugs page
16:59:17 <jlaska> #info Draft/promote a recommended test checklist for rawhide testers (for example see User:Jlaska/Draft2) - this is something spot discussed a while ago and adamw drafted an idea.  I think it would be worthwhile to reference from the QA/Joining page ... ideas?
16:59:19 <adamw> the docs team have a page explaining how to contribute to the release notes
16:59:53 <adamw> jlaska: we could probably stick it up as a page and just link it from one of the existing sections on Joining
16:59:55 <adamw> off the top of my head
17:00:08 <jlaska> adamw: yeah true ... I was looking for a single page that had pointed to the instructions and helped advise which place was relevant for different bugs
17:00:36 <jlaska> #info #  'How to debug <component> problems?' -- Determine whether guides can be offered by Fedora QA as a result of a test day/feature process. - Viking-Ice already discussed this previously
17:00:46 <adamw> jlaska: the problem with that is, what would drive people to the central page?
17:01:01 <jlaska> just like all other debug guides and doc pages ... we would
17:01:13 <jlaska> in our IRC discussions, mailing list interactions etc
17:01:13 <adamw> jlaska: well i don't see a natural flow, a real role for this central page
17:01:21 <adamw> hmm, i see
17:01:28 <jlaska> you don't .. cause you are the natural flow for this stuff! :D
17:01:37 <adamw> well that's what i'm trying to explain in a crappy way =)
17:01:41 <jlaska> hehe
17:02:02 <jlaska> I can take that action on that one for the release
17:02:06 <adamw> usually i can see a sorta natural 'place' for a new wiki page, where it would fit into the flow of reference from page to page...i'm not sure i can see a real good place for something new here...maybe beland would have an idea how to structure it
17:02:41 <jlaska> oooh, yeah he has a way with those kinds of wiki changes
17:03:00 <jlaska> I can reach out to beland for guidance/ideas if that makes sense
17:03:04 <adamw> sure
17:03:15 <adamw> maybe he can come up with a mockup that'd make me go 'ohhh yeah'
17:03:20 <jlaska> okay
17:04:23 <jlaska> #action jlaska to reach out to beland for guidance/ideas on how to document the process (or point to existing documentation) for how bugs are noted (common_bugs, release notes, install guide etc...)  How to determine which bugs land in which place
17:05:10 <jlaska> #topic F-13 QA Recommendations - communication
17:05:13 <jlaska> #link https://fedoraproject.org/wiki/Fedora_12_QA_Retrospective#Communication_4
17:05:20 <jlaska> I'm not going to spend time on this section, we discussed it earlier
17:05:55 <jlaska> so ... any other thoughts/concerns on the recommendations
17:05:58 <jlaska> I think we have a lot of good ideas
17:06:22 <jlaska> and I'd like for us to look back on this page after F-13 ... and say ... great, we did X% of what we said we would
17:06:30 <jlaska> regardless of what the value of X is
17:06:43 <jlaska> okay ... let's move on to open discussion
17:06:49 <jlaska> #topic Open discussion - <your topic here>
17:08:18 <adamw> nothing here
17:08:23 <jlaska> <crickets>
17:08:39 <jlaska> well, happy new year everyone and welcome back :)
17:08:46 <jlaska> I'm sure folks are still digging out from under email
17:08:49 <Viking-Ice> I propose that Fedora QA uses it's extensive forces and influences in the whole world to get a softer toilet paper to public toilets and conduct massive everyday testing on those paper..
17:09:04 <Viking-Ice> in the year 2010
17:09:15 <jlaska> Viking-Ice: I'd like to see that test plan!
17:09:31 <kparal> :))
17:09:51 <Viking-Ice> I need to redraw it I accidentally flushed it down the drain..
17:10:01 <jlaska> well, on that note  ...
17:10:04 <jlaska> #endmeeting