15:00:12 <jlaska> #startmeeting Fedora QA Meeting
15:00:12 <zodbot> Meeting started Mon May 17 15:00:12 2010 UTC.  The chair is jlaska. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:12 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:00:16 <jlaska> #meetingname fedora-qa
15:00:16 <zodbot> The meeting name has been set to 'fedora-qa'
15:00:29 <jlaska> #topic Gathering crit-mass
15:01:06 * kparal jumps
15:01:40 <jlaska> kparal: howdy
15:01:50 <jlaska> anyone else lurking for the QA meeting?
15:01:54 * maxamillion is here-ish
15:02:02 * wwoods appears in a puff of smoke
15:02:10 <maxamillion> we re-changed $dayjob meeting and now I have internets
15:02:12 <jlaska> maxamillion: wwoods: heyo!
15:02:40 <jlaska> adamw Viking-Ice robatino?
15:02:50 <jlaska> I'll wait another minute or two before starting
15:03:15 * Viking-Ice rides in on ash cloud..
15:03:28 <jlaska> Viking-Ice: heh
15:04:15 <jlaska> okay let's get started
15:04:25 <jlaska> #topic Previous meeting follow-up
15:04:43 <jlaska> This is easy, I've got nothing listed from last week for follow-up
15:04:48 <jlaska> #info no action items from last week
15:04:52 <jlaska> anything I missed?
15:05:32 <jlaska> well, if so ... raise it during open discussion
15:05:39 <jlaska> #topic Fedora 13 RC#3 test status
15:05:56 <jlaska> No doubt you've all read/heard about the 1 week slip
15:06:09 <jlaska> #info Fedora 13 RC#3 is the current release candidate
15:06:29 <jlaska> Test results available at http://fedoraproject.org/wiki/Category:Fedora_13_Final_RC_Test_Results
15:07:12 <jlaska> assuming I'm counting correctly ... we have 5 tests remaining on the installation matrix (4 - i386, 1 - x86_64)
15:07:33 <jlaska> The desktop matrix is empty, which means 32 outstanding tests
15:07:58 <jlaska> I'm unclear whether test tests from RC#2 can apply against RC#3
15:08:13 <jlaska> nonetheless, I'll try to pick off a few GNOME tests later today
15:09:09 <jlaska> #info Because of all this testing, we have a queue of 14 CommonBugs? that need to be addressed
15:09:18 <jlaska> #link https://bugzilla.redhat.com/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=CommonBugs%3F&sharer_id=141215
15:09:45 <jlaska> #action jlaska and adamw to empty the CommonBugs? queue
15:09:53 * jeff_hann says hello
15:09:57 <jlaska> jeff_hann: welcome :)
15:10:05 <jlaska> that's all I have on my radar for F-13-RC3 testing
15:10:06 <jeff_hann> :)
15:10:54 <jlaska> bug#580226 was moved from CLOSED/ERRATA -> ASSIGNED earlier today.  But the reporters problem was different from the failure originally reported.
15:11:04 <jlaska> Any other issues that need to be on the radar?
15:11:49 <jlaska> The go/no_go meeting is scheduled for tomorrow.  However, there is no reason we can't give the "GO" sooner if test results ared completed and all good
15:12:08 <jlaska> if no other F-13-RC3 issues, I'll move on ...
15:12:52 <jlaska> okay, next topic
15:12:54 <jlaska> #topic Fedora 13 QA Retrospective
15:13:30 <jlaska> You've heard me talk about it, perhaps 5 million times already ...
15:13:53 <jlaska> #info Please contribute your thoughts/concerns on Fedora 13 testing to https://fedoraproject.org/wiki/Fedora_13_QA_Retrospective
15:14:21 <jlaska> Once F-13 is behind us, I plan to organize the feedback and work with folks to come up with some tangible improvements for Fedora 14 Testing
15:14:35 * jskladan attends the meeting from bed :)
15:14:52 <jlaska> jskladan: ouch :(   welcome!
15:15:07 <jlaska> okay everyone quiet now, no more talking about jskladan :)
15:15:25 <jskladan> :-D i knew so :))
15:15:36 <jlaska> hehe
15:15:45 <jlaska> alright, next topic was a carry over from last week
15:15:52 <jlaska> #topic QA Summer of Code
15:16:08 <kparal> we already missed the deadline
15:16:14 <jlaska> Last week, kparal put out a request for feedback for anyone with experience or interest in a QA summer of code
15:16:21 <jlaska> kparal: the deadline was May 13, right?
15:16:25 <kparal> right
15:16:51 <jlaska> kparal: I take it no feedback received?
15:17:03 <kparal> no
15:17:07 <jlaska> booo :(
15:17:54 <quaid> well, the students have until this week to propose
15:17:58 <jlaska> well, if nothing else, we have a landing place for QA project ideas
15:18:13 <jlaska> #link https://fedoraproject.org/wiki/QA:Summer_of_Code_Ideas
15:18:15 <quaid> so if there was a student to do the proposal, it can still be on any idea(s)
15:18:35 <quaid> we are looking strong for a bigger, better run this Sep to Feb
15:18:43 <quaid> (southern hemisphere summer)
15:18:45 <kparal> well, we are short of mentors I'm afraid
15:19:17 <jlaska> quaid: oh, so there's another round planned Sep - Feb
15:19:18 <quaid> ok, you can watch how this summer goes and see if it helps you attract mentors
15:19:23 <quaid> jlaska: yes
15:19:25 <kparal> but we can take it as a test run this time, and have something better prepared for the next time
15:19:54 <quaid> and I am working with RHT partner marketing people to put our proposal in front of many partners who may be interested in being sponsors, so more bigger betterness :)
15:20:01 <jlaska> oooh
15:20:01 <quaid> kparal: +1
15:20:05 <kparal> quaid: maybe Dec - Feb?
15:20:11 <quaid> that's about what the _whole_ project islike
15:20:15 <quaid> kparal: for coding, yes
15:20:21 <kparal> I see
15:20:39 <quaid> I think the preparations -- open for ideas, etc. has to start in Sep so we aren't so rushed this time :)
15:20:44 <quaid> we haven't worked on the schedule yet, though.
15:21:12 <quaid> jlaska: so, if we have friends who work Fedora QA and work for companies that benefit from QA advancements ...
15:21:40 <quaid> let's see if we can work that company as a sponsor from multiple angles.
15:21:57 * quaid *cough* s/390 *cough*
15:22:10 <jlaska> quaid: hah!  I think you coughed up a mainframe
15:22:45 <jlaska> quaid: I can start asking around in that space and connect you with the right people?
15:23:28 <jlaska> kparal: anything we need to track here, or anything that you'd like to see completed in advance of this next round?
15:23:29 <quaid> sure
15:23:42 <quaid> and I'll be talking through all the partner level at some point ~August
15:23:44 <jlaska> perhaps I can complete the idea stubs I posted, but haven't fleshed out yet
15:23:50 <quaid> good call
15:24:15 <kparal> jlaska: I think that's too far ahead, to create an action point several months forward :)
15:24:42 <jlaska> kparal: hah, no kidding :)
15:24:58 <jlaska> okay then ... we'll drop this topic for now ... and hopefully pick this back up again in the end of the summer
15:25:02 * jlaska makes note
15:25:45 <jlaska> kparal: quaid: thanks
15:26:24 <jlaska> alright ... it's that time again folks
15:26:27 <jlaska> #topic Open discussion - <Your topic here>
15:26:35 <wwoods> the problem is making the QA tasks seem interesting, when QA is boring by design
15:26:38 <wwoods> heh
15:27:02 <jlaska> wwoods: exactly, I don't think we're looking for people to push buttons and run through pre-planned test execution
15:27:20 <jlaska> this should be fun project ideas that will either grow, or facilitate our QA community in some way
15:27:29 <wwoods> yeah obviously. there's some interesting stuff but none of it is *sexy* really
15:27:46 <wwoods> that's the Curse of QA: the people are sexy, the code isn't
15:27:53 <wwoods> it is our burden to bear.
15:27:56 <jlaska> well, let's make it sexy!
15:28:04 <wwoods> heh
15:28:06 <jlaska> there's a blog title in the making here
15:28:16 <wwoods> anyway, so the bodhi watcher
15:28:16 <kparal> lets give away hot chicks instead of money as a reward
15:28:35 <maxamillion> I've thought of ways to bring in more testers and I've talked a little with jlaska about it but now is a good time to bring it up
15:28:38 <jlaska> kparal: I assume you're talking about this http://letthemeatlentils.files.wordpress.com/2009/12/20_kent_chicks.jpg
15:28:48 <jlaska> #topic Open discussion - make QA sexy
15:28:50 <kparal> jlaska: almost D:
15:29:09 <maxamillion> I've thought about working with the design team to come up with "special" swag that we make and give away to testers who devote their time
15:29:19 <wwoods> http://git.fedorahosted.org/git/?p=autoqa.git;a=blob;f=hooks/post-bodhi-update/watch-bodhi-requests.py;hb=wwoods
15:29:27 <wwoods> err oops
15:29:32 <wwoods> disregard for the moment!
15:29:41 <maxamillion> one thing I was thinking of is a modification to the Fedora splat t-shirt that says something like "construction crew" on the back
15:29:47 <wwoods> maxamillion: hah nice
15:29:50 * jlaska notes wwoods is next in queue for the microphone
15:30:27 <jlaska> maxamillion: do we have a way to gather data on bodhi karma feedback?
15:30:28 <kparal> we should have some small rewards, yes
15:30:32 <maxamillion> or maybe a t-shirt that has something to do with a fedora logo squishing a bug
15:30:39 <maxamillion> jlaska: I'm not entirely sure
15:30:44 <jlaska> #idea small rewards process for QA volunteers
15:30:58 <wwoods> there's other measures of QA awesomeness
15:31:02 <kparal> maybe show the most active people in some "top statistics page" of QA contributors?
15:31:03 <jlaska> #idea work with design team for Fedora branded QA swag
15:31:16 * ianweller starts lurking at the sound of statistics
15:31:17 <jlaska> wwoods: definitely, that was just one I've been missing recently
15:31:19 <wwoods> I'd love it if we had some way for devs to hit a button for "the person who reported this bug did an awesome job"
15:31:29 <maxamillion> ianweller: you have an unique fascination with statistics
15:31:46 <maxamillion> wwoods: that'd be nice
15:31:52 <jlaska> wwoods: ah, right ... so a way for maintainers to say "This person did a nice job"
15:32:11 <kparal> collecting stars?
15:32:13 <wwoods> yeah. I mean it's a little nervous-making because it introduces a gameable element
15:32:15 <jlaska> #idea maintainer driver kudos -- a way for developers/maintainers to recognize a quality job
15:32:27 <jlaska> wwoods: that's fine ... all stats are like that
15:32:32 <wwoods> oh, true, good point
15:32:39 <jlaska> which I think is why we'd need multiple vectors on this stuff
15:32:51 <wwoods> anyway: In my view, doing a good job of actually tracing a bug to its cause
15:32:53 <jlaska> bugs, QA wiki namespace contributions, mailing list assistance ...
15:32:58 <jlaska> wwoods: right on
15:33:06 <wwoods> is possibly the most valuable thing a QA contributor can do
15:33:39 <jlaska> I think we've got some of this captured on the retrospective already ... so this is good
15:33:40 <maxamillion> wwoods: I think that depends on the situation and what the bug is
15:34:08 <wwoods> (note that "QA contributor" here is a slightly broader audience than, say, "QA developer", which includes those of us working on test frameworks and other craziness)
15:34:31 <wwoods> maxamillion: well, yeah. I mean there's plenty of other very, very valuable contributions and not every bug can be traced like that
15:34:40 <wwoods> but that is definitely a skill we want to encourage and reward
15:35:05 <wwoods> if there's one thing I hear from developers *constantly* it's: "god I wish the people reporting these bugs could actually explain what was going on"
15:35:06 <jlaska> I'll update the retrospective after this meeting with some of these ideas
15:35:09 <maxamillion> wwoods: agreed
15:35:14 <jlaska> but feel free to note things yourself (all)
15:35:45 * jlaska looking at a new wwoods entry around improving bug reporting... yay!
15:36:04 <jlaska> #topic Open discussion - <your topic here>
15:36:10 <jlaska> who is next?
15:36:28 <wwoods> me!
15:36:37 <jlaska> wwoods: autoqa?
15:36:50 <jlaska> #topic Open discusssion - autoqa update from wwoods
15:36:50 <wwoods> yeah, specifically the post-bodhi-update hook
15:36:58 <wwoods> so here's that link again
15:37:01 <jlaska> wwoods: take it away!
15:37:03 <wwoods> http://git.fedorahosted.org/git/?p=autoqa.git;a=blob;f=hooks/post-bodhi-update/watch-bodhi-requests.py;hb=wwoods
15:37:26 <wwoods> that's the current post-bodhi-update watcher. It's not quite finished but it's 95% there
15:38:02 <wwoods> The basic purpose of this thing is to fire autoqa tests whenever a maintainer uses bodhi to request an update be moved into the testing or stable repos
15:38:23 <wwoods> unfortunately current bodhi has some limitations that we need to work around so there's a couple things to keep in mind
15:38:48 <wwoods> this watcher *can* miss an update, if rel-eng pushes it into testing/stable before the watcher notices it
15:39:12 <wwoods> but if rel-eng force-pushes the update without waiting for test results, they probably have a good reason for it
15:39:31 <wwoods> so they'll be responsible for its safety in those cases anyway
15:39:47 * jlaska goes #info crazy
15:39:51 <maxamillion> lol
15:40:10 <jlaska> #info The basic purpose of [post-bodhi-update] is to fire autoqa tests whenever a maintainer uses bodhi to request an update be moved into the testing or stable repos
15:40:48 <jlaska> #info Some limitations exist for the watcher (see minutes and link above for details)
15:40:57 <wwoods> second problem: unfortunately critpath updates will appear *without* their target repo listed
15:41:25 <wwoods> originally (as the comments say) I was going to just fire the tests anyway, and we'd just have to test against both sets of repos
15:41:36 <kparal> wwoods: what about the future changes to bodhi that mean that all updates should be accepted "in sets"? will that require heavy changes in your script?
15:42:04 <wwoods> kparal: not sure what you mean by "in sets", but it shouldn't require changes to the watcher
15:42:14 <wwoods> the watcher's job is to notice when the contents of that set change
15:42:33 <wwoods> the tests themselves can choose to either a) test all the updates in the set of proposed updates,
15:42:40 <wwoods> or b) test the individual new update(s)
15:43:13 <kparal> that sounds cool, I will have to check it out
15:43:26 <wwoods> e.g. depcheck needs to test the entire set of proposed updates
15:43:34 <kparal> same for package sanity
15:43:45 <wwoods> right. but a simpler test that only needs to inspect the package
15:43:53 <wwoods> e.g. a test that checked to make sure the package is signed with the correct key
15:44:05 <wwoods> could just test the individual package
15:44:08 * jlaska would like that test
15:44:15 * jlaska <hearts> package signature test
15:44:52 <kparal> so, when we can expect final release of the watcher? :)
15:44:54 <wwoods> anyway, to finish my thought: critpath is a problem, and I'm not sure how to solve it
15:45:20 <kparal> do we have to wait for some bodhi changes?
15:45:38 <wwoods> there's three choices:
15:45:42 <wwoods> 1) fire test with target repo unset, let the tests decide what to do
15:46:07 <wwoods> 2) fire two tests, one for each target repo, let the users/testers decide which results they care about
15:46:33 <wwoods> 3) ignore the update until it has a target repo (i.e. when it gets the required karma), and then fire the test as normal
15:47:02 <kparal> my questions is why it doesn't have target repo set from the beginning?
15:47:15 <jlaska> is this a bodhi-1.0 design decision?
15:47:19 <wwoods> long story short: because current bodhi is kind of dumb, and they're rewriting it, and that's gonna take a while
15:47:35 <wwoods> so this watcher is similarly kind of dumb, and will be rewritten when bodhi2 appears
15:47:48 <kparal> I don't like neither 1) nor 2), and I'm not sure how 3) works
15:48:06 <wwoods> this is basically just supposed to be a "good enough" solution to get us something to work with until the stuff we need goes into bodhi 2
15:48:16 <jlaska> As a proventester, I'd want to see these test results before applying my karma.  So I'm not sure about option #3
15:48:44 <kparal> especially about critpath packages, right
15:48:44 <wwoods> kparal: we just keep a list of updates that appeared with 'request == None', and keep checking until they have a valid 'request' value
15:48:54 <wwoods> yeah, that's the problem
15:49:12 <wwoods> so basically, this watcher is supposed to be temporary so we just need to recognize and accept its limitations
15:49:14 <jlaska> wwoods: is there an option that leaves the tests unchanged when we move to the new bodhi watcher?
15:49:21 <wwoods> and try to choose whatever is the least bad option
15:49:28 <wwoods> and work toward making it work correctly later
15:49:38 <wwoods> yes, choice #2
15:49:45 <wwoods> or #3, actually
15:49:57 <wwoods> but I don't like delaying testing until after karma is applied
15:50:10 <wwoods> which is why #2 was my proposed solution, even though I don't like it
15:50:38 <jlaska> +1 on #2 -- noting your same regrets
15:50:52 <wwoods> I think #2 is the least bad option
15:51:09 <wwoods> if anyone has a better idea I'd love to hear it, but this is what we have to work with right now
15:51:38 <wwoods> anyway given that decision, I've got a couple more things to finish up and we should have the first working version in a day or two
15:51:51 <kparal> ok, great
15:52:03 <wwoods> so give it the rest of the week to shake some bugs out, and I'd say we'll have a functioning hook next week
15:52:13 <jlaska> wahooo!
15:52:18 <wwoods> still need to write the autoqa hook code and the test templates
15:52:30 <wwoods> and a hello-bodhi proof-of-concept test
15:52:40 <wwoods> but yeah, end of next week is my guess
15:52:42 <jlaska> Our watchers need to have icons associated with them.  Post-bodhi-update would be a hydra
15:53:05 <jlaska> wwoods: good stuff, thanks for the update
15:53:16 <jlaska> anything else on autoqa post-bodhi-update?
15:54:07 <jlaska> #topic Open discussion - <your topic here>
15:54:16 <jlaska> anything not already covered?
15:54:31 <jlaska> If not, I'll close out the meeting in 2 minutes
15:56:08 <jlaska> 30 seconds ...
15:56:18 * jlaska queues the Jeopardy theme music
15:56:47 <jlaska> Okay gang, thanks for a great meeting!
15:57:02 <jlaska> As always, I'll follow-up to the list with minutes
15:57:10 <jlaska> feel free to raise additional topics not discuss there as well
15:57:13 <jlaska> #endmeeting