16:00:17 #startmeeting Fedora QA Meeting 16:00:18 Meeting started Mon Mar 22 16:00:17 2010 UTC. The chair is jlaska. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:19 Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:00:20 #meetingname fedora-qa 16:00:21 The meeting name has been set to 'fedora-qa' 16:00:28 #topic Gathering 16:00:49 * kparal appears 16:00:50 alright folks, who do we have around for the QA meeting? 16:01:16 kparal: jskladan howdy gang 16:01:32 morning 16:01:44 adamw: heyo 16:01:46 * jskladan likes meeting time 16:02:17 * Southern_Gentlem 16:02:40 Southern_Gentlem: hey there! 16:02:45 * jlaska waits a few more minutes 16:05:34 Oxf13: said he'd be out for the meeting 16:05:40 is wwoods around? 16:06:04 haven't seen him (or at least he did not respond to my ping) 16:06:39 okay, well ... let's get moving, I don't want to keep you folks longer than we need to 16:06:52 I'll be walking through the proposed agenda at - http://lists.fedoraproject.org/pipermail/test/2010-March/089489.html 16:06:59 #topic Previous meeting follow-up 16:07:12 First up ... 16:07:17 #info adamw to check-in with wwoods on tooling needs for priv esc. test 16:07:27 well, I sent an email 16:07:31 he hasn't replied yet :) 16:07:33 I saw adam reached out on this, wwoods || adamw have you guys had a chance to talk further? 16:08:28 okay ... I spoke to wwoods on Friday and he was putting a lot of brain cycles into depcheck ... I'll keep this on for follow-up next week 16:08:32 okay 16:08:51 #action wwoods and adamw to discuss tooling needs for priv esc. test 16:09:14 This next topic is related to our agenda today, so we don't need to dive too deep on it 16:09:18 #info maxamillion seeking input from the Mentors group for guidance on mentor responsibilities 16:09:41 * jlaska seeing if maxamillion is around on #fedora-qa 16:10:16 sorry I'm late 16:10:23 maxamillion: hey there, no worries 16:10:52 hey maxa 16:10:58 maxamillion: just following up on your check-in with the Mentors group for guidance on mentor responsibilities 16:11:36 I think this will touch on a larger topic I was hoping we could discuss later on, so we can move discussion to that point if you like 16:11:37 jlaska: lemme find the archive link .... I got 1 response that I'd like to put up for discussion 16:11:44 yeah, lets do that 16:12:03 maxamillion: okay ... 16:12:30 #info got 1 response and will discuss later during QA meeting 16:12:38 its probably going to be a long discussion so I think the "previous meeting follow-up" might not be the best spot to bring it up :) 16:12:45 maxamillion: heh, okay :) 16:12:52 * Viking-Ice half in half out.. 16:13:08 Viking-Ice: howdy 16:13:16 #topic F13 Test Status 16:13:44 just a quick update on where things are with F13 Beta 16:14:03 We had the pre-beta acceptance drop last week - https://fedoraproject.org/wiki/Test_Results:Fedora_13_Pre-Beta_Acceptance_Test_1 16:14:12 nothing horribly broken detected from that test run 16:14:34 #info Oxf13 is working with dlehman to identify an anaconda build to deliver for the Beta 'test compose' 16:15:01 Once that's available ... we'll update rhe's announcement to the list 16:15:14 Folks are encouraged to post test results to https://fedoraproject.org/wiki/Category:Fedora_13_Beta_TC_Test_Results 16:15:34 in the upcoming test news front ... 16:16:05 #info assuming positive test results - Beta 'release candidate' planned for this Thursday (2010-03-25) 16:16:31 alongside the planned F13 testing, just a reminder on the test day front ... 16:16:57 thanks to all for participating in the gnome-disk-utility test day last week (https://fedoraproject.org/wiki/Test_Day:2010-03-18_Palimpsest) 16:16:59 * wwoods here 16:17:02 * maxamillion downloads the latest nightly build to fire up some tests 16:17:10 wwoods: hey there 16:17:15 this week we have ... 16:17:18 yep, disk test day went well 16:17:21 i'll send a recap to the list soon 16:17:27 12 bug reports, 2 already fixed 16:17:41 adamw: sweet, thanks ... I was really stoked at the improved UI 16:17:50 yes, it looks great 16:17:52 kudos to davidz 16:18:21 #info Thursday (2010-03-25) we have a Printing test day - https://fedoraproject.org/wiki/Test_Day:2010-03-25_Printing 16:18:52 I don't think anyone from the QA team is involved in organizing this event? Mostly twaugh driving here? 16:19:36 it'd be pretty cool if we had the Test Day SOPs so well-defined that QA involvement wasn't required to run a Test Day 16:20:21 * maxamillion would be willing to help but doesn't actually own a printer so thinks it would be a situational mismatch 16:20:23 wwoods: sarcasm? :) 16:21:18 jlaska: yeah, I haven't done anything on it yet, and the page looks good 16:21:26 the wiki looks fairly well defined for the printing event, I might just check-in with twaugh to see if they need anything from us 16:21:30 kudos to twaugh 16:21:48 right on! 16:22:12 alright, next week we have 2 test days ... hosted by QA 16:22:23 #info 2010-03-30 - SSSD by Default - https://fedoraproject.org/wiki/Test_Day:2010-03-30_SSSDByDefault 16:22:33 #info 2010-04-01 - ABRT - https://fedoraproject.org/wiki/Test_Day:2010-04-01_ABRT 16:22:47 jlaska: no, not at all - is that already the case? because awesome 16:23:09 Both have have a QA and devel pair established, so I believe they are moving forward nicely 16:23:19 wwoods: right on ... pass it along ... https://fedoraproject.org/wiki/QA/SOP_Test_Day_management 16:23:30 that's all I have for F-13 testing ... 16:23:38 it's just full of test runs and test days 16:23:40 oh that's awesome! 16:23:41 did I miss anything? 16:24:02 I'll send notes / make edits if there's anything missing. anyway, please continue 16:24:08 wwoods: kudos to adamw for that 16:24:35 okay ... moving on to next topic ... 16:24:49 #topic Updates testing brainstorm 16:25:16 I was hoping to spend time today discussing what we want (or think we need) out of an updates testing workflow 16:25:30 so much of our time is spent testing the branched release 16:26:06 but with the increased exposure of using bodhi for branched (and already released updates), I would like to start collecting ideas so we can document a roadmap/plan 16:26:26 there are a lot of efforts underway here already ... 16:26:41 kparal has an acceptance test plan in draft -- https://fedoraproject.org/wiki/User:Kparal/Proposal:Package_update_acceptance_test_plan 16:27:07 maxamillion has been putting out feelers for defining the 'proventesters' group -- http://lists.fedoraproject.org/pipermail/test/2010-March/088980.html 16:27:24 but right now ... what we document to our testers is https://fedoraproject.org/wiki/QA/Updates_Testing 16:27:30 so these are the things I think we need: a policy/sop for the 'proventesters' group, and a guide to providing updates-testing feedback for a) branched and b) stable releases, explaining what actually should be tested and how feedback should be given 16:27:36 and I got 1 reponse --> http://lists.fedoraproject.org/pipermail/mentors/2010-March/000177.html 16:27:41 response even ... 16:28:08 adamw: +1 16:28:09 #info we need a policy/sop for the 'proventesters' group 16:28:30 #info we need a guide to providing updates-testing feedback for branched + released explaining what should be tested and how to give feedback 16:28:52 we can leverage the fedora-easy-karma script 16:29:03 kparal: have a 'usage' message in it? 16:29:23 adamw: sorry? 16:29:34 related to your 'how to give feedback' ... lmacken and wwoods have talked about providing a wiki structure for testers (and contributors) to expand on that provides test instructions on a per-package basis 16:29:35 kparal: just wondering how you meant to 'leverage' it 16:29:43 We can document how best to use it 16:29:58 and mention it in the guide in the first place :) 16:30:02 jlaska: that sounds rather like the 'how_to_debug' pages? 16:30:19 adamw: similar, in that it's on the wiki and has a structure around the pages 16:30:59 what does it mean to test an update? 16:30:59 okay 16:31:07 does that mean you've run through a documented series of tests 16:31:12 or just validated the listed bugs are resolved? 16:31:17 or something else? 16:31:24 good point 16:31:31 yes, that's what I was concerned with 16:32:04 i think we need general guidance on that which applies to all packages; having specific instructions for some is also good, but the priority should be to define exactly what the acceptance testing is supposed to check 16:32:46 adamw: great point ... so continue to lock down the details drafted in https://fedoraproject.org/wiki/User:Kparal/Proposal:Package_update_acceptance_test_plan ? 16:32:48 do you think we should work that up ourselves, or go back to fesco for guidance? 16:33:04 I think that's hard to write down because updates generally do one of two things: either it fixes a previous bug or adds an enhancement 16:33:16 #info we need general guidance on that which applies to all packages; having specific instructions for some is also good, but the priority should be to define exactly what the acceptance testing is supposed to check 16:33:21 jlaska: sort of, but I think that's mostly focused on automated tests right now? 16:33:25 and in the event of a bug fix, you need to test that the bug is fixed but also test that there aren't any regressions 16:33:35 maxamillion: there can be multiple criteria for different types of updates, it doesn't need to be one-size-fits-all 16:33:41 adamw: yeah, just the tests required to accept the update for additional testing 16:33:52 maxamillion: just needs to be clear enough that you can always tell what you should be testing for any given update 16:34:00 adamw: that's the *acceptance* plan 16:34:01 adamw: true, maybe I just don't entirely understand how to capture that into a procedure 16:34:15 jlaska: right, i'm thinking more about what people should be looking for at the bodhi stage. we can certainly put the criteria into that document, though. 16:34:27 adamw: I'm with ya 16:34:29 acceptance plans are (kind of by definition) simple and fast 16:34:33 which lends them to automation 16:34:41 wwoods: well, the point is 'acceptance into what' 16:34:47 actual testing. 16:34:51 right 16:34:53 what I'm hoping to drill out here is what happens after we've accepted the update 16:35:01 so either we expand the scope of that document or we have two 16:35:05 wwoods: adamw: exactly, just what you guys are talking about 16:35:12 the point of an Acceptance Test Plan is to quickly decide whether or not the thing is even *testable* 16:35:17 i don't think that question is super-important, though, the important thing is to nail down all the actual information 16:35:24 then we can look at how exactly to arrange it 16:35:33 sure, let's get more ideas ... 16:35:54 and once something passes the acceptance testing, it's accepted for actual testing, and then moves on to another (separate) test plan 16:36:10 wwoods: sure, so if you want it to stay as an acceptance plan, we need a separate document for what we're discussing now (defining what people should look at when deciding whether to mark an update as good or bad) 16:36:31 for now i was just thinking about hashing out the actual criteria on a mailing list thread 16:36:35 in short: acceptance tests are designed to be the first-line test in a series of test plans. they're simple, very automatable, and not intended to be exhaustive. 16:36:37 we can write it down all fancy-like later =) 16:36:48 I think you're missing the point I'm trying to make here 16:36:54 i may be. what is it? 16:37:09 does it help for us to continue engaging in discussion to detail the different types of updates (e.g. https://fedoraproject.org/wiki/Stable_Release_Updates_Proposal) ? 16:37:28 * jlaska holds question until wwoods is done 16:37:30 we need a separate document for what we're discussing now, because an Acceptance Test Plan is, *by definition*, not supposed to be the sole test plean 16:37:33 err test plan 16:37:53 wwoods: sure. I got that. that's why I said that if we want it to be an acceptance plan, it can't cover this stuff and we need a new one. 16:38:27 so does it help to think of this discussion as just a "Package Update test plan" 16:38:32 an Acceptance Test Plan is always, by definition, the first in a series of test plans, so anything beyond the scope of "can this thing actually be tested properly" should be moved into subsequent plan(s) 16:38:50 if we're thinking in formal QA terms, sure, that sounds about right 16:39:00 I'm lost now. are we talking about acceptance test plan in general or about the work the proventesters will be doing (mainly critical path stuff)? 16:39:07 I think we all agree on that point, but I don't want to get bogged down in what final document this takes form in 16:39:23 the basic existence of an Acceptance Plan implies the intent to create more separate plans. 16:39:26 kparal: that's a good point, i think we're talking about both, and you're right that under the fesco-approved policy they'll be distinct 16:39:28 kparal: I'd like ideas around what testing needs to occur for updates in general 16:39:48 ok .... so: AcceptancePlan -> UpdatePlan -> Testing -> Karma -> CheckedForStable -> Bodhi 16:39:51 ? 16:40:00 I just wanted to make sure we're all clear on this point, 'cuz it keeps coming up 16:40:12 but yeah. point belabored. 16:40:13 maxamillion: er, the UpdatePlan is how you do the Testing, and how can it go from Karma to Bodhi? :) 16:40:54 adamw: yeah, basically that's the piece of the puzzle we need to sort out .... which is more or less the root of this conversation and what we need outlined for the ProvenTester group 16:40:56 erm, so can we reboot? do we want to talk about exactly what candidate updates should be tested for in this meeting, or do we just want to agree that's what we need to define and do it on the list? 16:41:11 adamw: thank you, the later was my intent 16:41:14 adamw: at least that's how I understand it 16:41:28 #info focus for this meeting - agree that's what we need to define and do it on the list 16:41:44 so right now ... I've got that we need to ... 16:41:47 i'm happy to write up a draft to get the ball rolling on discussion for that 16:41:53 if anyone would like that 16:41:54 create proventesters group 16:42:20 maxamillion: you've been looking into supporting details around mentoring people into this group? 16:42:45 maxamillion: are we expecting another draft of the proventesters policy/sop proposal? 16:43:14 * jskladan has to go - see you around tomorrow, gang! 16:43:23 jskladan: see you tomorrow :) 16:43:45 jlaska: yes, but I haven't been able to find much ... I only got one response to my query with the mentors and I didn't feel it was the information I was hoping for 16:44:25 adamw: yeah, I'm going to need to touch things up and write up a proposal for how the mentor program will be handled, but I worry that will spawn into a rather large topic of its own 16:44:58 oh fun! i like topics. 16:45:10 as adamw pointed out, I think we can work the details of that out of the meeting ... unless there are specific roadblocks you needed to walk through here? 16:45:47 nope, none in particular 16:46:20 alright ... so we've got defining 'proventesters' group, and drafting a policy/sop around detailing the group 16:46:25 so should we take an action item for maxamillion to provide the next draft of the provenpackagers end of things, and me to provide a draft for the update test plan end of things? 16:46:25 what's after that? 16:46:47 * jlaska recommends excessive use of #action and #info 16:47:01 who has control of meetbot? 16:47:12 I think anyone can add #action and #info tags, no? 16:47:18 yes 16:47:54 #action maxamillion will work on second draft of provenpackagers proposal 16:48:08 #action adamw will provide a draft for the update test plan 16:48:20 okay, so I'm still trying to wrap my brain around the big picture ... 16:48:23 * maxamillion doesn't think meetbot picked those up .... 16:48:32 doesn't meetbot normally echo the command or something? 16:48:37 not for those 16:48:41 oh ok 16:48:49 :) 16:49:09 alright, so we have proventesters and an update test plan ... are those the only 2 things standing in the way of having a defining updates testing workflow? 16:49:22 are there other things we need on the radar, perhaps further down the line? 16:49:33 s/defining/defined/ 16:49:43 not that I can think of ... but I'm sure there's something I missed from my inbox or I'm just simply out of the loop on 16:49:59 or maybe I'm not and that'd kick hind parts :) 16:50:23 i think that's enough to work on for now anyway 16:50:29 agreed 16:50:40 once we get those done there's some joining-up to do in the wiki but i can't think of anything else major 16:50:48 +1 16:50:56 okay 16:51:06 I have to run, have to grab a bite to eat before a meeting 16:51:17 adamw: I'll need to bug you after meeting to get a sense for hte scope of your test plan ... you might be knocking out a lot of stuff with just that 16:51:23 maxamillion: okay, thanks for joining 16:51:36 jlaska: always happy to be here :) 16:52:10 wwoods: kparal: any other points not raised that you'd like to consider as eventual tasks? 16:52:41 not currently 16:54:14 wwoods: anything else on your radar for this topic? 16:55:11 alright, in the interest of time ... let's move on to open-discussion 16:55:42 #topic open discussion - change meeting time? 16:56:25 once Europe changes their clocks for daylight savings, this meeting will be a bit late for kparal and jskladan I believe 16:56:40 well, 6 pm 16:57:15 should I send out a http://whenisgood.net/ for a new meeting time 16:57:19 there is this proposal to change the meeting time one hour back during the summer time (15.00 UTC) and revert it back once the winter time comes (16.00 UTC) 16:57:32 that way the meeting time would be always the same for all of us 16:57:54 but it would require announcing changes every half a year 16:57:55 UTC would change, but the localtime would remain the same 16:58:03 i don't really mind 16:58:04 jlaska: thanks for clarification 16:58:54 adamw: so that puts it @ 8am pacific? 16:59:47 alright, I'll follow-up to the list, but I don't have objections to that ... I don't think we're going to confuse lots of people with that. The UTC / DST change already confuses people 16:59:52 I think adamw is farthest in the west of all of us? 17:00:05 #action jlaska to post meeting time change request to test@l.fp.org 17:00:13 #topic open discussion - 17:00:13 Except for me since i'm on 0 GMT 17:00:14 yeah, 8am. 8am's fine 17:00:34 Viking-Ice: oh true 17:00:44 anything else not discuss that people would like to bring up? 17:00:54 + check with 0xf13 think he has always had problem with the meeting time 17:01:06 Viking-Ice: will do, thanks 17:01:48 I'll close out the meeting in another minute, unless there are other topics raised 17:02:51 alrighty ... that'll do it for today then 17:02:59 thanks everyone, I'll follow-up with minutes to the list 17:03:03 #endmeeting