16:01:25 #startmeeting Fedora QA Meeting 16:01:25 Meeting started Mon Nov 7 16:01:25 2011 UTC. The chair is adamw. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:01:25 Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:01:28 #meetingname fedora-qa 16:01:28 The meeting name has been set to 'fedora-qa' 16:01:52 * jskladan tips his hat 16:01:52 #chair j_dulaney 16:01:52 Current chairs: adamw j_dulaney 16:01:56 morning folks, who's around? 16:02:01 #topic roll call 16:02:02 * tflink is present 16:02:04 * Cerlyn is here 16:02:06 * brunowolff is 16:02:07 * pschindl is here 16:02:10 * kparal is not here 16:02:17 * j_dulaney is here 16:02:22 kparal: still away? :) 16:02:38 * kparal is partying with hot chicks in Carribean 16:02:48 or whatever is the spelling 16:02:49 It would seem that brunowolff just _is_ 16:03:10 bruno, indisputably, is. 16:03:17 kparal: and you're still on IRC? /me wonders about you 16:03:42 oh yes, fedora qa meetings are my weakness 16:03:48 * j_dulaney wondered about kparal already 16:04:38 * adamw feels bad for the hot chicks 16:05:25 #topic Fedora 16 release planning 16:05:33 * kparal converting all hot chicks to using Fedora 16:05:39 alright, this is just a kinda general f16 topic 16:05:43 make sure we tie up any loose ends 16:05:59 thanks again to everyone for their great work on validation, of course 16:06:39 we do have the livecd-tools updates to get out for f14, f15 and f16: https://lists.fedoraproject.org/pipermail/test-announce/2011-November/000340.html 16:06:56 i also need to get with hughsie about fixing f14's preupgrade 16:07:02 and we have common bugs to do 16:07:13 can anyone think of anything else? 16:07:22 * j_dulaney will test the livecd-tools for F15 16:07:24 commonbugs? 16:08:21 already mentioned 16:08:41 huh, I missed it 16:08:43 oh well 16:08:58 you're fired! 16:09:02 :D 16:09:05 man, you're starting early this week 16:09:39 I was on the phone at work. 16:09:43 adamw: I see the weekly firings are going to continue past F16 going gold 16:09:59 But with all the layoffs, who is going to test Fedora? 16:10:00 weekly? you're _lucky_ if it's only weekly. 16:10:07 Cerlyn: you are! 16:10:10 adamw, of course 16:10:18 no sleep at all 16:10:21 hehe 16:10:25 adamw: well, there was a 2 week span where I didn't get fired 16:10:30 adamw: I'm sure I can get rbergeron or jsmith to have you fired 16:10:33 man, i must've been slacking 16:10:45 anyhoo 16:11:08 the commonbugs list is pretty long: https://bit.ly/fedora-commonbugs-proposed 16:11:17 so if people can pitch in and help clear it that'd be great 16:11:48 other than that i guess we're good on this topic, yay - finally f16 is more or less behind us 16:12:04 oh, missed one thing - the retrospective 16:12:49 the retrospective is at https://fedoraproject.org/wiki/Fedora_16_QA_Retrospective : just as a reminder, it's where you can put thoughts about the f16 cycle, stuff that went well and stuff that didn't go so well, ideas for how to do things better next time 16:13:03 some time soon i'll go through and pull out action items from it and turn them into trac tickets, and summarize it for the list 16:13:26 so if you have any thoughts about how to make life less stressful next time, add them on there! 16:14:16 alrighty, moving on... 16:14:21 #topic proven tester process 16:15:11 so this one is interesting. at their meeting last week, fesco more or less decided they want to abandon the proven testers process 16:15:17 you can see the meeting log at http://meetbot.fedoraproject.org/fedora-meeting/2011-10-31/fedora-qa.2011-10-31-15.03.log.html 16:16:14 mjg notified devel list, too: https://lists.fedoraproject.org/pipermail/devel/2011-November/158881.html 16:16:31 they did not actually directly notify QA in any way, but never mind. 16:16:46 * nirik would like to interject. ;) 16:16:50 interject away! 16:16:54 we didn't decide anything. ;) 16:16:56 that first link goes to a QA meeting log, not to fesco :) 16:17:05 sigh, my link skills suck 16:17:06 one more try 16:17:12 we wanted to wait a week for feedback from all groups... including qa. ;) 16:17:16 http://meetbot.fedoraproject.org/fedora-meeting/2011-10-31/fesco.2011-10-31-17.00.log.html 16:17:46 The thought was that proventesters were not that helpfull, and we should continue critpath and all just with no proventesters... s/proventester/logged in karma submitter/ 16:17:49 nirik: ah, okay. of course, it helps to get feedback from qa if you actually tell qa. =) 16:17:58 it means proventester group would be cancelled with no replacement, do I read that correctly? 16:18:00 sorry... we've all been busy. ;( 16:18:33 kparal: well, replaced by 'anyone with a fas account' I guess... 16:18:49 that means no replacement, same process as for all the other packages 16:18:55 kparal: as nirik says, we'd essentially keep the current process except proventesters wouldn't exist and we'd just accept karma from any logged-in user as if it were pt karma as far as the current scoring goes 16:19:07 kparal: critpath would still need higher total karma to be approved - 2 vs 1 16:19:24 do we accept karma from anonymous users for non-critpath? 16:19:29 I mean no FAS 16:19:31 The thought was that proventesters haven't really made that much difference, so it seems like overhead to have to manage it and try and get people to become them. 16:19:46 kparal: nope. anon still doesn't count. 16:19:56 ok 16:19:58 kparal: no, we don't use the numeric karma from anon users for any package (critpath or non-critpath) 16:20:14 Is there any proof one way or another that proventesters actually test things in a proven way that justifies trying to train people? 16:20:25 nirik: my only worry is as posted to the list: two looks like a very small number but it is not zero, and this is a game where the difference between two and zero could be a substantial one 16:20:27 so, I don't think this is something we need to do instantly... if everyone would like more time to look at stats and mull it over thats cool with me. ;) 16:20:37 proventesters are not specially trained :) 16:20:41 Cerlyn: I try to do so 16:20:47 Cerlyn: they confirmed they read the guidelines. which might be more than ordinary testers 16:20:48 Cerlyn: the devel list post from mjg59 had some stats... 16:21:35 #link http://lists.fedoraproject.org/pipermail/test/2011-October/104084.html 16:21:50 there's also a (rather informal) process whereby people who submit updates and get dumb feedback from proventesters can come and complain to us about it, and the pt gets educated. that's happened a few times. 16:22:10 true. 16:22:25 nirik: i also think there have been cases where we _could have used_ proventester karma to prevent some updates going out 16:22:25 although we could also educate 'normal' testers. ;) 16:22:31 but didn't, because negative karma is currently handled poorly 16:22:42 yeah, could be... 16:23:21 adamw: negatice karma could certainly be improved 16:23:34 Even if we keep proventestor 16:23:39 +s 16:23:44 nirik: there've been a couple of cases where updates have gone out in cases like '+1 proventester, +1 regular user, +1 regular user, -1 proventester, SUBMITTED TO STABLE!, -1 proven tester, -1 proven tester, -1 regular user, PUSHED TO STABLE!' 16:24:03 j_dulaney: yeah, to a degree they're separate issues, but if we had better handling of negative karma it may have changed the pt stats. 16:24:19 crit path packages actually should have a thorough test guide which proventesters can follow - right now they are just normal testers that *try* to take things a bit more serious but often lack the detail knowledge nethertheless 16:24:23 there was a proposal, for e.g., to disallow any update with negative karma from a proven tester from going through 16:24:43 red_alert: yeah, we've been working on the test guide side of things for a while, but it's one of the victims of limited resources 16:25:26 adamw: That would be a good way to handle things 16:25:27 going back to negative karma - and there's also the possibility of blocking any update which is submitted after passing the criteria but, before being pushed, drops to failing the criteria 16:25:55 Indeed 16:25:55 so i think it might be worth considering ways of improving the handling of negative karma before deciding to dump the pt concept. especially since it's something that would be hard to get back 16:26:24 if we ever cancel it we need to be damn sure it's okay to cancel it, because if we cancel proventesters then six months later put it back in, people might feel like they were being jerked around. 16:26:37 yeah, not sure how much can be done in the current 1.x bodhi framework, but perhaps. 16:26:48 I think that we're missing part of the why this was proposed, though 16:26:52 the process as its done right now doesn't work (i.e. is no improvement to not having pts) but having pts with a better process would surely add to the overall quality 16:26:53 sure, I don't think we want to be hasty... 16:27:18 ie maintainers getting frustrated with their packages stuck in testing due to lack of proventester action 16:27:23 even though proventesters is trival to join, it's a hoop... so many people just don't bother. 16:27:42 tflink: yeah, it's important to consider that 16:27:52 tflink: I think its more a lack of karma for packages in general, although I haven't seen the stats 16:28:13 well, it's mostly the critpath ones... 16:28:18 Cerlyn: nah, the proposals to 'revise' the proventester process always come from people who are frustrated that their updates take a long time to clear it 16:28:21 because non critpath maintainers can push after a while. 16:28:38 tflink: I know that during the latter half of the release process, I tend to focus more on the new release and less on testing updates 16:29:13 so if we object to getting rid of the proventester process, do we have any solutions to the root of their complaints? 16:29:16 For instance, right now I don't have a box with F15 16:29:29 Although I'll be reinstalling it here shortly on my test box 16:29:33 I'm guessing here, but I assume that there wouldn't be so much of a push if the delays weren't there 16:29:43 tflink: fair point 16:29:46 well, to some extent, the 'allowing to push after 2 weeks with no - karma' would help 16:30:07 fesco was considering that for a while, and our position was basically 'we'll get back to you after f16', aside from the pt meetings nirik's been running 16:30:32 Which I haven't been able to make due to class at that time 16:30:48 (which haven't really been all that well attended. ;) We did get a few things moving, but not much... 16:30:52 We could still keep the proven tester group for a while even if we relax the karma requirements. 16:31:24 nirik: the timing wasn't the greatest with the crazy that F16 release brought 16:31:33 true. 16:31:39 brunowolff: yep. 16:32:15 so i guess we don't have an entirely concrete response to the proposal - we have concerns but recognize the reasons people don't like the process 16:32:38 Maybe down the road we will count proven tester votes more strongly, but still enough enough normal karma to OK an update. 16:32:49 oh, one other thought on the stats 16:32:57 perhaps everyone could air their concerns on the devel list thread... 16:33:05 and we can see where we end up next week? 16:33:13 i assume the stats made the assumption that the proventester feedback on any update would still have been present but treated it no different from regular feedback 16:33:38 so, that's not necessarily a safe assumption: proventesters may feel a stronger 'responsibility to test', and if you cancel the process, they might stop doing so 16:33:38 yeah, although you would have to ask mjg59 for sure. 16:33:41 but that's hard to gauge. 16:33:45 nirik: sounds good 16:33:51 yeah, very hard to quantify 16:35:00 I know that until about Beta of a release, I make it part of my daily ritual to pull from updates testing and put everything through its paces 16:35:15 I know not how many others do that. 16:35:30 i think there were some stats on total pt feedback from luke recently too 16:35:36 * j_dulaney does not have everything that is in critpath installed, however 16:35:38 I would propose to have a QA/proventester meeting during FUDCon Blacksburg and try to come up with a beter pt-process there 16:35:55 it would certainly be a nice thing to do at a fudcon if enough interested parties were there, for sure 16:35:59 red_alert: Not a bad idea 16:35:59 if people would wait that long 16:36:09 yeah, it's a while from now... but possibly 16:36:28 getting rid of the pt-process doesn't sound very time-sensitive, though 16:36:45 It's not. But satisfying the concerns of maintainers is important. 16:37:17 The risk isn't so much individual packages. If people feel that the critpath process is unreasonable then the risk is that there's a backlash against the entire testing process. 16:37:19 maybe deactivate the process in the meantime, then? 16:37:22 mjg59: as noted, the two week delay might sate the baying hordes for now =) 16:37:28 Is there a need for this to be voted upon at the fesco meeting following this one? 16:37:43 mjg59: er, the two-week timeout thing for critpath 16:37:50 adamw: The two week delay is an admission of absolute failure 16:38:07 The process needs to work without that 16:38:30 i didn't say otherwise, i said it might avoid a major backlash, which is a different question. 16:38:49 add the two week delay admission until we had the chance to fix the process, then? because right now it seems to be total failure until fixed :) 16:38:56 i don't really want to re-open the whole debate about whether it's a good thing in this meeting, we have other topics to cov.er 16:39:11 Looking through the stats there are actually more cases of proventesters inappropriately blocking a package than there are of the pt process preventing bugs getting through 16:39:42 Which isn't to slight any of the individual proventesters involved 16:40:02 It just means it's questionable as to whether the requirement makes an overall positive difference to the quality of our updates 16:40:11 mjg59: by inappropriately block, do you mean by not providing karma or providing negative karma where it wasn't warranted? 16:40:11 yes, we already covered that at the start of the topic, 20 minutes ago 16:40:25 are we going to get anywhere new, here, or should we move on? 16:40:38 tflink: Providing negative karma due to either an unrelated issue or a misunderstanding of the package 16:41:08 adamw: I reckon moving on might be a good thing 16:42:00 moving along then - we clearly haven't reached a complete conclusion on this topic yet, and we're not going to do it in the next five minutes 16:42:17 as nirik suggested, please everyone with concerns, post your thoughts to the devel list thread, and fesco will consider them 16:42:52 #agreed group has various perspectives on the proven tester issue, we will respond to the devel list thread and follow developments with fesco 16:43:14 #topic fedora 17 pre-planning: anaconda GUI rewrite 16:43:25 so, you know what the end of the f16 cycle means - time to start worrying about f17! 16:43:33 yay! 16:43:41 * kparal expected some pause 16:43:46 trip to hawaii, etc 16:43:54 nevermind 16:44:08 kparal: i thought you were there already? 16:44:48 finally! I've been waiting for the f17 cycle ever since the last readiness meeting's end ;D 16:44:49 for the rest of you 16:44:57 for anyone who doesn't know, one of the major features planned for fedora 17 is a complete re-design of the anaconda GUI - not just a cosmetic change to how it looks, but the actual workflow is to be completely changed 16:45:01 red_alert: :) 16:45:18 adamw: how realistic is that is will be done in f17? 16:45:22 adamw, *crossing fingers* 16:45:24 it seems like a really big task 16:45:38 kparal: depends on the number of slips we accept? ;) 16:45:40 kparal: very, they're already coding it. 16:45:44 red_alert: correct answer ;) 16:45:52 ok 16:46:06 My wondering is how soon we can start testing 16:46:10 so, obviously we want to try and avoid f16-style crazy stress scenarios here, as far as possible 16:46:41 * j_dulaney is thinking that as soon as it starts hitting Rawhide nightlies, testing will begin for him 16:46:52 i've thought for a while we should work directly with the anaconda team to get involved through the whole process and get testing done as early and often as possible 16:47:14 +1 16:47:17 i'm happy to work on that, or if anyone else would like to, feel free :) 16:47:46 * tflink wonders if it would be better to start with one point of contact 16:48:04 yeah, i did say 'or' 16:48:16 I can't imagine that it'll help if multiple people start asking them about their testing plans 16:48:31 don't we just need to know which specific nightlies they consider worth of testing and then report bugs, that's it? :) 16:48:32 adamw: true, just thought that I would mention it. 16:49:11 red_alert: maybe, depends on how we want to proceed and what the anaconda team already has planned 16:49:14 maybe create a special tracker bug for it, too...or go with F17Blocker already 16:49:36 When I can try installing with only 512 MB of memory, I'll be looking at that. 16:49:48 Like I said, as soon as their stuff starts hitting nightlies, test it 16:49:49 * kparal notes we have some bugs already that can be assigned to F17Blocker. creating it now would help 16:50:06 red_alert: it helps to know when they're planning to land changes, what's expected to work, and probably give them input into use cases they may not have considered, on the basis of the test matrix 16:50:20 kparal: i thought i already had, but someone mentioned earlier that they weren't there, i'll do it in a bit 16:50:31 and make sure that we're covering all the changes 16:50:43 our current test matrix doesn't explicitly cover all the gui stuff 16:50:57 kparal: though anyone caan do it, again - process is documented at https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Trackers 16:51:02 adamw: I'll be having a bunch of free time starting in a month, so I can do the connecting 16:51:10 starting in a month is a month too late ;) 16:51:26 * j_dulaney can start now 16:51:54 The only real blocker of my time will be finals the first week of December 16:52:30 i think it might be best to have someone who's definitely going to have lots of time for it, which is why i was gonna volunteer 16:53:25 True 16:53:40 * nirik looks for more things for adamw to do since he has lots of time. ;) 16:53:50 nirik: i don't now ;) 16:53:59 too late. rats. ;) 16:54:05 #action adamw to co-ordinate with anaconda team on f17 re-write test planning 16:54:05 Especially since he's firing everyone 16:54:08 missed your spot! 16:54:16 j_dulaney: oh right, i didn't fire nirik yet! 16:54:21 I 16:54:53 so, tflink already had a few ideas that might help with testing - want to re-air them here tflink? 16:55:16 I'll monitor adamw's sleeping hours and tell nirik when I spot that he's got time available...i.e. as soon as he sleeps over 4h/night ;) 16:55:20 adamw: I can or we can wait to see what the anaconda team has planned 16:56:39 adamw: ah, please make sure to get their thoughts on having some anaconda/QA hackfest during fudcon...might be important for my sponsorship request ;) 16:56:50 the first was to make sure that we always have install isos available for testing 16:56:56 red_alert: ooh, yup. 16:57:15 tflink: Pull in rel-eng for that? 16:57:42 which has two parts, keeping a somewhat up-to-date rawhide tree that is stable so that we can test anaconda without worrying about the rest of rawhide 16:58:35 and the second, which is making it easier to build test isos with custom package sets for testing 16:58:58 adamw, so you want hackfest space a Fudcon What days 16:59:05 * j_dulaney wonders if we could just test new Anaconda with F16 sicne we know that's stable? 16:59:20 j_dulaney: that might be another option 16:59:29 kk4ewt: we'll deal with that later 17:00:05 :) 17:00:19 using the new anaconda with F16 will make upgrade testing rather invalid, though 17:00:27 tflink: Scientific method 17:00:33 Change only one variable 17:00:44 the other different idea is going to be more of a question - would the availability of hardware/VMs affect the amount of testing done on new anaconda? 17:00:44 And this is a huge variable to change 17:01:06 yeah, agreed on that one. upgrade testing could be separate 17:01:38 but I'm not sure about all the implications of trying to test F17 anaconda w/ F16 17:01:44 tflink: might elaborate on your second idea a bit. only quickly. =) 17:01:46 tflink: if I can get my hands on a larger hardrive for the new laptop, I'll have hardware to test on (and do VM testing, too 17:02:37 my idea was to offer test VMs to people who want to test. Testers would be able to access the GUI installer through a VNC-ish interface 17:03:24 question is, do we have anyone who's dying to help test but has no access to test hardware or a VM? 17:03:28 or is that...not the case? 17:03:49 * j_dulaney does not really have hardware right now 17:03:58 I believe that use case may be pretty valid 17:04:07 adamw: a better way to put it would be: would access to more VMs increase the amount of testing that people would do? 17:04:10 I've got a laptop capable of it, but I don't have a HD and can't afford one 17:04:27 We are supposed to have FESCo meeting here. Will the Fedora QA meeting end soon? 17:04:43 adamw: could you end your meeting please? 17:04:53 mmaslano: yeah, sorry, been trying :( 17:05:08 let's end this and move over to #fedora-meeting-1 to conclude 17:05:14 k 17:05:20 why does this happen, btw? didn't we used to have 2 hours between the meetings? 17:05:23 As sort of a joint issue, is FESCO going to comment on their election guidelines since j_dulaney is a nominee who doesn't meet the letter of the policy? 17:05:39 So I should stick around? 17:05:42 #agreed meeting to conclude in #fedora-meeting-1 17:05:42 adamw, we have the start of the meeting UTC based 17:05:46 #endmeeting