15:00:30 #startmeeting Fedora QA meeting 15:00:30 Meeting started Mon May 12 15:00:30 2014 UTC. The chair is adamw. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:30 Useful Commands: #action #agreed #halp #info #idea #link #topic. 15:00:34 #meetingname fedora-qa 15:00:34 The meeting name has been set to 'fedora-qa' 15:00:37 #topic Roll call 15:00:41 ahoyhoy folks, who's here? 15:01:43 * adamw watches the dust balls 15:02:11 * satellit listening 15:02:53 * tflink is here 15:03:27 let us contemplate the dust balls, together 15:04:38 roshi: ahoy? 15:05:00 * pwhalen is here 15:05:14 * kparal joins 15:05:55 * roshi is here 15:06:18 ahoyhoy folks 15:06:45 welcome to the most exciting ride of your life - oh no, wait, just the qa meeting. 15:07:04 #topic Previous meeting follow-up 15:07:28 #info "adamw to synthesize server, desktop and cloud test outlines to give a rough idea of required fedora.next test coverage and kickstart a discussion of how it should be organized" - not done yet, sorry. will work more on it this week 15:07:40 #action adamw to synthesize server, desktop and cloud test outlines to give a rough idea of required fedora.next test coverage and kickstart a discussion of how it should be organized 15:07:52 #topic Rawhide validation testing 15:08:10 so Gene's mailed this morning to say current rawhide images may be badly broken, anyone else seeing that? 15:08:15 i'm planning to take a look after the meeting 15:08:34 * kparal looking for bug description 15:08:57 * kparal found it 15:08:57 In what sense. I successfully rebooted two rawhide machines this morning, including the one I am typing this on? 15:10:16 (That was after updating to this morning's rawhide updates.) 15:10:45 brunowolff: looks like he was testing boot of daily images 15:10:53 I get aualligned poiner ..... INIT18 BOOT FAILURE with liveinst 15:11:14 unalligned* 15:11:20 minimal arm image from today boots okay, suffers from existing bugs, but boots 15:11:31 hum, okay. sounds like we have something to look into at least 15:11:35 need 2 cores to boot lives 15:12:35 satellit: you were having that 'unaligned pointer' problem last week too though, right? 15:12:39 i don't think gene was having trouble then 15:12:46 perhaps your issue is specific to virtualbox? 15:13:10 oh, hum, you reported one case of it with metal. hrm. 15:13:11 https://fedoraproject.org/wiki/Test_Results:Fedora_21_Rawhide_2014_05_Install yes no also in bare metal install with DVD 15:13:33 and gene says "after about May 7th', which ties in with your timeframe 15:13:33 ok 15:13:42 so probably you are both hitting the same thing 15:13:55 looks like it..... 15:14:07 so we should definitely look into that as a priority...i'll hit it after the meeting 15:14:15 thanks 15:14:20 I remember Josh needing to move the iso9660 driver back to core last week. Could that be what they're seeing? 15:14:37 (At least I think that was the driver.) 15:14:47 um. possibly? need more info, i guess =) 15:15:04 I'll see if I can find the message and come back here with a reference. 15:15:34 thanks 15:16:29 #topic Fedora.next status and where we go next 15:16:51 so I was hoping to have the 'skeleton test plan' for fedora.next that i'm working on available as a sort of jumping-off point for discussion this week...but we can float the topic without it, for sure 15:17:31 this topic is kinda meant to cover the ongoing discussion about how far we should be attempting to cover testing for the new Products and so on 15:17:34 hey, viking, good timing 15:17:57 Viking-Ice: you just missed: " so I was hoping to have the 'skeleton test plan' for fedora.next that i'm working on available as a sort of jumping-off point for discussion this week...but we can float the topic without it, for sure" 15:18:46 does anyone have any particular thoughts on the topic, or any specifics to discuss, or anything? or would folks rather wait for me to launch the trial balloon so we have something specific to shoot at? :) 15:19:16 I got nothing against waiting for you 15:19:27 This is the start of the devel thread I rememebered: https://lists.fedoraproject.org/pipermail/devel/2014-May/199058.html 15:19:32 brunowolff: thanks 15:21:45 i know viking strongly feels like 'fedora qa' should cover only base components and leave product QA to the relevant WGs / SIGs / whatever, my current inclination is that 'fedora qa' should probably be responsible for at least directing / overseeing testing on the major 'fedora products', but i really want to get the 'test plan' down to see just how much work that's going to be 15:21:48 adamw: with this 'skeleton test plan,' is it going to be similar to the old test plans that used to be written or are you talking matrices? 15:22:31 roshi: similar to the old test plans - the textbook definition of 'test plan', with scopes and responsibilities and blah blah, but the idea is really to sort of provide a concrete reference of how much work it's going to be 15:22:56 kk 15:23:11 i don't think we'd wind up actually using this as the working document for fedora 21 testing, i think it'll be more useful as a way to visualize what fedora-as-a-whole is going to need to get done.\ 15:23:52 adamw, my opinion is becoming irrelevant since I'm leaving the project but as I go I still think Fedora QA should be focusing more/better on the core/baseOS part be it comps groups or the specific output from the baseWG, ( the end result is the same ) if and when we would have manage to covered that properly we could extend providing resources to other products given that there exist enough QA community resources and expertise to do so ( as 15:23:52 in covering core/base comps grousp and or baseWG +. whatever else ) 15:25:20 roger 15:25:30 I have always felt how we done things ( past/present ) been somewhat inefficient 15:25:48 or in subtle terms have a huge room for improvement ;) 15:25:56 i like putting it that way better ;) 15:26:04 lol 15:26:06 there's always room for improvement! :) 15:26:26 well, sounds like i should come up with a more concrete basis for discussion, anyhow 15:26:41 i have the action item for that already 15:27:34 oh, also: big vote of thanks to roshi and franciscod for the cloud and workstation 'outlines' respectively 15:27:59 so, we'll come back to that when i have the plan ready... 15:28:04 #topic Open floor 15:28:08 the fact remains the same the best people to possess the in-depth knowledge to properly test are the ones producing the product ( in current case ) and the community surrounding that 15:28:13 oops, sorry 15:29:13 But the people doing the testing would almost certainly benefit from consulting with the people who have been QAing Fedora a long time. 15:29:23 right 15:30:05 i think however we wind up constituting things exactly, development and testing should always be as collaborative as possible... 15:30:10 I envisioned both the roles of QA and releng and other service sub community's in the project being more of an guiding force rather then implementing and or testing one 15:30:37 historically we have been faced with the same problem regarding test day's as we are now with products 15:31:14 these are target spesific testing which often requires specific skill set even hardware to complete 15:31:39 i do find it's sometimes useful on test days when people who know almost nothing about the thing under testing come and try to test things 15:31:56 it's a good place to become familiar 15:31:57 often the devs find they've made assumptions about how people will do stuff or how they'll have their systems set up that are not correct 15:32:20 but you're right that often you need specialist knowledge or config/hardware to really test out something extensively 15:32:53 whoever's administratively 'in charge' we certainly need to be getting the folks developing the .next products to be actively involved in testing them, i agree completely on that 15:33:13 none of this will remotely work otherwise :) 15:33:35 right but limiting it to test day which we never intended to do at least not me and James ) as opposed to days or weekends weeks even months then you wind up spending resources assisting the people not in the know how, teaching them about the software they are dealing with 15:34:17 you can take 389ds for example 15:34:25 we did try running longer 'test day' type events a few times...i found it was hard both to dedicate the time / effort to running them effectively, and keep testers' interest over the longer term 15:34:47 even graphics test week is just a heck of a lot of work to run 15:35:08 which is one of the bigger problem we are dealing with in the project 15:35:08 true - but it was good, imo 15:35:10 lack of proper tools 15:35:39 like what? 15:35:59 like everything 15:36:19 ranging from feature process handling to release cycle to proper documentation tool and even bugzilla 15:36:33 you've got to start somewhere 15:37:00 I've only known what we currently have - which is why I was asking 15:37:01 we equally less prepared infrastructure wize to deal with multiple product as we are policy wise etc 15:37:44 @dayjob we have infrastructure in place which surpasses the project ones in decades 15:37:48 proper documentation tool == not the wiki, is what you mean, right? 15:38:03 right 15:38:12 FOSS infrastructure we could use/emulate? 15:38:13 linking for example to documentation from within bug reports 15:39:24 you're also probably building a smaller product with more people. i often get the same feeling that fedora is run on duct tape and wd40, but sometimes you have to work mostly with what you have. we have folks working on better tooling, but you know the realistic timescales for it. 15:39:57 we have 2000 projects here with ca 20000 customers in 3 different country 15:40:18 ranging from service support to development, to project management etc 15:40:33 fedora has, what, 10k+ SRPMs, with millions of users in X different countries =) 15:40:57 our clients are everywhere but we have companies in three different countries 15:41:05 (well, counting users is hard, but at least hundreds of thousands.) 15:41:17 we sold what we owned in Denmark since it gave us bad ebita 15:41:45 i'm just saying, it's not realistic to apply the standards of commercial software development to a project like fedora; if we sit around waiting for an infrastructure of that type to show up, it's just not going to work 15:41:55 we're a shoestring-and-dreams kind of project, we have to function on that level 15:42:05 and there's always the question of cost-benefit 15:42:08 (first 'we' there is fedora, second 'we' is fedora qa) 15:42:48 and we can always make it better 15:43:28 if I was involved in RH management of the project 15:43:42 I would be investing in proper tools to gain as much out of the project as I could 15:44:05 regardless of license and given that the foundations are under constant attack these days 15:44:05 Viking-Ice: fwiw, we - the RH fedora qa folks - make that argument as often as we can 15:44:32 what's the difference of just give up on the whole idealism of "free and open" 15:44:36 Viking-Ice: tflink has a list as long as your arm of things he'd like to have the resources to get built 15:44:45 many the kind of infrastructure things you're talking about 15:45:13 yes but that requires man power to build ourselves 15:45:23 which we dont have 15:45:24 what we have now in terms of manpower and hardware etc is what RH is willing to provide 15:45:26 yeah. 15:45:46 now and indefinitely 15:45:48 what i'm saying is, we do go out and ask for the resources to build all that stuff, and what you see is what we get 15:46:01 I've been under the illusion we could somehow change that for the past 8 years 15:46:04 dam was I wrong 15:46:11 so we have to prioritize the work according to the resources...and that means right now, taskotron is priority #1, then we have a list of stuff for after that 15:46:28 it's easier to buy existing solution that building it from ground up ( if possible ) 15:46:40 s/that/then 15:46:51 * kparal looks up 15:46:58 well sure - it's 'easier' 15:47:09 i'd love to have RH approve five new full-time fedora qa tool hires next week, but it's probably not gonna happen. i'd love to have more folks with the ability and motivation to work in that area come along from the community, and maybe we could push harder for that too... 15:47:10 but so is buying a sandwich than making one 15:47:22 and the Atlantis products ( jira/confluence ) most certanly can and are cost free for open source projects 15:47:38 for the licenses, sure 15:47:48 but they have to be hosted and maintained 15:48:00 I know I maintain the one here 15:48:00 we can hardly run anything non-open-source inside Fedora infra 15:48:02 and you have to put in the huge amount of work to reorient the entire fedora workflow around them 15:48:15 that's a full time job including handling workflows etc 15:49:02 it would take team of no larger then four persons by me guesstimate for the project to maintain 15:49:17 all products 15:49:19 which is one of the concerns I have - we'd be increasing our (fedora's) cost significantly if we shift away from what RH is already using 15:50:04 Viking-Ice: how do you mean 'maintain all products'? 15:50:06 well RH BRNO has XP running in their receptions so I honestly dont think they care us much as they want to believe in open 15:50:08 adamw: wel, the new hire doesn't have ti be strictly from RH 15:50:25 adamw, update/upgrades built workflows post functions etc 15:50:45 we have a complete automated triage process in place 15:50:58 Viking-Ice: that gets into inside baseball, but let's say at least that the question of *fedora's* F/OSS principles is somewhat separate from RH's. 15:51:22 from what's been taken place used to be 15:51:42 Viking-Ice: really? I have to see that. AFAIK all PCs run Fedora or RHEL 15:51:42 misc: do you know of someone else interested in hiring people to work on fedora QA? if so, for mercy's sake, send me their contact details :) 15:51:57 we dont have the resources to built this, we have been under the illusion could since the project began 15:51:59 Viking-Ice: oh, there's one reception with XP - the one that's not ours 15:52:19 misc: like i said we could renew efforts to get more community help with tooling...it's always good to take a fresh push at that 15:52:22 kparal, and the first computer you see when you enter RH HQ in EU that XP ;) 15:52:44 RH HQ in EU is not Brno, but aht's I think not relevant to the topic of the meeting, no ? 15:53:21 Oh interesting I thought you guys where the biggest and most bad ass but yeah it's irrelevant 15:53:30 Viking-Ice: at least one of RH's brno offices is in a shared building - the reception you're thinking of is the *building* reception. we can't really go and tell the building management to use RH. although perhaps we should send in a salesperson. :P 15:53:42 Viking-Ice: it's landlord's 15:54:19 anyway lack of tooling 15:54:28 please ladies and gentlemen, keep focused (and forget that RH even exist) 15:54:36 number80: thanks 15:54:52 what kinds of gains do you think we'd see with different tooling? 15:55:02 * adamw notes 5 minute warning 15:55:23 automation, better oversight of various things we implement, better management in general you name it 15:55:42 we are in the shit hole with all of this 15:55:49 because I've almost always seen that people always work *in spite* of their tooling regardless of how good the tooling is 15:56:07 * adamw certainly doesn't work 'in spite of' git. 15:56:09 we are suffering from massive resources leakage in terms of contribution 15:56:20 and oversight how and where things stand 15:56:27 okay, new rule: all tools have to be as good as git. get on it, roshi/tflink/kparal 15:56:43 ^^ 15:56:46 * pingou ducks 15:56:51 we'll just put / in a repo - and go from there 15:56:52 i.e. really hard to use for newcommers? 15:56:58 I have yet to see us properly keep tab on how well our ideas are working out 15:57:15 global wide in the project 15:57:45 name me one before and after measurement us QA have ever done or just somewhere in the project? 15:58:41 how many people used it, did it improve things or make them worse, we have no fracking idea 15:58:45 Viking-Ice: depends what you mean. i've done numbers on anaconda and GNOME memory usage a few times./ 15:59:10 that's also really hard to get a good measure of 15:59:18 we do stats on graphics test weeks, we do stats on bodhi. 15:59:20 those are memory usages not ideas, features or proesses implemented in the project or distribution 15:59:36 to even establish a baseline for - but I would love to be able to get those metrics 16:00:15 we are not keeping track on any of this and the whole change process seems to be done more for the sake of change and Matt's gut feeling then any actually statistic in usage droppings for example 16:00:23 * satellit_e afk 16:00:31 think about it we are at our largest component base so far 16:00:32 well, now we're outside QA scope again. 16:01:01 just pointing out global things we as in QA are as bad at measuring things as anyone else in the project 16:01:32 any suggestions on how to measure them within QA? because I would love to 16:01:43 what is it you think we should be measuring? 16:01:50 measuring 'quality' is notoriously hard. 16:01:59 bah - we're over time anyhow at this point, can continue in #fedora-qa 16:02:06 exactly 16:02:28 i'll wrap up here in a minute, can continue in -qa as roshi said... 16:02:38 though i need to go get some breakfast (belly rumble) 16:03:55 thanks for the discussion, folks 16:03:58 #endmeeting