16:00:17 #startmeeting Fedora QA meeting 16:00:17 Meeting started Mon Feb 4 16:00:17 2013 UTC. The chair is adamw. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:17 Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:00:20 #meetingname fedora-qa 16:00:20 The meeting name has been set to 'fedora-qa' 16:00:22 here 16:00:26 #topic roll call 16:00:29 too early, dan, too early 16:00:30 * dan408 waves 16:00:37 * Viking-Ice here 16:00:38 * mkrizek is here 16:00:41 * satellit here 16:00:46 sorry i'll be later next time 16:00:49 * nb here 16:00:50 you better 16:00:51 * tflink is here 16:00:56 * kparal here 16:00:58 hey nb 16:01:02 * pschindl is here 16:01:14 hey tflink kparal satellit pschindl 16:01:22 adamw you should throw chairs at people 16:01:24 wow, full house 16:01:26 hey Viking-Ice 16:01:27 good point 16:01:32 where's jreznik 16:01:33 * kparal pokes jskladan 16:01:34 #chair tflink kparal satellit 16:01:34 Current chairs: adamw kparal satellit tflink 16:01:47 funny how all these people somehow don't show up for blocker review ;) 16:01:48 dan408: I think jreznik was on FOSDEM 16:01:54 oh yeah that's right 16:01:59 or maybe still is 16:02:06 I don't know how long that lasts 16:02:07 yeah fosdem is in full effect 16:02:26 * jskladan lurks for sure 16:02:34 fosdem finished yesterday 16:02:44 ( and yep, jrzeznik was there, do not remember when he left ) 16:02:48 so they're probably still hungover? 16:02:54 +1 for recovery day 16:03:11 * adamw still hungover from fudcon 16:03:25 you? 16:03:36 :) 16:03:50 okay, so - we don't have 'previous meeting followup' this week as there were no action items 16:04:00 unless anyone has something to pick up that's not in the rest of the agenda? 16:04:02 https://fedoraproject.org/wiki/QA/Meetings/20130204 16:04:12 well i added new nth process 16:04:17 but i didnt know if it was in the right place 16:04:21 like i told you the other day 16:04:36 we covered it last week 16:04:43 oh ok 16:05:07 https://fedoraproject.org/wiki/QA/Meetings/20130128#Blocker_process_revision 16:05:23 alrighty, then straight on to: 16:05:30 #topic Criteria revision 16:05:46 i was hoping to have something a bit more concrete for the meeting, but unfortunately not yet 16:06:07 I guess we need to add back all the advanced storage stuff in criteria that got "suspended" in f18 16:06:08 but the idea here is to revamp how we present the criteria somehow - the 'giant list of text bullet points' thing has probably gone as far as it can go 16:06:18 i like it 16:06:25 but would 16:06:26 acceptedblocker 16:06:26 acceptedfb 16:06:30 be seperate bugs 16:06:31 or keywords 16:06:35 Viking-Ice: yeah, we have some content issues too 16:06:45 dan408: this is about the criteria - https://fedoraproject.org/wiki/Fedora_18_Alpha_Release_Criteria etc 16:06:52 and finishing up the storage criteria from F18 16:06:58 tflink: damnit, stop remembering that 16:07:20 well this is going to change isnt it 16:07:21 it's not like I proposed anything concrete, either :-P 16:07:22 for f19 16:07:29 dan408: well that's what we're talking about :) 16:07:33 well 16:07:34 I would like to see some simplified criteria page digestible by general users. with links to lawyer-like descriptions 16:07:38 what's proposed in the new anaconda 16:07:52 i mean a lot of it depends on that 16:08:07 adamw: did you ever make a concrete proposal based on discussions from fudcon? 16:08:16 tflink: not yet unfortunately 16:08:39 we never quite got around to drawing up a plan at fudcon, we just kinda talked round the issues, as i recall - i might've whiteboarded some of my crazy thoughts but i think that was all 16:08:41 and a QA vocabulary with common QA terms 16:08:49 kparal: also nice 16:09:01 i really liked tflink's app 16:09:17 i've been planning to try and draft up a new way to present them and send it to the list, but to avoid duplication, has anyone else been working down that line? or planning to? 16:09:20 I'm not so sure everyone is familiar with QA terms ( think newcomers ) at least not the from the field 16:09:22 like i was saying the ability for discussion and voting would make it perfect 16:09:39 Viking-Ice: yeah i think you and kparal are onto something there, we should have a glossary of some kind 16:09:56 dan408: again, this is just about presentation of the criteria, we're not talking about the blocker proposal / voting stuff right now 16:10:18 adamw: nothing concrete or anything that would happen in the near future, no 16:10:35 adamw: i know just saying 16:10:45 adamw: I haven't intended to work on this in the near future, no 16:11:04 so nothing in my back drawer 16:11:08 okay, so we got some good 'ideas to put into the plan' already - the legalistic/simple distinction and the glossary 16:11:14 anyone have any other ideas for me to put in the pot? 16:11:22 tflink, had already mentioned an rename of QA to better follow the industry definition but that is a massive ( at least wiki ) work doing so for little no benefit as I see it 16:11:38 Viking-Ice: rename to what? 16:11:45 adamw: just to try to get a better understanding of exactly what's coming so we aren't changing criterion on the fly 16:11:52 similar to making things simplified, have a shorter "summary" so that they fit in IRC lines 16:11:59 dan408: that's on the content side, but sure 16:12:22 #info adamw will be trying to come up with a proposal for better presenting the release criteria 16:12:28 #info ideas to keep in mind: 16:12:46 well, either a shorter "summary" or a better way to reference the criteria - rewriting them on the fly during review meetings is not fun 16:12:47 adamw: also I'd like to see the criteria grouped - for example installation criteria, desktop criteria, networking criteria, etc 16:13:01 #info kparal - draw a line between simple explanations and lawyerish complex clauses 16:13:08 but it's highly possible sometimes a criterion would fall into several groups 16:13:18 #info kparal/viking-ice - define terms that we're familiar with but a general reader may not be 16:13:20 dates would be nice 16:13:32 #info tflink - ensure there's short text for each criterion for IRC meeting purposes 16:13:42 #info kparal - group the criteria (installation, desktop, networking...) 16:13:43 kparal: I suspect he is referencing a conversation about "QA" -> "Quality Assistance" instead of "Quality Assurance" but I could be wrong 16:14:00 dan408: you mean dates of when the criteria were added? 16:14:06 or what? 16:14:10 and also a link to a discussion about that particular criterion would be great to have attached. it'd be easy to understand why the criterion was created 16:14:31 #info kparal - link to background info on each criterion (esp. discussion when it was created) 16:14:31 tflink, kparal yup that 16:14:37 kparal: you and I have all the same ideas I think :) 16:14:52 adamw: 1) dates of when all features are supposed to be approved by FESCO 2) deliverables and timelines for each feature 3) QA criterion for each feature (namely critical ones) 16:15:12 okay, you're still off in the weeds frm the topic we're actually meant to be on... 16:15:27 we have a feature topic coming up later 16:15:43 anything else for the criteria presentation? 16:15:44 one more: tags. I hate when I search for PXE, but it is called "network boot" or similar 16:16:04 #info kparal - some mechanism for tagging criteria - "I hate when I search for PXE, but it is called "network boot" or similar" 16:16:06 after the criterion there can be a few tags like "[PXE]" 16:16:10 sorry, i guess im always thinking far too ahead 16:16:18 you gotta let us catch up with you, dan :) 16:16:29 step away from the delorean 16:16:35 inorite 16:16:40 im already running rawhide 16:16:49 is the criteria something we are going to add to the qa blocker proposal page then text with tags and archive the wiki page ? 16:17:10 Viking-Ice: possibly but not for F19, I don't think 16:17:18 Viking-Ice: well that's a possibility, but i was just planning to revise it within the wiki for now 16:17:49 well is there anything burning on a stove right now? 16:18:01 apart from my pancakes? nope 16:18:05 cool 16:18:21 #info criteria revision proposal is just wiki-based for now, but there's a possibility we could move the criteria into the blocker tracking app in future if it gives us useful results 16:18:32 #action adamw to draft up a proposal for revising the presentation of the release criteria 16:18:35 no dates no nothing set in stone yet? 16:19:01 dan408: for f19? fesco didn't finish the proposed feature list yet i don't think 16:19:03 here 16:19:05 #link https://fedoraproject.org/wiki/Releases/19/Schedule 16:19:06 which reminds me that I still haven't written anything down about ideas for changing the blocker tracking process as a whole yet (longer term than F19) 16:19:14 fesco is still voting on features 16:19:29 f19 branch end of feb 16:19:30 yeah, they're still reviewing features and AFAIK, are planning to look at proposed schedules on wednesday 16:19:43 they've said they're going to do the schedule when they're done with features, but they're committed to making sure we have a reasonable gap between 'schedule set' and 'alpha release' 16:19:48 nothing has been finalized yet, though 16:19:58 yeah, what's on that page right now is probably out of date 16:20:02 i proposed a spin, and i dont know if it ever made it to the spins list, jreznik told me the spins process is kind of fubar'd 16:20:12 okay, this is also not the spins meeting 16:20:23 i think we've got the criteria, next topic! 16:20:30 #topic Test case revision 16:20:44 dont we have a feature list of what has been accepted since last meeting which we can iterated through 16:21:05 Viking-Ice: there's a topic for that later, see agenda 16:21:18 so boblfoot added this, with the comment "Revising of test cases during the F18 Final Test Phase was less than optimal - how to avoid with F19?" 16:21:37 i guess the takeaway is we should try to make sure we're happy with the test case set by Alpha, as much as possible 16:22:04 as well as the criteria I would say 16:22:04 are there any areas people are particularly worried about wrt the test case coverage? 16:22:06 yup 16:22:24 no 16:22:31 * adamw gives tflink a donut 16:22:33 the only one I can think of is graphical upgrades, once the client is finished 16:22:56 we could probably add a couple other upgrade test cases if we wanted, now we don't have to test both anaconda and preupgrade 16:23:08 it might be nice to cover the encrypted upgrade case at least, and maybe a couple of package sets 16:23:31 #info adamw - maybe expand upgrade test case set a little 16:23:36 yeah, I think the main question is how to do that without duplicating the test case 10 times :) 16:23:42 :P 16:24:02 * dan408 proposes a wait and see atitude 16:24:03 but that's not a problem unique to upgrades 16:24:05 one thing i'd like to see but realistically may not have time to get done would be some kind of set of storage test cases 16:24:07 attitude* 16:24:17 I think some partitioning criteria could be improved, like "should not crash for invalid operations" in custom part, but not in guided part 16:24:29 there's no way we can cover everything, but it might be kinda nice to cover a few btrfs, raid, lvm, install-over-previous-partitions kinda thing 16:24:30 I think we need to separate test cases into two categories hw related and not hw related 16:24:34 they promised to fix a lot of things for 19 16:24:53 kparal: yeah, that's what tflink mentioned earlier, the partitioning criteria are kinda horrible - they're stuck in the middle of being revised, i was working on it last cycle and just never finished 16:25:05 * satellit liveinst still not fixed for Soas spin...: ( 16:25:09 Viking-Ice: they're categorized already but the categories are somewhat old and we could look at revising them 16:25:22 there is no point in me adamw tflink dan kparal conducting the same test cases at the same similar time since the outcome can be expected to be the same unless they rely on different hw 16:25:39 i refuse 16:25:58 i want to see what they propose first 16:26:33 dan408: the feature for anaconda changes for f19 has already been accepted, you can check it out. there aren't any giant changes besides the re-addition of enterprise storage. 16:26:47 lovely 16:26:48 #info viking-ice - identify hw-dependent and non-hw-dependent test cases 16:27:04 #info kparal - finish revising the partitioning criteria 16:27:19 #info adamw - come up with a set of common partitioning operation test cases 16:27:24 lots of nice ideas there :) 16:27:40 any others? 16:27:40 how about fixing UEFI 16:27:54 ah, SB criteria, anyone? 16:27:58 fixing UEFI? 16:27:59 not showing dd USB booted from in anaconda 16:28:25 kparal: arguably covered by "The installer must boot (if appropriate) and run on all primary architectures, with all system firmware types that are common on those architectures", but maybe explicit might be better 16:28:33 #info kparal - explicit Secure Boot criterion? 16:28:35 tflink: yes 16:28:44 dan408: care to be a bit more specific? 16:28:47 kparal: that's the kind of thing we may be able to do better with the new criteria presentation if i get it right 16:29:00 pjones: lots of users having issues with both UEFI and secureboot installs 16:29:13 dan408: how is that related to criteria? 16:29:18 nevermind 16:29:18 off topic! off topic! awooga. 16:29:32 dan408: They really ought to file bugs, but as noted that's not on topic 16:29:39 * jskladan_ sometimes hates all teh internetz providers... 16:29:44 man, spot the people with irc highlighting :) 16:29:45 kparal, nr1 test case for SB check if we have certificate that does not expire during the release life time ;) 16:30:27 Viking-Ice: shouldn't make any difference. validity dates in SB aren't supposed to be being checked. 16:30:29 okay, any more criteria ideas? 16:30:37 er, or test case. 16:30:38 pjones, ok 16:30:45 not really like i said play it by ear 16:30:47 (and the ones in the cert that signs shim aren't under our control anyway, so criteria won't really help.) 16:31:01 Viking-Ice: that seems a bit too specific and corner-casey to have an actual test case for it. I'm not saying it isn't an issue, just not sure it needs to be checked for every milestone's TC/RCs 16:31:48 okay then, moving along 16:32:02 oh 16:32:15 #info we should try and get criteria revision and test case revision done before Alpha as much as possible 16:32:18 #topic Fedora 19 Feature list review 16:32:27 https://fedoraproject.org/wiki/Releases/19/FeatureList 16:32:34 tflink: did you have anything prepared for this? 16:33:12 we somehow need to flag features we have already covered 16:33:19 there's a list on the agenda 16:33:22 https://fedoraproject.org/wiki/QA/Meetings/20130204#Fedora_19_Feature_list_review 16:33:25 "Previous List" 16:33:49 adamw, the previous list is just what we thought we should keep an eye out for not all the feature we covered 16:34:18 as in total features we covered last week vs a slimmed list what has been approves since then 16:34:48 it could be totally different per feature, and with features growing massively i mean we have to decide on a few basic criterion 16:35:04 Viking-Ice: right, sorry, should've prepared something 16:35:18 adamw: nothing specific, no. just figured it would be a good thing to bring up 16:35:27 https://fedoraproject.org/w/index.php?title=Releases/19/FeatureList&diff=321788&oldid=320463 looks like the change to FeatureList from last time 16:36:03 i can see a few things on there that look like good test day candidates 16:36:22 hmm the feature process should actually somehow tag which criteria each feature hit ( if any ) 16:36:23 how do we test 3d printing? 16:36:30 dan408: we don't! 16:36:30 we dont 16:36:45 awesome! 16:36:49 ReplaceMySQLwithMariaDB has been discussed on-list a bit - obviously significant, we don't have much infrastructure for testing it at present 16:36:59 adamw, not criteria 16:37:04 yeah 16:37:39 adamw, I would say it's ours to cover criteria items it's theirs ( feature owners ) to cover what's related to their feature 16:37:42 Viking-Ice: well the 'criterion' to me is pretty simple - anyone using mysql in f18 should get transparently migrated to mariadb in f19 and stuff should keep working 16:37:58 symlinks? 16:37:59 Viking-Ice: well, we're not _just_ doing release validation, qa's remit is broader 16:38:10 dan408: the technical side isn't our problem 16:38:16 adamw: that sounds huge, though - how would we approach it? 16:38:22 but yeah, that one doesn't likely affect the criteria directly 16:38:42 tflink: you expect me to have *solutions*? wrong number! i only point out problems. 16:38:55 i cant remember the last time i ran mysql anyways 16:38:58 adamw, have the feature owners provides us with proper testing/debugging instruction or is that just added workload writing that for us so we can properly cover those features? 16:39:26 Viking-Ice: https://fedoraproject.org/wiki/Features/ReplaceMySQLwithMariaDB#How_To_Test 16:39:29 so, not much. 16:40:18 I wonder if some generic mysql benchmark might work well here 16:40:21 #info https://fedoraproject.org/wiki/Features/ReplaceMySQLwithMariaDB is clearly a significant change but doesn't directly hit the release criteria and it's pretty hard for us to come up with a reasonably scoped test plan 16:40:41 ie, if the benchmark ran on both mysql and mariadb, that is a decent indication of compat. 16:40:44 adamw, yup we need to somehow flag "feature testing day" with only the need to point reporters to the relevant feature and follow what is instructed there 16:40:44 right 16:41:06 #info tflink - maybe if we can find a comprehensive mysql benchmark suite and check that runs, it would indicate a good level of compatibility 16:41:42 Viking-Ice: with the quality of 'testing instructions' we get i think having one day for that for all features would be unworkably broad :/ i think it works better to try and schedule test days for specific features of concern 16:41:47 we should definitely have a test day for this one for e.g. 16:41:53 if mariadb will replace mysql on upgrade we need to cover upgrade tests 16:42:00 i need to set up the test day stuff for f19 cycle, it's on my todo list for this week 16:42:04 Viking-Ice: it does, and yeah 16:42:21 i actually got mariadb in my rawhide update the other day so that part's already working in at least a simple case 16:42:34 you need to test it with live data 16:42:42 #info viking-ice - make sure we cover testing upgrade from f18 to f19 achieves the mariadb swap cleanly 16:43:06 samples ( live data samples ) 16:43:12 sure 16:43:30 has mysql been orphaned ? 16:44:00 I thought that one of the issues with that proposal was the part about not allowing mysql any more 16:44:01 if we still ship mysql in f19 I assume this effort is not necessary 16:44:10 or am I getting my features confused 16:44:31 tflink, there where some mixed signals with that 16:44:46 yeah, it seems to keep changing 16:44:53 i don't recall precisely what fesco decided 16:45:01 mysql is kind of dead 16:45:03 on one hand oracle offered to maintain mysql on the other it look like remi hhorak wanted them to fix some issues 16:45:15 and they might continue to maintain that 16:45:34 "AGREED: Feature accepted as proposed (including obsoletion as a transition mechanism). If new MySQL maintainers appear, feature owners are asked to make it possible to install the MySQL stand-alone server (only)" 16:45:57 that's from the last fesco meeting. 16:46:26 I think that all was being decided that no one was going to be picking up mysql and maintaining it 16:46:26 okay, another one that pops out is Features/SystemdPredictableNetworkInterfaceNames 16:47:03 i believe that at least it's been agreed to make the Shiny New Scheme use em0 for most 'default onboard' devices so that won't change from biosdevname, which is a win 16:47:26 so i guess we should make sure that happens, and make sure the new mechanism actually works, and i guess test that whatever's supposed to happen on upgrade happens 16:47:32 any other things people can think of to check there? 16:47:33 adamw, we covered that one last meeting it would be good to get a clear statement from fesco saying we will not be releasing fedora 19 with mysql 16:47:41 oh sorry 16:47:59 Viking-Ice: you could probably just ask for clarification on the devel list there 16:48:03 reply to the fesco minutes post 16:48:04 Viking-Ice: I thought that F19 would have mysql but that it was going away for 20 16:48:15 Viking-Ice: ah yeah, predictable names was covered indeed. 16:48:29 tflink: so clearly we do need clarification :) 16:48:40 adamw, iptables/firewalld/arptables etc + something like libvirt might be doing something with hardcoded interface are there first thing that pop up to mind 16:48:45 #info if mysql is to still be available in f19, check it can be installed and used 16:49:04 adamw, if mysql is still available we do nothing 16:49:06 Viking-Ice: yeah, be good to see what happens with firewalld on upgrade 16:49:26 vm's might be something to look at - IIRC, it was a bit of an issue with biosdevname 16:49:30 adamw, if mysql is not available we do migration tests ;) 16:50:04 #info on predictable network names - check behaviour of firewalld and libvirt, particularly on upgrades; check 'em0' is still used by new scheme 16:50:39 anyone see any other features that raise red flags? 16:51:25 kscreen, maybe? I assume that the kde folks will have that covered, though 16:51:37 worth keeping an eye on for kde validation 16:51:44 * satellit OEM install in anaconda (covered I think) 16:51:45 #info tflink - KScreen might be worth keeping an eye on for KDE validation 16:51:52 what about syslinux 16:51:58 oh, good one 16:52:27 #info https://fedoraproject.org/wiki/Features/SyslinuxOption also worth testing 16:52:30 potentially the share system certs 16:52:34 shared 16:52:46 it looks like it'll be pretty hidden, but if we have time it'd be good to check it works (syslinux) 16:52:47 we probably should add syslinux to criteria 16:52:55 Viking-Ice: i think maybe for the first release it's in there we don't need to 16:53:10 since it's only going to be available via kickstart or a secret anaconda parameter 16:53:20 it might be good as an optional test case though 16:53:25 well from my pov we should dot that from the start 16:53:38 do we really want to block the release for something that's basically a hidden feature, though? 16:53:55 installer 16:54:10 you won't be able to just pick syslinux in the installer interactively 16:54:20 no but ks 16:54:49 eh...personally i'd still vote against it as a criterion but you could draft it up for the list and see what people think 16:55:09 otherwise we can just test it and maybe consider bugs for freezeexception 16:55:15 sorry not following dont we have to cover all ks options 16:55:39 I don't think it was all, just a subset considered important enough to block over 16:55:42 right 16:55:47 with a yet to be determined subset 16:55:47 and by we I mean autoqa ( since ks snippes should be automated ) 16:55:53 actually that's something we need to put an action item on 16:56:00 * satellit also USB .ks ? 16:56:12 we agreed in f18 validation to take kickstart stuff on case-by-case basis, but draft up a proper criterion for f19 16:56:23 but in general we didn't want to block on every possible kickstart option, just some subset 16:56:29 does someone want to take the action item for that or should I? 16:57:02 guess that's me then! 16:57:10 #action adamw to draft up a kickstart criterion for f19 16:57:29 #info consider whether syslinux should come in the subset of blocking kickstart options 16:57:31 well I think autoqa should be able to cover ks installation completely 16:57:39 in an ideal world, sure 16:57:43 Viking-Ice: feel free to submit patches 16:57:47 in practice, we have enough trouble keeping the simplest subset working 16:58:12 it's an obvious area for development for autoqa, but we can't rely on it yet 16:58:19 due to anaconda breakage or how autoqa is handling it 16:58:36 ( the trouble keeping simplest subsystem working ) 16:58:39 i dunno if anyone's looked at exactly what's not working in rats lately 16:58:51 we only really have tflink and kparal for autoqa and they have other things to work on too :/ 16:58:54 last time remote syslog was broken in anaconda 16:59:08 anything that autoqa depends on would have to be an automatic blocker 16:59:16 regardless criteria 16:59:19 personally, I think that RATS is broken on a conceptual level, but that's me 16:59:21 we're getting close to our hour here 16:59:37 we can put followups to this on the agenda for next week if people want - any specific topics? 16:59:40 it's not like I have something to replace it right now 16:59:55 kparal, I would assume anaonda defaults the online systemd syslogger on for f19 so fetching that log should be easy via some curl magic 17:00:00 we've got lots of plans for autoqa work, but it all blocks on having the people to do them 17:00:13 what adamw said 17:00:43 okay, if anyone wants to add a specific topic from the above to next week's agenda, do go ahead and put it in the wiki 17:00:52 I can't speak for anyone else but I'd love to work on autoqa but the blocker tracking stuff is a higher priority ATM 17:01:07 let's finish up for this week 17:01:09 #topic open floor 17:01:13 anyone have things for open floor? 17:01:29 hm I probably should go ahead and file an rfe for that... 17:01:52 go for it 17:02:52 if there's nothing for open floor, setting fuse for 1 minute..hisssss 17:03:19 snakes?! 17:03:25 * tflink runs away 17:04:05 yesss...chase him, my pretties 17:04:11 ok, thanks for coming everyone! 17:04:25 all criteria / test case proposals welcomed if people want to work on anything that came up above :) 17:04:40 i may be out several days this week, btw, looking like a good snow week, so i may not get all my action items done 17:04:53 same time next week 17:04:55 #endmeeting