16:00:20 #startmeeting Fedora QA meeting 16:00:20 Meeting started Mon Feb 2 16:00:20 2015 UTC. The chair is adamw. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:20 Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:00:23 #meetingname fedora-qa 16:00:23 The meeting name has been set to 'fedora-qa' 16:00:27 #topic Roll call 16:00:27 * pschindl is here 16:00:28 ahoyhoy 16:00:44 adamw: good morning 16:00:52 * mkrizek is here 16:00:53 * kparal is here 16:00:53 * roshi is here 16:00:57 * tflink is here 16:01:28 * satellit listening 16:01:46 #chair satellit kparal 16:01:46 Current chairs: adamw kparal satellit 16:02:59 * smccann is here 16:04:15 * adamw was expecting a flaming-torch-and-pitchfork turnout today 16:04:40 :P 16:04:47 how man more people are needed for that? 16:04:52 * tflink knew he forgot something 16:05:05 * roshi is used to things like, dozen, couple, few... 16:05:05 I can make it to home depot and back in 20 minutes, though 16:05:18 well, not more people, just more flaming torches, pitchforks, and effigies of anaconda developers 16:05:57 I have a shovel and a turkey frier, can that work as a standing for my torch and pitchfork? 16:06:09 s/standing/stand in/ 16:06:18 it's the thought that counts 16:06:21 allllrighty 16:06:25 #topic Previous meeting follow-up 16:06:52 "kparal to check in on Preupgrade Assistant feature, what testing is desired and how heavily it will be promoted" - kparal? 16:06:53 * tflink duct tapes his camp stove to the end of a shovel 16:07:16 yeah, so I talked to... remembering name... 16:07:33 Petr Hracek 16:08:03 he would love to have this tool mention in official upgrade guidelines 16:08:04 * danofsatx is hear 16:08:18 hi dan 16:08:19 so if he succeeds, it will affect us 16:08:42 have you asked fesco about their take on it? 16:09:33 no I didn't, should I? I guess he should initiative such a ticket, it he wants to push it that way 16:10:38 just figured that sort of thing should be reviewed as part of the Change... 16:10:47 anyway, if I understood it correctly, it should be just a tool which collects some information about your current system and prints various helpful information about what's going to change in the upgraded system and how you'll be affected. so it should affect the system anyhow, change anything 16:10:55 assuming I understood it correctly 16:11:11 also it will need some content from "providers", which should be major system parts maintainers 16:11:18 like systemd I assume 16:11:18 i see 16:11:26 he said there's no content there yet 16:11:54 so, we should check it out, but it's not likely to actively break anything...and i guess worst case we can always just drop it from the docs/publicity 16:11:58 so this tool should not add significant QA effort, I assume. it's just a helper 16:12:24 thanks for that 16:12:28 documentation helper 16:12:33 sure 16:12:56 #info "kparal to check in on Preupgrade Assistant feature, what testing is desired and how heavily it will be promoted" - the Change owner wants to tool to be promoted in official docs, but its impact is low (just an information provider) - we should check in on it later in the cycle 16:13:35 "roshi and corey84 to try and work up some methodical testing plans for dnf" - roshi? nice small job like this should be done by now, right? :) 16:13:41 haha 16:14:01 oh yeah, and all the testing is done - I declare things Notionally Complete :p 16:14:44 awesome, activate Project Colada stage 2 16:15:08 I spent last week getting to know all the points yum is used throughout the whole project, narrowed it down to what we as QA will want to actually test 16:16:05 cool 16:16:15 the idea is to have a Project Wide overview, just so people from other teams can take a look and see if things are covered 16:16:20 it's a far from trivial change 16:16:21 are you writing any kind of notes as you go along that could go on a wiki page or anything? 16:16:54 for us, ABRT, FedUp and packageKit/Apper will be the big things 16:17:20 yeah, I have notes, but it's a scattered spreadsheet at the moment - not for human consumption 16:17:29 I'll try to get something written up this week 16:18:04 don't spend too much time on that, i just figured if they were in shareable form anyway it'd be nice 16:18:08 but most important thing is to get the job done... 16:18:27 I thought getting something written up was "done" ;P 16:18:43 #info "roshi and corey84 to try and work up some methodical testing plans for dnf" - this is a long-term job but so far roshi has been identifying areas where yum is currently used and trying to stake out a test scope 16:20:43 #info "adamw to revise the package set criterion proposal again and move forward with multiboot proposal" - I went ahead and put the package set proposal in place last week, multiboot is still pending but pjones signed off on the tl;dr summary in that last email i sent 16:20:52 we have a topic for this later too 16:21:28 #info "adamw to document 'efibootmgr -n' workaround for failure to boot windows from grub with SB enabled (#1170245)" - did that, https://fedoraproject.org/wiki/Common_F21_bugs#grub-secure-boot 16:23:48 #topic Anaconda "test day" planning 16:23:56 * amita joining late 16:24:00 hi amita 16:24:15 do you need article for Anaconda "test day" 16:24:19 so, dcantrell (the anaconda team's manager) asked me last week about putting together a 'test day' relating to anaconda CI 16:24:28 amita: probably not, but wait up while i explain :) 16:24:33 sure 16:24:40 * danofsatx thought anaconda was a bad word now 16:24:58 this is what they asked for: 16:25:09 "I'd like to see us work through as many of the test cases you guys use for Fedora QA as well as our own and:" 16:25:16 "* Turn them in to Jenkins test cases we can use in our CI environment" 16:25:24 "* File as many upstream DNF bugs as possible and tag with 'anaconda' to help them prioritize the work (requested by DNF)" 16:25:32 "* Try to get a better handle on the general usability of DNF as the installer backend in rawhide at the moment." 16:25:57 isn't their CI environment non-public? 16:25:57 i'm not totally sure how points 2 and 3 relate to point 1 there, but in any case it seems like they'll want fairly experienced release testing folks 16:26:13 let me see if we can grab someone from #anaconda to chat 16:26:56 on condition that no-one yells at them about passwords. :P 16:27:24 hi, dacntrell 16:27:49 dcantrell: so far i just explained what you asked for (i pasted the bullets from your first email), and tflink asked "isn't their CI environment non-public?" 16:28:20 yes, our CI environment is in the installer lab in Westford 16:28:33 but that really shouldn't affect a test day effort 16:28:45 what is the CI capable of, and how can we help to create those new test cases? 16:28:56 what I want from the outcome of the test day is an enumerated set of tests which we can put in to the CI system for continued use 16:29:27 kparal: right now it's most useful for automated install tests where we can define the test using kickstart 16:29:51 so basically you're looking for kickstarts that cover specific validation testing cases, and expected outcomes? 16:29:53 kparal: what I want from Fedora QA is what sort of tests you guys run through during testing. perhaps some of those can be defined as tests for our CI system 16:30:00 adamw: right 16:30:03 hi bcl 16:30:10 that's one part of what we're after 16:30:19 sure, we can do that. how do we know what you have already? 16:30:20 I would say the more important aspect of a test day right now is to hammer the DNF backend 16:30:36 right, i wasn't totally sure how those two bits of the scope added up 16:31:12 adamw: so think of the test day request more of "we want to hammer DNF and make sure it works, oh and a nice side effect of this is maybe we can collect some test cases to add to the CI environment" 16:31:22 when you say 'hammer the DNF back end' you want us to all get together and just do exploratory manual testing on it, messing around with repos and package sets and stuff? or were you thinking of coming up with CI tests for that during the event too? 16:31:22 is there any further thoughts/decision on python3 anaconda/dnf? 16:31:38 is dnf enabled by default or do we need to force it somehow? 16:31:44 kparal: I can collect what we have right now and post it somewhere, but there's not really much. that's why I'm wanting to get a list or description of what Fedora QA tests 16:31:58 kparal: DNF should be enabled by default right now, bcl can verify for me 16:32:09 kparal: since ~20141210 it's in use by default 16:32:10 yep, it has been on for a couple months now 16:32:25 nirik: the main reason for this test day coordination request is so we can work through dnf and python3 bugs in anaconda 16:32:43 dcantrell: basically our entire set of planned tests is https://fedoraproject.org/wiki/Test_Results:Current_Installation_Test . 16:32:46 is there a specific rawhide build you want to use for testing, or are we going to need to get a special image for the testday? 16:32:47 nirik: we're wanting Fedora QA's help before we hit the alpha freeze deadline 16:32:51 adamw: thanks 16:32:56 dcantrell: ok, so python3 will be landed this cycle? 16:33:06 dcantrell: or https://fedoraproject.org/wiki/Template:Installation_test_matrix , same list really. 16:33:07 or still TBD 16:33:31 nirik: that will be determined based on the outcome of this coordinated testing 16:33:32 dcantrell: do you have some specific date in mind for this test day? 16:33:54 to me it seems like the earlier the better 16:33:55 ok, so the testing will take place with python3 enabled anaconda / dnf then I assume? 16:33:55 kparal: as soon as possible 16:34:08 * roshi can help figure out some of the details and help get the wiki pages and whatnot around 16:34:18 * nirik doesn't think it should land this cycle, but perhaps it will all go smoothly. ;) 16:34:47 in an ideal world, we'd only change one thing at a time while testing :p 16:35:05 nirik: we don't have a py3 anaconda/blivet yet. 16:35:06 feb 5 - 10 is not optimal, conferences 16:35:15 So I'm not sure how that'll work. 16:35:30 Feb 12, perhaps? 16:35:43 bcl: well, it was dcantrell who just said you wanted to "work through python3 bugs in anaconda", which seems difficult without a python3 build of anaconda :P 16:35:48 yeah. also dnf-3 hasn't really gotten much testing... which makes all the years of testing dnf kinda not as useful 16:36:02 wait, sorry, I am combining python3 and dnf in my discussions here 16:36:06 the testing should be around DNF 16:36:10 dnf is the focus here 16:36:13 dcantrell: did you have an image in mind, or does it need to be created for the testday? 16:36:14 ah, ok. 16:36:34 roshi: assuming nightly generation is working again by then, we can probably use a nightly. 16:36:36 if dnf is not usable for the install test cases that Fedora QA cares about, we need to file bugs upstream and then use that information to make a decision as to whether or not dnf is ready for f22 16:36:54 so, does Feb 12 sound OK for you? 16:36:55 so far all of the "let's move to dnf" discussion has been made without any real empirical evidence as to the functionality 16:37:01 Feb 12 is fine with me 16:37:07 ok 16:37:33 sounds good 16:37:34 roshi: let's just run with rawhide for now 16:37:43 works for me :) 16:37:44 pschindl: Feb 12 for you? 16:37:51 creating a special build seems like too much extra work right now 16:37:52 thats after branching, but otherwise... 16:37:53 #info main focus of proposed 'test day' is to exercise DNF and check whether it's in good enough state to be used for F22, tests for anaconda's CI would be a useful by-product 16:38:09 dcantrell: i think we'll be on Branched by then, but that's just a note, it shouldn't make a practical difference 16:38:19 sure 16:38:25 #info we will aim to set this up for 2015-02-12 16:38:32 I think we QEcamp at tht time, not sure if that matters 16:38:45 amita: that's the 9th and 10th 16:38:59 i guess i'll take the fall for this one... 16:39:13 #action adamw to organize/publicize anaconda/DNF test day 16:39:15 kparal: Feb 12 is ok 16:39:27 ok 16:39:57 adamw: if you're going to send an announcement, please put it into fedocal as well 16:39:58 so as it seems we'll be aiming to do some broad manual testing, it probably will be useful to have publicity...amita, i'll get in touch once i've got a bit more concrete stuff in place so we can organize an article 16:40:04 kparal: thanks, good point 16:40:20 dcantrell: bcl: this all sound OK for you? anything else to add? 16:40:21 adamw, sure 16:40:46 kickstart testing is good as well, and easier to translate into test cases. 16:41:03 adamw: this is sounding good. we'll get coordinated on our side. problems we find we'll want to triage and get assigned to dnf or the right component. I expect this to be bumpy, so testers should be prepared for that 16:41:15 eh, i was kinda including that...writing a kickstart on the spot and firing it off is pretty much manual testing, for me. anyhow 16:41:21 bcl: just to make sure, there's no point in e.g. gui testing which can't be translated into kickstart, right? 16:41:23 we're always ready for bumby 16:41:30 kparal: i think we need to do that too. 16:41:46 in general sure, but for this CI effort... 16:42:04 gui testing is fine, but since the focus is on dnf, whatever we can use to drive the dnf backend is key. the %packages section in kickstart allows more customization than the gui, for instance 16:42:06 kparal: as i understood it it's primarily a 'check out dnf' effort and secondarily a 'produce CI tests' effort. 16:42:07 we should basically produce a handful of kickstarts? 16:42:20 ok 16:42:38 dcantrell: the GUI code itself hooks into the package manager library for a bunch of what it does, so we do need to test all the GUI interactions too. 16:43:06 dcantrell: at least a couple of the bugs we hit with DNF so far are in the GUI - see https://bugzilla.redhat.com/show_bug.cgi?id=1177002 for e.g. 16:43:28 adamw: fair enough, definitely should be inclulded 16:44:59 ok, we can bash out other details as we go along, i'll set up a trac ticket and a skeleton wiki page asap 16:45:03 thanks, bcl/dcantrell 16:45:08 related: can folks test dnf-3 as much as they can. Last I heard dnf maintainers wanted to switch to that by default soon. (so yes, if anaconda stays python2, it would use python2 dnf and everyone else command line will be using python 3 dnf). 16:45:27 good grief. 16:45:33 :) 16:45:59 adamw: np 16:46:20 #topic Release criteria / test case revisions 16:46:30 also, on the python3 subject, we are still on python2 primarily because of the libuser python module. it has not been updated to python3 yet 16:46:33 just an fyi 16:46:39 bcl: you might be interested in this too i guess, just to know what we have in scope for f22 blocking 16:47:19 hey, I'm not the f22 lead I just do rawhide right now ;) 16:47:24 so in case anyone had trouble following the 'package set' proposal as it went along, i'll give a quick summary of what i put in place so anyone can point out any problems 16:47:58 for alpha, we basically just have 'install out of the box' for all release blocking images - whatever the default package set, it should work 16:48:04 that's no different from f21 in practice 16:48:37 (except we'll now have the couple of new media: workstation netinst and 'generic' netinst. so more testing. yay.) 16:48:48 :D 16:49:27 yippe. I'm having trouble containing my excitemnet. 16:49:34 for Beta, we expect the default package set for each blocking image to be 'correct', and package set selection to work on the 'generic' image 16:49:37 danofsatx: :P 16:50:20 technically i think we never explicitly required those before, but it was kinda an oversight; if either had been wrong we'd probably have raised a blocker somehow or other 16:51:11 for Final, we expect minimal install from the 'generic' image with no updates repository enabled to work 16:51:20 sorry for being late, catching up with stuff after I came back from FOSDEM - we talked with mkolman and he actually started planning test day for anaconda/dnf 16:51:33 we *could* require more there, but since the Products have their own netinsts it seemed unnecessary 16:51:46 jreznik: ah, ok - i'll co-ordinate with him then...thanks for the heads-up 16:52:08 adamw: why no updates repo? 16:52:23 kparal: because the updates repo can be...updated 16:52:46 in practice i'm sure we'll also check that it works with the packages in 'updates' at release time and yell if it doesn't, but the thing that we can't fix post-release is the frozen package set 16:53:18 I meant that it will be empty until release, so we don't really need to insist in needs to be disabled in the criteria 16:53:19 and it *is* plausible that someone might set up some kind of large scale unattended install process that they really need to be sure will always work, and it'd make sense to use the frozen repo for that 16:53:49 kparal: that could possibly change (as we discussed in relation to the upgrade testing thing) and there's the timing of when exactly the installer stops using updates-testing to consider as well... 16:54:00 ok, fair enough 16:54:06 adamw, so like the historic fedora folder 16:54:52 so in practice i think that keeps the test coverage manageable... 16:55:26 i added a requirement in https://fedoraproject.org/wiki/QA:Testcase_Boot_default_install that the default package set be 'correct' (to check that criterion) 16:55:40 and i resurrected https://fedoraproject.org/wiki/QA:Testcase_Package_Sets_Minimal_Package_Install to cover the Final criterion 16:56:01 and added the two new images to the "Default boot and install" table in https://fedoraproject.org/wiki/Template:Installation_test_matrix 16:56:08 that's the extent of the changes so far as testing goes 16:56:42 does anyone see any issues with that stuff? 16:57:21 * roshi doesn't 16:57:35 but we can adjust as we go if we find something 16:57:37 sure 16:57:56 so other than that the multiboot revision is still on the go, i'll try and finish that this week 16:58:37 and there's a trac ticket with a bunch of places we still need test cases to match criteria, if anyone wants something to work on - https://fedorahosted.org/fedora-qa/ticket/397 16:58:54 but it's nearly top of the hour, so let's do a quick open floor then kick over to blocker review... 16:58:56 #topic Open floor 16:59:03 anyone have anything we didn't cover? 16:59:31 we could talk about using those QA Contact fields in bugzilla for blocker bug meeting purposes, but we have 1 minute left 16:59:32 * roshi doesn't 16:59:54 oh yeah, good call kparal 16:59:59 +1 from me 17:00:31 should we discuss this during blocker bug meeting instead? 17:02:06 works for me, only have a couple blockers to go through 17:02:10 should be a short meeting 17:02:23 yeah, sounds good 17:02:31 oh, i totally forgot the planned OpenQA topic 17:02:32 go me! 17:02:56 really ,, short blocker bug meeting.. surprise :) 17:02:57 jskladan is not here, we can discuss it next time 17:03:22 OpenQA ? 17:03:32 #info some of the folks in RH's Brno office have been working on automating some of the release validation tests with OpenQA, and they've got quite a bit up and running already - code can be found at https://bitbucket.org/rajcze/openqa_fedora_tools/src/d48b884aa1244f5acbb9cad052f5b8a765cc08ad/tools/openqa_trigger/?at=master . we'll talk about it more next time! 17:04:00 nice 17:04:32 amita: it's a test automation thingy that SUSE wrote, https://en.opensuse.org/openSUSE:OpenQA 17:04:47 http://os-autoinst.github.io/openQA/ 17:04:50 yeah, got it 17:04:57 amita: one of the SUSE folks got some fedora tests running in it as a 'fun project' over the holidays, and some of the brno guys were inspired to try it out too 17:05:11 note it's not the one at http://www.openqa.org/ that's dead :) 17:05:20 ok folks, thanks for coming everyone! 17:05:30 next up: blocker review in #fedora-blocker-review 17:05:50 #endmeeting