21:12:11 #startmeeting autoqa - will woods 21:12:11 Meeting started Sat Dec 5 21:12:11 2009 UTC. The chair is adamw_. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:12:11 Useful Commands: #action #agreed #halp #info #idea #link #topic. 21:12:16 fedora has problems: lots of things break, all the time: automated testing will help find and solve problems 21:12:44 breaking down the problem: what is rawhide, what should it do -> rawhide acceptance test plan 21:12:58 small enough problem to start solving: jfdi 21:13:16 hackfest for autoqa 0.1 at previous fudcon: simple repoclosure test - check if repositories are sane when they are generated 21:13:23 moving to existing autoqa design 21:13:36 "when something happens run some tests" 21:13:57 event hooks: events you can trigger tests on, and what tests need to know about the event 21:14:47 ?: rats - 15 items on the list - public? 21:15:12 wwoods: yes, it's on the wiki (google rawhide acceptance test plan) 21:15:13 defines what needs to be tested and why 21:15:15 8/14 items are automated 21:15:43 design: event watcher for each hook 21:15:43 hook can be: repo updating, tree build, package build etc 21:15:56 https://fedoraproject.org/wiki/Category:Rawhide_Acceptance_Test_Cases 21:15:58 watcher is run and tells you what has happened since the last time it was run 21:17:51 harness is a script called by the watchers when some event has happened 21:17:53 (autotest interface shown on screen, running post-koji-build tests) 21:17:55 tests are already being run every time a package is built 21:18:01 autotest is used as autoqa's harness: comes from google 21:18:05 current hooks: post-repo-update , post-tree-compose , post-koji-build 21:18:31 greg: does autoqa determine success and failure 21:18:37 wwoods: to some degree yes, but not extensively yeet 21:18:48 aiming to allow tests to be run depending on status of previous tests 21:19:02 current tests: post-repo-update: conflicts, repoclosure, rats_sanity 21:19:08 checks for conflicts, dependencies 21:19:32 post-koji-build: rpmlint, rpmguard - check for problems and significant changes in new packages 21:19:53 autoqa python library: useful python modules that can be used across multiple watchers and tests 21:20:24 israwhidebroken.com: planned to be a specialized autoqa reporting system which implements a web page which tells you whether rawhide is broken 21:20:55 will be used to keep 'last known good' rawhide images 21:20:55 goal is to never have non-functional rawhide installation images 21:21:03 (screenshot of test israwhidebroken.com page on screen) 21:21:32 greg: is there enough data in the tests to know who broke something? 21:22:00 does israwhidebroken currently correctly show that rawhide is broken? ;) 21:22:11 wwoods: not automatically - for some tests you could tell (e.g. comps.xml validity), for some it would be difficult 21:22:23 er, i dunno :) hand raise? 21:23:03 * nirik can raise his hand, but I am not in the same country so I doubt it would do much good. ;) 21:23:15 nirik: it's not currently running because there's no rawhide install images because of nofrozenrawhide 21:23:21 ah, ok. 21:23:26 autoqa needs to talk to installer team about nofrozenrawhide 21:23:30 (that's the reply from will :>) 21:23:34 * nirik notes rawhide hasn't composed the last 2 days... 21:24:40 coming soon: public autotest instance 21:24:40 post-bodhi-update hook: for testing update and candidate update packages 21:24:40 coming soonish: easy autoqa for packagers: add tests right in pkgcvs so they can do it integrated with the package maintainer flow 21:24:41 e.g. font tests - could be added into each font package 21:27:38 post-iso-build hook: for testing installation of newly generated nightly live images e.g. 21:27:38 use Fedora Message Bus to replace watcher scripts 21:27:38 also allows two-way communication 21:27:38 the test can communicate failure back to koji (e.g.) and package blocked 21:27:40 coming some day: multi-host testing (for network testing): no current idea of how to do this 21:27:46 functional testing for GUIs: have to be able to script GUI interactions 21:27:52 getting involved: may be Coming Soon, switching to second presenter 21:28:08 red hat internal automated testing: different cases 21:28:10 e.g. testing a new kernel across 30 different machines 21:28:35 terminology: 21:30:29 RHTS is internal system, Beaker is open source version 21:30:29 'recipe': instructions per specific test machine 21:30:29 e.g. a test requires over 4GB of memory: recipe allows it to be run on systems with over 4GB 21:30:29 can also define software requirements 21:30:31 recipeset is a group of recipes that are run at one time 21:30:35 tells scheduler that all the recipes have to run around the same time 21:30:37 also defines which test has which role - 'server', 'client' eg 21:30:41 job: container for recipe sets - they don't have to run together but are related 21:30:45 workflow: program that creates recipes and recipe sets 21:32:29 scheduler: processes recipe sets and schedules them for execution 21:32:29 matches systems and installed distros to recipes 21:32:31 red hat may have rhel3, rhel4, rhel5, rhel6 testing all at once - each test system may support different releases 21:32:34 job whiteboard: description of the overall job 21:32:36 recipe whiteboard: description of the recipe 21:32:46 beaker: invetory, reservations, provisioning, scheduler, reporting 21:34:15 inventory: can work on hardware info, arbirtrary key.vaalue pairs, installed software 21:34:15 reservations: test systems can be shared, not 100% dedicated to testing 21:34:20 systems can be private, the group doesn't want other groups to use those systems 21:34:32 ownership done via group ACLs 21:35:03 loaning: a group can lend a system to another group 21:35:25 provisioning: shows only releases that match the system, can reboot machines under power control 21:36:28 scheduler: task library - called 'tasks' as some are just for setup, not actual tests. any combination or order can be run on any system. always starts from a clean installation for reproducibility. wwoods: current fedora tests do not require that 21:41:47 jobs are described in XML and results use the same XML but with additional tags 21:41:47 -> can use results to generate jobs 21:41:47 reporting: only one report available, shows all reservations sorted by longest 21:41:47 used to see what's been on a machine for a long time 21:41:47 matrix report: shows a matrix of all current tests vertically, horizontally shows machines they're running on 21:41:47 not currently available in open source beaker, will be in future 21:41:49 entries are links to actual result xml 21:41:53 #topic design 21:41:55 fully automated: remote power control and clean installation - can always recover machines 21:41:59 cobbler for provision, fence scripts for power control, kobo (similar to koji) for command line interaction 21:42:06 tasks/tests not tied to a specific language - some tests do have language-specific code for checking if the machine is in a specific state 21:42:11 central scheduler: allows integration of remote labs behind firewall / NAT 21:42:13 scheduler contains info on all hardware on remote lab 21:42:17 say a test matches 15 machines: 5 are local and all busy, but if a remote lab with free systems that match checks in, the job will be sent to that lab 21:42:20 ('pull' communication) 21:42:26 each remote lab has a controller which communicates back to the central scheduler 21:42:30 qustion: does scheduler allow for time blocking for machines? 21:42:34 answer: no, but a remote lab controller can effectively do that by only checking in at certain times 21:43:05 active monitoring of console output: lab controller watches console output on test systems 21:43:15 looks for panics / anaconda failures 21:43:42 new system allows configuration of what happens in this case - e.g. email notify and leave machine for remote debugging 22:05:04 #endmeeting 22:05:11 nirik: ping ^^ 22:05:47 .addchair freenode #fudcon-room-2 jds2001 22:05:47 jds2001: Error: '#fudcon-room-2' is not a valid nick. 22:05:59 .help addchair 22:06:00 ianweller: (addchair ) -- Add a nick as a chair to the meeting. 22:06:05 .addchair freenode jds2001 #fudcon-foom-2 22:06:05 jds2001: (addchair ) -- Add a nick as a chair to the meeting. 22:06:25 .addchair #fudcon-room-2 freenode jds2001 22:06:25 jds2001: Chair added: jds2001 on (#fudcon-room-2, freenode). 22:06:31 #endmeeting