21:12:11 <adamw_> #startmeeting autoqa - will woods
21:12:11 <zodbot> Meeting started Sat Dec  5 21:12:11 2009 UTC.  The chair is adamw_. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:12:11 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
21:12:16 <adamw_> fedora has problems: lots of things break, all the time: automated testing will help find and solve problems
21:12:44 <adamw_> breaking down the problem: what is rawhide, what should it do -> rawhide acceptance test plan
21:12:58 <adamw_> small enough problem to start solving: jfdi
21:13:16 <adamw_> hackfest for autoqa 0.1 at previous fudcon: simple repoclosure test - check if repositories are sane when they are generated
21:13:23 <adamw_> moving to existing autoqa design
21:13:36 <adamw_> "when something happens run some tests"
21:13:57 <adamw_> event hooks: events you can trigger tests on, and what tests need to know about the event
21:14:47 <adamw_> ?: rats - 15 items on the list - public?
21:15:12 <adamw_> wwoods: yes, it's on the wiki (google rawhide acceptance test plan)
21:15:13 <adamw_> defines what needs to be tested and why
21:15:15 <adamw_> 8/14 items are automated
21:15:43 <adamw_> design: event watcher for each hook
21:15:43 <adamw_> hook can be: repo updating, tree build, package build etc
21:15:56 <comphappy> https://fedoraproject.org/wiki/Category:Rawhide_Acceptance_Test_Cases
21:15:58 <adamw_> watcher is run and tells you what has happened since the last time it was run
21:17:51 <adamw_> harness is a script called by the watchers when some event has happened
21:17:53 <adamw_> (autotest interface shown on screen, running post-koji-build tests)
21:17:55 <adamw_> tests are already being run every time a package is built
21:18:01 <adamw_> autotest is used as autoqa's harness: comes from google
21:18:05 <adamw_> current hooks: post-repo-update , post-tree-compose , post-koji-build
21:18:31 <adamw_> greg: does autoqa determine success and failure
21:18:37 <adamw_> wwoods: to some degree yes, but not extensively yeet
21:18:48 <adamw_> aiming to allow tests to be run depending on status of previous tests
21:19:02 <adamw_> current tests: post-repo-update: conflicts, repoclosure, rats_sanity
21:19:08 <adamw_> checks for conflicts, dependencies
21:19:32 <adamw_> post-koji-build: rpmlint, rpmguard - check for problems and significant changes in new packages
21:19:53 <adamw_> autoqa python library: useful python modules that can be used across multiple watchers and tests
21:20:24 <adamw_> israwhidebroken.com: planned to be a specialized autoqa reporting system which implements a web page which tells you whether rawhide is broken
21:20:55 <adamw_> will be used to keep 'last known good' rawhide images
21:20:55 <adamw_> goal is to never have non-functional rawhide installation images
21:21:03 <adamw_> (screenshot of test israwhidebroken.com page on screen)
21:21:32 <adamw_> greg: is there enough data in the tests to know who broke something?
21:22:00 <nirik> does israwhidebroken currently correctly show that rawhide is broken? ;)
21:22:11 <adamw_> wwoods: not automatically - for some tests you could tell (e.g. comps.xml validity), for some it would be difficult
21:22:23 <adamw_> er, i dunno :) hand raise?
21:23:03 * nirik can raise his hand, but I am not in the same country so I doubt it would do much good. ;)
21:23:15 <adamw_> nirik: it's not currently running because there's no rawhide install images because of nofrozenrawhide
21:23:21 <nirik> ah, ok.
21:23:26 <adamw_> autoqa needs to talk to installer team about nofrozenrawhide
21:23:30 <adamw_> (that's the reply from will :>)
21:23:34 * nirik notes rawhide hasn't composed the last 2 days...
21:24:40 <adamw_> coming soon: public autotest instance
21:24:40 <adamw_> post-bodhi-update hook: for testing update and candidate update packages
21:24:40 <adamw_> coming soonish: easy autoqa for packagers: add tests right in pkgcvs so they can do it integrated with the package maintainer flow
21:24:41 <adamw_> e.g. font tests - could be added into each font package
21:27:38 <adamw_> post-iso-build hook: for testing installation of newly generated nightly live images e.g.
21:27:38 <adamw_> use Fedora Message Bus to replace watcher scripts
21:27:38 <adamw_> also allows two-way communication
21:27:38 <adamw_> the test can communicate failure back to koji (e.g.) and package blocked
21:27:40 <adamw_> coming some day: multi-host testing (for network testing): no current idea of how to do this
21:27:46 <adamw_> functional testing for GUIs: have to be able to script GUI interactions
21:27:52 <adamw_> getting involved: may be Coming Soon, switching to second presenter
21:28:08 <adamw_> red hat internal automated testing: different cases
21:28:10 <adamw_> e.g. testing a new kernel across 30 different machines
21:28:35 <adamw_> terminology:
21:30:29 <adamw_> RHTS is internal system, Beaker is open source version
21:30:29 <adamw_> 'recipe': instructions per specific test machine
21:30:29 <adamw_> e.g. a test requires over 4GB of memory: recipe allows it to be run on systems with over 4GB
21:30:29 <adamw_> can also define software requirements
21:30:31 <adamw_> recipeset is a group of recipes that are run at one time
21:30:35 <adamw_> tells scheduler that all the recipes have to run around the same time
21:30:37 <adamw_> also defines which test has which role - 'server', 'client' eg
21:30:41 <adamw_> job: container for recipe sets - they don't have to run together but are related
21:30:45 <adamw_> workflow: program that creates recipes and recipe sets
21:32:29 <adamw_> scheduler: processes recipe sets and schedules them for execution
21:32:29 <adamw_> matches systems and installed distros to recipes
21:32:31 <adamw_> red hat may have rhel3, rhel4, rhel5, rhel6 testing all at once - each test system may support different releases
21:32:34 <adamw_> job whiteboard: description of the overall job
21:32:36 <adamw_> recipe whiteboard: description of the recipe
21:32:46 <adamw_> beaker: invetory, reservations, provisioning, scheduler, reporting
21:34:15 <adamw_> inventory: can work on hardware info, arbirtrary key.vaalue pairs, installed software
21:34:15 <adamw_> reservations: test systems can be shared, not 100% dedicated to testing
21:34:20 <adamw_> systems can be private, the group doesn't want other groups to use those systems
21:34:32 <adamw_> ownership done via group ACLs
21:35:03 <adamw_> loaning: a group can lend a system to another group
21:35:25 <adamw_> provisioning: shows only releases that match the system, can reboot machines under power control
21:36:28 <adamw_> scheduler: task library - called 'tasks' as some are just for setup, not actual tests. any combination or order can be run on any system. always starts from a clean installation for reproducibility. wwoods: current fedora tests do not require that
21:41:47 <adamw_> jobs are described in XML and results use the same XML but with additional tags
21:41:47 <adamw_> -> can use results to generate jobs
21:41:47 <adamw_> reporting: only one report available, shows all reservations sorted by longest
21:41:47 <adamw_> used to see what's been on a machine for a long time
21:41:47 <adamw_> matrix report: shows a matrix of all current tests vertically, horizontally shows machines they're running on
21:41:47 <adamw_> not currently available in open source beaker, will be in future
21:41:49 <adamw_> entries are links to actual result xml
21:41:53 <adamw_> #topic design
21:41:55 <adamw_> fully automated: remote power control and clean installation - can always recover machines
21:41:59 <adamw_> cobbler for provision, fence scripts for power control, kobo (similar to koji) for command line interaction
21:42:06 <adamw_> tasks/tests not tied to a specific language - some tests do have language-specific code for checking if the machine is in a specific state
21:42:11 <adamw_> central scheduler: allows integration of remote labs behind firewall / NAT
21:42:13 <adamw_> scheduler contains info on all hardware on remote lab
21:42:17 <adamw_> say a test matches 15 machines: 5 are local and all busy, but if a remote lab with free systems that match checks in, the job will be sent to that lab
21:42:20 <adamw_> ('pull' communication)
21:42:26 <adamw_> each remote lab has a controller which communicates back to the central scheduler
21:42:30 <adamw_> qustion: does scheduler allow for time blocking for machines?
21:42:34 <adamw_> answer: no, but a remote lab controller can effectively do that by only checking in at certain times
21:43:05 <adamw_> active monitoring of console output: lab controller watches console output on test systems
21:43:15 <adamw_> looks for panics / anaconda failures
21:43:42 <adamw_> new system allows configuration of what happens in this case - e.g. email notify and leave machine for remote debugging
22:05:04 <comphappy> #endmeeting
22:05:11 <ianweller> nirik: ping ^^
22:05:47 <jds2001> .addchair freenode #fudcon-room-2 jds2001
22:05:47 <zodbot> jds2001: Error: '#fudcon-room-2' is not a valid nick.
22:05:59 <ianweller> .help addchair
22:06:00 <zodbot> ianweller: (addchair <channel> <network> <nick>) -- Add a nick as a chair to the meeting.
22:06:05 <jds2001> .addchair freenode jds2001 #fudcon-foom-2
22:06:05 <zodbot> jds2001: (addchair <channel> <network> <nick>) -- Add a nick as a chair to the meeting.
22:06:25 <jds2001> .addchair #fudcon-room-2 freenode jds2001
22:06:25 <zodbot> jds2001: Chair added: jds2001 on (#fudcon-room-2, freenode).
22:06:31 <jds2001> #endmeeting