16:08:23 #startmeeting Fedora QA Meeting - https://fedoraproject.org/wiki/QA/Meetings/20091026 16:08:23 Meeting started Mon Oct 26 16:08:23 2009 UTC. The chair is jlaska. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:08:23 Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:08:31 #chair adamw 16:08:31 Current chairs: adamw jlaska 16:08:45 #topic Gathering 16:09:14 hey gang, show of $limb ... who do we have joining today? 16:09:57 * jlaska notes ... Oxf13 indicated he would arrive late 16:10:01 * skvidal worries about this $limb thing 16:10:02 * kparal here 16:10:32 kparal: howdy 16:11:04 Viking-Ice: adamw: wwoods: and anyone else I'm forgetting 16:11:11 * Viking-Ice here.. 16:11:17 * wwoods is here 16:11:25 greetings gents 16:12:16 adamw might be running late, so just going to kick things off and he run things when he returns 16:12:25 skvidal: don't be afraid :) 16:12:44 I put adamw's proposed agenda on the wiki, and plan to just walk the list (see https://fedoraproject.org/wiki/QA/Meetings/20091026) 16:12:48 starting with ... 16:12:52 #topic Previous meeting follow-up 16:13:32 I took an action item to rename the existing debug pages to the new format "How to debug problems" 16:13:41 so I think we're good there 16:13:42 https://fedoraproject.org/wiki/Category:Debugging 16:13:52 there was 1 page left that I was unclear how folks wanted that 16:14:27 #info So the only open item on my list is ... should [[KernelBugTriage]] be renamed to [[How to debug kernel problems]]? 16:14:59 I'll follow up with cebbert before renaming that page 16:15:23 besides, I think that page is more about traiging and not so much debugging, but could be wrong 16:15:41 that's all I had recorded for last week, any other items before we dive into the agenda? 16:16:33 Okay ... then first topic up ... 16:16:39 #topic Viking_Ice locale test suggestion 16:16:48 #link https://www.redhat.com/archives/fedora-test-list/2009-October/msg00503.html 16:17:22 Viking-Ice: you raised this on f-t-list this past week, sounds like you felt this is something that could be automated? 16:17:40 yup give me sec.. 16:17:43 sure 16:20:01 Viking-Ice: want us to come back to this topic? 16:20:13 yup 16:21:05 alright ... I'll move that to the end 16:21:39 the next topic is around a discussion that milos Jakubicek and adamw had ... I'll save that for when Adamw returns 16:22:01 #topic Marcela Maslanova test day proposal 16:22:09 morning - sorry, I overslept 16:22:37 adamw: welcome morning ... 16:22:50 you need a few minutes? Or should we jump back to the FTBFS topic? 16:23:07 go for it 16:23:29 okay ... just a quick update on the current topic 16:23:49 pknirsch is will be providing a formal summary of last weeks power management test day 16:24:21 but I also received some updates from Marcela Maslanova around his experiences creating a test day rpm with automated scripts 16:24:37 not sure if everyone had a chance to participate in this last event ... there's always time to jump in 16:24:49 #link https://fedoraproject.org/wiki/Test_Day:2009-10-22 16:25:04 *her* experience :) 16:25:23 kparal: doh, thank you :) 16:26:02 Just a summary for me ... the test day was awesome from a preparation stand-point 16:26:20 you download a test day rpm ... and each of the tests was a script provided by the package 16:26:42 so, the only thing I wanted to raise here was just around building on this process 16:26:57 I know that not all test day topics are as easily automated 16:27:34 but certainly need help from folks in identifying ways to automate future test days in much the same format that Marcela and team have 16:27:48 installer might be one of those easy topics (w/ virt) 16:28:07 I suspect X might not be very easy 16:28:30 anyhow ... unless any other thoughts there ... let's jump back to FTBFS 16:28:37 #topic Milos Jakubicek suggestion on FTBFS bugs 16:28:41 adamw: you want to take it away? 16:29:02 well it's milos' idea, but I can explain what he told me 16:29:09 go for it 16:29:16 #link http://mjakubicek.fedorapeople.org/need-rebuild.html 16:29:36 that's a list of remaining packages that did not rebuild in the f12 mass rebuild, with most false positives taken out (there's still a couple in there) 16:30:07 milos' suggestion is that we make sure FTBFS (fails-to-build-from-scratch, it's a debian term) bugs are filed for each one 16:30:36 aha ... was trying to parse that acronym ... thx 16:30:54 we didn't get into any details of how exactly to arrange it, but it seemed like a decent idea to me 16:31:04 i suggested he come to the meeting to discuss it but obviously couldn't make it 16:31:30 I saw him earlier in #fedora-qa ... but he dropped off 16:32:10 here now. 16:32:34 hi oxf13 16:32:42 so, that's the idea, pretty much. if anyone has any thoughts go ahead 16:32:58 are these things that should be lined up prior to F-12 GA? 16:33:25 if they weren't rebuilt ... what's the fallout (packages that contain .fc11 in the name)? 16:33:52 there's no terrible fallout, but making sure all packages build is good policy 16:34:15 esp given a recent self-hosting blog post 16:34:19 it's possible some of them won't work any more till they get rebuilt, and if you leave it too long you wind up with a giant backlog of crusty packages (i speak from experience) 16:34:35 * Viking-Ice ready now :) 16:34:42 doing it before f12 ga would be nice but not essential, they can always be pushed as updates and anyway it'd be good for future releases too 16:34:56 I'd be curious for Oxf13's take ... I know Jesse was closely involved with the mass rebuild 16:35:18 if they're non crit-path packages, I have no issue with tagging them up until we hit RC 16:35:45 * jlaska wonders if python-bugzilla can help here 16:35:48 i think they all are, looking at the list, they're mostly pretty obscure 16:35:53 yeah 16:36:11 I don't see anything wrong with filing bugs to track this stuff, that's the suggestion right? 16:36:27 jlaska: it may, except it's useful to provide a note of the build failure message in ftbfs bugs, which wouldn't be straightforward to automate. 16:36:27 yes 16:36:38 adamw: good point 16:36:40 adamw, http://fedoraproject.org/wiki/FTBFS FYI 16:37:01 although the idea is specifically to do it as a 'project' within QA, not just to say 'filing these bugs is a good idea'. though I didn't get milos' thoughts on specifically how he wanted to arrange it. 16:37:05 mether: ah thanks 16:37:28 Matt Domsch does a mass rebuild now and then as a QA measure 16:37:32 that seems to claim that bugs 'will' be filed, implying it happens automatically 16:38:09 he has scripts that do that 16:38:16 and he autofiles bugs for the failures. 16:38:24 oh, yeah, milos did point to that same F12FTBFS list, but noted that there are 39 bugs on it 16:38:25 using python-bugizlla 16:38:38 whereas the list of packages he provided is rather longer than 39 16:38:38 his pass happened a month or so before my pass 16:38:52 #link https://bugzilla.redhat.com/showdependencytree.cgi?id=511348&hide_resolved=1 16:39:22 anyway, i'm sensing we need milos to explain his idea more fully either at next week's meeting or on the ml :) 16:39:32 yeah, let's carry the topic fwd? 16:39:52 ok 16:39:54 definitely open to the idea, but nothing tangible is coming to me other than one person walking down a list? 16:40:57 yeah, I haven't read the email yet 16:41:00 it could be done as a group thing, but i think we should clarify whether it needs to happen and why his list is longer than the bugs filed by matt's thingy and so on 16:41:17 his list is longer because more things failed in my pass 16:41:19 and I didn't autofile bugs 16:41:44 ah, ok. 16:42:11 anyway, i'll take an action item to ask him to follow-up at next week's meeting or on the list 16:42:38 #action adamw will ask Milos to follow-up at next week's meeting or on the list on the FTBFS topic 16:43:04 okay, shall we jump back to Viking-Ice's topic? 16:43:20 #topic Viking_Ice locale test suggestion 16:43:30 #link https://www.redhat.com/archives/fedora-test-list/2009-October/msg00503.html 16:43:44 basically this is what I had in mind.. 16:43:45 Viking-Ice: take it away my good man 16:43:54 http://fpaste.org/GuOk/ 16:44:14 * adamw -> biobreak 16:44:23 A test case that checks if keyboard layout is correct after install reports if not 16:44:56 Viking-Ice: would we then want to iterate over installing using different lang/keymap settings ... and run this script? 16:45:17 #link http://fpaste.org/GuOk/ 16:45:36 I dont think we need to parse all of them 16:46:04 a single test should suffice and we should be able to add this to a current installation test 16:46:30 Viking-Ice: for this test to fail now, you need to install with a non US lang and keymap? 16:47:17 in the ks file set the keyboard to a none us layout ( keyboard is-latin1 ) 16:48:24 jlaska: it's the other way around it fails if it's not what we defined in keyboard 16:48:25 Viking-Ice: seems like there is a enough to write a wiki Test_Case: for this 16:48:41 why ? 16:48:51 this should be automated 16:48:51 gotta have a test before you automate 16:49:02 Seems like it should be yeah 16:49:28 but if we just write automation with no background or indication of the steps required, it's not really clear for people outside the group why that test exists 16:49:28 how do you wiki-fi for autoqa ? 16:49:48 well, for israwhidebroken ... we have https://fedoraproject.org/wiki/QA:Rawhide_Acceptance_Test_Plan 16:50:07 once we have a test case, for me, the next step is determining where it's best to automate this 16:50:28 guess it would be added here https://fedoraproject.org/wiki/QA:Anaconda_package_install_test_case 16:50:33 whether it's a generally useful thing that affects how usable rawhide is - 'israwhidebroken' 16:50:35 but ok 16:50:50 Viking-Ice: no, that's a different test case. 16:51:02 test cases should be single purpose, not a clump of different tests in one. 16:51:10 lili and rhe are currently looking into automating the current test matrix ... I think your case would fit perfectly there 16:51:19 Basic Functionality <-- then ? 16:51:31 No. RATS is supposed to be a super-simple basic test 16:51:31 it would be a new test case, with its own name. 16:51:41 ah ok 16:51:42 you're talking about something for a more exhaustive test plan 16:52:06 yeah ... I'd see this case being added to the list here https://fedoraproject.org/wiki/QA:Fedora_12_Install_Test_Plan 16:52:11 it's called the Acceptance Test Plan because it sets out the criteria for judging whether we can even accept a rawhide tree for *real* testing 16:52:19 you're talking about defining one of the cases for the *real* testing 16:52:24 which is awesome and exciting 16:52:46 but, yeah, falls outside of the rawhide acceptance test plan 16:52:57 Viking-Ice: that test matrix you see floating aroudn is something that would come into play after wwoods's rats plan 16:53:13 and it's light on validating any i18n tests ... it could use this 16:53:25 ok what's usually done with the output from those test ( OK fail ) mailed etc.. 16:53:53 That's what they're investigating now ... very *early* stages 16:53:55 the tests in the test matrix are done manually, you see Liam Li send out mails every so often for that testing 16:54:09 depends on how you write the test. right now it gets stuffed into the autotest UI; we have the RATS tests also sending email when they complete 16:54:15 you can decide what you want it to do 16:54:22 autoqa tests are run regularly and results mailed to a special mailing list at present, but automating tests is more complex and hasn't been done for all tests yet 16:54:55 and I think you've got a good test case here to cover recent non-US lang/keymap failures 16:55:19 so it'll be a bit before we have the full test matrix automated 16:55:31 so I'd start with getting the test case defined and added to the install test plan 16:55:33 how do others feel? 16:55:51 worksforme 16:55:56 this checks for what is essentially what maintainer asked for on bug 530452 16:55:58 Bug https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=530452 high, low, ---, jmccann, MODIFIED, Gnome sets the keyboard layout to USA after every log in 16:56:35 Viking-Ice: yeah, we certainly need more testing there 16:57:00 so what additional steps do we take from here.. 16:57:20 basically, create a wiki test case and propose adding it to the current install test matrix 16:57:28 ok 16:57:34 I think that's the first step 16:58:07 the follow-on steps will be around automating that large matrix ... I don't have any updates on that front, other than that's a topic Liam and Rui have started investigating 16:58:20 Viking-Ice: but this way, if we've go tthe test case ... it won't get lost 16:58:40 anyone want to take an action item to create the test case and request addition to the matrix? 16:59:15 guess me I suppose with guided hand :) 16:59:31 hehe ... thanks! 16:59:35 Viking-Ice: it's pretty straightforward to create a test case, it's just a wiki page based on a template; edit any existing test case to see how it's done 16:59:35 or review patients :) 16:59:51 #link https://fedoraproject.org/wiki/Template:QA/Test_Case 16:59:54 Viking-Ice: but feel free to ping me if you're having trouble 17:00:31 #action Viking-Ice investigating creating a test case for bug#530452 ... and adding to F12 install matrix 17:00:33 Bug https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=530452 high, low, ---, jmccann, MODIFIED, Gnome sets the keyboard layout to USA after every log in 17:00:50 Viking-Ice: fedora-qa tickets will come into play here soon 17:01:00 once we have an idea on how to proceed on the matrix automation front 17:01:22 alright, any other updates on this topic, or shall we jump to autoqa? 17:02:11 okay ... autoqa time 17:02:14 #topic AutoQA Updates from wwoods and kparal 17:02:27 wwoods: kparal: what's up in your realm? 17:02:44 ok, news from the rpmguard world: 17:02:54 I have posted a blog about rpmguard 17:02:54 http://kparal.wordpress.com/2009/10/21/rpmguard-print-important-differences-between-rpms/ 17:02:54 and I also wrote to some mailing lists. 17:02:59 maybe you have noticed 17:03:13 #link http://kparal.wordpress.com/2009/10/21/rpmguard-print-important-differences-between-rpms 17:03:18 there was some feedback, skvidal posted me a link to his mungingdiff 17:03:18 http://skvidal.fedorapeople.org/misc/mungingdiff.py 17:03:18 but I'm not sure if there are parts inside which I haven't already implemented in rpmguard 17:03:41 also alexey torkhov suggested to leverage rpmsodiff in my tool. I have created a ticket for that, because I have decided to look at it a little later: 17:03:41 https://fedorahosted.org/autoqa/ticket/75 17:04:14 the upcoming task is to look at the watcher wwoods is working on and integrate rpmguard into autoqa to have it up and running 17:04:28 #info the upcoming task is to look at the watcher wwoods is working on and integrate rpmguard into autoqa to have it up and running 17:04:34 so, yup - we've got a basic watcher for koji 17:04:52 it's ready to be run and picks up every new build in a set of koji tags that we specify 17:05:28 so, for my script the important part would be to find out the version and the link of the previous package release. to have something to compare 17:05:57 right - the tests for that hook will likely get a package NVR and/or package name as input 17:06:13 but I'm writing a library to take e.g. a package NVR and tag and give you the previous released versions of that package 17:06:26 or rather, I have written that tool 17:06:30 is it in the git atm? 17:06:35 and it will be available for use in post-koji-build tests 17:06:35 not yet 17:06:40 sweet 17:06:42 looking forward to it 17:06:48 hrm. 17:06:57 I need to rearrange the autoqa library stuff a little bit so the tests can share code like this better 17:06:58 it might make more sense to compare to the previous shipped package, rather than the previous built package 17:07:09 Oxf13: I said "previously released" 17:07:16 (would make it a lot easier to discover too) 17:07:19 wwoods: oh sorry, missed that 17:07:39 very good point, nice 17:07:44 to go into more detail: you give it a package name (e.g. "preupgrade") and a tag (e.g. "dist-f12-updates-candidate") 17:07:54 #info basic watcher for koji in place. picks up every new build in a set of koji tags that we specify 17:08:05 we have some data that indicates which other tags to check for that 17:08:21 i.e. "dist-f12-updates-candidate" -> "dist-f12" and "dist-f12-updates" (iirc) 17:08:23 wwoods: (not to distract, but wouldn't dist-f12-updates(-testing) be more appropriate? not all -candidate builds get shipped) 17:08:31 #info wwoods has a library soon to be added to git repo which takes a package NVR and tag and provides the previous released versions of that package 17:08:31 wow, n/m. 17:08:33 so then it asks koji for the NVR of the packages with those specific tags 17:08:45 right, the point is that the watcher sees the build land in -candidate 17:09:15 and the test can use this library call if it needs to know the corresponding version for -updates or -updates-testing or the GA 17:09:22 wwoods: cool, so it walks the tag inheritance chain? 17:09:32 not explicitly, no 17:09:58 we have to keep a mapping in autoqa that tells us which "parent" tags to check for a given -candidate 17:10:06 we already have a mapping like this in the other watchers 17:10:25 wwoods - keepin' it simple 17:10:25 e.g. https://fedorahosted.org/autoqa/browser/hooks/post-repo-update/watch-repos.py 17:10:46 right on 17:10:50 so I'm thinking we need to move the data from line 30-76 17:11:00 which is also in https://fedorahosted.org/autoqa/browser/hooks/post-tree-compose/watch-composes.py 17:11:26 and keep a shared autoqa.repoinfo data structure 17:11:32 yeah, that sounds good 17:11:42 +1 17:11:42 we could make a watcher library 17:11:48 which may be provided by a config file (so we don't have to update the package just to tweak the inheritance/URLs/etc) 17:11:59 that includes the mapping, and any other common stuff, then each watcher script could import the library 17:12:03 Oxf13: yep, that's the plan - gonna have an autoqa library for the server-side stuff (watchers etc) 17:12:08 rock 17:12:16 it's like... real software development 17:12:17 and one for the test stuff (like the things in rats.py and virtguest.py) 17:12:37 (and the proposed koji-interfacing stuff I mentioned for post-koji-build) 17:12:45 Oxf13: maybe we need something to automatically check whether there's any bugs in it =) 17:12:56 adamw: crazy talk. 17:13:04 I should roadmap out this plan but I was upgrading my workstation to F12B last week 17:13:29 but yeah now that I think of it, I'm going to give myself an action to do that 17:13:44 * jlaska pulls out the meetbot action pen 17:14:13 wwoods: are these post irb.com things? 17:14:15 #action wwoods to add roadmap/tickets for reworking shared autoqa code (watchers, test helper methods, etc) into a proper library 17:14:25 oh thanks ... u did it for me :) 17:14:28 jlaska: yes, this is a different milestone in my opinion 17:15:08 #info update from mmcgrath on hardware the hardware delivery ... looking at Nov 20 17:15:18 oh also! I'd like to mention http://wwoods.fedorapeople.org/files/critical-path/critpath.py 17:15:34 which was rewritten to be dead stupid simple to use 17:15:45 you run that script, it solves critpath and writes critpath.txt 17:16:06 #link http://wwoods.fedorapeople.org/files/critical-path/critpath.py 17:16:07 I do believe jwb was going to integrate that with the rawhide compose 17:16:14 Oxf13: excellent! 17:16:20 so that we'd have a written out critpath just like we do the depsolve and repodiff 17:16:26 yes. need to get back to that, but i've been sick 17:16:30 Oxf13: so that output would be in the mash log dir for each rawhide compose? 17:16:42 which would give us a static url, http://kojipkgs.fedoraproject.org/mash/rawhide/logs/critpath.txt 17:16:49 jlaska: yes 17:16:52 delicious! 17:16:55 after warren's confusion using the months-old deplist.txt I had sitting in my people.fp.o page I want to make sure everyone knows how this works 17:17:00 "mash/rawhide" is a symlink that gets updated each compose 17:17:01 oh man that would be totally aces 17:17:11 it will be dated, but yeah 17:17:25 thanks jwb 17:17:25 I'll do whatever I can to help get that integrated 17:17:40 dated as in it'll have a date on it 17:17:43 not "it'll be kind of old" 17:17:44 right? 17:17:59 wwoods, you did a lot already. i just need to take your script and add it to the releng git repo, etc. yes, dated as in 'critpath-.txt' 17:18:17 jwb: dated? What will be dated? 17:18:20 hrm. 17:18:29 the file itself 17:18:29 jwb: it might be better to not date it, so that we have a static URL 17:18:38 it'll already be in a dated directory 17:18:42 it'll be in a datestamped *directory*.. yeah 17:18:43 oh, true 17:18:51 ok, wfm 17:19:04 i was still going off of the 'will be on mirrors' idea 17:19:08 #info jwb working to integrate critpath script into daily rawhide compose process 17:19:59 wwoods: kparal: great updates thx ... anything else to note for the logs? 17:20:33 oh 17:20:38 autoqa-devel! 17:20:45 ah yes 17:20:54 https://fedorahosted.org/mailman/listinfo/autoqa-devel 17:21:10 and atodorov gets points for being the first poster 17:21:22 I don't know if it needs this or not ... 17:21:28 #link https://fedorahosted.org/mailman/listinfo/autoqa-devel 17:21:31 blah, more lists I should probably be on 17:22:07 * jeff_hann here...sorry I'm late 17:22:33 jeff_hann: thanks for joining ... just about to go to open floor 17:22:35 yeah, Yet Another Mailing List, but honestly we avoided it as long as we possibly could 17:22:50 and eventually determined that we really need a mailing list after all 17:23:15 Thanks for the updates wwoods and kparal ... lots of progress 17:23:40 so ... I'm going to skip ahead and jump back on the agenda 17:23:55 #topic Upcoming QA events - 2009-10-29 - i18n Test Day 17:23:59 #link https://fedoraproject.org/wiki/Test_Day:2009-10-29 17:24:08 #info Looking for volunteers 17:24:27 Rui and Jens pulled together the test day wiki page. I think it looks pretty good 17:24:46 but we could use more volunteers to help with i18n testing this Thursday 17:25:10 if you're interested, feel free to add a watch to the wiki page 17:25:19 or add yourself to the cc list of the tracking ticket 17:25:24 also poke any chinese/japanese/korean speakers you know :) 17:25:31 #link https://fedorahosted.org/fedora-qa/ticket/25 17:25:37 adamw: yes please! 17:26:03 I'm personallyn not well versed in i18n verification ... so more eyes are needed to review the plan and propose changes 17:26:06 and of course ... help test :) 17:26:25 #topic Upcoming QA events - 2009-10-30 - Blocker Bug Day #2 17:26:50 Kudos to adamw and Oxf13 for walking through the list not once, but twice last week 17:26:54 we are shooting for this blocker bug meeting to end slightly before the heat death of the universe 17:27:02 #link https://bugzilla.redhat.com/showdependencytree.cgi?id=473303&hide_resolved=1 17:27:10 adamw: ouch, no kidding :) 17:27:25 recap from last meeting ... 17:27:28 #link https://www.redhat.com/archives/fedora-test-list/2009-October/msg00579.html 17:27:42 Under the column ... how can I help ... 17:28:13 add block:F12Blocker for issues you feel affect a fair number of users, or are just plain embarrasing 17:28:35 If you are assigned or on the cc list of a blocker list bug ... you can help in advance by ensuring the bug is up-to-date 17:28:40 adamw: anything else you can think of? 17:29:09 not really - more people coming to the meeting to help review the bugs is always welcome 17:29:34 agreed! 17:29:48 adamw: something about more eyes right? :) 17:30:00 more eyes make all arguments longer, yes 17:30:11 bingo! 17:30:16 okay, moving along ... 17:30:25 #Topic Open discussion - 17:31:10 anything else not covered that is of interest? 17:31:23 concerns, questions, complaints ... favorite color? 17:31:44 Who all is going to FUDCon ? 17:31:55 just for information value, I have several kernel bugs I'm working on 17:32:02 I'll be at FUDCon 17:32:34 wwoods: good, I want to get people talking about the message bus, and what kind of data you'd need from it for autoqa et al 17:32:36 I am 17:32:53 Jeff_S: wrong meeting, but did you get in touch with rjune regarding kernel triage in the end? 17:32:55 grr 17:33:00 jeff_hann: ^^^ 17:33:00 adamw: no :) 17:33:12 Jeff_S: go away, WrongJeff =) 17:33:16 lol 17:33:18 heh 17:33:34 so I signed up to do an update on AutoQA, but since wwoods will be there, perhaps wwoods should take over that pitch? 17:33:44 :) 17:33:48 will do that 17:34:08 Oxf13: sure, I can do that 17:34:26 Oxf13: we're planning for a blitzkrieg of autoqa information for the FUDCon 17:34:40 we're hoping to be able to give a status update and demonstrate a prototype for how maintainers can add post-build tests right in pkgcvs 17:34:45 rock 17:35:26 okay gang ... I think we can put a fork in this meeting 17:35:52 thanks everyone for joining and providing your $0.02 17:36:11 As usual, minutes will be on the mailing list soon 17:36:26 thanks jlaska 17:36:39 #endmeeting