16:00:06 #startmeeting Fedora QA Meeting 16:00:06 Meeting started Mon Dec 13 16:00:06 2010 UTC. The chair is jlaska. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:06 Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:00:10 #meetingname fedora-qa 16:00:10 The meeting name has been set to 'fedora-qa' 16:00:22 #topic Gathering life forms 16:00:37 Hi all ... show of hands for those joining the QA meeting today 16:00:53 or cilia, whichever is appropriate 16:00:55 * jskladan here 16:00:57 * mkrizek is here 16:01:05 * Alam here 16:01:06 yo! 16:01:33 * brunowolff I am here in case someone has a question about the not VERIFIED bug queries 16:01:57 * kparal hellos everyone 16:02:00 * fenrus02 waves 16:02:05 As an aside the fedora meeting page says the meeting is at 1500. I thought I missed it. 16:02:14 brunowolff: oh thanks ... lemme update that 16:02:22 * jlaska waits 60 seconds to start meeting 16:03:05 who are we missing? Viking-Ice wwoods robatino? 16:03:32 alright, let's get moving 16:03:39 #topic Previous Meeting follow-up 16:04:09 I have no action items from last week 16:04:17 we have some ongoing topics, but those will be covered in the agenda 16:05:02 I continue to delay the QA retrospective recommendations work due to other conflicts 16:05:12 thankfully ... that's not stopping some of you from moving forward on big topics for F15 16:05:49 So if you're feeling down about being behind on a task ... you can join me in #behind :) 16:06:18 alright, moving into the agenda 16:06:19 * Viking-Ice here 16:06:23 howdy! 16:06:25 #topic Call for Test Days 16:06:31 * phuzion is here 16:06:39 Just re-iterating last weeks call for topics 16:06:41 phuzion: hello :) 16:06:46 * adamw just sent out an email with that topic to devel-announce... 16:07:05 #link http://lists.fedoraproject.org/pipermail/devel-announce/2010-December/000733.html 16:07:10 adamw: that's great, thank you 16:07:23 adamw: you scheduled quite a few events last week already 16:07:34 well, mostly just several recurrences of two or three :) 16:07:57 the xorg-x11-drv week ... our bread'n'butter ;) 16:08:19 Alam mentioned another l10n/i18n event 16:08:32 I'll ping jens, rhe and Alam about that for F15 16:08:48 iirc, jens and rhe hosted a similar (and successful) event for F14 16:08:55 that'd be cool yep 16:09:07 the one for f14 was great, it identified some bugs we got marked as NTH and fixed 16:09:14 probably wouldn't have been caught otherwise 16:09:40 #action jlaska to ping jens+rhe and Alam for thoughts on a F15 l10n/i18n test day 16:10:03 The RetraceServer guys were interested in hosting something mid-march 16:10:35 but nothing final yet ... I've got an action item to ping them too 16:11:00 adamw: do you think systemd will likely land on the schedule somewhere? 16:11:12 should do yep 16:11:40 i'm waiting to hear from lennart and/or harald 16:11:56 adamw: sweet, I should have known you already reached out on that :) 16:12:03 Some links for those reading the logs ... 16:12:05 #link https://fedoraproject.org/wiki/Releases/15/FeatureList 16:12:30 #link http://fedoraproject.org/wiki/QA/Fedora_15_test_days 16:12:39 adamw: anything else to note on this topic? 16:12:45 harald's on vacation, i'm hoping lennart will get back soon 16:12:53 nope, i've mostly thrown in the things i had on my list 16:13:44 adamw: thanks for moving that process forward 16:13:50 npnp 16:13:58 #topic Bodhi patch 16:14:09 * fcami__ waves 16:14:15 hey, there he is :) 16:14:19 sorry, just got back from a meeting 16:14:22 * wwoods is lurking 16:14:27 wwoods: hey 16:14:44 so fcami posted a patch to update the comment that bodhi posts into bugzilla when an update is available for testing 16:14:58 then I sent an updated patch to jlaska/adamw yesterday 16:15:01 #link http://fedorahosted.org/fedora-infrastructure/ticket/701 16:15:28 the latest patch integrates all the comments done by bugzappers during the last meeting 16:15:35 fcami__: for next steps ... I think that patch needs review from the bodhi-devel team, and then figure out what to apply it against? 16:15:46 is lmacken lurking? 16:15:49 yes, if you're all OK with that 16:16:05 sure 16:16:13 just looking at the patch... 16:16:23 why are there two slightly different messages in different places? 16:16:27 sorry, I should have posted it to some mailing list 16:16:34 is the patch available for lmacken to review as well? 16:16:37 hmmm, they shouldn't be different 16:17:06 * adamw tries to fpaste it 16:17:10 one seems to have just a bit more text 16:17:13 the two places are 1/ what happens when you edit a bodhi action, adding bugs 2/ when you create a bodhi action with bugs 16:17:29 #link http://fpaste.org/F3Pv/ 16:17:30 ^^^ patch 16:17:31 oh. no, actually, they are the same 16:17:36 i just was missing where it wrapped a line :) 16:17:50 so looks fine 16:17:54 oy ok :) 16:17:57 is there a bodhi-devel list? How do we get this on the right radar? 16:18:10 except i still thought it would be good to have it give a note to log in before leaving feedback 16:18:27 I can add that, yes 16:18:37 there is bodhi@fedorahosted IIRC 16:18:48 I'll CC lmacken 16:18:58 fcami__: ah yeah ... https://fedorahosted.org/mailman/listinfo/bodhi 16:19:29 fcami__: also, is this patch against the current bodhi code base, or the bodhi-2.0 stuff? 16:19:53 jlaska, whatever's in master in the git repo. maybe I should dig more. 16:20:29 fcami__: I think lmacken can probably add some data there ... since he'll likely be the one accepting the patch (I think) 16:20:37 * jlaska sees you already started the discussion https://fedorahosted.org/pipermail/bodhi/2010-December/000543.html 16:21:01 yes, I did that, that's the previous version 16:21:17 what's the next step? 16:21:19 nirik replied to let me know there was an infra ticket 16:21:27 next step is to update the patch with the log in requirement 16:22:03 cool, and then we should track progress on the thread you started on bodhi@lists.fedorahosted.org ? 16:22:13 yeah, I think so 16:22:26 I'll CC test@lists.fedoraproject.org 16:22:28 as I did last time 16:22:41 okay ... sounds like you've got all the ducks in a row :) 16:23:11 shall we check-in on this again next week? 16:23:25 yeah, probably so 16:23:34 okay 16:23:34 I should update the patch tonight 16:23:38 thanks fcami__ 16:23:40 and then I'll try to reach lmacken directly 16:23:50 np jlaska 16:24:06 because I'd like to know if he's OK with the general idea 16:24:16 * fcami__ gives the mic back 16:24:29 alright ... bruno's next 16:24:41 #topic Improve tracking untested blocker bugs 16:24:50 This is a follow-up on ... 16:24:52 #link http://fedorahosted.org/fedora-qa/ticket/89 16:25:22 brunowolff created a draft wiki page with some stock queries we can use to keep track of blocker bugs that get CLOSED but are never VERIFIED 16:25:25 #link https://fedoraproject.org/wiki/User:Bruno#Mockup_for_QA_-_Tracking_bug_queries 16:26:30 brunowolff: that looks about what we needed. I don't know if we have a pressing need for tracking the NTH bugs that aren't VERIFIED 16:26:40 so if we were looking to slim up the links ... that might be one option 16:27:16 adamw: any thoughts? 16:27:22 The one possibly significant limitation is that only bugs directly dependent on each tracker are checked. 16:27:40 we don't need to track the NTH bugs, but hey, it doesn't hurt either. 16:27:41 The general idea here I think is that we want to incorporate review of these bugs into the blocker bug SOP somehow 16:27:48 If there are normal bugs depending on other normal bugs that aren't verified, that won't get caught. 16:27:51 yeah, that limitation is unfortunate 16:28:07 especially when we use component or team blockers, like the anaconda and kde and virt trackers 16:28:12 brunowolff: yeah, I'm used to that limitation ... so that doesn't bother me too much 16:28:17 just as long as we call it out somewhere 16:28:55 so ... how would we respond to bugs on these lists? 16:29:23 i kinda thought the aim of the tracker was to ensure that such bugs don't exist 16:29:25 my initial thought, is these lists would serve as extra data ... helping us decide whether some serious looking Blocker bugs never were tested 16:29:35 so it's to be used to identify process fail more than some sort of ongoing thing 16:29:36 adamw: yeah, you got it 16:29:53 I don't know if we'll be able to knock out *all* bugs that show up on these lists 16:30:04 although i suppose when we mark bugs as blockers early in the cycle and they just get fixed in the normal process of releasing we may skip verified 16:30:05 but that's a laudable goal 16:30:14 adamw: true true 16:30:37 howabout for F15 ... we periodically review the list during Blocker meetings, just like you do with NTH bugs? 16:30:44 we could, sure 16:30:44 looking for any ... "Oh shoot!" moments? 16:30:51 the only drawback with that is...longer blocker meetings 16:30:51 :P 16:30:58 yeah, true true 16:30:59 hmm 16:31:42 the other option is firing the list of test@lists.fp.org each week ... calling out issues if needed? 16:31:46 yeah 16:31:52 so it's list spam vs. meeting time 16:31:56 pick which you hate more 16:31:57 heh :) 16:32:15 we can move it to another meeting 16:32:18 this one 16:32:40 I feel like it's at least worth 5 minutes of a meeting to quickly scan the list 16:32:47 I'll be happy to do the scanning during the meetings in F15 16:32:52 sure 16:32:53 if it's useless ... oh well 16:32:55 let's see how it goes 16:32:58 yeah 16:33:15 should I integrate brunowolff's links into the blocker meeting SOP page? 16:33:47 otherwise ... I'll forget when we are in the thick of a release :) 16:34:06 if we're going to do it in the blocker meeting, sure 16:34:10 document the whole process is the goal 16:34:18 brunowolff: unless you have other recommendations, I can try to sanely merge your links into the blocker meeting SOP 16:34:31 #link http://fedoraproject.org/wiki/QA:SOP_Blocker_Bug_Meeting 16:34:54 #action jlaska to merge [[User:Bruno#Mockup_for_QA_-_Tracking_bug_queries]] into [[QA:SOP_Blocker_Bug_Meeting]] 16:35:04 That's fine. If you have any further questions for me, you can bug me by email. 16:35:14 brunowolff: will do, thanks for the bz help :) 16:35:24 #topic Fedora test case mgmt requirements 16:35:38 Just a quick update on #topic 16:35:39 #link http://fedorahosted.org/fedora-qa/ticket/152 16:35:54 Hurry drafted an initial requirements page to start collecting/organizing data 16:35:59 #link https://fedoraproject.org/wiki/Rhe/tcms_requirements_proposal 16:36:23 Please do add comments/ideas/suggestions to the Talk: page 16:36:43 I plan to add some, hopefully constructive, feedback later this week 16:37:21 Alright, next topic ... 16:37:28 #topic CritPath test case development 16:37:33 #link http://fedorahosted.org/fedora-qa/ticket/154 16:37:58 adamw: how's it coming along on the critpath work? 16:38:36 any updates/blockers/concerns/question/jokes ... 16:38:45 nope, just waiting on me to do something 16:39:00 till's feedback was good, i'll go ahead incorporating that 16:39:26 cool. I don't know if it helped, but there's plenty of stuff we can get through the mediawiki API 16:39:33 i'm going to write up one or two example (but useful...) test cases, in the appropriate name / category layout, then mail the list again explaining the proposed system and linking to the examples 16:39:35 lemme know if you want me to do any queries for you 16:39:47 adamw: sounds like a plan 16:40:39 #info still experimenting with some mock-ups/examples 16:40:56 #info Will mail the list explaining proposed system and linking to examples 16:41:05 alright ... kparal, are you ready to rock? 16:41:21 thanks adamw :) 16:41:23 jlaska: ready to rock the boat :) 16:41:25 #topic AutoQA Update 16:41:35 jlaska: i don't really need the queries, it's bodhi/f-e-k that need them :) 16:41:44 adamw: right on 16:42:13 adamw: I meant if you wanted feedback as to whether your examples are queryable as intended etc... 16:42:33 * kparal waiting a while 16:42:43 kparal: another busy week for AutoQA ... what's the latest? 16:42:57 Hello gang, even though I didn't want to be as wordy as last time, I'm afraid I will be :) Here are the latest news: 16:43:12 1. clumens' anaconda_storage test has been reviewed propertly (regarding AutoQA part of the test) and it is ready to be merged into master, once clumens deems it's ready. Therefore it should be very probably part of the next release [1]. Enjoy automatic Anaconda tests in the near future, hooray! 16:43:13 [1] https://fedoraproject.org/wiki/AutoQA_Roadmap 16:43:40 (Hey, I have written it in advance, so it would be quicker, but feel free to stop me after any section if you have some comments to it) 16:44:21 2. The patch representing second attempt for solving ticket #205 [2] ("Provide a support for sending comments into Bodhi") has landed into autoqa-devel [3]. Big thanks to mkrizek. If no further concerns, it will be merged into master very soon. That means we will be able to send our test results as comments into Bodhi. Example is here: [4]. Currently this is planned for depcheck and for upgradepath test (not enabled yet). 16:44:21 [2] https://fedorahosted.org/autoqa/ticket/205 16:44:21 [3] https://fedorahosted.org/pipermail/autoqa-devel/2010-December/001435.html 16:44:21 [4] https://admin.fedoraproject.org/updates/python-rpmfluff-0.3-6.fc14 16:45:16 hehe ... you kept dmalcolm on his toes in that update :) 16:45:24 yes we did :) 16:45:51 but he agreed. maybe he didn't know what to expect 16:46:11 nice workflow with that patch set guys ... proposing, incorporating feedback, providing examples etc... 16:46:23 it's lovely stuff 16:46:47 3. lmacken enabled us to use staging Bodhi instance for AutoQA development purposes (mainly he turned off email notifications). Kudos to him for that. There is currently also an invalid certificate problem, but that should be solved in some time. Meanwhile the process of (mis)using staging Bodhi server from AutoQA is described at [5]. 16:46:47 [5] https://fedorahosted.org/pipermail/autoqa-devel/2010-December/001438.html 16:47:39 "insecure=True" ... that gives you a warm feeling :) 16:47:47 :) 16:48:05 alright, next on 16:48:07 4. mkrizek and I worked on ticket #241 [6] ("Add support for a staging server"). The patch is not posted yet (waiting for previous patch to land in master), but the code is available at mkrizek-staging branch. It should allow us have configurable Koji/Bodhi server URLs and configurable options whether to send emails (3 different kinds, actually) and Bodhi comments. 16:48:07 [6] https://fedorahosted.org/autoqa/ticket/241 16:48:40 oh nice, that'll be a handy change 16:49:25 (and expect much more documentation in autoqa.conf, as a bonus) 16:49:34 hooray documentation! 16:49:35 heh 16:49:44 btw ... who gets the credit for the killer __docstring__'s ? 16:49:56 jlaska: which ones? 16:50:17 in the bodhi feedback patchset ... mkrizek has some really good docstrings for all the methods 16:50:34 jlaska: the praise goes to mkrizek I believe 16:50:56 I just asked him to document it well, he did the rest :) 16:51:07 nice! 16:51:32 ok, and last but not least: 16:51:33 * mkrizek flattered 16:51:38 5. jskladan works on a different kind of a koji watcher. His work is available in the 'new_koji_watcher' branch. His work should enable us to hook up depcheck properly into our framework. It will also obsolete the post-bodhi-update watcher. I'll give him a word to say some key concepts of our new watcher and why we even need it. 16:52:10 * jskladan steps forward 16:52:24 OK, gang. We found ourselves in need of some updates on the watchers front. 16:52:26 * kparal hands out the mic to jskladan 16:52:40 The required changes were 16:52:40 1) Make use of the -pending tags in Koji: https://fedorahosted.org/autoqa/ticket/228 16:52:40 2) Create 'batch' scheduler: https://fedorahosted.org/autoqa/ticket/204 16:52:50 ad watching -pending tags) 16:53:08 At the moment koji watcher is querying Koji for the list of recently built packages (based on the time of build). 16:53:08 This is a bit unsatisfactory - we can (and we do) miss some spots in the testing chain: 16:53:22 1) package Foo is built at date XYZ. It gains tag dist-f14-updates-candidate 16:53:22 2) koji-watcher founds out "ha, new package, let's test it" 16:53:22 3) tests are OK 16:53:22 4) package Foo gains dist-f14-updates-testing-pending tag 16:53:22 5) we'd like to run tests like depcheck on it, but because the 'built at' date, which the actual watcher checks is not altered, we miss the change 16:53:42 So we decided to use different querying model, based on the tagHistory() in Koji. 16:54:01 (kids, _do_ try this at home it's awesome :) `koji list-tag-history --tag=dist-f14-updates-pending`) 16:54:15 * jlaska tries 16:54:25 which effectively tells us which packages were 'pushed' to the tags we care about. 16:54:42 * jskladan hopes that there's no typo :) 16:54:49 ad "batch scheduling": 16:55:06 so this adds a new 'hook' name that tests would need to code to (for batch updates)? 16:55:21 yes, exactly 16:55:24 We also found us in need of 'batch' scheduling - e.g. we don't want to run depcheck for every package built, but we'd like just to inform autoqa "hey, there is new stuff in dist-f14-updates-pending tag, run tests". 16:56:05 So there is brand-spanking-new 'watch-koji-builds-batch' watcher. 16:56:13 jskladan, wwoods have you guys worked out how depcheck would integrate with this stuff? 16:56:33 which groups the updates according to the tag, and sends the whole 'batch' of updates at once 16:56:46 jlaska: I have not yet spoken with Will 16:57:07 jlaska: but I've tried it out in my testing environment, and it seems to be working like charm 16:57:30 jlaska: even though I'm not really sure about the differences in post-bodhi-update & post-koji-build 16:57:42 how do you mean? 16:57:56 (e.g. if we can miss something by watching only koji-tags, and amending the post-bodhi hook as such) 16:58:23 because at the moment, it seemed to me, that watching the tags is more straightforward for depecheck 16:58:34 because we can easily determine the repo 16:58:43 (or the tag, if you want) 16:59:04 from my understanding, it does seem to better align with the use of hte koji -pending tag 16:59:56 yes, this is also my point of view - I just wanted to make sure, that by setting the post-bodhi aside (at least for depcheck), we're not gonna miss any updates 17:00:18 can run the old watcher, and your watcher side-by-side to compare? 17:00:22 s/can/can we/ 17:01:14 the koji & bodhi watcher? sure can, but I belive that it suffers from the same problem as the old koji watcher - e.g. it's reacting to 'builds' instead of 'tag changes' 17:01:36 jskladan: I'll have to take it offline ... I think I'm getting confused :) 17:01:48 eh, my bad 17:02:04 this is a bit longer description than I expected :) 17:02:10 wwoods and jskladan ... can you guys sync up on the watcher and depcheck integration? 17:02:34 to put it straight: the plan is to effectively disable post-bodhi, and replace it with post-koji watcher over the 'pending' tags 17:02:40 oop sorry, got waylaid by something, catching up on scrollback 17:03:27 in the mean time - if you're interested in the new koji watcher, feel free to explore the git: 17:03:42 and post comments on autoqa-devel (or IRC) 17:03:45 using tag-history is definitely the right approach 17:03:59 jskladan: nice ... I've got plenty of catch-up work to do on autoqa-devel :) 17:04:36 but batching updates.. I don't think that's necesssary or desirable 17:05:11 since the plan is to move to a messagebus as soon as it's feasible 17:05:20 and the messagebus is going to send individual messages for each event 17:05:50 it's not going to batch the messages, so we don't want to build batching into the infrastructure and then drop it later 17:06:12 let's consult it after the meeting 17:06:25 this is ticket #204 17:06:43 https://fedorahosted.org/autoqa/ticket/204 17:06:51 yeah, was going to suggest continuing this in #fedora-qa after the meeting 17:07:16 what else on the autoqa front? 17:07:17 right - this is just a quick summary for the meeting notes 17:07:37 wwoods: thank you, that makes my minute gathering life easier! 17:07:40 ticket #204 involves batching updates from the watcher 17:07:54 a related problem is ticket #208, which concerns simultaneous depcheck runs 17:07:58 err, sorry 17:08:03 that's ticket #248 17:08:07 https://fedorahosted.org/autoqa/ticket/248 17:08:50 I've tried to give some details on the proposed solutions there 17:08:59 there's been a ton of change recently on the AutoQA front ... so if we need a side-meeting to talk through some of that ... let's do it 17:10:00 let's discuss post-meeting 17:10:05 anyway there's some complex, thorny issues with timing and tagging for depcheck (and other tests in this path) 17:10:07 * jsmith likes that idea 17:10:15 wwoods: kparal: I think you both are talking from different POV's in ticket#248 17:10:16 we should definitely discuss it further 17:10:35 I'm also working on a blog post to try to explain what the problems are and how we can fix 'em 17:10:46 not finished yet, but here's a diagram I'm working on: 17:10:50 http://wwoods.fedorapeople.org/images/repos-draft.png 17:10:55 Thanks everyone, but this was supposed to be a summary, erm :) 17:11:08 wwoods: we will discuss if after the meeting 17:11:08 heh, ah well 17:11:22 but yes. Further Discussion Outside The Meeting Is Suggested 17:11:24 no worries ... sounds like it identified a topic we need to drill down on outside of the meeting 17:11:34 wwoods: nice diagram :) 17:11:41 thanks jskladan for the problem description 17:12:07 and this was also my last topic in the "AutoQA news" 17:12:21 * jskladan is shocked by the outcome :-D but looking forward to discussing this further :) 17:12:53 thanks for the autoqa updates all 17:12:58 alright ... open discussion time 17:13:07 #topic Open discussion - 17:13:24 we've gone a bit over today ... are there any other items we need to review here? 17:13:47 * jlaska waits 60 seconds 17:14:27 20 seconds ... 17:14:36 nothing from me 17:14:45 5 seconds ... 17:14:54 Alright gang ... thanks for your time and updates today 17:15:05 I'll follow-up to the list later today with minutes 17:15:08 #endmeeting