08:34:20 #startmeeting bug triage 08:34:20 Meeting started Fri Jul 11 08:34:20 2014 UTC. The chair is hagarth. Information about MeetBot at http://wiki.debian.org/MeetBot. 08:34:20 Useful Commands: #action #agreed #halp #info #idea #link #topic. 08:34:34 * lalatenduM is here 08:35:22 lalatenduM: oh, right, we're just getting started, you didnt miss anything :) 08:35:41 ndevos, :) 08:35:43 right, I missed sending out this on gluster-devel 08:36:09 #topic bug backlog 08:36:18 here's our list of bugs - http://goo.gl/Q1cBHD 08:36:21 Title: Report: Component / Status (at goo.gl) 08:37:19 the most important question - how do we triage this list? 08:37:38 1015 in total!! 08:37:44 I have some scripts that can automate things a little 08:37:48 lalatenduM: right 08:38:07 First look at modified. Most of those could possibly be closed. 08:38:16 do we want to triage bugs which are logged 3 to 4 major releases back? 08:38:17 at least I can check what bugs should have been CLOSED, or in some other state, depending on any patches posted/merged 08:38:24 kshlm: good idea 08:38:36 ndevos, kshlm agree 08:38:44 ndevos, when you say you can check, do you mean manually? 08:38:46 hagarth: ndevos does it by some script 08:39:18 krishnan_p: no, list all bugs, and compare with details from gerrit - it's a script 08:39:28 ndevos, OK 08:39:40 output from earlier this week: 08:39:43 #link http://paste.fedoraproject.org/117188/14050678 08:39:46 Title: #117188 Fedora Project Pastebin (at paste.fedoraproject.org) 08:39:47 ndevos: how long would the script need to complete? 08:40:08 hagarth: a run takes about 30 minutes, maybe more? 08:40:28 and that is only for the listing and checking of bugs/patches, not updating 08:40:57 ndevos: hmm, once the candidates are identified, we can possibly do a quick manual scan? 08:41:00 my other scripts that I used to move 3.5.1 bugs to ON_QA and later to CLOSED just take a list of bugs and modify them 08:41:37 I propose we do this: 08:41:42 hagarth: yeah, some checks for changes that are proposed, select bugs to change, run a 2nd script to make the changes 08:41:56 1. move all modified to closed 08:42:25 2. check manually the results of script that ndevos will execute 08:42:46 2. (contd.) and update bugs accordingly ? 08:43:07 does this seem ok to everybody? 08:43:09 I dont like 1 08:43:30 users filed a bug, a patch got posted to master, and we'll close it? 08:43:31 hagarth, we can't do 1. we should close a bug against a release, for future reference 08:43:46 ndevos, krishnan_p: ok 08:43:57 I think we should only close bugs with a 'fixed in version' 08:44:08 ndevos: agree 08:44:10 ndevos, that brings the problem of how long we wait for user to ack the fix 08:44:42 krishnan_p: should we bring about an aging policy? 08:44:49 krishnan_p: not really, we can set the version to the tag that contains the patch, if nobody confirms the fix, we close it when there is a release 08:45:13 ndevos, that sounds like a good protocol to follow. 08:45:21 hagarth, yes we should have a aging policy 08:45:37 so what's the process here? 08:45:43 krishnan_p: I've moved 3.5.1 bugs to ON_QA when the beta was released, and I've marked them CLOSED when the final release was done - VERIFIED or not 08:46:13 ndevos, I got it. It makes sense. 08:46:31 example: https://bugzilla.redhat.com/show_bug.cgi?id=1071191#c3 08:46:32 Bug 1071191: high, medium, ---, ndevos, CLOSED CURRENTRELEASE, [3.5.1] Sporadic SIGBUS with mmap() on a sparse file created with open(), seek(), write() 08:46:34 bug moves from ASSIGNED -> POST after patches are posted, POST -> MODIFIED after patches are accepted, MODIFIED -> QA when a release is done and the fixed in version gets updated, ON_QA -> CLOSE when GA of a release happens 08:46:45 would this be our process? 08:47:00 yes 08:47:11 is everybody fine with this? 08:47:14 yes 08:47:15 Yes. 08:47:19 yes 08:47:23 yes 08:47:30 yes 08:47:35 I think we should document this in gluster.org 08:47:49 lalatenduM: IIRC, you have a bugzilla workflow/process page right? 08:47:50 #link http://www.gluster.org/community/documentation/index.php/Bug_report_life_cycle 08:47:51 Title: Bug report life cycle - GlusterDocumentation (at www.gluster.org) 08:48:11 hagarth, yup, we need to update http://www.gluster.org/community/documentation/index.php/Bug_report_life_cycle 08:48:12 hagarth, one question, does it need to be manual or ndevos's script does it? 08:48:13 Title: Bug report life cycle - GlusterDocumentation (at www.gluster.org) 08:48:30 atinmu: some of the transitions are going to be manual I suppose 08:48:51 for example, the post -> modified transition cannot be done automatically 08:48:58 atinmu: I think we should rely on developers setting the correct status of the bugs 08:49:07 since there might be multiple fixes for a particular bug 08:49:14 ndevos: right 08:49:29 hagarth : my short experience in upstream says people don't move the bugs to proper state, just for eg, even if a patch is submitted the bug stays in either NEW or ASSIGNED state 08:49:36 ndevos: will you take care of updating the bug_report_life_cycle page? 08:49:53 hagarth: yeah, I can 08:49:58 atinmu: any ideas on how we can get better there? 08:50:13 #action ndevos to update bugzilla process page 08:50:44 hagarth : so I was wondering whether we can have something automated like whenever a patch is submitted against a bug the bug state should move to "POST" state depending on the release version, can we achieve it? 08:51:08 atinmu, I think ndevos script does that 08:51:12 atinmu: thats not so easy, we dont know in advance how many patches for a bug would be needed 08:51:23 we could have the bug-id jenkins job check the bug status and ask the dev to move the bug. 08:51:36 s/the bug/the status of the bug/ 08:51:53 kshlm: either by posting a comment in the bug or in gerrit? 08:51:53 atinmu: yeah, I consider ASSIGNED -> POST & POST -> MODIFIED to be difficult transitions to automate 08:52:12 ndevos, gerrit. 08:52:22 kshlm: +1 08:52:26 The devs obivously will be looking at it. 08:52:39 kshlm: so we need the bug to be in POST state for a patch to be accepted? 08:53:05 hagarth, that seems like a good check. Don't know if kshlm meant that 08:53:24 hagarth: no, just a reminder in the gerrit comments, asking the dev to check and correct the status of the BUG in case all patches have been posted 08:53:26 We could make it rigid if we want, but what I meant was more of an FYI kinda thing. 08:53:54 yes, giving out a link in the warning would help 08:54:07 to bug-id 08:54:09 raghug: right, the link to the wiki page :) 08:54:20 kshlm, I won't consider that rigid. Anything that is not part of the workflow is more likely to be missed out 08:54:23 oh, yes and the bug 08:54:36 krishnan_p: agree with you 08:55:06 how about doing this? we make it FYI now and see how the process works 08:55:18 if it doesn't work well, we can make it a requirement 08:55:21 +1 to FYI 08:55:26 hmm, now I think about it, maybe merging of changes should only be possible for a bug in POST 08:55:43 that should prevent merging partial fixes 08:55:53 hmm.. 08:55:55 ndevos: But we want partial fixes? 08:56:15 pranithk: not really... 08:56:16 ndevos: For example I would hate to raise one bug per spurious failures fix I submitted... Its too much process 08:57:07 pranithk, would it be OK if you were expected to move the bug status to POST for the first patch you submitted to be merged? 08:57:10 ndevos: One more good reason for same bug for a set of patches is when you backport you know the complete set in just one bug. which is easy 08:57:29 pranithk, ndevos : whats the problem in having the bug state in POST even if u r sending multiple patches? 08:57:30 krishnan_p: That probably is not a bad idea. Moving to POST is fine I guess. 08:58:02 pranithk: one bug per issue makes it better for tracking... 08:58:07 pranithk, I think ndevos was recommending that we don't merge patches that are submitted against a bug that is not in POST state 08:58:24 ndevos: its a overkill for cases like pranith mentioned 08:58:32 krishnan_p: Yes that should be fine. 08:58:40 for regular bugs, I agree with you 08:58:42 but that opens up another problem, what would be the trigger to move these kind of bugs to MODIFIED? 08:58:52 raghug: but such fixes should be very few in number? 08:59:09 ndevos, did I rephrase you to pranithk correctly? 08:59:26 I dont like bugs like 'NetBSD port' that get new patches every time 08:59:40 krishnan_p: yep! 08:59:45 atinmu: I was hoping that POST -> MODIFIED can happen after patch acceptance 09:00:10 ndevos: Imagine these patches need to be backported to 3.5. It is extremely easy to find the set of patches. Even in gerrit the topic will be same 09:00:12 the problem is, how can we say the 'NetBSD port' would be functional, if we keep on adding changes to that bug? 09:00:45 ndevos: I guess the dev can move it to modified? 09:00:50 hagarth : thats for a regular bug, what if its an umbrella bug? 09:01:12 atinmu: +1, I guess we need a good distinction 09:01:16 we can some flags for ongoing bugs like netbsd port or coverity fix 09:01:22 s/can/can add/ 09:01:23 pranithk: but after a bug went to MODIFIED, no more patches should be posted for that bug 09:01:33 ndevos: Yes. 09:01:41 pranithk, while having one bug for related fixes help to determine relationship between fixes. But it makes it difficult for future developers/users to make a connection between the fix and a bug 09:01:44 lalatenduM: if there is a provision for it, its a good idea 09:01:53 so that we can automate the process 09:02:13 yup, and the script can ignore these bugs 09:02:13 krishnan_p: Like Atin mentioned it is an umbrella bug. 09:02:29 we can flag patches which need manual transition 09:02:33 pranithk, what do you mean when you say umbrella bug? 09:02:35 s/patches/bugs 09:02:37 krishnan_p: like netbsd port/spurious failures etc 09:02:43 lalatenduM: great idea 09:02:56 krishnan_p, like coverity fix umbrella bug 09:02:58 krishnan_p, take a coverity BZ as an example 09:03:00 lalatenduM: +1 09:03:27 I think we should propose this on gluster-devel to see how everyone feels about the process change 09:03:29 lalatenduM: that works, and in fact I think the script checks for keyword=Tracking and skips those 09:03:32 lalatenduM, +1 09:03:41 ndevos, awesoem! 09:04:03 any volunteers to propose this change on gluster-devel? 09:04:26 * lalatenduM vote for ndevos :) 09:04:33 pranithk, I think we should see if there are alternative ways of handling related issues via bugzilla 09:04:42 anyway he is doing most of it 09:04:46 lalatenduM: I am afraid that we might be overloading him :) 09:05:08 he already has more than one AIs here.. 09:05:26 hagarth, in that case I can be his minion :) 09:05:38 lalatenduM: awesome! 09:05:45 thanks lalatenduM! 09:05:52 lalatenduM++ :) 09:05:56 np :) 09:06:00 hmm, glusterbot? 09:06:03 #action lalatenduM to propose new gerrit - bugzilla workflow on gluster-devel 09:06:06 lalatenduM: :) 09:06:13 ndevos: karma not loaded for this channel 09:06:20 raghug, :) 09:06:32 cool, this is good progress 09:07:06 #topic triaging process 09:07:42 hagarth: We need more components and default owners. 09:08:00 pranithk: hchiramm_ is working on getting the default owner assignment right 09:08:12 pranithk: do you have a list of components that we need? 09:08:23 hagarth: put an action item on me 09:08:34 if you have suggestions on default owner assignment per component, please route them to hchiramm_ 09:08:38 pranithk: sure 09:08:53 #action pranithk to determine new components needed in bugzilla 09:09:02 hagarth: will hchiramm__ also be able to add new and modify components? 09:09:21 ndevos: no, he is just a router to bugzilla admins for us 09:09:44 ndevos: feel free to reach out to bugzilla admin directly too, we need not always have it go through hchiramm_ 09:10:02 now for the more important question 09:10:05 hagarth: hmm, but at least the Fedora admins have those permissions, I think we (as Gluster community) should be able to get them too 09:10:21 ndevos: I tried it but got a NO from bz-admin 09:10:29 ndevos: anyway let us try again 09:10:35 hagarth: ok, maybe I'll try it too then :) 09:10:40 ndevos: cool 09:10:57 Sorry but I just remembered this. ppai talked to me about a tool called Zuul which is being used in ovirt. This performs the kindof checks we want for our review/bug workflow. 09:11:04 how do we triage incoming bugs? 09:11:38 hagarth, yeah , it is dfficult without support from others 09:12:00 hagarth, also what is the role of the triager 09:12:10 should we have a window of time devoted to this? like this IRC meeting on a bi-weekly basis? 09:12:28 hagarth: what does triaging process consist of? 09:12:30 lalatenduM: ensure that new bugs get looked into and are assigned to the right developer 09:12:42 hagarth: That is too late. All the debugging info could be lost by then? 09:12:45 lalatenduM: also ensure that bugs are in the right state 09:12:48 pranithk: agree 09:13:10 can each of us take ownership for a few components? 09:13:20 hagarth, just putting them assigned , does not help much, 09:13:21 hagarth: that would be a better idea 09:13:25 and do triaging on an ongoing basis? 09:13:28 what I experiance till now 09:13:43 lalatenduM: new -> assigned can also come with some initial analysis of the problem 09:13:50 lalatenduM: ^ 09:14:03 hagarth, ok, may be I was missing that 09:14:17 then it is better comp owner triage bugs 09:14:26 lalatenduM: +1 09:14:46 raghug, ndevos, krishnan_p, kshlm, atinmu: your thoughts on this? 09:15:00 hagarth: component owner sounds a natural choice 09:15:16 since (s)he would have a better understanding 09:15:26 I think the sheer volume of bugs that get opened on glusterd, both correctly and incorrectly, makes me wonder if kshlm and my triage rates would be enough 09:15:26 any triaging helps, get basic information/debugging and an understanding of the problem - it does not need to come from the component owner 09:15:29 raghug, we have a page in gluster wiki for bug triage 09:15:39 krishnan_p: you should also take atin's help I guess 09:15:46 but yeah as krishnan_p mentions I am worried about keeping track of it 09:15:55 hagarth: I can take up replicate/locks/io-threads/io-stats/index/client/server to begin with 09:16:04 pranithk: cool 09:16:10 pranithk, Like ndevos mentioned triaging doesnt need in depth understanding 09:16:26 krishnan_p: we can look at scaling out when the volume becomes higher 09:16:26 hagarth: dht on me 09:16:41 I feel like it has to be done by multiple team, individuals can't take the complete burden 09:16:46 krishnan_p: But the bug would be difficult if it misses some critical info. For example in afr, changelog xattrs etc 09:17:01 pranithk: right 09:17:16 pranithk: anyone that triages can ask for assistance from the component owner? 09:17:23 ndevos: +1 09:17:25 atinmu: the volume of bugs on a regular basis is fairly low in glusterfs 09:17:26 ndevos, +1 09:17:27 pranithk, I dont deny that. In an ideal world I would like to triage all the bugs that enter glusterd component. Only that I haven't and doubt I will be able to :) 09:17:49 krishnan_p: Don't worry I will help ya 09:17:50 I noticed that triaging bugs takes lot of time 09:18:01 atinmu: on an average, we get < 5 bugs per day logged from folks outside the development team. 09:18:02 pranithk, thanks! 09:18:20 hagarth, the real trouble is triaging the backlog. 09:18:23 hagarth, I thought the count is much higher 09:18:28 lalatenduM: over time, the time spent will reduce 09:18:44 atinmu: no, I have bugzilla queries everyday to let me know what happened in the previous day 09:18:46 hagarth, yes , to start with we need good size triage teeam 09:18:49 atinmu: DOn't count the ones cloned from downstream. We don't really need to triage them :-) 09:19:01 pranithk, +1 09:19:10 pranithk: do you mean RHS by downstream? 09:19:15 hagarth: Yes sir 09:19:23 pranithk: there could be several downstreams to GlusterFS :) 09:19:30 hagarth, do u think it will be better to change the default assignee list to gluster-devel or a mailing list of people who are component maintainers ? 09:19:46 hagarth: ah! 09:20:10 :0 09:20:11 hchiramm_: I am open to inputs on that one 09:20:21 hchiramm__: thinking.. 09:20:36 hchiramm_: a mailinglist per component (or a group of) might be helpful 09:20:36 may be different email alias 09:20:47 hchiramm__: why not different alias? 09:20:50 lalatenduM: +1 09:20:54 gluster-bugs@gluster.org 09:21:04 nfs-bugs@gluster.org 09:21:08 ndevos, +1. A mailing list for a group of components sounds good. 09:21:19 ndevos, +1 09:21:21 lalatenduM, pranithk gluster-bugs@redhat.com is the one we are moving into 09:21:23 * ndevos does not want to see all glusterd bugs :) 09:21:24 actually we can have just one to begin with 09:21:27 Depending on the size of incoming bugs 09:21:52 lalatenduM, a blanket email address is not solving any problem 09:22:00 krishnan_p: +1 09:22:05 one list works too, bugzilla sets all possible headers in emails 09:22:11 ndevos, you can choose ignore everythng else except ur comp 09:22:15 ndevos, mailing list per component looks an overhead.. Isnt it ? 09:22:16 ndevos: Now you are talking 09:22:19 ndevos, what happened to you ignoring glusterd bugs ;) 09:22:27 krishnan_p: I think ndevos' idea is better 09:22:29 here are some numbers 09:22:30 :) 09:22:31 hchiramm_, what is the overhead 09:22:39 krishnan_p, then we will have a meeting like this again :) 09:22:49 we had 26 new bugs logged over the last 5 days - it boils down to around 5-6 new bugs / day 09:22:51 hchiramm_, I didnt follow that 09:23:11 hagarth, that might increase to 20 as we grow :) 09:23:20 I mean per day 09:23:30 lalatenduM, that means we are in trouble ;P 09:23:31 hagarth, then we should be able to triage the new bugs on daily basis, not sure about the backlog though, probably gradually we should clean them 09:23:36 lalatenduM: Hopefully devs also increase if we are growing :-) 09:23:48 pranithk, agree 09:23:53 so what is the consensus here? one list to start with and proliferate as needed? 09:24:01 yes 09:24:04 maybe starte with one list, and we'll see if glusterd needs its own list after a while ;) 09:24:10 ndevos, pranithk there are 45 components listed for GlusterFS 09:24:13 ndevos, lol :) 09:24:21 45 :O 09:24:24 I can help setup bug whines for your components (if you want to check your own bugs only) 09:24:25 hagarth: yes one list. But like ndevos said '[component-name]' prepended to the bug description 09:24:25 are u proposing 45 mailing lists ? 09:24:37 hagarth, that will help 09:24:43 pranithk: I mean email headers, not [tags] 09:24:46 pranithk: the component should be good enough for that 09:24:55 pranithk, if it is possible, nothin like it 09:25:07 if you need help with bug whines, please ping me after the meeting 09:25:14 another qn 09:25:20 45 mailing lists, no problem. You don't need to subscribe to all of them, unless you are pranithk :) 09:25:24 how do we handle the existing backlog? 09:25:27 hagarth, will do. 09:25:48 first degree of cleanup by ndevos's scripts ;) 09:25:50 pranithk: see http://paste.fedoraproject.org/117202/70700140/ for an example of the headers 09:25:52 Title: #117202 Fedora Project Pastebin (at paste.fedoraproject.org) 09:26:06 then I think it would be appropriate for each of us to triage our own components 09:26:27 it is going to be a laborious process, but let us get done with that 09:26:32 ndevos: got it 09:26:42 krishnan_p, yup :) 09:27:04 hchiramm_: can you please post the components in a wiki? we can then assign ownership for triaging there 09:27:22 ok.. will do 09:27:49 #action hchiramm_ to post list of components in wiki 09:27:49 hagarth, how do we set goals for the triage exercise. For eg. do we decide by when we want to finish triaging or do we have a goal of reducing the un-triaged bugs in the list? 09:28:09 hagarth, on a related note, how do we differentiate triaged vs un-triaged bugs? 09:28:09 krishnan_p, good question :) 09:28:21 krishnan_p: I think it will help to set a deadline for ourselves here 09:28:23 krishnan_p: I was just wondering about that too :) 09:28:29 New->assigned? 09:28:36 raghug: +1 09:29:05 hagarth, raghug, does moving to assigned mean that I commit to work on the bug in the near future. 09:29:06 ? 09:29:08 hagarth: Can we measure this? Amount of time it took for the bug to be moved from New->Assigned. We will know the components that need extra hand based on this? 09:29:18 can somebody find the wiki page on bug triaging 09:29:22 krishnan_p: +1, I dont know if I can assign bug with good confidence 09:29:24 krishnan_p: No you have the initial info and the info to debug the issue 09:29:29 I am not able to find it in the new website 09:29:29 krishnan_p: no, at least that we have done triaging of the bug 09:29:51 if we want that granularity, we need another state :) 09:30:03 lalatenduM: http://www.gluster.org/community/documentation/index.php/Bug_triage 09:30:04 Title: Bug triage - GlusterDocumentation (at www.gluster.org) 09:30:07 does this definition seem ok to everyone? (new -> assigned means triaging is complete) 09:30:21 hagarth, then we need to make this information public as well. 09:30:22 can we add new states? 09:30:24 ndevos, thanks 09:30:34 krishnan_p: sure, let us have it in a wiki page. 09:30:35 hagarth: I'm not sure to who I would assign bugs... 09:30:39 Assigned would seem to mean someone is working on it. 09:30:51 hagarth, I don't want to mislead the user that the bug is actively being worked on. 09:30:58 hagarth: if a state can be introduced, it would be better I think 09:31:09 * ndevos does not know how busy the others (santosh, and who else?) are 09:31:12 if that is complex, we can set the keyword triaged in whiteboards 09:31:14 How about a triaged label/flag? 09:31:16 what can't we have a flag called "triaged" ? 09:31:17 raghug: no new states please 09:31:23 pranithk: :) 09:31:30 raghug: new states are difficult to introduce 09:31:39 hagarth: whiteboard is good enough 09:31:55 hagarth: triaged in the whiteboard works for me 09:32:01 cool 09:32:02 hagarth, +1 09:32:17 +1 for triage in whiteboard 09:32:22 hagarth: should we also need ack from the assignee? 09:32:26 #action Triaged bugs will have triaged in whiteboard :) 09:32:29 ndevos: trick is to clone the bug and file it on RHS as well and it will magically be assigned by the managers and you will get the fix on upstream ;-) 09:32:31 or is it too process heavy? 09:32:39 raghug: you need not assign it 09:32:47 pranithk: hehe, yeah, that works quite well :D 09:32:54 New with internal white board 09:32:58 we seem to be running out of time here 09:33:06 yeah, got a meeting now 09:33:11 shall we re-convene in a week's time and check how we are doing? 09:33:16 raghug: not internal whiteboard, just the normal one 09:33:21 maybe we can get some of the process stuff going by then 09:33:29 ndevos: ok 09:33:30 hagarth, yeah 09:33:34 ndevos: right, whiteboard only 09:33:54 hagarth, OK 09:33:58 in fact, there is a Triaged keyword 09:34:09 ndevos: is it possible to set a default whiteboard status? 09:34:11 cool, I will send out one more invite for next week. thanks every body and see you all then! 09:34:19 ndevos: I like the keyword Triaged 09:34:19 I think it's easier to use the keyword fields 09:34:20 we need to update http://www.gluster.org/community/documentation/index.php/Bug_triage, with what ever we decided here 09:34:29 it would help searching untriaged bugs 09:34:43 raghug: +1 09:34:54 lalatenduM: let us just use keyword then, will help us from not typing Triaged in white board :) 09:35:08 ttyl 09:35:10 no typos! 09:35:19 hagarth, np :) 09:35:23 #endmeeting