19:00:29 <nirik> #startmeeting proventesters (2011-09-21)
19:00:29 <zodbot> Meeting started Wed Sep 21 19:00:29 2011 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:29 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:00:29 <nirik> #meetingname proventesters
19:00:29 <nirik> #topic init process/agenda
19:00:29 <zodbot> The meeting name has been set to 'proventesters'
19:00:37 <nirik> any proventesters around today?
19:00:45 <OldFart> yes.
19:00:56 <tflink> yep
19:01:05 <nirik> cool.
19:01:16 * nirik will wait a few minutes and see if we get any more folks.
19:01:43 <mcloaked> I am here -
19:01:53 <nirik> welcome
19:01:56 <Cerlyn> I'm here
19:02:34 * Southern_Gentlem 
19:02:44 <nirik> excellent. A few folks. ;) Anyone here _not_ currently a proventester?
19:02:55 * jsmith is not
19:03:10 <nirik> jsmith: cool. It's easy to become one...
19:03:20 <jsmith> nirik: I imagine it is :-)
19:03:30 <jsmith> Don't mind me though... I'm mostly just lurking
19:03:43 <nirik> ok, let me do some intro materal and then we can move on to real business...
19:03:49 <nirik> #topic What's required of proventesters?
19:03:50 <nirik> http://fedoraproject.org/wiki/Proven_tester
19:04:01 <nirik> So, the above wiki page is the main landing page about proventesters.
19:04:48 <nirik> so, you can find info there about joining and what proventesters need to do.
19:05:09 <nirik> basically to join you have to agree that you have read and understand what you are supposed to do. ;)
19:05:37 <nirik> Any questions on process or the like? should be able to answer them by reading that page... (and it's links)
19:06:33 <nirik> ok, I'd like to talk process/tools for a minute.
19:06:37 <nirik> #topic Resources
19:06:50 <nirik> If folks aren't currently using them or know about them:
19:06:51 <mcloaked> Quick question: do you know how many proventesters there are now?
19:07:03 <nirik> mcloaked: yeah, I looked this morning. There's 80 people in the group.
19:07:11 <Cerlyn> A better question might be how many are active
19:07:20 <nirik> yeah, thats much harder to determine. ;(
19:07:37 <tflink> we could grab recent bodhi comments and compare to the list of members
19:07:45 <nirik> * fedora-easy-karma makes it easy to submit karma on updates if you have a bunch installed. Very useful.
19:07:54 <mcloaked> True - though does bodhi keep a track count of comments/karma additions?
19:08:01 <nirik> * bodhi has rss feeds if you want to know about just one packges updates.
19:08:13 <nirik> mcloaked: not sure.
19:08:25 <tflink> mcloaked: not sure what it keeps track of by user, I'm usually grabbing data for updates
19:08:34 <nirik> * bodhi command line can be great for pulling down updates that aren't in testing yet.
19:08:37 * brunowolff is here, but was doing work stuff when the meeting started.
19:08:59 <nirik> There's also an overall comments rss feed
19:09:21 <Southern_Gentlem> ?brb
19:09:24 <nirik> https://admin.fedoraproject.org/updates/rss/rss2.0?comments=True
19:09:34 <mcloaked> Yes often getting rpms that have yet to reach the repo is useful.
19:10:04 <tflink> either way, it is doable - even if that does involve some ugly code that pulls a crapton of data from bodhi :)
19:11:05 <nirik> bodhi also has a metrics page: https://admin.fedoraproject.org/updates/metrics/?release=F16
19:11:53 <nirik> So, I thought I would focus on f14 today... but I'm happy to discuss anything else folks feel is related and important...
19:12:57 * nirik listens to crickets.
19:13:01 <nirik> #topic F14
19:13:31 <nirik> so, there's currently 12 pending security updates in f14
19:13:50 <nirik> and 15 updates in critpath
19:14:01 <nirik> (there is some overlap there)
19:14:29 <nirik> Does anyone here have f14 instances still? ;)
19:14:29 * nirik has a vm
19:14:42 <tflink> my laptop is still running F14
19:14:46 <OldFart> I have F14
19:14:48 * Cerlyn has VMs and XOs
19:14:56 <Southern_Gentlem> ?can there be a mailing list other than testing that gets all the noise  from qa
19:15:10 <brunowolff> I don't really have any f15 instances. Just an old laptop, that I only use for special stuff and will upgrade probably the next time I touch it.
19:15:31 <brunowolff> Everything I normally use is f16 or f17 now.
19:15:39 <nirik> Southern_Gentlem: what email/info do you want to get thats lost in the noise currently?
19:16:01 <OldFart> The last time I looked at F14 security updates, I didn't feel qualified to test them.
19:16:08 <brunowolff> And I mostly have older hardware, so haven't been too interested in setting up VMs.
19:16:52 <nirik> some of them would seem to be pretty easy... like the kernel...
19:17:19 <nirik> but others are more esoteric.
19:17:19 <nirik> Would having more test plans help?
19:17:42 <tflink> for some things, yes
19:17:54 <OldFart> Yeah, the kernel, but the fixes are not known to me.  I can +1 from casual use.  Is that OK?
19:17:56 <tflink> I've run across more than one update that I haven't the foggiest idea how to test
19:18:11 <Southern_Gentlem> nirik,  stuff for f14 and f15 that is waiting for testing
19:18:13 <Cerlyn> A large part of my testing is not so much based on test plans, but trying to replicate the issue that was fixed
19:18:39 <tflink> how to hit the code that was changed to verify that it doesn't explode, rather
19:19:08 <nirik> Southern_Gentlem: so the updates-testing report? could you filter on that? or perhaps we could add a topic for it to make it easier to filter on?
19:19:11 <Cerlyn> yes; it doesn't make sense in my view to test the kernel for example if I don't have a computer with the updated module
19:19:17 <mcloaked> For kernel I still think it is useful to get comments if there are no regressions, but it would be helpful to have notes to remind people to install not just the appropriate kernel rpm but also the related perf, kernel-headers etc (I know perf has gone from the kernel builds for f16)?
19:19:32 <nirik> OldFart: yes. You don't need to verify everything, you need to verify that it shows no regressions for you and isn't breaking.
19:19:43 <Southern_Gentlem> nirik,  to have the updates testing report sent to a proven testers list
19:20:23 <nirik> that seems like kinda a waste for an entire list, IMHO
19:21:00 <Southern_Gentlem> nirik,  i think the reason stuff is not getting tested is no notification system for the proven testers
19:21:32 <Southern_Gentlem> we all get busy doing our things and that falls through the cracks
19:21:35 <OldFart> Used to be mail notifications, but too many complained about spam....
19:21:36 <nirik> for critpath packages, remember you simply need to test: http://fedoraproject.org/wiki/Critical_path_action#Actions
19:22:16 <nirik> OldFart: right.
19:22:41 <mcloaked> If a proventester is subscribed to the test list then a list of critpath and security updates gets sent out fairly regularly?
19:22:47 <nirik> I am sure if we started mailing proventesters on every updates push people would complain again.
19:22:48 <Southern_Gentlem> nirik,  or a compromise is that the package has sat there with no testing and the maintainer sends it to a mailing list
19:22:55 <nirik> mcloaked: yes, every push
19:23:19 <OldFart> Feasible to have an opt in mailing list just for testers?
19:23:26 <nirik> Subject: Fedora N updates-testing report
19:23:34 <nirik> OldFart: I would call the 'test' list that.
19:23:53 <OldFart> OK...
19:24:03 <Cerlyn> Goes back to my email earlier yesterday/today on wanting to be able to subscribe to package updates like one can for bug reports and commits.  (I was trying to phrase it in ways that won't overload infrastructure)
19:24:19 <nirik> Cerlyn: would the rss feeds meet that need?
19:24:38 <nirik> https://admin.fedoraproject.org/updates/rss/rss2.0?package=ipython&release=F16
19:24:49 <Cerlyn> nirik: The full updates list per release would, if an RSS reader exists which can filter
19:24:58 <Cerlyn> I don't want to subscribe to every sugar* package out there
19:25:10 <mcloaked> By the way if you add yourself to the bugzilla report for a package you are interested in then you will get mail for new comments - and that will usually include mail reporting when a fix package has been built.
19:25:12 <Cerlyn> that's a burden on infrastructure, and I was an infrastructure person
19:25:44 <Cerlyn> mcloaked: But does the package owner have to approve those memberships?  I was trying to figure out the system
19:25:44 <nirik> right, so you want a collection of packages, but in one feed or the like...
19:25:51 <Southern_Gentlem> test list to me is more for QA and (yes proventesters can follow that if they wish be it looks like alot of spam for people who want to test updates for current releases
19:26:03 <brunowolff> This is from back a bit, but perf isn't gone, the sub-package name changed to kernel-tools.
19:26:47 <mcloaked> Cerlyn: I think the package owner does not need to approve if you add yourself to the bug cc: list
19:27:02 <nirik> Cerlyn: no approval needed, just a bugzilla account.
19:27:24 <nirik> Cerlyn: you can also add yourself in pkgdb to packages you are interested in to be cc'ed on all bug reports against that package.
19:27:33 <nirik> (get ready for lots of email on some packages though)
19:27:43 <Cerlyn> It's a side topic but I noticed a lot of packages didn't have a QA contact in pkgdb
19:27:55 * Viking-Ice sneaks in...
19:27:58 <nirik> Cerlyn: I don't think that field is used for anything. ;(
19:28:19 <Cerlyn> bugzilla uses it IIRC
19:28:27 <mcloaked> brunowolff: thanks - about perf->kernel-tools
19:28:56 <nirik> Southern_Gentlem: I guess if there's enough interest for a new list for the updates emails, we could try it... perhaps we should ask on the test list if there is enough interest?
19:29:55 <Southern_Gentlem> example
19:29:57 <Southern_Gentlem> http://lists.fedoraproject.org/pipermail/test/2011-September/thread.html
19:30:29 <Southern_Gentlem> you have so scroll past the first page to see anything about f15 or f14 updates needing tested
19:30:46 <nirik> Cerlyn: it's set to extras-qa@fedoraproject.org for all fedora bugs... which goes to... /dev/null. ;)
19:31:13 <Cerlyn> One potentially also could generate a report on packages which have not received feedback after some time, and email that, as a warning that the packager can now push at will
19:31:17 <nirik> Southern_Gentlem: yeah, but couldn't you filter through it for the ones you want? or you mean many people aren't subscribed and are looking at the web archive?
19:32:30 <Southern_Gentlem> nirik,  so why dont separate the list so just the released releases updates that need testing and not devel
19:32:40 <Viking-Ice> if you are part of the QA community you should be subscribed and all QA task requested should be posted to the test list not -devel
19:33:15 <Southern_Gentlem> nirik,  i dropped the test list because of 60 messages being delivered to get the ones that deal with me
19:33:21 <nirik> Southern_Gentlem: well, I fear splitting that out would just result in another list that not many people would subscribe to, since most folks interested are already subscribed to test.
19:33:32 <nirik> but I could be wrong.
19:33:53 <tflink> either way, sounds like time to ask for input from test@ ?
19:33:59 <mcloaked> nirik: my feeling is that you are right...
19:34:01 <Southern_Gentlem> nirik,  stuff isnt getting tested tells me that most feel the test list is to noisey
19:34:05 <Viking-Ice> not really we dont want split which which results in divided sub community
19:34:17 <Southern_Gentlem> Viking-Ice,  you already have one
19:34:21 <nirik> yeah, lets ask for input on the test list... and see if there's enough folks who want another list...
19:34:24 <Viking-Ice> bugzapper
19:34:44 <Viking-Ice> if that's the route you are heading have at it
19:35:10 <nirik> so, moving back to f14. ;) What can we do to better handle these? would folks like to go thru one by one in the meeting? or should we look at hard to test ones? easy to test ones?
19:35:16 <Viking-Ice> we struggle hard to not divide the zapper stuff was due to autoqa only discussion on QA
19:35:17 <mcloaked> There could be other reasons - like people are busy, or for install testing when there are major issues giving bad or unbootable installs it can slow things down.
19:35:44 <nirik> I think it's mostly that people don't know they can test/provide feedback
19:36:00 <nirik> visibility is just low.
19:36:10 <tflink> you mean among proventesters or among fedora users?
19:36:22 <mcloaked> Is there any chance that f14 is being sidelined by people trying to test later (eg f16) packages and getting short of time?
19:36:28 <nirik> tflink: fedora users.
19:36:50 <tflink> I think part of it is worry that they'll tank their system
19:37:03 <nirik> I know as a package maintainer when I fix a bug, ofter the reporter will say "great, works here" in the bug, but won't add karma unless I specifically ask them to, and often not even then.
19:37:14 * athmane is late
19:37:22 <mcloaked> tflink: do testers mainly use production systems in their testing or spare machines?
19:37:31 <tflink> have no idea
19:37:38 <nirik> mcloaked: yes, f14 suffers from many things:
19:37:48 <tflink> I know that my F14 system is a production machine that I use every day
19:37:52 <nirik> * fedora is a fast moving distro, so most of the energy is on f16+
19:38:10 <nirik> * the people running f14 just want it to keep working, so don't want to test dangerous things.
19:38:18 <OldFart> I have F14, F15, F16 and rawhide on one machine...no VMs and try to test when I see the updates testing report.  But,
19:38:20 <nirik> * f14 people are less likely to be contributors I suspect...
19:38:35 <mcloaked> In my own case for potentially "tanking" tests I use a machine that won't matter if it breaks - but for production machines I will test packages selectively.
19:38:54 <OldFart> most often I am not familiar enough with the package or, if familiar, with the bug or security fix.
19:39:15 <athmane> imho f14 (fedora old stable in general) needs a less strict update policy
19:39:27 <nirik> OldFart: for that perhaps the test list or #fedora-qa could help... ie, someone who knows the package could help others test it
19:39:49 <mcloaked> OldFart: you can still install a package in critpath and check if there are no regressions - that is valid process.
19:39:49 <nirik> athmane: EBOYONDTHESCOPEOFTHISMEETING. ;)
19:40:15 <OldFart> Test cases would help....but they add a burden to the developer/maintainer.
19:41:10 <nirik> sure, but they are great to have...
19:41:22 <nirik> but most of them time currently, we don't have them
19:42:05 <nirik> as a side note, some of the f14 critpath updates should drop off the report soon... some of the obscure X drivers are dropping out of critpath
19:42:32 <nirik> qxl and openchrome
19:43:21 <mcloaked> On testcases - sometimes people could meet on here and discuss how to test a particular package perhaps? Might help where there is no testcase available?
19:43:41 <nirik> I would think these would be not too bad to verify critpath functionality on: kernel, gnupg2, curl, udev,
19:43:51 <nirik> mcloaked: absolutely.
19:44:09 <OldFart> +1
19:44:36 <nirik> that was the last item I had for today: should we bother meeting again? are meetups helpfull or useful in any way?
19:45:02 <athmane> there's a ticket for criticalpath test cases creation => https://fedorahosted.org/fedora-qa/ticket/154
19:45:13 <OldFart> Helpful....more should show up over time
19:45:17 <Viking-Ice> it should be mandantory for maintainers to provide how to debug their component and provide test cases
19:45:20 <mcloaked> Also I guess there is #fedora-qa for discussions?
19:46:11 <nirik> mcloaked: yep.
19:46:38 <nirik> athmane: good link...
19:46:47 <mcloaked> I guess that the log of meetings such as this can be seen when they are sent out to the test list - so they are useful even if only small numbers
19:46:54 <nirik> Viking-Ice: not really enforceable... but in an ideal world...
19:47:05 <athmane> we have ~15 test case of 432 critical path pkg
19:47:12 <tflink> forcing test cases != good test cases
19:47:37 <Southern_Gentlem> and no testing ==no testing
19:47:38 <tflink> does not necessarily lead to, anyways
19:47:39 <nirik> athmane: long way to go for sure.
19:48:10 <Viking-Ice> nirik, why not
19:48:14 <mcloaked> Also worth stressing that if testing leads to problems seen then it is important to file bz.... not just a comment in bodhi
19:48:16 * nirik just finished rebooting his f14 vm...
19:48:22 <nirik> mcloaked: +1
19:48:34 <Viking-Ice> nirik if you want your component in fedora you need to provide us with that info
19:48:43 <nirik> Viking-Ice: what do you do if they don't?
19:48:51 <nirik> and what if they are horrible at testing?
19:48:56 <Viking-Ice> dont introduce the component
19:49:12 <tflink> what do you do about the bad test cases that just introduce noise?
19:49:30 <Southern_Gentlem> all of this is not addressing the issues
19:49:32 <Viking-Ice> if the packager and or maintainer does not know how to debug and test his own component <---
19:49:51 <nirik> Southern_Gentlem: suggestions?
19:50:01 <tflink> knowing how to debug and being willing to write out test cases are not the same thing
19:50:22 <Viking-Ice> mcloaked, reporters have come and gone mostly because they dont get responded to in bugzilla
19:50:42 <mcloaked> For some kinds of test fails are quite tough to get useful debug info - eg boot crash - how to get backtrace, install crash - how to get bactrace - similar for serious kernel issues when they happen even if they are rare.
19:50:56 <Southern_Gentlem> nirik,  other than breaking out another list that has what needs testing nope
19:51:13 <nirik> Southern_Gentlem: ok, will float that idea to the test list... or did you want to?
19:51:17 <Southern_Gentlem> i see notification to proven testers as the bigest problem
19:51:40 <nirik> we didn't really see much more testing back when we spamed all proventesters I don't think...
19:51:46 <nirik> but there was likely fewer back then.
19:52:04 <Southern_Gentlem> nirik,  go ahead but i see most of the test list are testing devel and could care less about the currcent releases or we wouldnt be in this shape
19:52:28 <Viking-Ice> yeah somepeople have thrown updates-testers of that list
19:52:34 <mcloaked> Within bodhi you can get a list of security packages such as at https://admin.fedoraproject.org/updates/F14/security?_csrf_token=0834a0859819089ae1a83c6e8e96ed3b63a57934
19:52:38 <Viking-Ice> as in GA testers
19:52:45 <nirik> sure, for mentioned reasons there's less interest in the stable releases. But really there's not that many critpath/security ones.
19:53:01 <mcloaked> Might be nice if there was also a link to current critpath package updates per Fedora version too?
19:53:17 <nirik> I really think it wouldn't take too much for a group to process thru them... I think it's reachable.
19:53:51 <Southern_Gentlem> nirik,  i think a most a week email of list of packages waiting to be tested in the current releases would be easy
19:53:57 <Viking-Ice> still reporters/testers are faced with nonresponsive maintainers
19:54:04 <Viking-Ice> and EOL
19:54:07 <tflink> mcloaked: you mean other than https://admin.fedoraproject.org/updates/critpath?unapproved=True&release=F16
19:54:34 <Viking-Ice> seriously when are we going to target the root of the problem instead of working around it
19:54:41 <nirik> Southern_Gentlem: well, emailing the updates-testing reports to another list would be easy...
19:55:10 <Southern_Gentlem> nirik,  only when stuff needs to be tested no status after that
19:55:11 <mcloaked> tflink: Excellent - I was unaware of that construction!  I will certainly use it in the future.
19:55:12 <nirik> Viking-Ice: because there's no solution to that unless you intend to hire and task people with full time development/qa?
19:55:27 <Viking-Ice> drop the ownership model
19:55:34 <Viking-Ice> get the devs working together etc
19:55:49 <nirik> Southern_Gentlem: that would be a lot harder, as it would need bodhi code changes. ;(
19:55:50 <Viking-Ice> necessary and needed change
19:56:08 <nirik> Viking-Ice: beyond the scope of this meeting.
19:56:08 <mcloaked> tflink: It would be even nicer if that list of unapproved critpath packages was a link on the left side of the bodhi interface!
19:56:21 <nirik> Anyhow, 4 min left until next meeting...
19:56:29 <nirik> should we meet again next week at this time? another time? not at all?
19:56:30 <tflink> I'm not convinced that would work all that well, my experience is that when there's no ownership, nothing gets done
19:56:38 <Viking-Ice> nirik, yeah bury the root cause once more, nonresponsive maintainers are the root of alot of problems
19:56:48 <tflink> mcloaked: yeah, you have to click through a bunch of stuff to find that link
19:56:52 <Southern_Gentlem> nirik,  and only updates-testing for current releases not devel
19:57:02 <nirik> Southern_Gentlem: devel has no updates-testing. ;)
19:57:19 <Southern_Gentlem> nirik,  what does f16 go to at the moment
19:57:19 <OldFart> I'm ok with a meeting next week....prefer an hour earlier.
19:57:25 * nirik waits for votes. will close the meeting in a minute or two
19:57:30 <Southern_Gentlem> +1
19:57:34 <tflink> I'm also ok with next week
19:57:41 <mcloaked> I believe I can make next week same time
19:57:46 <nirik> next week at 18?
19:57:50 <athmane> +1
19:57:55 <Viking-Ice> +1
19:57:57 <Cerlyn> +
19:58:04 <Southern_Gentlem> +1
19:58:11 <OldFart> +1 can't stay for the hour
19:58:12 <mcloaked> +!
19:58:16 <nirik> #agreed will meet next week at 18UTC.
19:58:31 <nirik> Thanks for coming everyone... continue discussion over in #fedora-qa and on list...
19:58:37 <nirik> #endmeeting