19:43:04 #startmeeting 19:43:04 Meeting started Mon Mar 1 19:43:04 2010 UTC. The chair is mchua. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:43:06 Useful Commands: #action #agreed #halp #info #idea #link #topic. 19:43:17 oh yay, mel's talkin out loud and i get to see it LIVE! and in person! 19:43:37 For those listening in: I'm playing real quick with some brute-force python scripts to explore the idea of automatically gathering contributions for a FAS user from a bunch of different things FAS authenticates against. 19:44:10 this is for anthro-diana 's research, since she has the problem of trying to figure out who the "current active contributors" are, where we don't have one clear-cut way of distinguishing this 19:44:29 and since people can contribute in so many different ways we can't just say, like, "ok, everyone who edited the wiki is a contributor" because some people code, etc. 19:44:35 rbergeron: :) 19:44:41 * mchua documents evvvverything. 19:45:04 * rbergeron nods 19:45:07 this is conundrum 19:45:08 so my idea was to look at the different services that FAS auths into and see if you can write a little script to scrape and parse the different activity logs for each user 19:45:16 for instance, wiki edits 19:45:19 #topic wiki edits 19:45:30 You can see user contributions for any user on the wiki. 19:45:33 For instance 19:45:36 #link https://fedoraproject.org/wiki/Special:Contributions/Mchua 19:45:50 #info URL is https://fedoraproject.org/wiki/Special:Contributions/FAS_USERNAME 19:46:08 * rbergeron wonders if there are any wiki contributions that get done by an automagic bot by people who may not be around? 19:46:32 * rbergeron is thinking like... changes to some code base get their descriptions passed along to wiki, under someone's name 19:46:50 rbergeron: I don't think we have many (if any) of those kinds of bots on our wiki 19:46:58 just thinking out loud :) 19:47:08 and generally speaking, wiki rules are usually set (and I think Fedora is no exception here, though I'm not 100% sure) such that bots *must* be registered and tagged as such 19:47:12 rbergeron: it's good :) 19:47:24 since you can only log into the wiki via your FAS account, all wiki edits are done by someone's FAS account. 19:47:29 So this is a pretty good count afaict. 19:47:35 of "wiki activity by a person" 19:47:39 * rbergeron nods 19:47:50 (it also means that all wiki activity by a person *will* be logged under their FAS name because you can't edit anonymously, etc) 19:47:55 um... where was I? 19:47:56 Ah yes. 19:48:20 so if you had a list of FAS accounts, you could chug through with a script and scrape https://fedoraproject.org/wiki/Special:Contributions/FAS_USERNAME for each FAS_USERNAME in list. 19:48:30 so I'm going to hack a twill script together to do exactly that. 19:48:44 and it will be kludgetastic and we will get back lots of broken fugly HTML and it will be wonderfully inefficient. 19:48:48 wah-hah. 19:49:19 #topic twill 19:49:31 to do this, I'll be trying out twill, which is a little python library for web-browsing 19:49:36 what does "new accounts" under show contributions of new accounts mean? 19:49:47 rbergeron: hm? 19:49:50 is there some sort of defined time period 19:50:02 https://fedoraproject.org/wiki/Special:Contributions/FAS_USERNAME 19:50:04 first radio button 19:50:12 uh... /me tries 19:50:21 Oh, that's interesting. 19:50:33 That looks like it's the list of all contributions made by users with an account creation date less than $datetime 19:50:39 for some $datetime I don't know. 19:50:44 omg! all these new people have been contributing to release slogan ;) 19:50:46 yay 19:50:57 another thing is to have not only the contributions, but also a log: https://fedoraproject.org/wiki/Special:Log/Mchua if there something hase moved or changed you can script diffs, etc. out of it... 19:51:02 That's a guess, not necessarily the correct one :) 19:51:06 wonderer: oooo yes. 19:51:14 yeah, i'm not sure 19:51:19 wonderer: so you could get, not just number of edits, but number of lines of edits, etc 19:51:23 even finer granularity. 19:51:56 or whether edits are being made to pages that a lot of other people are contributing to at that time (they're "joining the conversation" vs working in their own little sandbox) 19:52:08 and... anthro-diana, you've got a data goldmine here if you can figure out how to script stuff to get it :) 19:52:21 uh, so, twill! 19:52:25 #link http://twill.idyll.org/ 19:52:44 I'm sure there's a better way to do this, btw 19:53:03 let's see if twill is packaged..... 19:53:08 * mchua does "yum search twill" 19:53:13 HAWT 19:53:20 some do it with curl, but i'm years out of scripting ;-) 19:53:21 #info twill package is called "python-twill" 19:54:00 wonderer: point - I've played with both a bit, and vaguely remember liking twill more, but I have no idea why, so it might be worth trying for a 2nd round 19:54:23 * mchua does "sudo yum install python-twill" and watches the dependencies come rolling in 19:54:58 if you have some script "go over the wiki and do X" and THEN "forward/compare/diff that with planet.fp.o" would be nice... 19:55:25 wonderer: yep yep. Idea would be to get a set of scripts that could extract this data for any user on any FAS-authenticated service, and then we could cross-compare... anything we wanted, really 19:55:37 wonderer: "do people tend to git commit a lot before they make a blog post?" etc 19:56:08 hello 19:56:37 mchua: the thing is that this idea not only works on FAS based 8because most things you get without logins) but also crawl over the blogs, the websites, etc. for further stuff... 19:56:44 hy nmarques. 19:57:53 wonderer: Exactly! 19:58:28 mchua: so, at the end it CAN be a thing like "user X has some about