15:03:08 #startmeeting 15:03:08 Meeting started Thu Nov 21 15:03:08 2013 UTC. The chair is tstclair. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:03:08 Useful Commands: #action #agreed #halp #info #idea #link #topic. 15:03:21 #meetingname BIG_DATA_SIG 15:03:21 The meeting name has been set to 'big_data_sig' 15:03:39 #meetingtopic Collect_Agenda_Items 15:03:42 morning 15:04:35 Morning folks, right now I figure we can collect agenda items. e.g. - For me I have reviews which have been held up for a while. 15:04:59 btw, congrats on getting tachyon in! 15:05:11 mattf, thank you ;-) 15:06:08 pmackinn, willb, mattf, rsquared - other agenda items? 15:06:08 No agenda items from me atm 15:06:11 agenda - if i read #fedora-bigdata correctly, it sounds like willb is still deep in sbt/scala, could you share some of the experience / existing issues w/ everyone? 15:06:15 nothing here 15:06:31 absolutely 15:07:22 #topic status 15:07:27 agenda - there were also a few new folks that showed up looking for ways to jump in, but i don't seem them in the channel atm, so maybe table a topic of figuring out short tasks for initial involvement 15:08:04 We could update the SIG page to add "On Boarding" 15:08:08 status - i've been poking a few reviews since it seems like we're developing a backlog 15:08:14 I could take that as an action item 15:10:35 status - I've been circling back to update mesos review package, however there has been a fundamental change in the startup routine for spawn'd processes which causes the whole testsuite to hang under mock. At this point I'm inclined to disable the tests entirely. (New issue w/latest rebase) 15:12:33 rsquared, pmackinn any new status updates to share with the whole SIG? 15:12:44 status - i emailed rbergeron about big data sig efforts making their way into f20 announcements, waiting to hear back 15:13:05 mattf, It would be nice if it were a headliner. 15:13:18 agreed 15:13:33 I'll get to my status when we come around to my topic 15:13:44 status - time-api in updates pending; working on datanucleus-core but new xmvn 1.3 has thrown a wrench into that effort 15:14:06 pmackinn, what's the issue? 15:14:54 willb, fire away whenever. 15:15:08 xmvn 1.3 doesn't like seeing system-scoped deps now, breaks install phase 15:15:34 status - There's some movement upstream to pull in some of the hadoop patches, and I've been working to support that effort. hbase is in spec creation phase. There are a number of jamon packages in review, but it looks like those are going to be picked up. Other hbase deps will need to be reviewed and approved before hbase and get in 15:16:31 rsquared, is there an hbase page with a link to the deps? I may be able to take a couple of reviews 15:16:57 rsquared, have you reached out to stevel in email or twitter? 15:17:18 * rsquared doesn't twitter 15:17:32 OK, I will start with a high-level overview (apologies to those of you who've already heard this) 15:17:46 tstclair: https://fedoraproject.org/wiki/SIGs/bigdata/packaging/Hbase#Missing_Java_Dependencies 15:18:23 We need sbt in Fedora to support most of the Scala ecosystem. From a Big Data SIG perspective, Spark and Shark are very interesting projects that we could package given sbt support, but there are a lot of other cool Scala projects that would be great in Fedora for a general audience. 15:18:43 mattf: You mean stevel in the hadoop team? 15:19:08 rsquared, yeah, looks like he's on a tear updating poms 15:19:45 From 30,000 feet, the big issues with sbt from a Fedora perspective are that it builds itself, needs to use Ivy to find its own dependencies (many of which also need sbt to build), and is designed to be totally self-contained (i.e. you download a launcher and it pulls down deps, including the Scala libraries and compiler from remote Ivy repositories). 15:20:36 The scala package itself was also broken for some time in Fedora, but I was able to get it working again and one of the maintainers built it with my spec and patches. 15:20:50 mattf: He's committed a few of the patches I submitted and I've intereacted over jira and some in e-mail via the common-dev mailing list when needed. 15:20:53 We don't have great Ivy support in Fedora at the moment, but I understand that Mikolaj is working to improve that. And we've seen Fedora do amazing things with similar language-specific ecosystems (maven, rubygems, python eggs, etc). For now, I've written a script that generates a local Ivy repository from Fedora's maven metadata and am using that to build sbt. 15:21:20 mattf: There's no real plan for hadoop releases so not sure how to push for inclusion of more difficult changes into a roadmap 15:22:16 Earlier this year, I did a lot of work to get sbt to build itself against a local Ivy repository, so I could use this script. (I also got a patch in to sbt that will make it easier to integrate with Fedora in future.) In the last couple of weeks, I've been ironing out problems with that work: moving dependency versions from F18->F19 and from sbt 0.12 to 0.13 (and thus scala 2.10 instead of scala 2.09), squashing random 15:22:16 dependencies on my environment (like jars that sbt had cached in my home dir), fixing issues that crop up in rpm builds but not in local builds, etc. 15:22:29 rsquared, there's a plan, just hard to ferret out. you should consider reaching out to him. 15:24:42 mattf: It didn't seem that way when discussing java 7. The impression I got was no one had any roadmap past the 2.2.0 release so they couldn't properly plan for java 7/jetty 9 inclusion 15:25:28 mattf: If there was a plan, no one with knowledge of it spoke up in the thread 15:26:28 So I've been building up a spec for sbt from my experience building it locally. It's been somewhat slow going because the workflow is "fire off a build and then add any transitively-carried Ivy dependencies that we didn't already have." Based on the volume of errors with each build, though, I suspect I'm very close to getting a bootstrap build of sbt under rpmbuild. 15:27:50 I should also note that the sbt upstream developer has been extremely helpful, even with non-sbt-specific issues like Ivy details 15:28:29 sounds like a tedious effort. anything we should do w/ the java sig (or others?) to make it less tedious? 15:29:14 yes, first-class Ivy support in Fedora (a la xmvn) is basically a must to make this supportable in the long term 15:29:49 right now, I'm at ~100 dependencies in the locally-generated Ivy repository 15:33:01 Are there any finite self contained items, 1st class Ivy support is a bit much. 15:33:21 I'm sure the java-sig guru's would be better suited for that one. 15:33:56 tstclair, it is already on Mikolaj's plate IIUC 15:34:44 but I think this case (along with other JVM build tools that rely on Ivy to lesser extents) is an important motivation 15:35:26 rsquared, I'll try to pickup a review today. 15:36:12 are there any other items folks wanted to discuss? 15:37:01 nothing here 15:37:12 tstclair: The jamon packages depend on each other iirc. Likely only 1 review can get done at a time until that review is built in rawhide. 15:37:28 rsquared, ack 15:38:28 ok if there are no more items, we can call it. Thanks everyone! 15:38:38 thanks tim! 15:38:41 thanks tstclair 15:38:52 #endmeeting