15:01:34 <tstclair> #startmeeting
15:01:34 <zodbot> Meeting started Thu Dec 19 15:01:34 2013 UTC.  The chair is tstclair. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:34 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:01:35 <mizdebsk> hello
15:01:45 <tstclair> #meetingname BIG_DATA_SIG
15:01:45 <zodbot> The meeting name has been set to 'big_data_sig'
15:02:00 <mattf> howdy
15:02:08 <tstclair> #meetingtopic status_update
15:02:26 <tstclair> #topic collection_agenda
15:02:27 <mattf> fedora 20 released w/ hadoop !
15:03:12 <tstclair> rsquared, I have 1 item for the agenda - Apparently someone from the field is having issues.
15:03:27 <mattf> tstclair, is that topic and meetingtopic right?
15:03:32 <rsquared> tstclair: Regarding the datanode?
15:04:01 <tstclair> mattf, - collecting agenda items atm.
15:04:21 <tstclair> feel free to chime in and we'll roll through the topics.
15:04:42 <tstclair> Agenda Item 2 - Review status.
15:04:55 <tstclair> Agenda Item 3 - Issues
15:05:15 <tstclair> Anything else folks want to discuss.
15:05:18 <tstclair> rsquared, yup
15:06:10 <mattf> willb and i went to the env & stack wg meeting this week, we could give a bit of an update
15:07:22 <tstclair> Kyle, lets roll through them and we can always add on.
15:07:30 <tstclair> tap complete..
15:07:44 <tstclair> Kyle -> k
15:08:02 <tstclair> #topic datanode_fail
15:08:26 <mattf> i wasn't able to reproduce it. i used a docker managed container instead of a full system though
15:08:28 <tstclair> rsquared, any insight here.  I've recently updated and been running for some time.
15:09:15 <rsquared> tstclair: I'd need to see the log.  I'd guess he didn't start the namenode after formatting it and before starting the datanode, but without more details I can't say much.  I was waiting for him to respond to matt's request for log info
15:10:06 <tstclair> agreed.
15:10:22 <tstclair> rsquared, he also made the off-the-cuff comment re:dep chain.
15:11:01 <tstclair> have we compared full dep-chain vs other distros?
15:11:48 <rsquared> I wonder if he formated the namenode but never ran hdfs-create-dirs
15:12:29 <rsquared> Not sure the dep chain comparison is all that relevant.  Fedora may package differently, and pulling in multiple hundreds of packages for a single entity isn't uncommon.
15:12:35 <rsquared> Even outside of java
15:12:36 <mattf> afaik hdfs-create-dirs wouldn't make a difference until he tries to run a job
15:12:53 <willb> Seems like a lot of things need to happen in order on initial setup; could we provide a script?
15:13:43 <tstclair> rsquared, ^
15:14:00 <rsquared> willb: We have one.   hdfs-create-dirs
15:14:08 <rsquared> It will format and setup the namenode.
15:14:20 <rsquared> Then you just need to start the services as you require
15:14:46 <mattf> ahh, kinda hdfs-setup, because it does the initial service start
15:15:00 <mattf> all other times you have to either set the service to start on boot or start it manually
15:16:08 <tstclair> 1-time pad.
15:16:25 <rsquared> What hdfs-create-dirs does it format the namenode then create dirs that are needed in hdfs.  It will start the namenode if needed or leave it running if it was found running.  I'm open to a different name if that'll help clear things up.
15:17:07 <rsquared> If hdfs-create-dirs starts the namenode to do the dir creation, it will shut it down afterwards.
15:17:14 <mattf> you know what would be nice, if it was integrated into the hdfs command
15:17:57 <mattf> hdfs namenode -format && hdfs namenode -populate-fs, or somesuch
15:18:26 <willb> philosophical point:  there probably shouldn't be n different ways to set this up without one obviously correct one
15:19:02 <mattf> +1 ^^
15:19:32 <mattf> when we looked at this before, all the distros seemed to just document a handful of steps that rsquared automated into hdfs-create-dirs
15:20:34 <rsquared> bigtop was moving to providing a script for setup as well.  It's probably in the 0.7.0 release.
15:21:13 <tstclair> I think at this point we're kind of splitting hairs, as none of us have repro'd and many are running.
15:21:15 <mattf> 0.7.0 was released, iirc
15:21:23 <mattf> tstclair, aye
15:21:42 <tstclair> #topic status
15:21:44 <rsquared> Yep.  I've looked at 0.7.0 but haven't verified the script is in their release.
15:22:38 <mattf> status - congrats on getting hadoop into F20, released this week!
15:23:23 <tstclair> update: I've been working on a blog post to outline MR w/tachyon, in that process I found several oddities that I have not seen before b/c my original smoke testing wasn't very rough.  In digging deeper I've started filing issues upstream.
15:24:33 <tstclair> I'll likely push out updates to package and blog once everything is working.
15:24:44 <tstclair> as expected.
15:24:52 <rsquared> status - Fought battles with rawhide and hopefully in the final few tests before submitting hbase for review
15:25:02 <willb> update: since last time, I got my patch for sbt upstream (to support Ivy 2.3.0) and have been grinding through my review and packaging backlog
15:25:08 <mattf> i tried out hadoop in a docker container and failed - the current docker containers don't support "system" containers, one where systemd is run, so there's no systemctl start ..., you have to manually start the services
15:25:40 <tstclair> mattf, I'd expect failure honestly.
15:25:43 <mattf> i also spun some f20 kvm instances w/ virt-install, that worked pretty well except i didn't give them enough memory and a basic pi job nearly took down my laptop
15:26:17 <pmackinn> status - 1) datanucleus-{core,api-jdo,rdbms} in rawhide, creeping into f20 2) hive spec development based on latest 0.12.0 3) toyed with mizdebsk ivy xmvn resolver from bz 1012612; ran into CL issues, will have another go at some point
15:26:17 <tstclair> Has anyone thought about the systemd <> Docker issue?
15:26:54 <mattf> supposedly systemd w/i a docker container can work w/ the new libvirt-lxc backend to docker, but it's not upstream yet
15:27:13 <tstclair> good-to-know
15:28:29 <mizdebsk> latest xmvn snapshots (not yet in rawhide) have initial integration with ivy; resolution of system artifacts from ant with ivy tasks or from sbt works
15:28:37 <tstclair> status update 2: working on patches with to update mesos build and enable c++11.  Conversed with folks around ecosystem, features, etc.
15:28:47 <mizdebsk> if i get some confirmation that it works for you too i'll release it in rawhide and maybe f20
15:29:12 <pmackinn> mizdebsk, got a test suite for that? didn't see one in gh.c
15:29:42 <willb> mizdebsk, it's on my list to check out as soon as I can, but I'd love to see it in F20 as well as rawhide
15:29:57 <mizdebsk> pmackinn: not commited yet
15:30:07 <pmackinn> mizdebsk, ack
15:31:13 <mizdebsk> willb: i checked it with bootstrapped sbt in rawhide and sbt was able to resolve stuff from xmvn
15:31:40 <willb> mizdebsk, that is great!
15:31:50 <mizdebsk> (i don't know how to ass extra libs to sbt classpath, so i put them in scala/lib dir)
15:31:59 <mizdebsk> s/ass/add
15:32:52 <willb> I'll look at integration
15:33:02 <willb> another update:  mattf and I attended the Env and Stacks WG meeting and raised some of the concerns that we've seen around SIG packaging efforts:  poor integration of non-"standard" language-specific dependency managers into Fedora, upstreams maintaining compatibility with older JVMs (and thus being tied to older libraries than are in Fedora), etc.  They are going to be looking at our SIG's use cases as they flesh out a plan
15:33:02 <willb> for better integrating new language ecosystems in Fedora.
15:34:59 <tstclair> #topic issues
15:35:06 <tstclair> anything blocking?
15:36:55 <tstclair> If nothing, then we can end a bit early today.?.?
15:38:45 <tstclair> Kyle, I think we'll call it then.  Thanks everyone!  Happy Holidays, & Happy New Year!
15:39:21 <pmackinn> Happy Holidays Kyle!
15:39:29 <tstclair> #endmeeting