Introduction

DataFrames are useful constructs for data scientists, popularized by R and pandas.

Implementations of these packages are not natively distributed - their DataFrames struggle with big data. bigdf is a dataframe on top of Apache Spark. It is written as an internal Scala DSL with the look and feel of Pandas to make it easy for those familiar with Pandas to use it.

A gentle(but outdated) introduction is available at:

http://www.slideshare.net/codeninja4086/df-38948475?utm_source=slideshow03&utm_medium=ssemail&utm_campaign=iupload_share_slideshow

How to get it

• Choose a repos directory (make one if it doesn't exist)

export $REPOS = ~/Documents/repos

• Install scala and sbt (brew instructions below)

$ brew install scala
$ brew install sbt

• clone the bigdf repo

$ cd $REPOS
$ git clone [email protected]:ayasdi/bigdf.git

• sbt update, package, and test(optional) in the bigdf directory.

$ cd bigdf
$ sbt update
$ sbt test
$ sbt assembly

• Start a shell and begin playing. You need spark 1.4.x

$ cd $REPOS
$ ./spark/bin/spark-shell --jars ./bigdf/target/scala-2.10/bigdf-assembly-*.jar 

• https://github.com/AyasdiOpenSource/bigdf/wiki/ for examples
• Look at DFTest.scala for usage

Use with ipython notebook

$cd $REPOS
$git clone [email protected]:mattpap/IScala.git
$cd IScala
$sbt
>++ 2.10.4
>assembly
>^D

$ipython profile create scala

Add the following line to the three files ~/.ipython/profile_scala/ipython*_config.py

c.KernelManager.kernel_cmd = ["java", "-jar", "$ISCALA_PATH/lib/IScala.jar", "--connection-file", "{connection_file}", "--parent"]"

To start notebook

$cd <my notebooks directory>
$ipython notebook --profile scala

To install spark and bigdf to your local mvn repos

$brew install maven
$mvn install:install-file -Dfile=$REPOS/bigdf-transforms/lib/spark-assembly-1.2.0-SNAPSHOT-hadoop1.0.4.jar -DgroupId=org.apache -DartifactId=spark-assembly_2.10 -Dversion=1.2.0-SNAPSHOT-hadoop1.0.4 -Dpackaging=jar
$ mvn install:install-file -Dfile=$REPOS/bigdf-transforms/bigdf/target/scala-2.10/bigdf-assembly-0.2.jar -DgroupId=com.ayasdi -DartifactId=bigdf-assembly_2.10 -Dversion=0.2 -Dpackaging=jar

To start using bigdf in a notebook

%libraryDependencies += "org.apache" %% "spark-assembly" % "1.2.0-SNAPSHOT-hadoop1.0.4"
%libraryDependencies += "com.ayasdi" %% "bigdf-assembly" % "0.2"
%resolvers += "Local Maven Repository" at "file:///Users/mohit/.m2/repository"
%update

import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import com.ayasdi.bigdf._

val sc = new SparkContext("local[*]", "bigdf-intro")
val df = DF(sc, "/Users/mohit/Downloads/faredata2013/trip_fare_1.csv", ',', true)