If you are running your job from a Spark CLI (for example, spark-shell, pyspark, spark-sql, spark-submit
), you can use the –-packages
command, which will extract, compile, and execute the necessary code for you to use the GraphFrames package.
For example, to use the latest GraphFrames package (version 0.3) with Spark 2.0 and Scala 2.11 with spark-shell, the command is:
> $SPARK_HOME/bin/spark-shell --packages graphframes:graphframes:0.3.0-spark2.0-s_2.11
If you are using a notebook service, you may need to install the package first. For example, the following section shows the steps to install the GraphFrames library within the free Databricks Community Edition (http://databricks.com/try-databricks).