You can build Spark from the source code, or you can download precompiled binaries from http://spark.apache.org. For a standard use case, binaries are good enough, and this recipe will focus on installing Spark using binaries.
At the time of writing, Spark's current version is 2.1. Please check the latest version from Spark's download page at http://spark.apache.org/downloads.html. Binaries are developed with the most recent and stable version of Hadoop. To use a specific version of Hadoop, the recommended approach is that you build it from sources, which we will cover in the next recipe.
All the recipes in this book are developed using Ubuntu Linux, but they should work fine on any POSIX environment. Spark expects Java to be installed and the JAVA_HOME environment variable set.
In Linux/Unix systems, there are certain standards for the location of files and directories, which we are going to follow in this book. The following is a quick cheat sheet:
Directory | Description |
| This stores essential command binaries |
| This is where host-specific system configurations are located |
| This is where add-on application software packages are located |
| This is where variable data is located |
| This stores the temporary files |
| This is where user home directories are located |
Here are the installation steps:
- Open the terminal and download the binaries using the following command:
$ wget http://d3kbcqa49mib13.cloudfront.net/spark-2.1.0-bin-hadoop2.7.tgz- Unpack the binaries:
$ tar -zxf spark-2.1.0-bin-hadoop2.7.tgz- Rename the folder containing the binaries by stripping the version information:
$ sudo mv spark-2.1.0-bin-hadoop2.7 spark- Move the configuration folder to the
/etcfolder so that it can be turned into a symbolic link later:
$ sudo mv spark/conf/* /etc/spark- Create your company-specific installation directory under
/opt. As the recipes in this book are tested on theinfoobjectssandbox, useinfoobjectsas the directory name. Create the/opt/infoobjectsdirectory:
$ sudo mkdir -p /opt/infoobjects- Move the
sparkdirectory to/opt/infoobjects, as it's an add-on software package:
$ sudo mv spark /opt/infoobjects/- Change the permissions of the
sparkhome directory, namely0755 = user:read-write-execute group:read-execute world:read-execute:
$ sudo chmod -R 755 /opt/infoobjects/spark- Move to the
sparkhome directory:
$ cd /opt/infoobjects/spark- Create the symbolic link:
$ sudo ln -s /etc/spark conf- Append Spark binaries path to
PATHin.bashrc:
$ echo "export PATH=$PATH:/opt/infoobjects/spark/bin" >> /home/hduser/.bashrc- Open a new terminal.
- Create the
logdirectory in/var:
$ sudo mkdir -p /var/log/spark- Make
hduserthe owner of Spark'slogdirectory:
$ sudo chown -R hduser:hduser /var/log/spark- Create Spark's
tmpdirectory:
$ mkdir /tmp/spark- Configure Spark with the help of the following command lines:
$ cd /etc/spark $ echo "export HADOOP_CONF_DIR=/opt/infoobjects/hadoop/etc/hadoop" >> spark-env.sh $ echo "export YARN_CONF_DIR=/opt/infoobjects/hadoop/etc/Hadoop" >> spark-env.sh $ echo "export SPARK_LOG_DIR=/var/log/spark" >> spark-env.sh $ echo "export SPARK_WORKER_DIR=/tmp/spark" >> spark-env.sh
- Change the ownership of the
sparkhome directory toroot:
$ sudo chown -R root:root /opt/infoobjects/spark