The previous chapter covered the details of the installation and development tool setup that is required for developing and running data processing applications using Spark. In most of the real-world applications, the Spark applications can become very complex with a really huge directed acyclic graph (DAG) of Spark transformations and Spark actions. Spark comes with really powerful monitoring tools for monitoring the jobs that are running in a given Spark ecosystem. The monitoring doesn't start automatically.
Tip
Note that this is a completely optional step for running Spark applications. If enabled, it will give a very good insight into the way the Spark applications are run. Discretion has to be used to enable this in production environments, as it can affect the response time of the applications.
First of all, there are some configuration changes to be made. The event logging mechanism should be turned on. For this, take the following steps:
$ cd $SPARK_HOME
...