Spark was developed as a general-purpose engine for large-scale data processing. It recently released its 1.0 version. Spark has two important features.

The first feature that Spark has is a resilient distributed dataset (RDD). This is a collection of elements partitioned across the nodes of a cluster, which can be operated on in parallel. A file on HDFS or any existing Scala collection can be converted to an RDD collection, and any operation on it can be executed in parallel. RDDs can also be requested to persist in memory, which leads to efficient parallel operations. RDDs have automatic fail-over support and can recover from node failures.

The second important feature of Spark is the concept of shared variables that can be used in any parallel operations. Spark supports two types of shared variables: broadcast variables and accumulators. Broadcast variables can be used to cache a value in memory on all the nodes, whereas accumulators are variables that can only be added up...