The first thing to do in order to access Spark is to create a SparkContext. The SparkContext initializes all of Spark and sets up any access that may be needed to Hadoop, if you are using that as well.
The initial object used to be a SQLContext, but that has been deprecated recently in favor of SparkContext, which is more open-ended.
We could use a simple example to just read through a text file as follows:
from pyspark import SparkContextsc = SparkContext.getOrCreate()lines = sc.textFile("B05238_04 Spark Total Line Lengths.ipynb")lineLengths = lines.map(lambda s: len(s))totalLength = lineLengths.reduce(lambda a, b: a + b)print(totalLength)
In this example:
- We obtain a
SparkContext - With the context, read in a file (the Jupyter file for this example)
- We use a Hadoop
mapfunction to split up the text file into different lines and gather the lengths - We use a Hadoop
reducefunction to calculate the length of all the lines - We display our results
Under Jupyter this looks like...