Book Image

Learning Jupyter 5 - Second Edition

Book Image

Learning Jupyter 5 - Second Edition

Overview of this book

The Jupyter Notebook allows you to create and share documents that contain live code, equations, visualizations, and explanatory text. The Jupyter Notebook system is extensively used in domains such as data cleaning and transformation, numerical simulation, statistical modeling, and machine learning. Learning Jupyter 5 will help you get to grips with interactive computing using real-world examples. The book starts with a detailed overview of the Jupyter Notebook system and its installation in different environments. Next, you will learn to integrate the Jupyter system with different programming languages such as R, Python, Java, JavaScript, and Julia, and explore various versions and packages that are compatible with the Notebook system. Moving ahead, you will master interactive widgets and namespaces and work with Jupyter in a multi-user mode. By the end of this book, you will have used Jupyter with a big dataset and be able to apply all the functionalities you’ve explored throughout the book. You will also have learned all about the Jupyter Notebook and be able to start performing data transformation, numerical simulation, and data visualization.
Table of Contents (18 chapters)
Title Page
Packt Upsell
Contributors
Preface
Index

First Spark script


Our first script reads in a text file and sees how much the line lengths add up to, as shown next. Note that we are reading in the Notebook file we are running; the Notebook is named Spark File Lengths, and is stored in the Spark File Lengths.ipynb file:

import pyspark
if not 'sc' in globals():
    sc = pyspark.SparkContext()
lines = sc.textFile("Spark File Line Lengths.ipynb")
lineLengths = lines.map(lambda s: len(s))
totalLengths = lineLengths.reduce(lambda a, b: a + b)
print(totalLengths)

 

In the print(totalLengths) script, we first initialize Spark, but only if we have not done so already. Spark will complain if you try to initialize it more than once, so all Spark scripts should have this if statement prefix.

The script reads in a text file (the source of this script), takes every line and computes its length, and then adds all the lengths together.

A lambda function is an anonymous (not named) function that takes arguments and returns a value. In the first case, given...