Book Image

Big Data Analysis with Python

By : Ivan Marin, Ankit Shukla, Sarang VK
Book Image

Big Data Analysis with Python

By: Ivan Marin, Ankit Shukla, Sarang VK

Overview of this book

Processing big data in real time is challenging due to scalability, information inconsistency, and fault tolerance. Big Data Analysis with Python teaches you how to use tools that can control this data avalanche for you. With this book, you'll learn practical techniques to aggregate data into useful dimensions for posterior analysis, extract statistical measurements, and transform datasets into features for other systems. The book begins with an introduction to data manipulation in Python using pandas. You'll then get familiar with statistical analysis and plotting techniques. With multiple hands-on activities in store, you'll be able to analyze data that is distributed on several computers by using Dask. As you progress, you'll study how to aggregate data for plots when the entire data cannot be accommodated in memory. You'll also explore Hadoop (HDFS and YARN), which will help you tackle larger datasets. The book also covers Spark and explains how it interacts with other tools. By the end of this book, you'll be able to bootstrap your own Python environment, process large files, and manipulate data to generate statistics, metrics, and graphs.

Related Content you might be interested in

Current Title:

Small book image
Big Data Analysis with Python
Ivan Marin | Ankit Shukla | Sarang VK
Apr, 2019 | 276 pages
Small book image
Become a Python Data Analyst
Alvaro Fuentes
Aug, 2018 | 178 pages
Small book image
Exploratory Data Analysis with Python Cookbook
Ayodele Oluleye
Jun, 2023 | 382 pages
Small book image
Mastering Numerical Computing with NumPy
Mert Cuhadaroglu | Umit Mert Cakmak
Jun, 2018 | 248 pages
Table of Contents (11 chapters)
Big Data Analysis with Python
Big Data Analysis with Python
Preface
Preface
Free Chapter
1
The Python Data Science Stack
The Python Data Science Stack
Introduction
Python Libraries and Packages
Using Pandas
Data Type Conversion
Aggregation and Grouping
Exporting Data from Pandas
Visualization with Pandas
Summary
2
Statistical Visualizations
Statistical Visualizations
Introduction
Types of Graphs and When to Use Them
Components of a Graph
Seaborn
Which Tool Should Be Used?
Types of Graphs
Pandas DataFrames and Grouped Data
Changing Plot Design: Modifying Graph Components
Exporting Graphs
Summary
3
Working with Big Data Frameworks
Working with Big Data Frameworks
Introduction
Hadoop
Spark
Writing Parquet Files
Handling Unstructured Data
Summary
4
Diving Deeper with Spark
Diving Deeper with Spark
Introduction
Getting Started with Spark DataFrames
Writing Output from Spark DataFrames
Exploring Spark DataFrames
Data Manipulation with Spark DataFrames
Graphs in Spark
Summary
5
Handling Missing Values and Correlation Analysis
Handling Missing Values and Correlation Analysis
Introduction
Setting up the Jupyter Notebook
Missing Values
Handling Missing Values in Spark DataFrames
Correlation
Summary
6
Exploratory Data Analysis
Exploratory Data Analysis
Introduction
Defining a Business Problem
Translating a Business Problem into Measurable Metrics and Exploratory Data Analysis (EDA)
Structured Approach to the Data Science Project Life Cycle
Summary
7
Reproducibility in Big Data Analysis
Reproducibility in Big Data Analysis
Introduction
Reproducibility with Jupyter Notebooks
Gathering Data in a Reproducible Way
Code Practices and Standards
Avoiding Repetition
Summary
8
Creating a Full Analysis Report
Creating a Full Analysis Report
Introduction
Reading Data in Spark from Different Data Sources
SQL Operations on a Spark DataFrame
Generating Statistical Measurements
Summary
Appendix
Appendix
Chapter 01: The Python Data Science Stack
Chapter 02: Statistical Visualizations Using Matplotlib and Seaborn
Chapter 03: Working with Big Data Frameworks
Chapter 04: Diving Deeper with Spark
Chapter 05: Missing Value Handling and Correlation Analysis in Spark
Chapter 6: Business Process Definition and Exploratory Data Analysis
Chapter 07: Reproducibility in Big Data Analysis
Chapter 08: Creating a Full Analysis Report

Getting Started with Spark DataFrames

To get started with Spark DataFrames, we will have to create something called a SparkContext first. SparkContext configures the internal services under the hood and facilitates command execution from the Spark execution environment.

Note

We will be using Spark version 2.1.1, running on Python 3.7.1. Spark and Python are installed on a MacBook Pro, running macOS Mojave version 10.14.3, with a 2.7 GHz Intel Core i5 processor and 8 GB 1867 MHz DDR3 RAM.

The following code snippet is used to create SparkContext:

from pyspark import SparkContext
sc = SparkContext()

Note

In case you are working in the PySpark shell, you should skip this step, as the shell automatically creates the sc (SparkContext) variable when it is started. However, be sure to create the sc variable while creating a PySpark script or working with Jupyter Notebook, or your code will throw an error.

We also need to create an SQLContext before we can start working with DataFrames. SQLContext in Spark...

Previous Section
End of Section 3
Next Section