Book Image

Big Data Analysis with Python

By : Ivan Marin, Ankit Shukla, Sarang VK
Book Image

Big Data Analysis with Python

By: Ivan Marin, Ankit Shukla, Sarang VK

Overview of this book

Processing big data in real time is challenging due to scalability, information inconsistency, and fault tolerance. Big Data Analysis with Python teaches you how to use tools that can control this data avalanche for you. With this book, you'll learn practical techniques to aggregate data into useful dimensions for posterior analysis, extract statistical measurements, and transform datasets into features for other systems. The book begins with an introduction to data manipulation in Python using pandas. You'll then get familiar with statistical analysis and plotting techniques. With multiple hands-on activities in store, you'll be able to analyze data that is distributed on several computers by using Dask. As you progress, you'll study how to aggregate data for plots when the entire data cannot be accommodated in memory. You'll also explore Hadoop (HDFS and YARN), which will help you tackle larger datasets. The book also covers Spark and explains how it interacts with other tools. By the end of this book, you'll be able to bootstrap your own Python environment, process large files, and manipulate data to generate statistics, metrics, and graphs.

Related Content you might be interested in

Current Title:

Small book image
Big Data Analysis with Python
Ivan Marin | Ankit Shukla | Sarang VK
Apr, 2019 | 276 pages
Small book image
Become a Python Data Analyst
Alvaro Fuentes
Aug, 2018 | 178 pages
Small book image
Exploratory Data Analysis with Python Cookbook
Ayodele Oluleye
Jun, 2023 | 382 pages
Small book image
Mastering Numerical Computing with NumPy
Mert Cuhadaroglu | Umit Mert Cakmak
Jun, 2018 | 248 pages
Table of Contents (11 chapters)
Big Data Analysis with Python
Big Data Analysis with Python
Preface
Preface
Free Chapter
1
The Python Data Science Stack
The Python Data Science Stack
Introduction
Python Libraries and Packages
Using Pandas
Data Type Conversion
Aggregation and Grouping
Exporting Data from Pandas
Visualization with Pandas
Summary
2
Statistical Visualizations
Statistical Visualizations
Introduction
Types of Graphs and When to Use Them
Components of a Graph
Seaborn
Which Tool Should Be Used?
Types of Graphs
Pandas DataFrames and Grouped Data
Changing Plot Design: Modifying Graph Components
Exporting Graphs
Summary
3
Working with Big Data Frameworks
Working with Big Data Frameworks
Introduction
Hadoop
Spark
Writing Parquet Files
Handling Unstructured Data
Summary
4
Diving Deeper with Spark
Diving Deeper with Spark
Introduction
Getting Started with Spark DataFrames
Writing Output from Spark DataFrames
Exploring Spark DataFrames
Data Manipulation with Spark DataFrames
Graphs in Spark
Summary
5
Handling Missing Values and Correlation Analysis
Handling Missing Values and Correlation Analysis
Introduction
Setting up the Jupyter Notebook
Missing Values
Handling Missing Values in Spark DataFrames
Correlation
Summary
6
Exploratory Data Analysis
Exploratory Data Analysis
Introduction
Defining a Business Problem
Translating a Business Problem into Measurable Metrics and Exploratory Data Analysis (EDA)
Structured Approach to the Data Science Project Life Cycle
Summary
7
Reproducibility in Big Data Analysis
Reproducibility in Big Data Analysis
Introduction
Reproducibility with Jupyter Notebooks
Gathering Data in a Reproducible Way
Code Practices and Standards
Avoiding Repetition
Summary
8
Creating a Full Analysis Report
Creating a Full Analysis Report
Introduction
Reading Data in Spark from Different Data Sources
SQL Operations on a Spark DataFrame
Generating Statistical Measurements
Summary
Appendix
Appendix
Chapter 01: The Python Data Science Stack
Chapter 02: Statistical Visualizations Using Matplotlib and Seaborn
Chapter 03: Working with Big Data Frameworks
Chapter 04: Diving Deeper with Spark
Chapter 05: Missing Value Handling and Correlation Analysis in Spark
Chapter 6: Business Process Definition and Exploratory Data Analysis
Chapter 07: Reproducibility in Big Data Analysis
Chapter 08: Creating a Full Analysis Report

Introduction

In the previous chapter, we learned the basic concepts of Spark DataFrames and saw how to leverage them for big data analysis.

In this chapter, we will go a step further and learn about handling missing values in data and correlation analysis with Spark DataFrames—concepts that will help us with data preparation for machine learning and exploratory data analysis.

We will briefly cover these concepts to provide the reader with some context, but our focus is on their implementation with Spark DataFrames. We will use the same Iris dataset that we used in the previous chapter for the exercises in this chapter as well. But the Iris dataset has no missing values, so we have randomly removed two entries from the Sepallength column and one entry from the Petallength column from the original dataset. So, now we have a dataset with missing values, and we will learn how to handle these missing values using PySpark.

We will also look at the correlation between the variables in the Iris dataset...

Previous Section
End of Section 2
Next Section