Book Image

Big Data Analysis with Python

By : Ivan Marin, Ankit Shukla, Sarang VK
Book Image

Big Data Analysis with Python

By: Ivan Marin, Ankit Shukla, Sarang VK

Overview of this book

Processing big data in real time is challenging due to scalability, information inconsistency, and fault tolerance. Big Data Analysis with Python teaches you how to use tools that can control this data avalanche for you. With this book, you'll learn practical techniques to aggregate data into useful dimensions for posterior analysis, extract statistical measurements, and transform datasets into features for other systems. The book begins with an introduction to data manipulation in Python using pandas. You'll then get familiar with statistical analysis and plotting techniques. With multiple hands-on activities in store, you'll be able to analyze data that is distributed on several computers by using Dask. As you progress, you'll study how to aggregate data for plots when the entire data cannot be accommodated in memory. You'll also explore Hadoop (HDFS and YARN), which will help you tackle larger datasets. The book also covers Spark and explains how it interacts with other tools. By the end of this book, you'll be able to bootstrap your own Python environment, process large files, and manipulate data to generate statistics, metrics, and graphs.

Related Content you might be interested in

Current Title:

Small book image
Big Data Analysis with Python
Ivan Marin | Ankit Shukla | Sarang VK
Apr, 2019 | 276 pages
Small book image
Become a Python Data Analyst
Alvaro Fuentes
Aug, 2018 | 178 pages
Small book image
Exploratory Data Analysis with Python Cookbook
Ayodele Oluleye
Jun, 2023 | 382 pages
Small book image
Mastering Numerical Computing with NumPy
Mert Cuhadaroglu | Umit Mert Cakmak
Jun, 2018 | 248 pages
Table of Contents (11 chapters)
Big Data Analysis with Python
Big Data Analysis with Python
Preface
Preface
Free Chapter
1
The Python Data Science Stack
The Python Data Science Stack
Introduction
Python Libraries and Packages
Using Pandas
Data Type Conversion
Aggregation and Grouping
Exporting Data from Pandas
Visualization with Pandas
Summary
2
Statistical Visualizations
Statistical Visualizations
Introduction
Types of Graphs and When to Use Them
Components of a Graph
Seaborn
Which Tool Should Be Used?
Types of Graphs
Pandas DataFrames and Grouped Data
Changing Plot Design: Modifying Graph Components
Exporting Graphs
Summary
3
Working with Big Data Frameworks
Working with Big Data Frameworks
Introduction
Hadoop
Spark
Writing Parquet Files
Handling Unstructured Data
Summary
4
Diving Deeper with Spark
Diving Deeper with Spark
Introduction
Getting Started with Spark DataFrames
Writing Output from Spark DataFrames
Exploring Spark DataFrames
Data Manipulation with Spark DataFrames
Graphs in Spark
Summary
5
Handling Missing Values and Correlation Analysis
Handling Missing Values and Correlation Analysis
Introduction
Setting up the Jupyter Notebook
Missing Values
Handling Missing Values in Spark DataFrames
Correlation
Summary
6
Exploratory Data Analysis
Exploratory Data Analysis
Introduction
Defining a Business Problem
Translating a Business Problem into Measurable Metrics and Exploratory Data Analysis (EDA)
Structured Approach to the Data Science Project Life Cycle
Summary
7
Reproducibility in Big Data Analysis
Reproducibility in Big Data Analysis
Introduction
Reproducibility with Jupyter Notebooks
Gathering Data in a Reproducible Way
Code Practices and Standards
Avoiding Repetition
Summary
8
Creating a Full Analysis Report
Creating a Full Analysis Report
Introduction
Reading Data in Spark from Different Data Sources
SQL Operations on a Spark DataFrame
Generating Statistical Measurements
Summary
Appendix
Appendix
Chapter 01: The Python Data Science Stack
Chapter 02: Statistical Visualizations Using Matplotlib and Seaborn
Chapter 03: Working with Big Data Frameworks
Chapter 04: Diving Deeper with Spark
Chapter 05: Missing Value Handling and Correlation Analysis in Spark
Chapter 6: Business Process Definition and Exploratory Data Analysis
Chapter 07: Reproducibility in Big Data Analysis
Chapter 08: Creating a Full Analysis Report

Introduction

If you have been part of the data industry for a while, you will understand the challenge of working with different data sources, analyzing them, and presenting them in consumable business reports. When using Spark on Python, you may have to read data from various sources, such as flat files, REST APIs in JSON format, and so on.

In the real world, getting data in the right format is always a challenge and several SQL operations are required to gather data. Thus, it is mandatory for any data scientist to know how to handle different file formats and different sources, and to carry out basic SQL operations and present them in a consumable format.

This chapter provides common methods for reading different types of data, carrying out SQL operations on it, doing descriptive statistical analysis, and generating a full analysis report. We will start with understanding how to read different kinds of data into PySpark and will then generate various analyses and plots on it.

Previous Section
End of Section 2
Next Section