Book Image

Big Data Analysis with Python

By : Ivan Marin, Ankit Shukla, Sarang VK
Book Image

Big Data Analysis with Python

By: Ivan Marin, Ankit Shukla, Sarang VK

Overview of this book

Processing big data in real time is challenging due to scalability, information inconsistency, and fault tolerance. Big Data Analysis with Python teaches you how to use tools that can control this data avalanche for you. With this book, you'll learn practical techniques to aggregate data into useful dimensions for posterior analysis, extract statistical measurements, and transform datasets into features for other systems. The book begins with an introduction to data manipulation in Python using pandas. You'll then get familiar with statistical analysis and plotting techniques. With multiple hands-on activities in store, you'll be able to analyze data that is distributed on several computers by using Dask. As you progress, you'll study how to aggregate data for plots when the entire data cannot be accommodated in memory. You'll also explore Hadoop (HDFS and YARN), which will help you tackle larger datasets. The book also covers Spark and explains how it interacts with other tools. By the end of this book, you'll be able to bootstrap your own Python environment, process large files, and manipulate data to generate statistics, metrics, and graphs.

Related Content you might be interested in

Current Title:

Small book image
Big Data Analysis with Python
Ivan Marin | Ankit Shukla | Sarang VK
Apr, 2019 | 276 pages
Small book image
Become a Python Data Analyst
Alvaro Fuentes
Aug, 2018 | 178 pages
Small book image
Exploratory Data Analysis with Python Cookbook
Ayodele Oluleye
Jun, 2023 | 382 pages
Small book image
Mastering Numerical Computing with NumPy
Mert Cuhadaroglu | Umit Mert Cakmak
Jun, 2018 | 248 pages
Table of Contents (11 chapters)
Big Data Analysis with Python
Big Data Analysis with Python
Preface
Preface
Free Chapter
1
The Python Data Science Stack
The Python Data Science Stack
Introduction
Python Libraries and Packages
Using Pandas
Data Type Conversion
Aggregation and Grouping
Exporting Data from Pandas
Visualization with Pandas
Summary
2
Statistical Visualizations
Statistical Visualizations
Introduction
Types of Graphs and When to Use Them
Components of a Graph
Seaborn
Which Tool Should Be Used?
Types of Graphs
Pandas DataFrames and Grouped Data
Changing Plot Design: Modifying Graph Components
Exporting Graphs
Summary
3
Working with Big Data Frameworks
Working with Big Data Frameworks
Introduction
Hadoop
Spark
Writing Parquet Files
Handling Unstructured Data
Summary
4
Diving Deeper with Spark
Diving Deeper with Spark
Introduction
Getting Started with Spark DataFrames
Writing Output from Spark DataFrames
Exploring Spark DataFrames
Data Manipulation with Spark DataFrames
Graphs in Spark
Summary
5
Handling Missing Values and Correlation Analysis
Handling Missing Values and Correlation Analysis
Introduction
Setting up the Jupyter Notebook
Missing Values
Handling Missing Values in Spark DataFrames
Correlation
Summary
6
Exploratory Data Analysis
Exploratory Data Analysis
Introduction
Defining a Business Problem
Translating a Business Problem into Measurable Metrics and Exploratory Data Analysis (EDA)
Structured Approach to the Data Science Project Life Cycle
Summary
7
Reproducibility in Big Data Analysis
Reproducibility in Big Data Analysis
Introduction
Reproducibility with Jupyter Notebooks
Gathering Data in a Reproducible Way
Code Practices and Standards
Avoiding Repetition
Summary
8
Creating a Full Analysis Report
Creating a Full Analysis Report
Introduction
Reading Data in Spark from Different Data Sources
SQL Operations on a Spark DataFrame
Generating Statistical Measurements
Summary
Appendix
Appendix
Chapter 01: The Python Data Science Stack
Chapter 02: Statistical Visualizations Using Matplotlib and Seaborn
Chapter 03: Working with Big Data Frameworks
Chapter 04: Diving Deeper with Spark
Chapter 05: Missing Value Handling and Correlation Analysis in Spark
Chapter 6: Business Process Definition and Exploratory Data Analysis
Chapter 07: Reproducibility in Big Data Analysis
Chapter 08: Creating a Full Analysis Report

Big Data Analysis with Python

Copyright © 2019 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Authors: Ivan Marin, Ankit Shukla and Sarang VK

Technical Reviewer: Namachivayam D

Managing Editor: Vishal Kamal Mewada

Acquisitions Editor: Kunal Sawant

Production Editor: Nitesh Thakur

Editorial Board: David Barnes, Ewan Buckingham, Simon Cox, Manasa Kumar, Shivangi Chatterji, Alex Mazonowicz, Douglas Paterson, Dominic Pereira, Shiny Poojary, Saman Siddiqui, Erol Staveley, Ankita Thakur and Mohita Vyas

First Published: April 2019

Production Reference: 1080419

ISBN: 978-1-78995-528-6

Published by Packt Publishing Ltd.

Livery Place, 35 Livery Street

Birmingham B3 2PB, UK

Previous Section
Next Chapter