Book Image

Big Data Analysis with Python

By : Ivan Marin, Ankit Shukla, Sarang VK
Book Image

Big Data Analysis with Python

By: Ivan Marin, Ankit Shukla, Sarang VK

Overview of this book

Processing big data in real time is challenging due to scalability, information inconsistency, and fault tolerance. Big Data Analysis with Python teaches you how to use tools that can control this data avalanche for you. With this book, you'll learn practical techniques to aggregate data into useful dimensions for posterior analysis, extract statistical measurements, and transform datasets into features for other systems. The book begins with an introduction to data manipulation in Python using pandas. You'll then get familiar with statistical analysis and plotting techniques. With multiple hands-on activities in store, you'll be able to analyze data that is distributed on several computers by using Dask. As you progress, you'll study how to aggregate data for plots when the entire data cannot be accommodated in memory. You'll also explore Hadoop (HDFS and YARN), which will help you tackle larger datasets. The book also covers Spark and explains how it interacts with other tools. By the end of this book, you'll be able to bootstrap your own Python environment, process large files, and manipulate data to generate statistics, metrics, and graphs.
Table of Contents (11 chapters)
Big Data Analysis with Python
Preface

Chapter 01: The Python Data Science Stack


Activity 1: IPython and Jupyter

  1. Open the python_script_student.py file in a text editor, copy the contents to a notebook in IPython, and execute the operations.

  2. Copy and paste the code from the Python script into a Jupyter notebook:

    import numpy as np
    
    def square_plus(x, c):
        return np.power(x, 2) + c
  3. Now, update the values of the x and c variables. Then, change the definition of the function:

    x = 10
    c = 100
    
    result = square_plus(x, c)
    print(result)

    The output is as follows:

    200

Activity 2: Working with Data Problems

  1. Import pandas and NumPy library:

    import pandas as pd
    import numpy as np
  2. Read the RadNet dataset from the U.S. Environmental Protection Agency, available from the Socrata project:

    url = "https://opendata.socrata.com/api/views/cf4r-dfwe/rows.csv?accessType=DOWNLOAD"
    df = pd.read_csv(url)
  3. Create a list with numeric columns for radionuclides in the RadNet dataset:

    columns = df.columns
    id_cols = ['State', 'Location', "Date Posted", 'Date Collected', 'Sample Type', 'Unit']
    columns = list(set(columns) - set(id_cols))
    columns
  4. Use the apply method on one column, with a lambda function that compares the Non-detect string:

    df['Cs-134'] = df['Cs-134'].apply(lambda x: np.nan if x == "Non-detect" else x)
    df.head()

    The output is as follows:

    Figure 1.19: DataFrame after applying the lambda function

  5. Replace the text values with NaN in one column with np.nan:

    df.loc[:, columns] = df.loc[:, columns].applymap(lambda x: np.nan if x == 'Non-detect' else x)
    df.loc[:, columns] = df.loc[:, columns].applymap(lambda x: np.nan if x == 'ND' else x)
  6. Use the same lambda comparison and use the applymap method on several columns at the same time, using the list created in the first step:

    df.loc[:, ['State', 'Location', 'Sample Type', 'Unit']] = df.loc[:, ['State', 'Location',g 'Sample Type', 'Unit']].applymap(lambda x: x.strip())
  7. Create a list of the remaining columns that are not numeric:

    df.dtypes

    The output is as follows:

    Figure 1.20: List of columns and their type

  8. Convert the DataFrame objects into floats using the to_numeric function:

    df['Date Posted'] = pd.to_datetime(df['Date Posted'])
    df['Date Collected'] = pd.to_datetime(df['Date Collected'])
    for col in columns:
        df[col] = pd.to_numeric(df[col])
    df.dtypes

    The output is as follows:

    Figure 1.21: List of columns and their type

  9. Using the selection and filtering methods, verify that the names of the string columns don't have any spaces:

    df['Date Posted'] = pd.to_datetime(df['Date Posted'])
    df['Date Collected'] = pd.to_datetime(df['Date Collected'])
    for col in columns:
        df[col] = pd.to_numeric(df[col])
    df.dtypes

    The output is as follows:

    Figure 1.22: DataFrame after applying the selection and filtering method

Activity 3: Plotting Data with Pandas

  1. Use the RadNet DataFrame that we have been working with.

  2. Fix all the data type problems, as we saw before.

  3. Create a plot with a filter per Location, selecting the city of San Bernardino, and one radionuclide, with the x-axis set to the date and the y-axis with radionuclide I-131:

    df.loc[df.Location == 'San Bernardino'].plot(x='Date Collected', y='I-131')

    The output is as follows:

    Figure 1.23: Plot of Date collected vs I-131

  4. Create a scatter plot with the concentration of two related radionuclides, I-131 and I-132:

    fig, ax = plt.subplots()
    ax.scatter(x=df['I-131'], y=df['I-132'])
    _ = ax.set(
        xlabel='I-131',
        ylabel='I-132',
        title='Comparison between concentrations of I-131 and I-132'
    )

    The output is as follows:

    Figure 1.24: Plot of concentration of I-131 and I-132