3. Learning What Your Data Truly Holds | Python Business Intelligence Cookbook

Sign In Start Free Trial

Book Overview & Buying
Table Of Contents
Feedback & Rating

Python Business Intelligence Cookbook

By : Dempsey

3.6 (9)

Python Business Intelligence Cookbook

3.6 (9)

By: Dempsey

Overview of this book

The amount of data produced by businesses and devices is going nowhere but up. In this scenario, the major advantage of Python is that it's a general-purpose language and gives you a lot of flexibility in data structures. Python is an excellent tool for more specialized analysis tasks, and is powered with related libraries to process data streams, to visualize datasets, and to carry out scientific calculations. Using Python for business intelligence (BI) can help you solve tricky problems in one go. Rather than spending day after day scouring Internet forums for “how-to” information, here you’ll find more than 60 recipes that take you through the entire process of creating actionable intelligence from your raw data, no matter what shape or form it’s in. Within the first 30 minutes of opening this book, you’ll learn how to use the latest in Python and NoSQL databases to glean insights from data just waiting to be exploited. We’ll begin with a quick-fire introduction to Python for BI and show you what problems Python solves. From there, we move on to working with a predefined data set to extract data as per business requirements, using the Pandas library and MongoDB as our storage engine. Next, we will analyze data and perform transformations for BI with Python. Through this, you will gather insightful data that will help you make informed decisions for your business. The final part of the book will show you the most important task of BI—visualizing data by building stunning dashboards using Matplotlib, PyTables, and iPython Notebook.

Preface

Preface

What this book covers

What you need for this book

Who this book is for

Sections

Conventions

Reader feedback

Customer support

Free Chapter

1. Getting Set Up to Gain Business Intelligence

1. Getting Set Up to Gain Business Intelligence

Introduction

Installing Anaconda

Learn about the Python libraries we will be using

Installing, configuring, and running MongoDB

Installing Rodeo

Starting Rodeo

Installing Robomongo

Using Robomongo to query MongoDB

Downloading the UK Road Safety Data dataset

2. Making Your Data All It Can Be

2. Making Your Data All It Can Be

Importing a CSV file into MongoDB

Importing an Excel file into MongoDB

Importing a JSON file into MongoDB

Importing a plain text file into MongoDB

Retrieving a single record using PyMongo

Retrieving multiple records using PyMongo

Inserting a single record using PyMongo

Inserting multiple records using PyMongo

Updating a single record using PyMongo

Updating multiple records using PyMongo

Deleting a single record using pymongo

Deleting multiple records using PyMongo

Importing a CSV file into a Pandas DataFrame

Renaming column headers in Pandas

Filling in missing values in Pandas

Removing punctuation in Pandas

Removing whitespace in Pandas

Removing any string from within a string in Pandas

Merging two datasets in Pandas

Titlecasing anything

Uppercasing a column in Pandas

Updating values in place in Pandas

Standardizing a Social Security number in Pandas

Standardizing dates in Pandas

Converting categories to numbers in Pandas for a speed boost

3. Learning What Your Data Truly Holds

3. Learning What Your Data Truly Holds

Creating a Pandas DataFrame from a MongoDB query

Creating a Pandas DataFrame from a CSV file

Creating a Pandas DataFrame from an Excel file

Creating a Pandas DataFrame from a JSON file

Creating a data quality report

Generating summary statistics for the entire dataset

Generating summary statistics for object type columns

Getting the mode of the entire dataset

Generating summary statistics for a single column

Getting a count of unique values for a single column

Getting the minimum and maximum values of a single column

Generating quantiles for a single column

Getting the mean, median, mode, and range for a single column

Generating a frequency table for a single column by date

Generating a frequency table of two variables

Creating a histogram for a column

Plotting the data as a probability distribution

Plotting a cumulative distribution function

Showing the histogram as a stepped line

Plotting two sets of values in a probability distribution

Creating a customized box plot with whiskers

Creating a basic bar chart for a single column over time

4. Performing Data Analysis for Non Data Analysts

4. Performing Data Analysis for Non Data Analysts

Performing a distribution analysis

Performing categorical variable analysis

Performing a linear regression

Performing a time-series analysis

Performing outlier detection

Creating a predictive model using logistic regression

Creating a predictive model using a random forest

Creating a predictive model using Support Vector Machines

Saving a predictive model for production use

5. Building a Business Intelligence Dashboard Quickly

5. Building a Business Intelligence Dashboard Quickly

Creating reports in Excel directly from a Pandas DataFrame

Creating customizable Excel reports using XlsxWriter

Building a shareable dashboard using IPython Notebook and matplotlib

Exporting an IPython Notebook Dashboard to HTML

Exporting an IPython Notebook Dashboard to PDF

Exporting an IPython Notebook Dashboard to an HTML slideshow

Building your First Flask application in 10 minutes or less

Creating and saving your plots for your Flask BI dashboard

Building a business intelligence dashboard in Flask

Index

Index

Customer Reviews

3.6 (9)

5 star

33.3%

4 star

11.1%

3 star

33.3%

2 star

22.2%

1 star

0%

Creating a Pandas DataFrame from a JSON file

Along with CSV, JSON is another commonly found format for datasets, especially when extracting data from web APIs.

How to do it…

To create a Pandas DataFrame from a JSON file, first import the Python libraries that you need:
```
import pandas as pd
```
Next, define a variable for the JSON file and enter the full path to the file:
```
customer_json_file = 'customer_data.json'
```
Next, create a DataFrame from the JSON file using the read_json() method provided by Pandas. Note that the dates in our JSON file are stored in the ISO format, so we're going to tell the read_json() method to convert dates:
```
customers_json = pd.read_json(customer_json_file,
   convert_dates=True)
```
Finally, use the head() command to see the top five rows of data:
```
customers_json.head()
```

How it works…

After importing Pandas and defining a variable for the full path to our JSON file, we use the read_json() method provided by Pandas to create a DataFrame from our JSON file.

read_json() takes a number of...

Visually different images

CONTINUE READING

83

Tech Concepts

36

Programming languages

73

Tech Tools

Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.

Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.

50+ new titles added per month and exclusive early access to books as they are being written.

Python Business Intelligence Cookbook

Search

Your notes and bookmarks