7. Statistical Data Analysis with Incanter | Clojure Data Analysis Cookbook - Second Edition

Sign In Start Free Trial

Book Overview & Buying
Table Of Contents

Clojure Data Analysis Cookbook - Second Edition - Second Edition

By : Eric Richard Rochester

4 (2)

Clojure Data Analysis Cookbook - Second Edition

4 (2)

By: Eric Richard Rochester

Overview of this book

This book is for those with a basic knowledge of Clojure, who are looking to push the language to excel with data analysis.

Preface

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Free Chapter

1. Importing Data for Analysis

1. Importing Data for Analysis

Introduction

Creating a new project

Reading CSV data into Incanter datasets

Reading JSON data into Incanter datasets

Reading data from Excel with Incanter

Reading data from JDBC databases

Reading XML data into Incanter datasets

Scraping data from tables in web pages

Scraping textual data from web pages

Reading RDF data

Querying RDF data with SPARQL

Aggregating data from different formats

2. Cleaning and Validating Data

2. Cleaning and Validating Data

Introduction

Cleaning data with regular expressions

Maintaining consistency with synonym maps

Identifying and removing duplicate data

Regularizing numbers

Calculating relative values

Parsing dates and times

Lazily processing very large data sets

Sampling from very large data sets

Fixing spelling errors

Parsing custom data formats

Validating data with Valip

3. Managing Complexity with Concurrent Programming

3. Managing Complexity with Concurrent Programming

Introduction

Managing program complexity with STM

Managing program complexity with agents

Getting better performance with commute

Combining agents and STM

Maintaining consistency with ensure

Introducing safe side effects into the STM

Maintaining data consistency with validators

Monitoring processing with watchers

Debugging concurrent programs with watchers

Recovering from errors in agents

Managing large inputs with sized queues

4. Improving Performance with Parallel Programming

4. Improving Performance with Parallel Programming

Introduction

Parallelizing processing with pmap

Parallelizing processing with Incanter

Partitioning Monte Carlo simulations for better pmap performance

Finding the optimal partition size with simulated annealing

Combining function calls with reducers

Parallelizing with reducers

Generating online summary statistics for data streams with reducers

Using type hints

Benchmarking with Criterium

5. Distributed Data Processing with Cascalog

5. Distributed Data Processing with Cascalog

Introduction

Initializing Cascalog and Hadoop for distributed processing

Querying data with Cascalog

Distributing data with Apache HDFS

Parsing CSV files with Cascalog

Executing complex queries with Cascalog

Aggregating data with Cascalog

Defining new Cascalog operators

Composing Cascalog queries

Transforming data with Cascalog

6. Working with Incanter Datasets

6. Working with Incanter Datasets

Introduction

Loading Incanter's sample datasets

Loading Clojure data structures into datasets

Viewing datasets interactively with view

Converting datasets to matrices

Using infix formulas in Incanter

Selecting columns with $

Selecting rows with $

Filtering datasets with $where

Grouping data with $group-by

Saving datasets to CSV and JSON

Projecting from multiple datasets with $join

7. Statistical Data Analysis with Incanter

7. Statistical Data Analysis with Incanter

Introduction

Generating summary statistics with $rollup

Working with changes in values

Scaling variables to simplify variable relationships

Working with time series data with Incanter Zoo

Smoothing variables to decrease variation

Validating sample statistics with bootstrapping

Modeling linear relationships

Modeling non-linear relationships

Modeling multinomial Bayesian distributions

Finding data errors with Benford's law

8. Working with Mathematica and R

8. Working with Mathematica and R

Introduction

Setting up Mathematica to talk to Clojuratica for Mac OS X and Linux

Setting up Mathematica to talk to Clojuratica for Windows

Calling Mathematica functions from Clojuratica

Sending matrixes to Mathematica from Clojuratica

Evaluating Mathematica scripts from Clojuratica

Creating functions from Mathematica

Setting up R to talk to Clojure

Calling R functions from Clojure

Passing vectors into R

Evaluating R files from Clojure

Plotting in R from Clojure

9. Clustering, Classifying, and Working with Weka

9. Clustering, Classifying, and Working with Weka

Introduction

Loading CSV and ARFF files into Weka

Filtering, renaming, and deleting columns in Weka datasets

Discovering groups of data using K-Means clustering

Finding hierarchical clusters in Weka

Clustering with SOMs in Incanter

Classifying data with decision trees

Classifying data with the Naive Bayesian classifier

Classifying data with support vector machines

Finding associations in data with the Apriori algorithm

10. Working with Unstructured and Textual Data

10. Working with Unstructured and Textual Data

Introduction

Tokenizing text

Finding sentences

Focusing on content words with stoplists

Getting document frequencies

Scaling document frequencies by document size

Scaling document frequencies with TF-IDF

Finding people, places, and things with Named Entity Recognition

Mapping documents to a sparse vector space representation

Performing topic modeling with MALLET

Performing naïve Bayesian classification with MALLET

11. Graphing in Incanter

11. Graphing in Incanter

Introduction

Creating scatter plots with Incanter

Graphing non-numeric data in bar charts

Creating histograms with Incanter

Creating function plots with Incanter

Adding equations to Incanter charts

Adding lines to scatter charts

Customizing charts with JFreeChart

Customizing chart colors and styles

Saving Incanter graphs to PNG

Using PCA to graph multi-dimensional data

Creating dynamic charts with Incanter

12. Creating Charts for the Web

12. Creating Charts for the Web

Introduction

Serving data with Ring and Compojure

Creating HTML with Hiccup

Setting up to use ClojureScript

Creating scatter plots with NVD3

Creating bar charts with NVD3

Creating histograms with NVD3

Creating time series charts with D3

Visualizing graphs with force-directed layouts

Creating interactive visualizations with D3

Index

Index

Smoothing variables to decrease variation

We saw in the last chapter how to use Incanter Zoo to work with time series data and how to smooth values using a running mean. However, sometimes we'll want to smooth data that doesn't have a time component. For instance, we may want to track the usage of a word throughout a larger document or set of documents.

Getting ready

For this, we'll need usual dependencies:

(defproject statim "0.1.0"
  :dependencies [[org.clojure/clojure "1.6.0"]
                 [incanter "1.5.5"]])

We'll also require those in our script or REPL:

(require '[incanter.core :as i]
         '[incanter.stats :as s]
         '[incanter.charts :as c]
         '[clojure.string :as str])

For this recipe, we'll look at Sir Arthur Conan Doyle's Sherlock Holmes stories. You can download this from Project Gutenberg at http://www.gutenberg.org/cache/epub/1661/pg1661.txt or http://www.ericrochester.com/clj-data-analysis/data/pg1661.txt.

How to do it…

We'll look at the distribution of baker over...

Visually different images

CONTINUE READING

83

Tech Concepts

36

Programming languages

73

Tech Tools

Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.

Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.

50+ new titles added per month and exclusive early access to books as they are being written.

Clojure Data Analysis Cookbook - Second Edition

Search

Your notes and bookmarks