Book Image

Clojure for Data Science

By : Garner
Book Image

Clojure for Data Science

By: Garner

Overview of this book

The term “data science” has been widely used to define this new profession that is expected to interpret vast datasets and translate them to improved decision-making and performance. Clojure is a powerful language that combines the interactivity of a scripting language with the speed of a compiled language. Together with its rich ecosystem of native libraries and an extremely simple and consistent functional approach to data manipulation, which maps closely to mathematical formula, it is an ideal, practical, and flexible language to meet a data scientist’s diverse needs. Taking you on a journey from simple summary statistics to sophisticated machine learning algorithms, this book shows how the Clojure programming language can be used to derive insights from data. Data scientists often forge a novel path, and you’ll see how to make use of Clojure’s Java interoperability capabilities to access libraries such as Mahout and Mllib for which Clojure wrappers don’t yet exist. Even seasoned Clojure developers will develop a deeper appreciation for their language’s flexibility! You’ll learn how to apply statistical thinking to your own data and use Clojure to explore, analyze, and visualize it in a technically and statistically robust way. You can also use Incanter for local data processing and ClojureScript to present interactive visualisations and understand how distributed platforms such as Hadoop sand Spark’s MapReduce and GraphX’s BSP solve the challenges of data analysis at scale, and how to explain algorithms using those programming models. Above all, by following the explanations in this book, you’ll learn not just how to be effective using the current state-of-the-art methods in data science, but why such methods work so that you can continue to be productive as the field evolves into the future.
Table of Contents (12 chapters)
11
Index

Comparative visualizations of electorate data

Let's look now at a dataset from another general election, this time from Russia in 2011. Russia is a much larger country, and its election data is much larger too. We'll be loading two large Excel files into the memory, which may exceed your default JVM heap size.

To expand the amount of memory available to Incanter, we can adjust the JVM settings in the project's profile.clj. The a vector of configuration flags for the JVM can be provided with the key :jvm-opts. Here we're using Java's Xmx flag to increase the heap size to 1GB. This should be more than enough.

  :jvm-opts ["-Xmx1G"]

Russia's data is available in two data files. Fortunately the columns are the same in each, so they can be concatenated together end-to-end. Incanter's function i/conj-rows exists for precisely this purpose:

(defmethod load-data :ru [_]
  (i/conj-rows (-> (io/resource "Russia2011_1of2.xls")
                   (str)
                   (xls/read-xls))
               (-> (io/resource "Russia2011_2of2.xls")
                   (str)
                   (xls/read-xls))))

In the preceding code, we define a third implementation of the load-data multimethod to load and combine both Russia files.

Note

In addition to conj-rows, Incanter-core also defines conj-columns that will merge the columns of datasets provided they have the same number of rows.

Let's see what the Russia data column names are:

(defn ex-1-29 []
  (-> (load-data :ru)
      (i/col-names)))

;; ["Code for district"
;; "Number of the polling district (unique to state, not overall)"
;; "Name of district" "Number of voters included in voters list"
;; "The number of ballots received by the precinct election
;; commission" ...]

The column names in the Russia dataset are very descriptive, but perhaps longer than we want to type out. Also, it would be convenient if columns that represent the same attributes as we've already seen in the UK election data (the victor's share and turnout for example) were labeled the same in both datasets. Let's rename them accordingly.

Along with a dataset, the i/rename-cols function expects to receive a map whose keys are the current column names with values corresponding to the desired new column name. If we combine this with the i/add-derived-column data we have already seen, we arrive at the following:

(defmethod load-data :ru-victors [_]
  (->> (load-data :ru)
       (i/rename-cols
        {"Number of voters included in voters list" :electorate
         "Number of valid ballots" :valid-ballots
         "United Russia" :victors})
       (i/add-derived-column :victors-share
                             [:victors :valid-ballots] i/safe-div)
       (i/add-derived-column :turnout
                             [:valid-ballots :electorate] /)))

The i/safe-div function is identical to / but will protect against division by zero. Rather than raising an exception, it returns the value Infinity, which will be ignored by Incanter's statistical and charting functions.