First of all, let's proceed to introduce all the settings you need in order to create a fully working data science environment to test the examples and experiment with the code that we are going to provide you with.
Python is an open source, object-oriented, cross-platform programming language that, compared to its direct competitors (for instance, C++ and Java), is very concise. It allows you to build a working software prototype in a very short time. Did it become the most used language in the data scientist's toolbox just because of this? Well, no. It's also a general-purpose language, and it is very flexible indeed due to a large variety of available packages that solve a wide spectrum of problems and necessities.
There are two main branches of Python: 2 and 3. Although the third version is the newest, the older one is still the most used version in the scientific area, since a few libraries (see http://py3readiness.org for a compatibility overview) won't run otherwise. In fact, if you try to run some code developed for Python 2 with a Python 3 interpreter, it won't work. Major changes have been made to the newest version, and this has impacted past compatibility. So, please remember that there is no backward compatibility between Python 3 and 2.
In this book, in order to address a larger audience of readers and practitioners, we're going to adopt the Python 2 syntax for all our examples (at the time of writing this book, the latest release is 2.7.8). Since the differences amount to really minor changes, advanced users of Python 3 are encouraged to adapt and optimize the code to suit their favored version.
Novice data scientists who have never used Python (so, we figured out that they don't have it readily installed on their machines) need to first download the installer from the main website of the project, https://www.python.org/downloads/, and then install it on their local machine.
Tip
This section provides you with full control over what can be installed on your machine. This is very useful when you have to set up single machines to deal with different tasks in data science. Anyway, please be warned that a step-by-step installation really takes time and effort. Instead, installing a ready-made scientific distribution will lessen the burden of installation procedures and it may be well suited for first starting and learning because it saves you time and sometimes even trouble, though it will put a large number of packages (and we won't use most of them) on your computer all at once. Therefore, if you want to start immediately with an easy installation procedure, just skip this part and proceed to the next section, Scientific distributions.
Being a multiplatform programming language, you'll find installers for machines that either run on Windows or Unix-like operating systems. Please remember that some Linux distributions (such as Ubuntu) have Python 2 packeted in the repository, which makes the installation process even easier.
To open a python shell, type
pythonin the terminal or click on the Python icon.Then, to test the installation, run the following code in the Python interactive shell or REPL:
>>> import sys >>> print sys.version_info
If a syntax error is raised, it means that you are running Python 3 instead of Python 2. Otherwise, if you don't experience an error and you can read that your Python version has the
attribute major=2, then congratulations for running the right version of Python. You're now ready to move forward.
To clarify, when a command is given in the terminal command line, we prefix the command with $>. Otherwise, if it's for the Python REPL, it's preceded by >>>.
We mentioned that the two most relevant Python characteristics are its ability to integrate with other languages and its mature package system that is well embodied by PyPI (the Python Package Index; https://pypi.python.org/pypi), a common repository for a majority of Python packages.
The packages that we are now going to introduce are strongly analytical and will offer a complete Data Science Toolbox made up of highly optimized functions for working, optimal memory configuration, ready to achieve scripting operations with optimal performance. A walkthrough on how to install them is given in the following section.
Partially inspired by similar tools present in R and MATLAB environments, we will together explore how a few selected Python commands can allow you to efficiently handle data and then explore, transform, experiment, and learn from the same without having to write too much code or reinvent the wheel.
NumPy, which is Travis Oliphant's creation, is the true analytical workhorse of the Python language. It provides the user with multidimensional arrays, along with a large set of functions to operate a multiplicity of mathematical operations on these arrays. Arrays are blocks of data arranged along multiple dimensions, which implement mathematical vectors and matrices. Arrays are useful not just for storing data, but also for fast matrix operations (vectorization), which are indispensable when you wish to solve ad hoc data science problems.
Website: http://www.numpy.org/
Version at the time of print: 1.9.1
Suggested install command:
pip install numpy
As a convention largely adopted by the Python community, when importing NumPy, it is suggested that you alias it as np:
import numpy as np
We will be doing this throughout the course of this book.
An original project by Travis Oliphant, Pearu Peterson, and Eric Jones, SciPy completes NumPy's functionalities, offering a larger variety of scientific algorithms for linear algebra, sparse matrices, signal and image processing, optimization, fast Fourier transformation, and much more.
Website: http://www.scipy.org/
Version at time of print: 0.14.0
Suggested install command:
pip install scipy
The pandas package deals with everything that NumPy and SciPy cannot do. Thanks to its specific object data structures, DataFrames and Series, pandas allows you to handle complex tables of data of different types (which is something that NumPy's arrays cannot do) and time series. Thanks to Wes McKinney's creation, you will be able to easily and smoothly load data from a variety of sources. You can then slice, dice, handle missing elements, add, rename, aggregate, reshape, and finally visualize this data at your will.
Website: http://pandas.pydata.org/
Version at the time of print: 0.15.2
Suggested install command:
pip install pandas
Conventionally, pandas is imported as pd:
import pandas as pd
Started as part of the SciKits (SciPy Toolkits), Scikit-learn is the core of data science operations on Python. It offers all that you may need in terms of data preprocessing, supervised and unsupervised learning, model selection, validation, and error metrics. Expect us to talk at length about this package throughout this book. Scikit-learn started in 2007 as a Google Summer of Code project by David Cournapeau. Since 2013, it has been taken over by the researchers at INRA (French Institute for Research in Computer Science and Automation).
Website: http://scikit-learn.org/stable/
Version at the time of print: 0.15.2
Suggested install command:
pip install scikit-learn
A scientific approach requires the fast experimentation of different hypotheses in a reproducible fashion. IPython was created by Fernando Perez in order to address the need for an interactive Python command shell (which is based on shell, web browser, and the application interface), with graphical integration, customizable commands, rich history (in the JSON format), and computational parallelism for an enhanced performance. IPython is our favored choice throughout this book, and it is used to clearly and effectively illustrate operations with scripts and data and the consequent results.
Website: http://ipython.org/
Version at the time of print: 2.3
Suggested install command:
pip install "ipython[notebook]"
Originally developed by John Hunter, matplotlib is the library that contains all the building blocks that are required to create quality plots from arrays and to visualize them interactively.
You can find all the MATLAB-like plotting frameworks inside the pylab module.
Website: http://matplotlib.org/
Version at the time of print: 1.4.2
Suggested install command:
pip install matplotlib
You can simply import what you need for your visualization purposes with the following command:
import matplotlib.pyplot as plt
Tip
Downloading the example code
You can download the example code files from your account at http://www.packtpub.com for all the Packt Publishing books you have purchased. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.
Previously part of SciKits, statsmodels was thought to be a complement to SciPy statistical functions. It features generalized linear models, discrete choice models, time series analysis, and a series of descriptive statistics as well as parametric and nonparametric tests.
Website: http://statsmodels.sourceforge.net/
Version at the time of print: 0.6.0
Suggested install command:
pip install statsmodels
Beautiful Soup, a creation of Leonard Richardson, is a great tool to scrap out data from HTML and XML files retrieved from the Internet. It works incredibly well, even in the case of tag soups (hence the name), which are collections of malformed, contradictory, and incorrect tags. After choosing your parser (basically, the HTML parser included in Python's standard library works fine), thanks to Beautiful Soup, you can navigate through the objects in the page and extract text, tables, and any other information that you may find useful.
Version at the time of print: 4.3.2
Suggested install command:
pip install beautifulsoup4
Developed by the Los Alamos National Laboratory, NetworkX is a package specialized in the creation, manipulation, analysis, and graphical representation of real-life network data (it can easily operate with graphs made up of a million nodes and edges). Besides specialized data structures for graphs and fine visualization methods (2D and 3D), it provides the user with many standard graph measures and algorithms, such as the shortest path, centrality, components, communities, clustering, and PageRank. We will frequently use this package in Chapter 5, Social Network Analysis.
Website: https://networkx.github.io/
Version at the time of print: 1.9.1
Suggested install command:
pip install networkx
Conventionally, NetworkX is imported as nx:
import networkx as nx
The Natural Language Toolkit (NLTK) provides access to corpora and lexical resources and to a complete suit of functions for statistical Natural Language Processing (NLP), ranging from tokenizers to part-of-speech taggers and from tree models to named-entity recognition. Initially, the package was created by Steven Bird and Edward Loper as an NLP teaching infrastructure for CIS-530 at the University of Pennsylvania. It is a fantastic tool that you can use to prototype and build NLP systems.
Website: http://www.nltk.org/
Version at the time of print: 3.0
Suggested install command:
pip install nltk
Gensim, programmed by Radim Řehůřek, is an open source package that is suitable for the analysis of large textual collections with the help of parallel distributable online algorithms. Among advanced functionalities, it implements Latent Semantic Analysis (LSA), topic modeling by Latent Dirichlet Allocation (LDA), and Google's word2vec, a powerful algorithm that transforms text into vector features that can be used in supervised and unsupervised machine learning.
Website: http://radimrehurek.com/gensim/
Version at the time of print: 0.10.3
Suggested install command:
pip install gensim
PyPy is not a package; it is an alternative implementation of Python 2.7.8 that supports most of the commonly used Python standard packages (unfortunately, NumPy is currently not fully supported). As an advantage, it offers enhanced speed and memory handling. Thus, it is very useful for heavy duty operations on large chunks of data and it should be part of your big data handling strategies.
Website: http://pypy.org/
Version at time of print: 2.4.0
Download page: http://pypy.org/download.html
Python won't come bundled with all you need, unless you take a specific premade distribution. Therefore, to install the packages you need, you can either use pip or easy_install. These are the two tools that run in the command line and make the process of installation, upgrade, and removal of Python packages a breeze. To check which tools have been installed on your local machine, run the following command:
$> pip
Alternatively, you can also run the following command:
$> easy_install
If both these commands end with an error, you need to install any one of them. We recommend that you use pip because it is thought of as an improvement over easy_install. By the way, packages installed by pip can be uninstalled and if, by chance, your package installation fails, pip will leave your system clean.
To install pip, follow the instructions given at https://pip.pypa.io/en/latest/installing.html.
The most recent versions of Python should already have pip installed by default. So, you may have it already installed on your system. If not, the safest way is to download the get-pi.py script from https://bootstrap.pypa.io/get-pip.py and then run it using the following:
$> python get-pip.py
The script will also install the setup tool from https://pypi.python.org/pypi/setuptools, which also contains easy_install.
You're now ready to install the packages you need in order to run the examples provided in this book. To install the generic package <pk>, you just need to run the following command:
$> pip install <pk>
Alternatively, you can also run the following command:
$> easy_install <pk>
After this, the package <pk> and all its dependencies will be downloaded and installed. If you're not sure whether a library has been installed or not, just try to import a module inside it. If the Python interpreter raises an ImportError error, it can be concluded that the package has not been installed.
This is what happens when the NumPy library has been installed:
>>> import numpy
This is what happens if it's not installed:
>>> import numpy Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: No module named numpy
In the latter case, you'll need to first install it through pip or easy_install.
Note
Take care that you don't confuse packages with modules. With pip, you install a package; in Python, you import a module. Sometimes, the package and the module have the same name, but in many cases, they don't match. For example, the sklearn module is included in the package named Scikit-learn.
Finally, to search and browse the Python packages available for Python, take a look at https://pypi.python.org.
More often than not, you will find yourself in a situation where you have to upgrade a package because the new version is either required by a dependency or has additional features that you would like to use. First, check the version of the library you have installed by glancing at the __version__ attribute, as shown in the following example, numpy:
>>> import numpy >>> numpy.__version__ # 2 underscores before and after '1.9.0'
Now, if you want to update it to a newer release, say the 1.9.1 version, you can run the following command from the command line:
$> pip install -U numpy==1.9.1
Alternatively, you can also use the following command:
$> easy_install --upgrade numpy==1.9.1
Finally, if you're interested in upgrading it to the latest available version, simply run the following command:
$> pip install -U numpy
You can alternatively also run the following command:
$> easy_install --upgrade numpy