Book Image

Python Data Science Essentials

By : Alberto Boschetti, Luca Massaron
Book Image

Python Data Science Essentials

By: Alberto Boschetti, Luca Massaron

Overview of this book

The book starts by introducing you to setting up your essential data science toolbox. Then it will guide you across all the data munging and preprocessing phases. This will be done in a manner that explains all the core data science activities related to loading data, transforming and fixing it for analysis, as well as exploring and processing it. Finally, it will complete the overview by presenting you with the main machine learning algorithms, the graph analysis technicalities, and all the visualization instruments that can make your life easier in presenting your results. In this walkthrough, structured as a data science project, you will always be accompanied by clear code and simplified examples to help you understand the underlying mechanics and real-world datasets.
Table of Contents (13 chapters)

Testing and validating

After loading our data, preprocessing it, creating new useful features, checking for outliers and other inconsistent data points, and choosing the right metric, we are finally ready to apply some machine learning algorithm that, by observing a series of examples and pairing them with their outcome, is able to extract a series of rules that can be successfully generalized to new examples by correctly guessing their resulting outcome. This is the supervised learning approach where a series of specialized algorithms that are fundamental to data science is used. How can we correctly apply the learning process in order to achieve the best generalizable model for prediction?

There are some best practices to be followed. Let's proceed step by step, by first loading the dataset that we will be working on in the following example:

In: from sklearn.datasets import load_digits
digits = load_digits()
print digits.DESCR
X =
y =

The digit dataset contains...