Book Image

Python Data Science Essentials

By : Alberto Boschetti, Luca Massaron
Book Image

Python Data Science Essentials

By: Alberto Boschetti, Luca Massaron

Overview of this book

The book starts by introducing you to setting up your essential data science toolbox. Then it will guide you across all the data munging and preprocessing phases. This will be done in a manner that explains all the core data science activities related to loading data, transforming and fixing it for analysis, as well as exploring and processing it. Finally, it will complete the overview by presenting you with the main machine learning algorithms, the graph analysis technicalities, and all the visualization instruments that can make your life easier in presenting your results. In this walkthrough, structured as a data science project, you will always be accompanied by clear code and simplified examples to help you understand the underlying mechanics and real-world datasets.
Table of Contents (13 chapters)


If you have run the previous experiment, you may have realized that:

  1. Both the validation and test results vary as samples are different

  2. The chosen hypothesis is often the best one, but this is not always the case

Unfortunately, relying on the validation and testing phases of samples brings uncertainty along with a strong reduction of the learning examples for training (the fewer the examples, the more the variance of the obtained model).

A solution is to use cross-validation, and Scikit-learn offers a complete module for cross-validation and performance evaluation (sklearn.cross_validation).

By resorting to cross-validation, you'll just need to separate your data into a training and test set, and you will be able to use the training data for both model optimization and model training.

How does cross-validation work? The idea is to divide your training data into a certain number of partitions (called folds) and train your model as many times as the number of partitions, keeping...