Book Image

Python Data Analysis Cookbook

By : Ivan Idris
Book Image

Python Data Analysis Cookbook

By: Ivan Idris

Overview of this book

Data analysis is a rapidly evolving field and Python is a multi-paradigm programming language suitable for object-oriented application development and functional design patterns. As Python offers a range of tools and libraries for all purposes, it has slowly evolved as the primary language for data science, including topics on: data analysis, visualization, and machine learning. Python Data Analysis Cookbook focuses on reproducibility and creating production-ready systems. You will start with recipes that set the foundation for data analysis with libraries such as matplotlib, NumPy, and pandas. You will learn to create visualizations by choosing color maps and palettes then dive into statistical data analysis using distribution algorithms and correlations. You’ll then help you find your way around different data and numerical problems, get to grips with Spark and HDFS, and then set up migration scripts for web mining. In this book, you will dive deeper into recipes on spectral analysis, smoothing, and bootstrapping methods. Moving on, you will learn to rank stocks and check market efficiency, then work with metrics and clusters. You will achieve parallelism to improve system performance by using multiple threads and speeding up your code. By the end of the book, you will be capable of handling various data analysis techniques in Python and devising solutions for problem scenarios.
Table of Contents (23 chapters)
Python Data Analysis Cookbook
Credits
About the Author
About the Reviewers
www.PacktPub.com
Preface
Glossary
Index

Unit testing your code


If code doesn't do what you want, it's hard to do reproducible data analysis. One way to gain control of your code is to test it. If you have tested code manually, you know it is repetitive and boring. When a task is boring and repetitive, you should automate it.

Unit testing automates testing and I hope you are familiar with it. When you learn unit testing for the first time, you start with simple tests such as comparing strings or numbers. However, you hit a wall when file I/O or other resources come into the picture. It turns out that in Python we can mock resources or external APIs easily. The packages needed are even part of the standard Python library. In the Learning to log for robust error checking recipe, we logged messages to a file. If we unit test this code, we don't want to trigger logging from the test code. In this recipe, I will show you how to mock the logger and other software components we need.

Getting ready

Familiarize yourself with the code under test in log_api.py.

How to do it...

The code for this recipe is in the test_log_api.py file of dautil. We start by importing the module under test and the Python functionality we need for unit testing:

from dautil import log_api
import unittest
from unittest.mock import create_autospec
from unittest.mock import patch

Define a class that contains the test code:

class TestLogApi(unittest.TestCase):

Make the unit tests executable with the following lines:

if __name__ == '__main__':
    unittest.main()

If we call Python functions with the wrong number of arguments, we expect to get a TypeError. The following tests check for that:

    def test_get_logger_args(self):
        mock_get_logger =       create_autospec(log_api.get_logger, return_value=None)
        mock_get_logger('test')
        mock_get_logger.assert_called_once_with('test')

    def test_log_args(self):
        mock_log = create_autospec(log_api.log, return_value=None)
        mock_log([], 'test')
        mock_log.assert_called_once_with([], 'test')

        with self.assertRaises(TypeError):
            mock_log()

        with self.assertRaises(TypeError):
            mock_log('test')

We used the unittest.create_autospec() function to mock the functions under test. Mock the Python logging package as follows:

    @patch('dautil.log_api.logging')
    def test_get_logger_fileConfig(self, mock_logging):
        log_api.get_logger('test')
        self.assertTrue(mock_logging.config.fileConfig.called)

The @patch decorator replaces logging with a mock. We can also patch with similarly named functions. The patching trick is quite useful. Test our get_logger() function with the following method:

    @patch('dautil.log_api.get_logger')
    def test_log_debug(self, amock):
        log_api.log({}, 'test')
        self.assertTrue(amock.return_value.debug.called)
        amock.return_value.debug.assert_called_once_with(
                'Inside the log function')

The previous lines check whether debug() was called and with which arguments. The following two test methods demonstrate how to use multiple @patch decorators:

    @patch('dautil.log_api.get_distribution')
    @patch('dautil.log_api.get_logger')
    def test_numpy(self, m_get_logger, m_get_distribution):
        log_api.log({'numpy.version': ''}, 'test')
        m_get_distribution.assert_called_once_with('numpy')
        self.assertTrue(m_get_logger.return_value.info.called)

    @patch('dautil.log_api.get_distribution')
    @patch('dautil.log_api.get_logger')
    def test_distutils(self, amock, m_get_distribution):
        log_api.log({'distutils.version': ''}, 'test')
        self.assertFalse(m_get_distribution.called)

How it works...

Mocking is a technique to spy on objects and functions. We substitute them with our own spies, which we give just enough information to avoid detection. The spies report to us who contacted them and any useful information they received.

See also