Chapter 1. From Data to Decisions – Getting Started with Analytic Applications

From quarterly financial projections to customer surveys, analytics help businesses to make decisions and plan for the future. While data visualizations such as pie charts and trend lines using spreadsheet programs have been used for decades, recent years have seen a growth in both the volume and diversity of data sources available to the business analyst and the sophistication of tools used to interpret this information.

The rapid growth of the Internet, through e-commerce and social media platforms, has generated a wealth of data, which is available faster than ever before for analysis. Photographs, search queries, and online forum posts are all examples of unstructured data that can't be easily examined in a traditional spreadsheet program. With the proper tools, these kinds of data offer new insights, in conjunction with or beyond traditional data sources.

Traditionally, data such as historical customer records appear in a structured, tabular form that is stored in an electronic data warehouse and easily imported into a spreadsheet program. Even in the case of such tabular data, the volume of records and the rate at which they are available are increasing in many industries. While the analyst might have historically transformed raw data through interactive manipulation, robust analytics increasingly requires automated processing that can scale with the volume and velocity of data being received by a business.

Along with the data itself, the methods used to examine it have become more powerful and complex. Beyond summarizing historical patterns or projecting future events using trend lines derived from a few key input variables, advanced analytics emphasizes the use of sophisticated predictive modeling (see the goals of predictive analytics, as follows) to understand the present and forecast near and long-term outcomes.

Diverse methods for generating such predictions typically require the following common elements:

While predictive modeling techniques can be used in powerful analytic applications to discover complex relationships between seemingly unrelated inputs, they also present a new set of challenges to the business analyst:

In this book, we will show you how to address these challenges by developing analytic solutions that transform data into powerful insights for you and your business. The main tasks involved in building these applications are:

Throughout this volume, we will use open-source tools written in the Python programming language to build these sorts of applications. Why Python? The Python language strikes an attractive balance between robust compiled languages such as Java, C++, and Scala, and pure statistical packages such as R, SAS, or MATLAB. We can work interactively with Python using the command line (or, as we will use in subsequent chapters, browser-based notebook environments), plotting data, and prototyping commands. Python also provides extensive libraries, allowing us to transform this exploratory work into web applications (such as Flask, CherryPy, and Celery, as we will see in Chapter 8, Sharing Models with Prediction Services), or scale them to large datasets (using PySpark, as we will explore in future chapters). Thus we can both analyze data and develop software applications within the same language.

Before diving into the technical details of these tools, let's take a high-level look at the concepts behind these applications and how they are structured. In this chapter, we will: