Book Image

Principles of Data Science - Second Edition

By : Sinan Ozdemir, Sunil Kakade, Marco Tibaldeschi
Book Image

Principles of Data Science - Second Edition

By: Sinan Ozdemir, Sunil Kakade, Marco Tibaldeschi

Overview of this book

Need to turn programming skills into effective data science skills? This book helps you connect mathematics, programming, and business analysis. You’ll feel confident asking—and answering—complex, sophisticated questions of your data, making abstract and raw statistics into actionable ideas. Going through the data science pipeline, you'll clean and prepare data and learn effective data mining strategies and techniques to gain a comprehensive view of how the data science puzzle fits together. You’ll learn fundamentals of computational mathematics and statistics and pseudo-code used by data scientists and analysts. You’ll learn machine learning, discovering statistical models that help control and navigate even the densest datasets, and learn powerful visualizations that communicate what your data means.
Table of Contents (17 chapters)
16
Index

Technologies for machine learning projects

In this section, we are going to learn about some of the most famous and useful technologies that you can use in machine learning and big data projects. It is very important to understand the difference between these technologies and how they relate to each other, as in some ways they are similar, but they have important differences.

We are going to look at the following:

  • Apache Spark
  • Databricks
  • Azure Databricks
  • MLlib

Apache Spark

In 2012, in order to surpass MapReduce computing limitations, Apache Spark and its RDD (more on this later) were released. In order to understand the change, we have to learn how the MapReduce paradigm works by comparing it to Spark.

MapReduce substantially uses a linear dataflow, because it reads data from the disk and maps a function across the data (this means that for each entry, a function is evaluated and a result is generated). The mapped result is reduced, and the reduced data is stored again on the disk. It is easy to...