Apache Spark Datasets are an extension of the DataFrame API that provide a type-safe object-oriented programming interface. This API was first introduced in the 1.6 release. Spark 2.0 version brought out unification of DataFrame and Dataset APIs. DataFrame becomes a generic, untyped Dataset; or a Dataset is a DataFrame with an added structure. The term "structure" in this context refers to a pattern or an organization of underlying data, more like a table schema in RDBMS parlance. The structure imposes a limit on what can be expressed or contained in the underlying data. This in turn enables better optimizations in memory organization as well as physical execution. Compile-time type checking leads to catching errors earlier than during runtime. For example, a type mismatch in a SQL comparison does not get caught until runtime, whereas it would be caught during compile time itself if it were expressed as a sequence of operations on Datasets. However, the inherent dynamic nature of...
Spark for Data Science
By :
Spark for Data Science
By:
Overview of this book
This is the era of Big Data. The words ‘Big Data’ implies big innovation and enables a competitive advantage for businesses. Apache Spark was designed to perform Big Data analytics at scale, and so Spark is equipped with the necessary algorithms and supports multiple programming languages.
Whether you are a technologist, a data scientist, or a beginner to Big Data analytics, this book will provide you with all the skills necessary to perform statistical data analysis, data visualization, predictive modeling, and build scalable data products or solutions using Python, Scala, and R.
With ample case studies and real-world examples, Spark for Data Science will help you ensure the successful execution of your data science projects.
Table of Contents (18 chapters)
Spark for Data Science
Credits
Foreword
About the Authors
About the Reviewers
www.PacktPub.com
Preface
Free Chapter
Big Data and Data Science – An Introduction
The Spark Programming Model
Introduction to DataFrames
Unified Data Access
Data Analysis on Spark
Machine Learning
Extending Spark with SparkR
Analyzing Unstructured Data
Visualizing Big Data
Putting It All Together
Building Data Science Applications
Customer Reviews