Book Image

Apache Spark 2: Data Processing and Real-Time Analytics

By : Romeo Kienzler, Md. Rezaul Karim, Sridhar Alla, Siamak Amirghodsi, Meenakshi Rajendran, Broderick Hall, Shuen Mei
Book Image

Apache Spark 2: Data Processing and Real-Time Analytics

By: Romeo Kienzler, Md. Rezaul Karim, Sridhar Alla, Siamak Amirghodsi, Meenakshi Rajendran, Broderick Hall, Shuen Mei

Overview of this book

Apache Spark is an in-memory, cluster-based data processing system that provides a wide range of functionalities such as big data processing, analytics, machine learning, and more. With this Learning Path, you can take your knowledge of Apache Spark to the next level by learning how to expand Spark's functionality and building your own data flow and machine learning programs on this platform. You will work with the different modules in Apache Spark, such as interactive querying with Spark SQL, using DataFrames and datasets, implementing streaming analytics with Spark Streaming, and applying machine learning and deep learning techniques on Spark using MLlib and various external tools. By the end of this elaborately designed Learning Path, you will have all the knowledge you need to master Apache Spark, and build your own big data processing and analytics pipeline quickly and without any hassle. This Learning Path includes content from the following Packt products: • Mastering Apache Spark 2.x by Romeo Kienzler • Scala and Spark for Big Data Analytics by Md. Rezaul Karim, Sridhar Alla • Apache Spark 2.x Machine Learning Cookbook by Siamak Amirghodsi, Meenakshi Rajendran, Broderick Hall, Shuen MeiCookbook

Related Content you might be interested in

Current Title:

Small book image
Apache Spark 2: Data Processing and Real-Time Analytics
Romeo Kienzler | Md. Rezaul Karim | Sridhar Alla | Siamak Amirghodsi | Meenakshi Rajendran | Broderick Hall | Shuen Mei
Dec, 2018 | 10 hours 16 minutes
No titles found
Table of Contents (23 chapters)
Title Page
Title Page
Copyright
Copyright
About Packt
About Packt
Contributors
Contributors
Preface
Preface
Free Chapter
1
A First Taste and What's New in Apache Spark V2
A First Taste and What's New in Apache Spark V2
Spark machine learning
Spark Streaming
Spark SQL
Spark graph processing
Extended ecosystem
What's new in Apache Spark V2?
Cluster design
Cluster management
Cloud-based deployments
Performance
Cloud
Summary
2
Apache Spark Streaming
Apache Spark Streaming
Overview
Errors and recovery
Streaming sources
Summary
3
Structured Streaming
Structured Streaming
The concept of continuous applications
Windowing
Increased performance with good old friends
How transparent fault tolerance and exactly-once delivery guarantee is achieved
Example - connection to a MQTT message broker
Summary
4
Apache Spark MLlib
Apache Spark MLlib
Architecture
Classification with Naive Bayes
Clustering with K-Means
Artificial neural networks
Summary
5
Apache SparkML
Apache SparkML
What does the new API look like?
The concept of pipelines
Model evaluation
CrossValidation and hyperparameter tuning
Winning a Kaggle competition with Apache SparkML
Summary
6
Apache SystemML
Apache SystemML
Why do we need just another library?
A cost-based optimizer for machine learning algorithms
Performance measurements
Apache SystemML in action
Summary
7
Apache Spark GraphX
Apache Spark GraphX
Overview
Graph analytics/processing with GraphX
Summary
8
Spark Tuning
Spark Tuning
Monitoring Spark jobs
Spark configuration
Common mistakes in Spark app development
Optimization techniques
Summary
9
Testing and Debugging Spark
Testing and Debugging Spark
Testing in a distributed environment
Testing Spark applications
Debugging Spark applications
Summary
10
Practical Machine Learning with Spark Using Scala
Practical Machine Learning with Spark Using Scala
Introduction
Configuring IntelliJ to work with Spark and run Spark ML sample codes
Running a sample ML code from Spark
Identifying data sources for practical machine learning
Running your first program using Apache Spark 2.0 with the IntelliJ IDE
How to add graphics to your Spark program
11
Spark's Three Data Musketeers for Machine Learning - Perfect Together
Spark's Three Data Musketeers for Machine Learning - Perfect Together
Introduction
Creating RDDs with Spark 2.0 using internal data sources
Creating RDDs with Spark 2.0 using external data sources
Transforming RDDs with Spark 2.0 using the filter() API
Transforming RDDs with the super useful flatMap() API
Transforming RDDs with set operation APIs
RDD transformation/aggregation with groupBy() and reduceByKey()
Transforming RDDs with the zip() API
Join transformation with paired key-value RDDs
Reduce and grouping transformation with paired key-value RDDs
Creating DataFrames from Scala data structures
Operating on DataFrames programmatically without SQL
Loading DataFrames and setup from an external source
Using DataFrames with standard SQL language - SparkSQL
Working with the Dataset API using a Scala Sequence
Creating and using Datasets from RDDs and back again
Working with JSON using the Dataset API and SQL together
Functional programming with the Dataset API using domain objects
12
Common Recipes for Implementing a Robust Machine Learning System
Common Recipes for Implementing a Robust Machine Learning System
Introduction
Spark's basic statistical API to help you build your own algorithms
ML pipelines for real-life machine learning applications
Normalizing data with Spark
Splitting data for training and testing
Common operations with the new Dataset API
Creating and using RDD versus DataFrame versus Dataset from a text file in Spark 2.0
LabeledPoint data structure for Spark ML
Getting access to Spark cluster in Spark 2.0
Getting access to Spark cluster pre-Spark 2.0
Getting access to SparkContext vis-a-vis SparkSession object in Spark 2.0
New model export and PMML markup in Spark 2.0
Regression model evaluation using Spark 2.0
Binary classification model evaluation using Spark 2.0
Multiclass classification model evaluation using Spark 2.0
Multilabel classification model evaluation using Spark 2.0
Using the Scala Breeze library to do graphics in Spark 2.0
13
Recommendation Engine that Scales with Spark
Recommendation Engine that Scales with Spark
Introduction
Setting up the required data for a scalable recommendation engine in Spark 2.0
Exploring the movies data details for the recommendation system in Spark 2.0
Exploring the ratings data details for the recommendation system in Spark 2.0
Building a scalable recommendation engine using collaborative filtering in Spark 2.0
14
Unsupervised Clustering with Apache Spark 2.0
Unsupervised Clustering with Apache Spark 2.0
Introduction
Building a KMeans classifying system in Spark 2.0
Bisecting KMeans, the new kid on the block in Spark 2.0
Using Gaussian Mixture and Expectation Maximization (EM) in Spark to classify data
Classifying the vertices of a graph using Power Iteration Clustering (PIC) in Spark 2.0
Latent Dirichlet Allocation (LDA) to classify documents and text into topics
Streaming KMeans to classify data in near real-time
15
Implementing Text Analytics with Spark 2.0 ML Library
Implementing Text Analytics with Spark 2.0 ML Library
Introduction
Doing term frequency with Spark - everything that counts
Displaying similar words with Spark using Word2Vec
Downloading a complete dump of Wikipedia for a real-life Spark ML project
Using Latent Semantic Analysis for text analytics with Spark 2.0
Topic modeling with Latent Dirichlet allocation in Spark 2.0
16
Spark Streaming and Machine Learning Library
Spark Streaming and Machine Learning Library
Introduction
Structured streaming for near real-time machine learning
Streaming DataFrames for real-time machine learning
Streaming Datasets for real-time machine learning
Streaming data and debugging with queueStream
Downloading and understanding the famous Iris data for unsupervised classification
Streaming KMeans for a real-time on-line classifier
Downloading wine quality data for streaming regression
Streaming linear regression for a real-time regression
Downloading Pima Diabetes data for supervised classification
Streaming logistic regression for an on-line classifier
Other Books You May Enjoy
Other Books You May Enjoy
Leave a review - let other readers know what you think
Index
Index

Debugging Spark applications

In this section, we will see how to debug Spark applications that are running locally (on Eclipse or IntelliJ), standalone or cluster mode in YARN or Mesos. However, before diving deeper, it is necessary to know about logging in the Spark application.

Logging with log4j with Spark recap

As stated earlier, Spark uses log4j for its own logging. If you configured Spark properly, Spark gets logged all the operation to the shell console. A sample snapshot of the file can be seen from the following figure:

Figure 16: A snap of the log4j.properties file

Set the default spark-shell log level to WARN. When running the spark-shell, the log level for this class is used to overwrite the root logger's log level so that the user can have different defaults for the shell and regular Spark apps. We also need to append JVM arguments when launching a job executed by an executor and managed by the driver. For this, you should edit the conf/spark-defaults.conf. In short, the following...

Previous Section
End of Section 4
Next Section