MLlib library
MLlib is a scalable machine learning library built on top of Spark. The machine learning library is composed of two distinct packages, which are [17:03]:
org.apache.spark.mllib
: RDD-based library of some common machine learning algorithms. This package will be deprecated in future releases.org.apache.spark.ml
: Library of machine learning algorithms that leverages datasets and data frames structures. The package supports tasks pipeline and stages that are described and illustrated in the next section.
Overview
The main components of the MLlib package are as follows:
Classification algorithms, including logistic regression, Naïve Bayes, and support vector machines
Clustering and unsupervised learning techniques such as K-means
L1 and L2 regularization
Optimization techniques such as gradient descent, logistic gradient and stochastic gradient descent, and L-BFGS
Linear algebra such as singular value decomposition
Data generator for K-means, logistic regression, and support vector machines...