SMACK is about a full stack for pipeline data architecture--it's Spark, Mesos, Akka, Cassandra, and Kafka. Further on in the book, we will also talk about the most important factor: the integration of these technologies.
Pipeline data architecture is required for online data stream processing, but there are a lot of books talking about each technology separately. This book talks about the entire full stack and how to perform integration.
This book is a compendium of how to integrate these technologies in a pipeline data architecture.
We talk about the five main concepts of pipeline data architecture and how to integrate, replace, and reinforce every layer:
- The engine: Apache Spark
- The actor model: Akka
- The storage: Apache Cassandra
- The message broker: Apache Kafka
- The hardware scheduler: Apache Mesos:
Figure 1.1 The SMACK pipeline architecture
Spark is a fast and general engine for data processing on a large scale.
The Spark goals are:
- Fast data processing
- Ease of use
- Supporting multiple languages
- Supporting sophisticated analytics
- Real-time stream processing
- The ability to integrate with existing Hadoop data
- An active and expanding community
Here is some chronology:
- 2009: Spark was initially started by Matei Zaharia at UC Berkeley AMPLab
- 2010: Spark is open-sourced under a BSD license
- 2013: Spark was donated to the Apache Software Foundation and its license to Apache 2.0
- 2014: Spark became a top-level Apache Project
- 2014: The engineering team at Databricks used Spark and set a new world record in large-scale sorting
As you are reading this book, you probably know all the Spark advantages. But here, we mention the most important:
- Spark is faster than Hadoop: Spark makes efficient use of memory and it is able to execute equivalent jobs 10 to 100 times faster than Hadoop's MapReduce.
- Spark is easier to use than Hadoop: You can develop in four languages: Scala, Java, Python, and recently R. Spark is implemented in Scala and Akka. When you work with collections in Spark it feels as if you are working with local Java, Scala, or Python collections. For practical reasons, in this book we only provide examples on Scala.
- Spark scales differently than Hadoop: In Hadoop, you require experts in specialized Hardware to run monolithic Software. In Spark, you can easily increase your cluster horizontally with new nodes with non-expensive and non-specialized hardware. Spark has a lot of tools for you to manage your cluster.
- Spark has it all in a single framework: The capabilities of coarse grained transformations, real-time data-processing functions, SQL-like handling of structured data, graph algorithms, and machine learning.
It is important to mention that Spark was made with Online Analytical Processing (OLAP) in mind, that is, batch jobs and data mining. Spark was not designed for Online Transaction Processing (OLTP), that is, fast and numerous atomic transactions; for this type of processing, we strongly advise the reader to consider the use of Erlang/Elixir.
Apache Spark has these main components:
- Spark Core
- Spark SQL
- Spark Streaming
- Spark MLIB
- Spark Graph
The reader will find that each Spark component normally has several books. In this book, we just mention the essentials of Apache Spark to meet the SMACK stack.
In the SMACK stack, Apache Spark is the data-processing engine; it provides near real-time analysis of data (note the word near, because today processing petabytes of data cannot be done in real time).
Akka is an actor model implementation for JVM, it is a toolkit and runtime for building highly concurrent, distributed, and resilient message-driven applications on the JVM.
The open source Akka toolkit was first released in 2009. It simplifies the construction of concurrent and distributed Java applications. Language bindings exist for both Java and Scala.
It is message-based and asynchronous; typically no mutable data is shared. It is primarily designed for actor-based concurrency:
- Actors are arranged hierarchically
- Each actor is created and supervised by its parent actor
- Program failures treated as events are handled by an actor's supervisor
- It is fault-tolerant
- It has hierarchical supervision
- Customizable failure strategies and detection
- Asynchronous data passing
- Parallelized
- Adaptive and predictive
- Load-balanced
Apache Cassandra is a database with the scalability, availability, and performance necessary to compete with any database system in its class. We know that there are better database systems; however, Apache Cassandra is chosen because of its performance and its connectors built for Spark and Mesos.
In SMACK, Akka, Spark, and Kafka can store the data in Cassandra as a data layer. Also, Cassandra can handle operational data. Cassandra can also be used to serve data back to the application layer.
Cassandra is an open source distributed database that handles large amounts of data; originally started by Facebook in 2008, it became a top-level Apache Project from 2010.
Here are some Apache Cassandra features:
- Extremely fast
- Extremely scalable
- Multi datacenters
- There is no single point of failure
- Can survive regional faults
- Easy to operate
- Automatic and configurable replication
- Flexible data modeling
- Perfect for real-time ingestion
- Great community
Apache Kafka is a distributed commit log, an alternative to publish-subscribe messaging.
Kafka stands in SMACK as the ingestion point for data, possibly on the application layer. This takes data from one or more applications and streams it across to the next points in the stack.
Kafka is a high-throughput distributed messaging system that handles massive data load and avoids back pressure systems to handle floods. It inspects incoming data volumes, which is very important for distribution and partitioning across the nodes in the cluster.
Some Apache Kafka features:
- High-performance distributed messaging
- Decouples data pipelines
- Massive data load handling
- Supports a massive number of consumers
- Distribution and partitioning between cluster nodes
- Broker automatic failover
Mesos is a distributed systems kernel. Mesos abstracts all the computer resources (CPU, memory, storage) away from machines (physical or virtual), enabling fault-tolerant and elastic distributed systems to be built easily and run effectively.
Mesos was build using Linux kernel principles and was first presented in 2009 (with the name Nexus). Later in 2011, it was presented by Matei Zaharia.
Mesos is the foundation of several frameworks; the main three are:
- Apache Aurora
- Chronos
- Marathon
In SMACK, Mesos' task is to orchestrate the components and manage resources used.