In this section, we will talk about how messaging systems play important role in a big data application.
Let's understand the different layers in a big data application:
- Ingestion layer: The input data required for the processing gets ingested in some storage system. There can be many sources of data for which the same or different processing needs to be done.
- Processing layer: This contains the business logic that processes the data received in the ingestion layer and applies some transformation to make it into a usable form. You can call it converting raw data to information. There can be multiple processing applications for the same or different data. Each application may have its different processing logic and capability.
- Consumption layer: This layer contains data processed by the processing layer. This processed data is a single point of truth and contains important information for business decision makers. There can be multiple consumers who can use the same data for different purposes or different data for the same purpose.
Streaming applications would probably fall into the second layer--the processing layer. The same data can be used by many applications simultaneously, and there can be different ways of serving this data to the application. So, applications can be either streaming, batch, or micro-batch. All these applications consume data in different ways: streaming applications may require data as a continuous stream and batch applications may require data as batches. However, we have already said that there can be multiple sources for this data.
We can see multiple producer and multiple consumer use cases here, so we have to go for a messaging system. The same message can be consumed by multiple consumers so we need to retain the message until all the consumers consume it. How about having a messaging system that can retain the data until we want it, provides a high degree of fault tolerance, and provides a different way of consuming data streams, batches, and micro-batches?
Streaming applications will simply consume the data from the messaging queue that they want and process it as needed. However, there is one problem. What if the message received by the streaming application fails, what if there are a lot of such messages? In such cases, we may want to have a system that will help us provide those messages based on the request and reprocess them.
We need a messaging system that immediately tells the streaming application that, Something got published; please process it. The following diagram helps you understand a messaging system use case with a streaming application:
The preceding figure explains the following points:
- Streaming application 1 has subscribed to Topic 1, which means any event published to topic 1 will be immediately available to Streaming Application 1.
- Streaming Application 1 processes the event and stores them into two destinations; one is a database and other is Topic 2 of the messaging system. Here, the the streaming application acts as the producer for Topic 2. Remember there can be other applications that may consume the event from Topic 1.
- Streaming application 2 has subscribed to Topic 2, which will immediately receive the event when it gets published to Topic 2. Remember that there can be other applications that can publish the event to either Topic 1 or Topic 2.
- Streaming Application 2 processes the event and stores it in the database.
In streaming application, each stream or message has its own importance; something will be triggered based on the type or nature of the message. There can be a scenario where one streaming application processes the event and passes it to another streaming application for further processing. In this case, they both need to have a medium of communication. Remember that the application should care about what it wants to do rather than how to send the data somewhere. This is the best use case for a publish/subscribe messaging system as it would ensure that a message published by the producer will reach to all the applications who have subscribed to it.
Concluding our discussion on messaging systems, these are the points that are important for any streaming application:
- High consuming rate: Streaming data sources can be click-stream data or social media data where the rate of message producing is too high. Stream applications may or may not be required to consume at a similar rate. We may want to have a messaging queue that can consume data at a higher rate.
- Guaranteed delivery: Some streaming applications cannot afford to lose messages; we need a system that guarantees the delivery of messages to the streaming application whenever needed.
- Persisting capability: There can be multiple applications consuming similar data at a different rate. We may want to have a messaging system that retains data for a period of time and serves the data to a different application asynchronously. This helps in decoupling all the applications and designing micro service architecture.
- Security: Some applications may want to have security over the data that they consume; you may not want to share some data with other applications consuming from the same messaging system. You want to have a system that ensures such security.
- Fault tolerance: Applications never want to have a system that does not deliver messages or data whenever they need. We want to have a system that guarantees fault tolerance and serves messages irrespective of the failure of the server serving the data before.
There are many other points that force us to go for a messaging system that has at least the capabilities mentioned earlier. We will discuss how Kafka is different from other messaging systems, and meets the requirement of a messaging system for a streaming application, in upcoming chapters.