A typical enterprise will have a number of systems that must typically communicate with each other in order to implement a well-defined business process. A question that is frequently tackled for this reason is how to implement the communication channel between these types of systems? For example, consider the following diagram:
The question mark in the preceding picture denotes the communication media for the six systems that are illustrated. In the diagram, we can think of these separate systems as the components of one large system and the problem stays the same. Before discussing the various alternatives for integration, a number of key factors are considered, as follows:
Loose coupling: At what degree do the different systems depend on each other or can operate independently?
Real-time workload processing: How fast is the communication between the systems?
Scalability: How does the entire system scale when more systems are added and the workload demands an increase?
Maintainability: How hard it is to maintain the integrated systems?
Extensibility: How easy it is to integrate new systems?
Let's assume that the systems communicate directly via some kind of remote procedure calls as shown in the following diagram:
This implies that separate communication links must be established between each pair of systems, which leads to tight coupling, a lot of effort to maintain all of the links, reduced scalability, and reduced extensibility (for every new system that is added, a few more communication links with other systems must be created). However, real-time communication requirements might be met with some additional effort to design the communication links.
A second approach is to use a shared file system in order to exchange files between the systems that are being integrated, as illustrated in the following diagram:
A shared file system is used to provide the communication medium. Each system may export data to a file that can be imported and used by other systems. The fact that each system may support its own data format leads to the fact that each system must have a particular mechanism to import data from every other system that it needs to communicate with. This, on other hand, leads to the same problems that are described in the case of direct communication. Moreover, real-time communication requirements might be more difficult to establish and reading or writing data from disk is also an expensive operation.
A third option is to use a shared database as shown in the following diagram:
Here, all the systems should depend on the same database schema. Although this reduces coupling between systems and improves extensibility (new systems must conform to a single database schema in order to integrate with other systems), real-time workload processing is still an issue. Scalability and maintainability depend directly on the type of database that is being used and they could turn out to be weak factors especially if it is a relational database (this may not be the case if NoSQL solutions are applicable for the particular use case).
Messaging comes to the rescue when considering the problems that arise when applying the previous approaches. Consider the following diagram for the Enterprise Messaging System:
A message is the central unit of communication used in enterprise messaging systems. A message typically consists of the following:
A message header: It provides metadata about the message such as encoding, routing information, and security-related information
A message body: It provides the actual data that is carried by the message, represented in a proper format
The messaging system itself provides mechanisms to validate, store, route, and transform messages that are being sent between the different systems. Each system is responsible for crafting its own message that is transferred via the messaging system (also called the messaging broker) to other systems that are connected to the broker and configured to receive that particular type of message. Each system may create a message in a proper format that is specified by the protocol of the message broker—meaning that the system is only coupled with that particular protocol. If the broker implements a protocol that is based on a well-recognized standard, then this would further decouple the systems from that particular message broker implementation.
Real-time workload processing is typically quite fast as the particular protocol that is implemented by different messaging brokers is optimized to process message data in a reliable and secure manner with minimal overhead. Most messaging solutions provide a number of facilities that allow easy configuration, management, and monitoring; thus, simplifying maintainability. Clustering support is also considered by most implementations due to scalability and reliability requirements and increasing workload demands. Integrating new systems is a matter of implementing a mechanism for direct communication with the message broker.
In case the different systems provide different implementations of messaging protocols (such as REST, SOAP, JSON-RPC, JMX, AMQP, and many others), a messaging system could further provide various adapters for the different protocols as well as extended mechanisms for routing and transformation of different types of messages—this extended functionality also categorizes the message brokers as Enterprise Service Bus (ESB) solutions. One major drawback of an ESB is that it must implement all the communication requirements of all systems that are being integrated by the ESB, otherwise workarounds must be used in order to implement direct communication between the integrated systems (thus, neglecting the usage of an ESB).
There are a variety of scenarios where messaging systems may be applied, such as the following:
Financial services: High rate real-time trade transactions handled between different systems
Social networking: Activity streams and event propagation between different components in a social network
E-mailing: Sending e-mail notifications or digests periodically to a large number of users
Processing large volumes of data upon request, such as image rendering
Chat services
Propagation of events throughout a system
Any type of real-time system integration (system of systems)
As you can see, messaging solutions can be applied to a variety of scenarios that typically involve a number of systems that must communicate in a timely manner or perform a large number of time-consuming tasks. Messaging solutions are also extensively being deployed by Cloud providers in order to provide messaging as a service for Cloud-based applications.
A wide variety of open source and property messaging solutions are available for use, which are based on the multitude of use cases. The choice of a messaging broker depends on a number of factors, as shown in the following:
Supporting tools, documentation and services: These are tools for management and monitoring of the broker along with possible options to receive support, typically the support is guaranteed only for commercial brokers. For open source, this depends on the activeness of the community.
Ease of configuration: This shows how easy it is to set up and configure the message broker.
Functionality: The features implemented by the solution and their coverage of the usage scenario. Here the supported protocols for message transfer play a key role in the decision.
The cost and licensing model.
A messaging system provides patterns for communication between system endpoints.
In a point-to-point communication model, there is exactly one sender and one receiver of a message. In case there are multiple senders that are applicable for the purpose of receiving the message, only one of them succeeds. Such receivers are also referred to as competing consumers, indicating that any of them are eligible to receive the message. The sender does not receive a response in a point-to-point model.