Smoke signals, couriers, carrier pigeons, and semaphores: if this was a riddle, you would think of messages right away. Humanity has always had the need to connect with each other, finding new ways to defy the challenge posed by the distance between the different groups of people who need to communicate. We've come a long way with modern technologies, but essentially the basics remain. Senders, recipients, and messages are the core of all our communication infrastructures.
Software applications have the same needs; systems need to exchange messages with each other. They sometimes need to be sure that the message sent has reached its destination. They sometimes need to receive an immediate response, but not all the time. In some cases, they may even need to receive more than one response. Based on these different needs, different styles of communication between systems have emerged.
All this can be explained with the help of the following figure:
The request-response style of interaction
This request-response style of interaction is the most common style; a system (acting as a client) interacts with a remote system (acting as a server) via a synchronous interaction with a remotely exposed communication endpoint. Whether it takes the form of a remote procedure call, a web service invocation, or consumption of a resource, the model is the same: one system sends a message to another and waits for the remote party to respond synchronously. Systems communicate with each other in a point-to-point manner.
The advantages of this approach are met with inconveniences. On one hand, programmers get a simple programming model as everything happens in a procedural fashion. On the other hand, the tight coupling between both parties has a deep impact on the architecture of the whole system as it is hard to evolve, hard to scale, and so on.
One-way interaction with message queuing
Enter the one-way style of interaction, where systems interact with each other in an asynchronous fashion via the transmission of messages, and generally through the intermediary of relaying parties known as message brokers. In this scheme, commonly referred to as messaging or message queuing, systems play the role of message publishers (producers) and message consumers. They publish a message to a broker on which they rely on to deliver it to the intended consumer. If a response is required, it will eventually come at some point on time through the same mechanism, but reversed (the consumer and producer roles will be swapped).
The advantage of the messaging approach is that systems are loosely coupled. They don't need to know exactly where they are located; a mere name is enough to reach them. Systems can, therefore, be evolved in an independent manner with no impact on each other as the reliability of message delivery is entrusted to a broker. This is demonstrated in the following figure:
Message enabling a loosely coupled architecture
Indeed, the architecture represented in the preceding figure allows the following:
The publishers or consumers fail without impacting each other
The performance of each side to leave the other side unaffected
The number of instances of publishers and consumers to grow and reduce and to accommodate their workload in complete independence
The publishers are unaware of the location and technology of the consumers and vice-versa
The main downside of this approach is that programmers cannot rely on the mental model of procedural programming where things immediately happen one after another. In messaging, things happen over time, so systems must be programmed to deal with it.
If all this is a little blurry, let's use an analogy of a well-known protocol: Simple Mail Transfer Protocol (SMTP). In this protocol, e-mails are published (sent) to an SMTP server. This initial server then stores and forwards the e-mail to the next SMTP server, and so on until the recipient e-mail server is reached. At this point, the message is queued in an inbox, waiting to be picked up by the consumer (typically, via POP3 or IMAP). With SMTP, the publisher has no idea when the e-mail will be delivered or whether it will eventually be delivered at all. In case of a delivery failure, the publisher can be notified of issues later down the line. The only sure fact is that the broker has successfully accepted the message it had initially sent.
Furthermore, if a response is needed, it will arrive asynchronously using the same delivery mechanism but with the publisher and consumer roles reversed. The entire process is demonstrated in the following figure:
The e-mail infrastructure as an analogy for message queuing
With these fundamental notions established, let's now delve into the messaging protocol that we are going to consider in this book: Advanced Message Queuing Protocol (AMQP).
The Advanced Message Queuing Protocol (AMQP) is an open standard that defines a protocol for systems to exchange messages. AMQP defines not only the interaction that happens between a consumer/producer and a broker, but also the over-the-wire representation of the messages and commands that are being exchanged. Since it specifies the wire format for messages, AMQP is truly interoperable—nothing is left to the interpretation of a particular vendor or hosting platform. And since it is open, the AMQP community has flourished with broker and client implementations in a wide range of languages.
Note
The AMQP 0-9-1 specification can be downloaded at http://www.rabbitmq.com/resources/specs/amqp0-9-1.pdf.
Let's look at the following list of core concepts of AMQP, which we will revisit in detail in the upcoming chapters:
Broker: This is a middleware application that can receive messages produced by publishers and deliver them to consumers or to another broker.
Virtual host: This is a virtual division in a broker that allows the segregation of publishers, consumers, and all the AMQP constructs they depend upon, usually for security reasons (such as multitenancy).
Connection: This is a physical network (TCP) connection between a publisher/consumer and a broker. The connection only closes on client disconnection or in the case of a network or broker failure.
Channel: This is a logical connection between a publisher/consumer and a broker. Multiple channels can be established within a single connection. Channels allow the isolation of the interaction between a particular client and broker so that they don't interfere with each other. This happens without opening costly individual TCP connections. A channel can close when a protocol error occurs.
Exchange: This is the initial destination for all published messages and the entity in charge of applying routing rules for these messages to reach their destinations. Routing rules include the following: direct (point-to-point), topic (publish-subscribe) and fanout (multicast).
Queue: This is the final destination for messages ready to be consumed. A single message can be copied and can reach multiple queues if the exchange's routing rule says so.
Binding: This is a virtual connection between an exchange and a queue that enables messages to flow from the former to the latter. A routing key can be associated with a binding in relation to the exchange routing rule.
Overview of the concepts defined by the AMQP specification
You may have a message-queuing background and are by now wondering what are the main differences between AMQP and another protocol that you know. Here is a quick comparison of some of the main features:
Java Message Service (JMS): Unlike AMQP, this only defines the wire protocol for a Java programming interface and not messages. As such, JMS is not interoperable and only works when compatible clients and brokers are used. Moreover, unlike AMQP, it does not define the commands necessary to completely configure messaging routes, leaving too much room for vendor-specific approaches. Finally, in JMS, message producers target a particular destination (queue or topic), meaning the clients need to know about the target topology. In AMQP, the routing logic is encapsulated in exchanges, sparing the publishers from this knowledge.
MQ Telemetry Transport (MQTT): This is an extremely lightweight message-queuing protocol. MQTT focuses only on the publish-subscribe model. Like AMQP, it is interoperable and is very well suited for massive deployments in embedded systems. Like AMQP, it relies on a broker for subscription management and message routing. RabbitMQ can speak the MQTT protocol—thanks to an extension.
ØMQ (also known as ZeroMQ): This offers messaging semantics without the need for a centralized broker (but without the persistence and delivery guarantees that a broker provides). At its core, it is an interoperable networking library. Implemented in many languages, it's a tool of choice for the construction of high-performance and highly-available distributed systems.
Process inboxes: Programming languages and platforms such as Erlang or Akka offer messaging semantics too. They rely on a clustering technology to distribute messages between processes or actors. Since they are embedded in the hosting applications, they are not designed for interoperability.
Multiple commercial and open source implementations of AMQP are available. Often, existing messaging brokers have been extended with an AMQP adapter, like in the case of ActiveMQ.
The open source broker we will look at in detail for this book has been built from the ground to support AMQP. So let's now turn our focus on RabbitMQ.