Book Image

Learning NServiceBus Sagas

By : Richard L Helton
Book Image

Learning NServiceBus Sagas

By: Richard L Helton

Overview of this book

Table of Contents (13 chapters)

A brief introduction to ESBs

NServiceBus is an Enterprise Service Bus. An ESB, a software architecture model, is usually a framework that is used for designing and implementing the end-to-end pieces of service-oriented architecture (SOA). SOA is a software architecture model based on distinct pieces of software, called services, providing application functionality as services to other applications. ESBs are common methods for designing SOAs. NSB is the most popular C# ESB. An ESB is a bus that brings a common communication mechanism together between services. Services are an abstract concept of a managed, self-contained process with messaging used to talk to the loosely-coupled, asynchronous, message-exchanging, monitored, managed, scalable, reliable, durable, and standard services.

SOA principles are widely implemented in the industry in the form of web services as web service messaging, usually across HTTPS sending XML-like messaging defined in Web Service Definition Language (WSDL). ESBs, implementing SOAs, decouple the frontend services from backend services. Some of the benefits of decoupling services include the ability to work on the services dependently and separately, which includes adding updates to one without affecting the others. The interfaces between the services are messages, usually in XML, that define the data to be exchanged between messages. An example of how decoupling is useful is: updating interfaces of banks and accounting systems may need to be performed, and during these updates, the working of frontend is totally autonomous to the services that are affected. Decoupling expedites testing, maintenance, reduction of coding side-effects, and assists in breaking down business logic into discrete pieces as services.

ESBs extend the use of SOAs, as they provide a common communication medium, known as a bus, to transmit messages in a managed form that is centrally monitored. Unlike a Service Broker, the ESB ensures message integrity so that the message arrives correctly and the transaction is centrally managed. ESBs can use multiple means to communicate messages, even in the form of web services, but they use a common framework to monitor the endpoints, messaging, and services. Otherwise, the pieces are more desperate and cannot be used collectively. Depending on the ESB used, it may collectively manage these pieces. ESBs are also event driven and rely on message queuing, so that when the bus receives a message from a service, it routes it to the next appropriate service to process the business logic. Here's a common bus in an ESB, managing messages to different endpoints:

The preceding diagram show very high-level interfaces, so we can drill down into the protocols of the services themselves.

By using NSB, we can rapidly develop end-to-end applications in Visual Studio. There are also tools to deploy services, monitor all endpoints, and check the integrity of messages from end to end to ensure proper SOA. ESBs have many pieces of cross-cutting functionality that improve the quality of software.

The purpose of the cross-cutting quality of software is to ensure maintainability, security, high availability, reporting, alerting, scalability, performance, and other operational integrity features. An ESB such as NSB helps by sharing the common framework for a service bus. For instance, I may want to encrypt only the SSN of a message from end to end. By using the same base code and framework, we can encrypt and decrypt at various endpoints' SSN pieces with the same codebase.

By using a framework such as NSB, we achieve unity in cross-cutting functionality. The use of NSB's service bus technology can bring many disparate communication protocols to act with a common methodology and workflow to use all the benefits of NSB, for the use of endpoint protocols such as the Secure File Transfer Protocol (SFTP), Microsoft Web stack for more Restful Web APIs, and Microsoft's WCF for web services. The queuing mechanisms, which are also endpoints, consist of Azure Services Bus or Azure Storage Queues, communicating through SQL queuing, Microsoft Message Queuing (MSMQ), ActiveMQ, and RabbitMQi. There are many endpoints, from queuing services to third-party vendors, which can be used in NSB.

Another instance of cross-cutting functionality is using the same code. Here are some of the benefits of using an ESB, such as NSB:

  • Message durability and error handling: Messages are guaranteed to be delivered. If an error occurs during message delivery, it can roll back and place the message in the error queue. There are first-level and second-level retries to get the message to work successfully.

  • Message queuing and fault tolerance: Messages persist in a queue until they are handled. If there is something wrong with the service, the message will remain in a persisted queue until the service is working successfully. No message will ever be lost.

  • High availability and high performance: Services can be cloned and clustered as needed to ensure bandwidth utilization of messages and services. These clones can be coded and configured based on meeting the service-level agreements (SLAs).

  • Extremely configurable: NSB is configured using the Configure.With() function that can easily change the persistence from RavenDB to the NHibernate ORM framework, or queuing from MSMQ, RabbitMQ, ActiveMQ, or SQL Queuing. Most of the code, if not all, except some configurations, will work the same regardless of the persistence and queuing models, as NSB takes care of the endpoint provisioning and mapping.

  • Service hosting: NSB takes care of the management and hosting of services using the NServiceBus.Host.exe file.

  • Central production monitoring: Particular's ServicePulse can monitor the production endpoint, message reporting, and service reporting.

  • ServiceInsight: Particular's ServiceInsight can provide a deep insight through visual canvases, endpoint views, message views, and a deeper understanding of saga's messages, endpoints, and services.

  • Rapid development: Particular's ServiceMatrix is a development studio extension for Visual Studio to provide rapid development through graphical canvases that generate C# code.

Let's look at a couple of practical needs for ESB designs.

  • Payment engine to a bank: A C# programmer could use the Microsoft Windows Communication Foundation (WCF) or Microsoft Web APIs to perform web services for an external bank to debit a credit card. There are many pieces that WCF will not perform to support cross-cutting concerns, such as retries and failures. NSB can host the service and provide durable messaging in MSMQ that performs message retries on the WCF interface. NSB will persist with messages if there is an interruption of web services to third-party services, such as a bank. This includes keeping track of message reporting and even the encryption of messages. So, while WCF can implement the web service piece, there are many pieces outside of the realm of WCF that only a product like NSB can provide cross-cutting functionality for.

  • An online ordering system: When ordering a product, you may be greeted with a page that spins and states Do not refresh, processing order. This is clearly a design where the website is tightly coupled to the ordering site, including the processing of the credit card. There are many automated processes that may disturb a web browser, or there may be some disconnect on the server side that may start the process over again. An ESB will always ensure that if you have to place your order again or had an interruption, your messaging is transactional and durable, ensuring integrity when charging for your order.


    For a website that is loosely coupled with an NSB design, the website could just send a message to the service queue without interrupting the web flow. A site that doesn't allow refreshing when processing an order depends on backend processes to finish, where the web flow may be interrupted. An NSB design can still give the user feedback on the status of their order without any interruption to the website.

Event-driven jobs

For scheduling jobs on Windows, the Windows Task Scheduler can do the job (see However, depending on the complexity of the job, the Windows Task Scheduler may not be enough to deal with it. For instance, if a holiday schedule needs to be checked before starting the job and if the job has dependencies on other events and services happening before it runs, you may run into limitations with the task scheduler. The Windows Task Scheduler is a great tool, but it is not event driven, as is an ESB. With ESBs, time is an event.

Even though a job may be easily done by Task Manager, you will not get the global management of services, the messaging of events, commands, and messages, and the feedback of the processing from a managed perspective, unless you use a framework such as NSB.

NSB extends the design to allow many facets of event-driven designing where building a simple website, scheduling a daily job, or sending payments can easily grow into something much larger in order to have, monitor, manage, extend, secure, and maintain great software quality. It is not uncommon that when these simple designs become complex, a rewrite is warranted. That is simply because software quality and extending the design were not taken into consideration ahead of time.

The ESB is an architectural software design pattern and framework for designing SOA architectures. However, there are many additional software design patterns that an ESB can bring to the table as well. These will be software design patterns that NSB supports, but not all ESBs support these patterns.

Additional SOA patterns

There are many additional software design and messaging patterns that NSB supports. Many such patterns can be found at The purpose of design patterns is to provide reusable frameworks that have been hardened through reuse in the industry, to rapidly deploy what's been designed before. NSB has several design patterns that are popular in any robust and solid ESB solution.

The publish-subscribe pattern

One of the biggest benefits of using ESB technology is the availability of the publish-subscribe message pattern (for more details, see The publish-subscribe pattern has been around for a very long time, and its simplicity is the cornerstone of many, many systems.

The pattern makes use of message queues, and NSB stores its messages through the use of queues, be it MSMQ, SQL queues, RabbitMQ, Azure queues, or ActiveMQ. The basic premise is that a message is published to a queue; in NSB, it uses the Bus.Publish(message) function that places an event message in a queue that does not need to be sent to specific receivers. There can be multiple output channels called subscribers listening in on the publisher. The benefit is that the services do not depend on any relationship with each other except the message, and can act independently of each other. This is a service which is decoupled. This means that dependencies are separated from each other.

Of course, normal NSB features still apply. For instance, the message will stay in the queue until the subscribers pull the message off the queue to handle it. For NSB, this means that the machine can be rebooted and the message will still be available in the queue until the subscribers process the message, which ensures that a message is never lost.

The subscribers will process the message, and there could be one or many subscribers. For instance, the publisher may put 10 different types of messages in its queue, and there could be a subscriber for each type of message.

The NSB framework has plugins for Windows Performance Monitor. In Service Level Agreement (SLA) code of the endpoint, we can specify that new services will be created or alerts will be sent if the service does not meet performance criteria. So if one of the subscribers is not performing well, it will force the other subscribers to take over, as they are performing better across multiple machines. Because of this, NSB is a highly available, durable-messaging, high-performance ESB engine that can be developed in a multitude of ways. A design of the publish-subscribe pattern appears as follows:

In NSB, there are special types of messages used specifically for publish-subscribe, called events. This forces the code to handle the message in a publish-subscribe scenario, and not in a request-reply scenario.

The publisher-subscriber mappings are done through the application configuration file. This will define the type of messages from the publisher queue that it will subscribe to, to handle the messages. When these programs execute, the subscription mappings will be further saved into a local database. By default, they are saved in RavenDB.

The request-reply pattern

The request-reply pattern is different from the publish-subscribe pattern as it sends a message to a directed endpoint.

The NSB framework will use the Bus.Send(message) function for request-reply. The message remains durable as it is still queued. The sender may or may not be available to handle the response, and the replier may not send a response, but the functionality in NSB is available to have these components easily added to handle the request-reply on their endpoints. The sender could easily be a website and the replier could send a response to update the website, for instance, when a credit card has been processed. The request-response pattern doesn't require subscription information to be stored, but it does require the endpoints to be defined along with the message types in the application configuration file. Sagas automatically store the sender's original information and reply directly to the requester, without storing extra information.

This pattern still has high availability and high performance, and still uses a multitude of topologies with message durability. Please see the scale out pattern in this chapter on how request-reply will cluster across machines.

The gateway pattern

There are cases when services may be partly stored on one part of an organization's LAN, and other services are stored on another LAN, and the only transport to each other is an HTTP or HTTPS tunnel to pass messages to NSB.

The main purpose of the gateway is to allow you to do the same durable fire-and-forget messaging that you are accustomed to with NServiceBus across physically separated sites, where sites are locations in which you run IT infrastructure and not websites.

The gateway only comes into play where you can't use normal LAN-to-LAN VPN tunnels or use internal LAN servers to communicate MSMQ to MSMQ. The purpose of the gateway is to create messages that communicate through HTTP, but it would be preferable to use HTTPS to ensure that messages are secured. The following architecture diagram represents a gateway built using the NSB:

The source code

In this section, we will be using the following gateway solutions:

  • Headquarter.Messages: This refers to the common messages for Headquarters, SiteA, and SiteB.

  • Headquarter: This will receive messages from http://localhost:25899/Headquarter/ and http://localhost:25899/Headquarter2/, and send messages to http://localhost:25899/SiteA/ and http://localhost:25899/SiteB/.

  • SiteA: This is a project that will receive updated price information from Headquarters across http://localhost:25899/SiteA/ and respond that it was successful to Headquarters across http://localhost:25899/Headquarter2/.

  • SiteB: This is a project that will receive updated price information from Headquarters across http://localhost:25899/SiteB/.

  • WebClient: This will have an Index.htm page to send a JSON script to http://localhost:25899/Headquarter/.

The preceding code solution is built using Visual Studio 2012 in Windows Server 2012, with MSMQ, DTC, NServiceBus references, and SQL Server 2012 Express LocalDB installed.

In a gateway, there are incoming channels and defined site keys to send outgoing messages to their sites. We can see in the application configuration file of the headquarters that the receiving channels for the headquarters are http://localhost:25899/Headquarter/ and http://localhost:25899/Headquarter2/.

There will be a set of site keys for sending sites that make up SiteA and SiteB.

      <Site Key="SiteA" Address="http://localhost:25899/SiteA/" ChannelType="Http" />
      <Site Key="SiteB" Address="http://localhost:25899/SiteB/" ChannelType="Http" LegacyMode="false" />
      <Channel Address="http://localhost:25899/Headquarter/" ChannelType="Http" />
      <Channel Address="http://localhost:25899/Headquarter2/" ChannelType="Http" Default="true" />

The Bus.SendToSites(new[] { "SiteA", "SiteB}) allows you to send messages to multiple sites as you can see in the preceding configurations. For instance, the parameter of SiteA will send the message to http://localhost:25899/SiteA/.

Going across alternate channels such as HTTP means that you lose the MSMQ safety guarantee of exactly one message. This means that communication errors resulting in retry can lead to receiving the same message more than once. To avoid burdening you with de-duplication, the NServiceBus gateway supports this out of the box. You just need to store the message IDs of all received messages so it can detect potential duplicates. De-duplication tables can be stored on the SQL Server using the NHibernate persistence configuration. This will be configured on the IBus using the .UseNHibernateGatewayDeduplication() method. Of course, settings always need to be applied in the App.config file to define the database connection. Here, we are connecting to the local SQLExpress instance.

  <connectionStrings><add name="NServiceBus/Transport" connectionString="cacheSendConnection=true"/>
    <add name="NServiceBus/Persistence" connectionString="Data Source=.\SQLEXPRESS;Initial Catalog=nservicebus;Integrated Security=True"/>
  <!-- specify the other needed NHibernate settings like below in appSettings:-->
    <!-- dialect is defaulted to MsSql2008Dialect, if needed change accordingly -->
    <add key="NServiceBus/Persistence/NHibernate/dialect" value="NHibernate.Dialect.MsSql2008Dialect"/>
    <!-- other optional settings examples -->
    <add key="NServiceBus/Persistence/NHibernate/connection.provider" value="NHibernate.Connection.DriverConnectionProvider"/>
    <add key="NServiceBus/Persistence/NHibernate/connection.driver_class" value="NHibernate.Driver.Sql2008ClientDriver"/>

This gateway pattern allows us to pass messages through HTTP/HTTPS to allow queuing across outside systems from the local LAN. Normally web services run through HTTP/HTTPS, but with NSB, the queuing process can also run through HTTP/HTTPS to outside servers, where all ports are blocked and only HTTP/HTTPS can be used.

The DataBus pattern

The DataBus is used to send large chunks of data or files across as an attachment because MSMQ is limited to 4 MB. For this reason, a reference can be passed on a local file to transfer data using the databus method. The message will provide a reference to a larger data file to be accessed that exceeds the message queue size due to size constraints.

In this section, we will use the gateway solution.

The path of the data bus has to be set in the configuration of the endpoint. We will be using a relative path to where the gateway project is running. Both SiteA and SiteB will also have relative paths. There will be a relative path to the binary file with a data bus subdirectory containing the files that will have a lot of data. The following is the code:

When we execute the gateway project, we can mock the SomeLargeString variable to simulate data larger than 4 MB, as shown in the following code:

If we execute the gateway project, it will create a message to the relative path of its binary file, save the message under data bus, and use it as a reference to send to SiteA and SiteB. Here, we see the message saved to the local relative path.

The data bus is very useful for processing files or chunks of data that are too large for MSMQ.

Timeout patterns

In ESB systems, the need for timers and timeouts cannot be underestimated. Many developers use the Microsoft Task Scheduler. The Microsoft Task Scheduler looks like this:

In NSB, we have the ability to use event-driven timer messages. This is the ability of a saga to start a function, run a command, or perform many other tasks, based on its timer. In the event-driven timer function, we can set a timeout for any time in the day, from seconds after the process starts, to minutes, hours, days, and even use a holiday table for the process. By having a managed service, the usage of timers can be more complex, for instance, adding holiday tables, not executing timers on weekends, and more business functions that task schedulers cannot handle. Also included in NSB, is the installation and management of the service itself. Time saved just in deploying services and having NSB set up DTC for the administrator may already pay for the NSB licenses.

The source code in this section will be in NSB Version 5.0 in the TimerSaga – v5 directory. Here we have the following projects:

  • AppCommon: This contains the ViewModel and context for the Windows forms.

  • TimerSubmit: This is a project that submits a timer message to the saga between 1-100 seconds.

  • TimerMessages: This contains common messages for the projects.

  • TimerSaga: This saga will perform the timeout and respond back to the TimerSubmit project. This could easily start a cron job, execute a program, or direct other services just as easily as responding back to the submitting program.

In this solution, we have a Windows form, where we enter a variable from 1 to 100, based on which the TimerSubmit project will generate a message to the Timer saga to create a timeout for those seconds. After those seconds expire, the Timer saga will handle the timeout message and respond back to the TimerSubmit window saying that the timer has expired. This is a simple exercise in an event-driven timer that could have many uses.

The program will start by submitting the number of seconds between 1-100 to TimerSubmit as shown in the following screenshot:

When we add a value between 1 and 100 seconds, the message will be sent to the queue of TimerSaga with the number of seconds and a RequestId to keep track of the message ID, as shown in the following screenshot:

The message's RequestId is used to map the message to the saga data. The saga is started by SubmitRequestCommand with the number of seconds in it. It will save the saga data in the TimerRequestData object, which will allow the original client data to respond to it. The number of seconds will be set in the timeout, and when the timeout expires, it will execute a TimeoutMessage instance that will be handled by the saga. The saga's object class definition that defines this mapping appears as follows:

The starting message SubmitRequestCommand will be handled by the saga's message handler in the following code:

This code will set the timer that will send a TimeoutMessage instance with the message's RequestId. All the messages that are part of the same starting message will have the same RequestId. This will keep track of which message started the next message, as well as map the saga data to save and retrieve it from the saga. The RequestId is handled like a primary key to the saga data that NSB will use to map the data to the messages. NSB handles the mapping, but we must define the unique identifiers and keep processing them in the messages. If this seems complicated, keep reading, as it is broken down in subsequent chapters.

When the timeout that we passed to the RequestTimeout<>() code is reached, an instance of a timeout message is passed back to the saga. If it was set for 10 seconds, after 10 seconds have passed, the following message handler in the saga is called to process the timeout message that we set in the timer. The Timeout message handler appears as follows:

This code will process the TimeoutMessage instance that we called state. We had mapped the RequestId in the version 5.0 Saga mapper code, ConfigureHowToFindSaga(SagaPropertyMapper<TimerRequestData> mapper) to map the message's RequestId to the data, so the state.RequestId will be the same as the Data.RequestId data instance of saga. The RequestId is checked to ensure that when we execute MarkAsComplete(), the correct saga store is deleted.

This can be broken down into simple steps:

  1. A message is sent to the saga with a timeout value.

  2. The saga values are stored and the timeout message is created, all using the same RequestId.

  3. The timeout expires.

  4. A timeout message is handled by the saga.

The saga data is deleted, and a message is sent to the originator saying that the timeout has been completed. In this message handler, we create a SubmitRequestReplyMessage instance with the RequestId, which will be the same RequestId as the message that started this process. We reply to the originator, which will send this message to the original TimerSubmit program.

When the TimerSubmit process receives the reply message, it will pop a Timer in Seconds window as follows:

This code has all the pieces of a simple saga. The code could be created with a combination of other frameworks as well, such as Quartz for the timer and TopShelf to create services, but NSB is a complete framework that provides end-to-end pieces of architecture and many patterns that have all the pieces.

The end message that showed the Timed status in the Windows form could easily be the execution of a job, another program, an e-mail of system status, a report of the system status, and many more timed tasks. The number of seconds used earlier was just a demonstration. The RequestTimeout() function could take seconds, minutes, hours, days, or could be executed at a specific time of day. We could execute the function with a weekend and holiday calendar and extend the functionality further. We could create a centralized saga timer to literally schedule all the tasks like Microsoft Task Scheduler, but create a managed service that can be monitored, managed, perform reporting, and improve functionality in a much further detailed solution.

Message mutation patterns

Message mutators allow you to change messages by plugging custom logic into a couple of simple interfaces. For instance, you can encrypt a part or all of a message. The encryption message mutator is part of the NServiceBus library, and can be used at any time. You can intercept the incoming message and then mutate it before sending it as an outgoing message. This is the process of changing messages as they leave a client and enter a server.

The source code

In this section, we will be using the MessageMutators solution with the following projects:

  • Client: The client will send messages to the server.

  • Server: This will receive the mutated message.

  • Messages: This is the message format being passed between client and server.

  • MessageMutators: This project will contain the mutation code to compress and decompress the messages in TransportMessageCompressionMutator.cs and validate the message annotation in ValidateMessageMutator.cs.

The client and server needs to be running. The client will prompt to send a good or bad message. The good message is compressed so that it will pass the 4 MB MSMQ buffer size, as shown in the following screenshot:

The queue's data will be validated and compressed from the client before processing it in the MSMQ. This is shown in the following diagram:

Then, the server will receive the message from MSMQ, but before processing it, this will decompress and validate the message. It will restore the message that the client mutated. This is shown in the following diagram:

This is just a simple compression and data annotation validation to ensure that MSMQ will process the message. One of the reasons for mutating the message may be the encryption of a credit card within a payment message.

Message encryption patterns

NSB supports the AES (Rijndael) encryption algorithm. AES stands for Advanced Encryption Standard. This is a symmetric key algorithm, so both the program encrypting data and decrypting data must share a secret key for their functioning. Visit for more information on AES.

Encrypting data will depend on the needs of the organization, but common items include passwords, financial information, or personal customer identification information. AES is the strongest symmetric encryption algorithm, and most languages, such as Java and C#, have API support for its use.

We know that part of the configuration on both sides will be a secret key.

The source code

In this section, we will be using the Encryption solution with the following projects:

  • Client: This will send an encrypted credit card message to the server

  • Server: This will receive the credit card message and decrypt it

  • Messages: This is the message format being passed between client and server

Both, the client and server must be running. The client will have a prompt to send messages to the server, as shown in the following screenshot:

After pressing Enter, we see that the message is encrypted on the server queue:

When running the server, NSB will decrypt the message before passing it to the server's message handler.

All that is really needed is to enable both ends for AES in the IBus by the configuration .RijindaelEncryptionService();. We set the part of the message that we want to encrypt by the public WireEncryptionString Secret { get;set; } where WireEncryptionString defines that the string will be encrypted. Also, the secret key has to be in the App.config file of both the client and server. The following screenshot shows the code:

The ScaleOut pattern

As mentioned earlier, one of the many benefits of using NSB is that you can distribute the load or the NSB services or processes. This is commonly known as scaling out the services. The idea is that you can deploy the same service across a farm or multiple servers. This is used to create an environment of high availability.

This model is a form of round-robin clustering, where a handler can distribute its workload to additional workers doing exactly the same work. A distributor is used with MSMQ. If an endpoint has a critical time set for performance and requires more processing help, this clustering is used to spawn off work to the same services living on other machines to share the load. If the machine processing the message crashes, the message rolls back to the queue and other machines can then process it.

Worker services send messages through a control queue saying that they are ready for work. The distributor stores these messages, and when it receives messages, it fetches them out of the available queues. All pending work stays in the distributor's queue so that messages can be timed for performance.

The saga design pattern

The saga design pattern starts with a message to the saga service. The saga service requires several components, as follows:

  • IAmStartedByMessages<IMessage>: The saga's actions are started by an incoming message. This message initializes the saga data object and creates the first elements, which needs to include a primary key from this message for subsequent lookups. The saga data elements are created in the message handler. This is shown in the following screenshot:

  • Saga<IContainSagaData>: The saga data is the data that is persisted. This data contains default values to be set by NSB for an ID, originator, and OriginalMessageId instance. Also, a unique ID should be used for the mapping of the message to data, and vice versa. This is shown in the following screenshot:

  • IHandleMessages<IMessage>: The saga doesn't do much unless it is handling messages. It can retrieve the saved data of a message that is mapped by the configuration. NSB handles the mapping of the data to the message by the unique ID, in this case, RequestId, that is defined in ConfigureHowToFindSaga().

  • ConfigureHowToFindSaga(): This function of the saga pattern is used to map the data to the messages. Usually, the mapping can be performance from a GUID or ID, but it must be a unique data type that can be stored as a primary key in a database. The messages that are mapped contain the same key, and when it passes in the saga, the data is found that matches the key. This function is called when the saga object is instanced.