Jasper meets RabbitMQ

For the most part, I’ve been focused on Jasper’s built in socket-based and HTTP transports because they’re compatible with the infrastructure that our older, existing systems at work already use. For pretty easy to understand reasons, other folks in the company are very apprehensive about depending on something new that they’ve never heard of (even though its forebears have been running happily in production for 4-5 years, but I digress). To have another transport option using more industry proven technology, I took a couple days and built out a new transport option in Jasper that uses the well known RabbitMQ message broker to do the actual work of moving message data from service to service.

Getting Started with RabbitMQ within Jasper

There really isn’t that much to do. Assuming that you have RabbitMQ itself stood up (I run it locally on OS X for development, but hosting it in Docker is pretty easy too) and a Jasper application, you just need to install the Jasper.RabbitMQ Nuget into your Jasper application project and configure listeners or publishing rules via “rabbitmq” Uris as shown in the samples below.

From there, you can direct Jasper to listen for messages from a RabbitMQ broker like this:

public class AppListeningToRabbitMQ : JasperRegistry
{
    public AppListeningToRabbitMQ()
    {
        // Port is optional if you're using the default RabbitMQ
        // port of 5672, but it's shown here for completeness
        Transports.ListenForMessagesFrom("rabbitmq://rabbitserver:5672/messages");
    }
}

I’ll discuss how to get around this restriction in a later section, but for right now this configuration implies that the messages received from this queue are coming from another Jasper application or at least an application that is using Jasper’s idioms for envelope metadata through headers.

The Uri structure is specific to Jasper just to identify the connection to a certain RabbitMQ broker by host name, port number, and RabbitMQ specific information like queue names, exchanges, and exchange types.

Likewise, message subscriptions or publishing rules are configured by the same Uri data, with an example publishing rule shown below (when Jasper sees the Uri scheme “rabbitmq,” it knows to delegate to the RabbitMQ transport internally):

public class AppPublishingToRabbitMQ : JasperRegistry
{
    public AppPublishingToRabbitMQ()
    {
        Publish.AllMessagesTo("rabbitmq://rabbitserver:5672/messages");
    }
}

Again, this simplistic example is assuming that there’s another Jasper application on the receiving side of the queue, or at least something that can understand the message metadata and serialized message data sent through the queue.

For the moment, I’ve just been testing with direct exchanges in RabbitMQ. It should be possible as is to use fan out exchanges with the current code, but I was unable to write a successful test proving that really worked locally. The usage of RabbitMQ right now is assuming that Jasper is responsible for the message routing. I’m going to leave the work for a later day, but the “seams” are already in place where we could replace Jasper’s built in message routing to use RabbitMQ topic or header exchanges to offload the actual routing logic to RabbitMQ.

More things to know:

• Jasper is using the RabbitMQ.Client library to interact with RabbitMQ. If you browse the Jasper documentation, you can see how Jasper.RabbitMQ exposes all the RabbitMQ.Client ConnectionFactory configuration so that you can use any necessary features or configuration for that library (like I don’t know, security?).
• The RabbitMQ transport can be used either in a “fire and forget,” non-durable manner, or as a durable transport that’s still backed up by Jasper’s support for durable messaging and the outbox pattern I demonstrated in my last post that obviates the need for distributed transactions between your application database and RabbitMQ itself.
• More below, but you can use the RabbitMQ transport to integrate with other applications that are not themselves Jasper applications or even .Net applications

 

What if Jasper isn’t on the other end?

I did a long, internal presentation on Jasper at work a couple weeks ago that I would charitably describe as a train wreck (Jasper itself did fine, but the presentation went soaring off the rails for me). One of the main axis of pushback about adopting Jasper was that it would lock us into the .Net platform. Besides the ability to just use HTTP services between applications, I tried to say that we’d still be able to do messaging between Jasper/.Net applications and say a Node.js application through RabbitMQ with just a little bit of potential effort to translate between message metadata.

At the point of receiving a message from RabbitMQ or enqueuing a message to RabbitMQ, there’s a little bit of mapping that takes Jasper’s Envelope object that contains the raw message and some tracking metadata, and maps that to the RabbitMQ.Client IBasicProperties model. If there’s a non-Jasper application on the other side of the queue, you may want to do some translation of the metadata using the IEnvelopeMapper interface or subclass the built in DefaultEnvelopeMapper as shown below:

public class CustomEnvelopeMapping : DefaultEnvelopeMapper
{
    public override Envelope ReadEnvelope(byte[] data, IBasicProperties props)
    {
        // Customize the mappings from RabbitMQ headers to
        // Jasper's Envelope values
        return base.ReadEnvelope(data, props);
    }

    public override void WriteFromEnvelope(Envelope envelope, IBasicProperties properties)
    {
        // Customize how Jasper Envelope objects are mapped to
        // the outgoing RabbitMQ message structure
        base.WriteFromEnvelope(envelope, properties);
    }
}

To use your custom envelope mapping, attach it by Uri again like this:

public class CustomizedRabbitMQApp : JasperRegistry
{
    public CustomizedRabbitMQApp()
    {
        Settings.Alter<RabbitMQSettings>(settings =>
        {
            // Retrieve the Jasper "agent" by the full Uri:
            var agent = settings.For("rabbitmq://server1/queue1");

            // Customize or change how Jasper maps Envelopes to and from
            // the RabbitMQ properties
            agent.EnvelopeMapping = new CustomEnvelopeMapping();
        });
    }
}

You may not even need to do this at all because at a minimum, the raw message data, the application id, the message type, and content type (think “application/json” etc) will map correctly to Jasper’s internal structure, but you’d just miss out on some correlation data and the ability to do workflows like saga id tracking and request/reply semantics.

I edited this section a couple times to try to filter out most of the snark and irritation on my part, but I’m sure some of that comes shining through.

 

Why not just use RabbitMQ?

In an internal presentation at work a couple weeks ago, I tried to use this slide to illustrate why we would still use Jasper with RabbitMQ instead of “just using RabbitMQ”:

Feel free to argue with the details of my Venn diagram up there, but the point I was trying to get across was that there were a lot of application concerns around messaging that simply aren’t part of what RabbitMQ or Kafka or Azure Service Bus does for you. I spent about 20 minutes on this, then demonstrated a lot of Jasper’s features for how it takes raw byte[] data and marshals that to the right command executor in your application with all the error handling, instrumentation, outbox, and middleware fixings you’d expect in a tool like Jasper. And at the end of all that, I got a huge dose of “why aren’t we just using Kafka?” Grrrrrr.

I intend to write a Jasper-flavored version of this question someday soon, but for now, I’ll recommend Sure, You Can Just Use RabbitMQ.