Tag Archives: Jasper

Jasper’s “Outbox” Pattern Support

Jasper supports the “outbox pattern,”  a way to achieve consistency between the outgoing messages that you send out as part of a logical unit of work without having to resort to two phase, distributed transactions between your application’s backing database and whatever queueing technology you might be using. Why do you care? Because consistency is good, and distributed transactions suck, that’s why.

Before you read this, and especially if you’re a coworker freaking out because you think I’m trying to force you to use Postgresql, Jasper is not directly coupled to Postgresql and we will shortly add similar support to what’s shown here for Sql Server message persistence with Dapper and possibly Entity Framework.

Let’s say you have an ASP.Net Core MVC controller action like this in a system that is using Marten for document persistence:

public async Task<IActionResult> CreateItem(
    [FromBody] CreateItemCommand command,
    [FromServices] IDocumentStore martenStore,
    [FromServices] IMessageContext context)
{
    var item = createItem(command);

    using (var session = martenStore.LightweightSession())
    {
        session.Store(item);
        await session.SaveChangesAsync();
    }
    
    var outgoing = new ItemCreated{Id = item.Id};
    
    await context.Send(outgoing);

    return Ok();
}

It’s all contrived, but it’s a relatively common pattern. The HTTP action:

  1. Receives a CreateItemCommand message from the client
  2. Creates a new Item document and persists that with a Marten document session
  3. Broadcasts an ItemCreated event to any known subscribers through Jasper’s IMessageContext service. For the sake of the example, let’s say that under the covers Jasper is publishing the message through RabbitMQ (because I just happened to push Jasper’s RabbitMQ support today).

Let’s say that in this case we need both the document persistence and the message being sent out to either succeed together or both fail together to keep your system and any downstream subscribers consistent. Now, let’s think about all the ways things can go wrong:

  1. If we keep the code the way that it is, what if the transaction succeeds, but the call to context.Send() fails, so we’re inconsistent
  2. If we sent the message before we persisted the document, but the call to session.SaveChangesAsync() failed, we’re inconsistent
  3. The system magically fails and shuts down in between the document getting saved and the outgoing message being completely enqueued — and that’s not that crazy if the system handles a lot of messages

We’ve got a couple options. We can try to use a distributed transaction between the underlying RabbitMQ queue and the Postgresql database, but those can be problematic and are definitely not super performant. We could also use some kind of compensating transaction to reestablish consistency, but that’s just more code to write.

Instead, let’s use Jasper’s support for the “outbox” pattern with Marten:

public async Task<IActionResult> CreateItem(
    [FromBody] CreateItemCommand command,
    [FromServices] IDocumentStore martenStore,
    [FromServices] IMessageContext context)
{
    var item = createItem(command);
    
    using (var session = martenStore.LightweightSession())
    {
        // Directs the message context to hold onto
        // outgoing messages, and persist them 
        // as part of the given Marten document
        // session when it is committed
        await context.EnlistInTransaction(session);
        
        var outgoing = new ItemCreated{Id = item.Id};
        await context.Send(outgoing);
        
        session.Store(item);
        
        await session.SaveChangesAsync();
    }

    return Ok();
}

The key things to know here are:

  • The outgoing messages are persisted in the same Postgresql database as the Item document with a native database transaction.
  • The outgoing messages are not sent to RabbitMQ until the underlying database transaction in the call to session.SaveChangesAsync() succeeds
  • For the sake of performance, the message persistence goes up to Postgresql with all the document operations in one network round trip to the database for just a wee bit of performance optimization.

For more context, here’s a sequence diagram explaining how it works under the covers using Marten’s IDocumentSessionListener:

Handling a Message w_ Unit of Work Middleware (1)

So now, let’s talk about all the things that can go wrong and how the outbox usage makes it better:

  • The transaction fails. No messages will be sent out, so there’s no inconsistency.
  • The transaction succeeds, but the RabbitMQ broker is unreachable. It’s still okay. Jasper has the outgoing messages persisted, and the durable messaging support will continue to retry the outgoing messages when the broker is available again.
  • The transaction succeeds, but the application process is killed before the outgoing message is completely sent to RabbitMQ. Same as the bullet point above.

 

Outbox Pattern inside of Message Handlers

The outbox usage within a message handler for the same CreateItemCommand in its most explicit form might look like this:

public static async Task Handle(
    CreateItemCommand command, 
    IDocumentStore store, 
    IMessageContext context)
{
    var item = createItem(command);


    using (var session = store.LightweightSession())
    {
        await context.EnlistInTransaction(session);

        var outgoing = new ItemCreated{Id = item.Id};
        await context.Send(outgoing);

        session.Store(item);

        await session.SaveChangesAsync();
    }
}

Hopefully, that’s not terrible, but we can drastically simplify this code if you don’t mind some degree of “magic” using Jasper’s cascading message support and Marten transaction middleware:

[MartenTransaction]
public static ItemCreated Handle(
    CreateItemCommand command,
    IDocumentSession session)
{
    var item = createItem(command);

    session.Store(item);
    return new ItemCreated{Id = item.Id};
}

The usage of the [MartenTransaction] attribute directs Jasper to apply a transaction against the IDocumentSession usage and automatically enlists the IMessageContext for the message in that session. The outgoing ItemCreated message returned from the action is sent out through the same IMessageContext object.

 

Advertisements

Jasper Command Line App Support you Wish Your Framework Already Had

Jasper is a new messaging and command runner framework targeting Netstandard2 my shop has been building as a replacement for part of the old FubuMVC framework. I wrote about the general vision and rationale here.

Earlier today I made a v0.7.0 release of Jasper and its related extensions. The pace of development has kicked back up because we’re getting ready to start doing load and chaos testing with our QA folks later this week and we’re already transitioning some smaller, low volume systems to Jasper. The highlights this time are:

  • A lot of optimization for the “cold start” time, especially if you’re using Jasper in combination with ASP.Net Core. I collapsed the ASP.Net Core support back to the core library, so this post is already obsolete.
  • The integration with ASP.Net Core is a lot tighter. For example, Jasper is now using the ASP.Net Core logging under its covers, the ASP.Net Core IHostedService, and just generally plays nicer when used in combination with ASP.Net Core applications.
  • Jasper now has some support for stateful sagas, but only with Marten-backed persistence. I’ll blog about this one soon, and there will be other saga persistence options coming fairly soon. Sql Server backed persistence at a bare minimum.
  • Finer grained control over how certain message types are published
  • Mild improvements to the Marten integration. Again, Jasper isn’t hard coupled to Marten and Postgresql, but it’s just been easy to prove out concepts with Marten first.
  • More command line usages that I’m showing in the rest of this post;)

Command Line Integration

First off, let’s say that you have a simple Jasper application that listens for incoming messages at a designated port configured with this class:

public class SubscriberApp : JasperRegistry
{
    public SubscriberApp()
    {
        // Listen for incoming messages via the
        // built in, socket transport in a 
        // fire and forget way at port 2222
        Transports.LightweightListenerAt(2222);
    }
}

To run your Jasper application as a console application, you can use the Jasper.CommandLine library as a quick helper that also adds some diagnostic commands you may find helpful during both development and deployment time. Using your SubscriberApp class above, you can bootstrap your application in a console application like this:

class Program
{
    static int Main(string[] args)
    {
        return JasperAgent.Run(args);
    }
}

Once that’s done, you can immediately run your application from the command line with dotnet run, which would give you some output like this:

Running service 'SubscriberApp'
Application Assembly: Subscriber, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
Hosting environment: Production
Content root path: [the IHostedEnvironment.ContentRootPath value]
Hosted Service: Jasper.Messaging.MessagingActivator
Hosted Service: Jasper.Messaging.NodeRegistration
Listening for loopback messages
Listening for messages at [url]/messages
Listening for messages at [url]/messages/durable

Active sending agent to loopback://replies/
Active sending agent to loopback://retries/
Handles messages:
            [Message Type]: [Handler Type and Handler Method Name]

Now listening on: [listener Uri]
Application started. Press Ctrl+C to shut down.

Other than a little bit of contextual information, it’s about what you would get with the ASP.Net Core command line support. If you’re not familiar with the dotnet cli, you can pass command line arguments to your Program.Main() ​method by using double dashes to separate arguments that apply to dotnet run from the arguments that get passed into your main method. Using the Oakton library for parsing Linux style command arguments and flags, your Jasper application can also respond to other commands and optional flags.

Knowing all that, this:

dotnet run -- -v

or

dotnet run -- --verbose

will run your application with console and debug loggers, and set the minimum log level in the ASP.Net Core logging to “Debug.”

Alternatively, you can also override the log level by:

dotnet run -- --log-level Information

or

dotnet run -- -l Trace

where the value is one of the values in the LogLevel enumeration.

To override the environment your application is running under, you can use this flag:

dotnet run -- --environment Development

or use the “-e” short version of that.

So what, what else do you got?

You can run a Jasper application, but there’s actually quite a bit more. If you type dotnet run -- ?, you can see the other available commands:

 

Screen Shot 2018-04-11 at 3.53.09 PM

The “export-json-schema” and “generate-message-types” commands are from an extension library that allows you to export JSON schema documents for the known message types or generate C# classes with the necessary Jasper message type identity from JSON schema documents. The command line support is extensible, allowing you to add prepackaged commands from addon Nugets or even be exposed from your own application. I’m going to leave that to a later post or *gasp*, updated documentation.

Preview the Generated Code

If you read my earlier post on Jasper’s Roslyn-Powered “Special Sauce,” you know that Jasper internally generates and compiles glue code to handle messages or HTTP requests. To help troubleshoot applications or just to understand the interplay between message handlers and any configured middleware, you can use this command to either list out the generated code or export it to a file:

dotnet run -- code -f export.cs

 

Check out the IoC Container Configuration

As a long time IoC tool author and user, I’m painfully aware that people run into issues with service registrations being incorrect or using erroneous lifecycles. To help ease those issues, Jasper allows you to see the service registrations of your entire application with this command:

dotnet run -- services

This is just displaying the output of the Lamar WhatDoIHave() report, similar to StructureMap’s WhatDoIHave() functionality.

Validate the System

As part of deployment or maybe even local development, you can choose to just start up the application, run all the registered environment checks, and verify that all the known message handlers and HTTP routes can be successfully compiled — with oodles of ugly red textual output if any check along the way fails. That’s done with dotnet run -- validate.

 

Manage Subscriptions

It’s admittedly kind of boring and I’m running out of time before I need to head home, but there is a dotnet run -- subscriptions command that you can use to manage message subscriptions at deployment or build time that’s documented here.

 

Next up:

I’ll get a decent, business facing example of Jasper’s stateful saga support.

 

Integrating Jasper into ASP.Net Core

Continuing a blog series on Jasper functionality:

  1. Jasper’s Configuration Story 
  2. Jasper’s Extension Model
  3. Integrating Marten into Jasper Applications
  4. Durable Messaging in Jasper 
  5. Integrating Jasper into ASP.Net Core (this one)
  6. Jasper’s HTTP Transport
  7. Jasper’s “Outbox” Support within ASP.Net Core Applications

There will be some need for completely headless services written with Jasper that rely strictly on TCP connections or yet to come queueing transports, but I expect that most of the systems at work where we’ll use Jasper will be within ASP.Net Core applications.

Moreover, as a nasty lesson learned from my hubristic attempts at creating a freestanding development ecosystem with FubuMVC, Jasper is meant to be merely a good citizen within the greater server side ASP.Net Core ecosystem. In regards to this blog post, that means using as much of the standard Hosting model as possible. For example, Jasper supports the IHostedService model from ASP.Net Core out of the box for long running background services or startup and shutdown actions.

As of Jasper 0.6, I pulled the HTTP support and ASP.Net Core integration into a separate Jasper.Http Nuget. This might feel like the tail wagging the dog, but I really only did this to optimize the core Jasper testing suite because bootstrapping ASP.Net Core on every integration test was slowing the automated build down too much. If I can find a way to optimize or at least parallelize much more of the bootstrapping with the messaging, I will consider merging things back together again later.

When Jasper is integrated into an ASP.Net Core system, it:

  • Adds more service registrations to the application
  • Bootstraps the JasperRuntime object and places that within the container so that the Jasper transports will be cleanly shut down when the IWebHost is disposed
  • Replaces the built in DI container with Lamar (Jasper only works with Lamar at this point)
  • Jasper also sneaks in some ASP.Net Core middleware to add its own routes into the application, which I’ll show off in the next post about Jasper’s HTTP messaging transport

All of this is documented in the Jasper Getting Started page and in the specific documentation for ASP.Net Core integration.

Longer term, I might try to move Jasper closer to the existing ASP.Net Core bootstrapping mechanisms.

Bootstrapping ASP.Net Core the Idiomatic Jasper Way

The first option is really about adding HTTP support to an idiomatic Jasper application. In this case, you just use the JasperHttpRegistry from the Jasper.Http library as the base class for your application definition like so:

public class AppWithMiddleware : JasperHttpRegistry
{
    public AppWithMiddleware()
    {
        // Do the normal stuff you do to configure
        // service registrations, configuration, and
        // messaging support

        Http.Configure(app =>
        {
            app.UseMiddleware<CustomMiddleware>();

            // Explicitly control the order in which the Jasper
            // middleware is placed within the ASP.Net Core
            // pipeline. 
            app.AddJasper();

            // Just to show how you can configure ASP.Net Core
            // middleware that runs after Jasper's RequestDelegate,
            // but do note that Jasper has its own default "not found"
            // behavior
            app.Run(c =>
            {
                c.Response.StatusCode = 404;

                return c.Response.WriteAsync("Not found");
            });
        });
    }
}

A couple things to note:

  • The Http property in the class shown above is just the IWebHostBuilder interface you’re already used to if you use ASP.Net Core today
  • If the call to IApplicationBuilder.AddJasper() is omitted, Jasper will add its own middleware to the very end of the pipeline
  • The HTTP bootstrapping in the idiomatic model is somewhat parallelized with the messaging support bootstrapping
  • I’d argue that this usage makes the ASP.Net Core StartUp conventional configuration model unnecessary, but you’re perfectly able to continue using that if you want.

I hope to do more optimizations to the cold startup time in the future for the idiomatic Jasper approach that would make this option be more attractive. Right now, the biggest reason to use this approach over the following is to be able to use Jasper’s console application harness and Storyteller integration.

 

Adding Jasper to an Existing ASP.Net Core System

You can also add Jasper to an existing ASP.Net Core system using its idiomatic bootstrapping approach. In this case, you still start with the JasperHttpRegistry base class from the Jasper.Http library, but you mostly use this to configure the messaging support:

public class SimpleJasperBusApp : JasperHttpRegistry
{
    public SimpleJasperBusApp()
    {
        // Enable the HTTP messaging transport
        Http.Transport.EnableListening(true);
        
        // Listen for TCP messages at port 2222
        Transports.LightweightListenerAt(2222);
    }
}

Then, to add the Jasper support to your ASP.Net Core application, you would add these calls:

var builder = new WebHostBuilder();
builder
    .UseKestrel()
    .UseUrls("http://localhost:3003")
    .UseStartup<Startup>()
    
    // This *has* to be the last call 
    // to your IWebHostBuilder
    .UseJasper<SimpleJasperBusApp>();


theHost = builder.Build();

theHost.Start();

I hate this from a usability perspective, but for right now, the call to UseJasper() has to be added after any other IStartUp registration including the UseStartup<T>() method. You still have the same ability to explicitly control the order of the Jasper middleware within your ASP.Net Core middleware pipeline.

 

 

 

Jasper v0.6: Better outbox usage, Lamar, ASP.Net Core integration changes

I’ve been a little distracted with the Lamar/BlueMilk work and Marten bug-fixing, but Jasper is still rolling along and about to get into production at work (a super early prototype is running in a low volume system now). I just pushed Jasper 0.6 to Nuget with some new improvements. The documentation has been updated and reflects the new changes described below.

Big changes:

  • Jasper uses Lamar for all IoC usage. I’m still very confident that the Jasper + Lamar (was “BlueMilk”) combination will lead to a very effective combination of flexibility and performance in the runtime pipeline. Lamar natively supports all the ASP.Net Core DI abstractions, so if you don’t care about any of its advanced features you don’t even need to care that it exists.
  • I broke the ASP.Net Core integration out into its own library, Jasper.Http. SeeAdding Jasper to an ASP.Net Core Application in the docs for the details.

    We’ve gone back and forth on whether Jasper is going to be modular or an easier to work with single library, but this one came down to the ASP.Net Core bootstrapping being somewhat expensive and making the main Jasper test suite be unnecessarily slow, so out it goes. I don’t think you’re going to notice the hit if all you do is bootstrap a single application in a test suite, but you sure do if you’re developing on Jasper itself and bootstrap and tear down 100+ applications during the integration tests;-)

  • The “outbox pattern” support was extended to cascading messages. This was an overdue improvement over its FubuMVC/FubuTransportation/RhinoServiceBus ancestors. I’ll have a blog post about this next week after every one is back from the MVP Summit.
  • IServiceBus was renamed to IMessageContext. This one will make a lot more sense with some documentation or a blog post on the outbox pattern work.
  • (Hopefully) Easier messaging support configuration. I tried hard to simplify the API underneath

Next Up…

Jasper is getting put into a production application at work within the month, and we’re doing some significant proof of concept work around using Consul for service discovery with Jasper applications and building out Octopus deployment steps for the dynamic subscriptions. Next week I’ll get back to blogging about Jasper’s integration with ASP.Net Core applications, the new HTTP transport option, and Jasper’s support for the “outbox” pattern.

Renaming BlueMilk to Lamar

BlueMilk was the early working name of a successor project to StructureMap that was originally ripped out of the new Jasper framework project. 

Most of the feedback on the name “BlueMilk” wasn’t positive. I wasn’t terribly attached to the name, so I’m officially renaming “BlueMilk” to “Lamar.” The first Nuget (v0.8) is published with the very latest work. I might be able to throw an OSS Friday sometime this week at finishing the remaining StructureMap features inside of Lamar and publishing some documentation, but we’ll see.

First, what the heck is Lamar? For most of you, it’s my intended successor to the venerable, well liked (by at least some people), and unfortunately slowpoke StructureMap library. It’s also the runtime code generation and compilation subsystem I pulled out of Jasper so folks could use that independently of Jasper.

For the moment, you can find way more information about Lamar under the “BlueMilk” tag on my blog.

What’s with the new name?

Other than Marten, most of my OSS efforts the past 2-3 years have really been working toward the Jasper framework we’re brewing up at work as a successor to FubuMVC. “Jasper” itself is just named after my ancestral hometown (Jasper, MO), and most of the other projects on the JasperFx organization are named after either other little towns around Jasper (Oakton, Alba) or local landmarks (Baseline). Fitting into that theme, Lamar is the next town up highway 71 and I’ve got plenty of family roots there as well.

Other notes that may only interest me:

  • Mirabeau B. Lamar was the 2nd president of the Republic of Texas, a hero of the TX revolution, and worlds of things in Texas are named after him. I didn’t know this until researching this post, but Lamar, MO is apparently named after him as well.
  • My wife is *this* close to completing a master’s program at Lamar University and I’m super proud of her
  • Lamar the town is the birthplace of Harry S. Truman
  • It was raided during the Civil War by Quantrill’s Raiders (think Jesse James)
  • Wyatt Earp was their first constable
  • I’m biased, but the Barton County Fair in Lamar may be the best small town fair in the entire state
  • If you’re a Modern Family friend, the Cameron character is supposedly from this area

BlueMilk is Ready for Early Adopters

This was renamed “Lamar” because the feedback on the name was, um, not good:)

 

EDIT 2/14/2018: And this already brought out a bug if you have a type that would need a closed generic type as an argument to its constructor. 0.7.1 will follow very shortly on Nuget.

 

BlueMilk is the name of a new OSS Inversion of Control tool I’m building specifically for usage in Jasper applications, but also as a higher performant replacement for StructureMap in Netstandard 2.0 applications going forward. To read more about what is genuinely unique about its internals and approach, see Jasper’s Roslyn-Powered “Special Sauce.”

I’m declaring BlueMilk 0.7 on Nuget right now as ready for enterprising, early adopter types to try out either on its own or within an ASP.Net Core application. At this point it’s passing all the ASP.Net Core compliance tests with a couple exceptions that I can’t possibly imagine being important in many cases (like the order in which created objects are disposed and a really strange way they order objects in a list when there’s mixed open and closed generic type registrations). It’s also supporting quite a few StructureMap features that I missed while trying to work with the built in DI container.

As I said in the introductory post, you can use BlueMilk as either a drop in replacement for the built in ASP.Net Core DI tool or as a faster subset of StructureMap for a much more richer feature set.

Current Feature Set

In most cases, the BlueMilk feature and API is identical to StructureMap’s and I’ll have to send you to the StructureMap documentation for more explanation.

Caveats

  • I’m wrestling with the 1st usage, warm up time just due to how long it takes Roslyn to bootstrap itself on its very first usage. What I’ve come up with so far is to have the dynamic classes for services registered as singletons or scoped be built on the initial startup, but allowing any other resolvers be built lazily the first time they are actually used. This is an ongoing struggle.
  • The lifecycle scoping is different than idiomatic StructureMap. I opted to give up and just use the ASP.Net team’s new definition for what “transient” and “scoped” means.
  • There is a dependency for the moment on a library called Baseline, that’s just my junk drawer of convenience extension methods (leftovers from FubuCore for anyone that used to follow FubuMVC). Before BlueMilk hits 1.0, I’ll internalize those extension methods somehow and eliminate that dependency

Using within ASP.Net Core Applications

You’ll want to pull down the BlueMilk.Microsoft.DependencyInjection package to get the ASP.Net Core bootstrapping shim — and blame the ASP.Net team for the fugly naming convention.

In code, plugging in BlueMilk is done through the UseBlueMilk() extension method as shown below:

var builder = new WebHostBuilder();
builder
    .UseBlueMilk()
    .UseUrls("http://localhost:5002")
    .UseKestrel()
    .UseStartup();

Pretty standard ASP.Net Core stuff. Using their magical conventions on the Startup class, you can do specific BlueMilk registrations using a ConfigureContainer(ServiceRegistry) method as shown below:

public class Startup
{
    public void ConfigureContainer(ServiceRegistry services)
    {
        // BlueMilk supports the ASP.Net Core DI
        // abstractions for registering services
        services.AddLogging();

        // And also supports quite a few of the old 
        // StructureMap features like type scanning
        services.Scan(x =>
        {
            x.AssemblyContainingType<SomeMarkerType>();
            x.WithDefaultConventions();
        });

     }

 // Other stuff we don't care about here
}

Durable Messaging in Jasper

My colleague Mike Schenk had quite a bit of input and contributions to this work. This continues a series of blog posts just trying to build up to the integration of durable messaging with ASP.Net Core:

  1. Jasper’s Configuration Story 
  2. Jasper’s Extension Model
  3. Integrating Marten into Jasper Applications
  4. Durable Messaging in Jasper (this one)
  5. Integrating Jasper into ASP.Net Core Applications
  6. Jasper’s HTTP Transport
  7. Jasper’s “Outbox” Support within ASP.Net Core Applications

 

Right now (0.5.0), Jasper offers two built in message transports using either raw TCP socket connections or HTTP with custom ASP.Net Core middleware. Either transport can be used in one of two ways:

  1. Fire and Forget” — Fast, but not backed by any kind of durable message storage, so there’s no guaranteed delivery.
  2. Store and Forward” — Slower, but make damn sure that any message sent is successfully received by the downstream service even in the face of system failures.

Somewhere down the line we’ll support more transport options like RabbitMQ or Azure Service Bus, but for right now, one of the primary design goals of Jasper is to be able to effectively do reliable messaging with the infrastructure you already have. In this case, the “store” part of durable messaging is going to be the primary, backing database of the application. Our proposed architecture at work would then look something like this:

Slide1

So potentially, we have multiple running instances (“nodes”) of each service behind some kind of load balancer (F5 in our shop), with each service having its own dedicated database.

For the moment, we’ve been concentrating on using Postgresql through Marten to prove out the durable messaging concept inside of Jasper, with Sql Server backing to follow shortly. To opt into that persistence, add the MartenBackedExtension extension from the Jasper.Marten library like this:

public class PostgresBackedApp : JasperRegistry
{
    publicPostgresBackedApp()
    {
        Settings.ConfigureMarten(_ =>
        {
            _.Connection("some connection string");
        });

        // Listen for messages durably at port 2301
        Transports.DurableListenerAt(2301);

        // This opts into Marten/Postgresql backed
        // message persistence with the retry and
        // message recovery agents
        Include<MartenBackedPersistence>();
     }
}

What this does is add some new database tables to your Marten configuration and directs Jasper to use Marten to persist incoming and outgoing messages before they are successfully processed or sent. If you end up looking through the code, it uses custom storage operations in Marten for better performance than the out of the box Marten document storage.

Before talking about all the ways that Jasper tries to make the durable messaging as reliable as possible by backstopping error conditions, let’s talk about what actually happens when you publish a message.

What Happens when You Send a Message?

When you send a message through the IServiceBus interface  or through cascading messages, Jasper doesn’t just stop and send the actual message. If you’re publishing a message that is routed by a durable channel, calling this code:

public async Task SendPing(IServiceBus bus)
{
    // Publish a message
    await bus.Send(new PingMessage());
}

will result in an internal workflow shown in this somewhat simplified sequence diagram of the internals:

Handling a Message w_ Unit of Work Middleware

The very first thing that happens is that each outgoing copy of the message is persisted to the durable storage, which is Postgresql in this post. Next, Jasper batches outgoing messages in a similar way to the debounce operator in Rx by outgoing Uri. If you’re using either the built in TCP or HTTP transports, the next step is to send a batch of messages to the receiving application. The receiving application in turn first persists the incoming messages in its persistence, and sends back an acknowledgement to the original sending application. Once the acknowledgement is received, the sending application will delete the outgoing messages just sent successfully from its message persistence and go on its merry way.

That’s the “store and forward” happy path. Now let’s talk about all the myriad ways things could go wrong and what Jasper tries to do to ensure that your messages get to where they are supposed to go.

Network Hiccups

How’s that for a scientific term? Sending message batches will occasionally fail due to the normal litany of temporary network issues. If an outgoing message batch fails, all the messages get their “sent attempts” count incremented in storage and they are added back into the local, outgoing sending agent queue to be tried again.

Circuit Breaker

Jasper’s sending agents implement a form of the Circuit Breaker pattern where a certain number of consecutive failures (it’s configurable) to send a message batch to any destination will cause Jasper to latch that sending agent. When the sending agent is latched, Jasper will not make any attempts to send messages to that destination. Instead, all outgoing messages to that destination will simply be persisted to the backing message persistence without any node ownership. The key point here is that Jasper won’t keep trying to send messages just to get runtime exceptions and it won’t allow the memory usage of your system to blow up from all the backed up, outgoing messages being in an in memory sending queue.

When the destination is known to be in a failure condition, Jasper will continue to poll the destination with a lightweight ping message just to ascertain if the destination is back up yet. When a successful ping is acknowledged by the destination, Jasper will unlatch the sending agent and begin sending outgoing messages.

Resiliency and Node Failover

If you are using the Marten/Postgresql backed message persistence, your application has a constantly running message persistence agent (it’s actually called SchedulingAgent) that is polling for persisted messages that are either owned by no specific node or persisted messages that are owned by an inactive node.

To detect whether a node is active, we rely on each node holding a session level advisory lock in the underlying Postgresql database as long as it’s really active. Periodically, Jasper will run a query to move any messages owned by an inactive node to “any node” ownership where any running node can recover both the outgoing and incoming messages. This query detects inactive nodes simply by the absence of an active advisory lock for the node identity in the database.

The message persistence agent also polls for the existence of any persisted incoming or outgoing messages that are not owned by any node. If it detects either, it will assign some of these messages to itself and pull outgoing messages into its local sending queues and incoming messages into its local worker queues to be processed. The message polling and fetching was designed to try to enable the recovery work to be spread across the running nodes. This process also uses Postgresql advisory locks as a distributed lock to prevent multiple running nodes from double dipping into the same persisted messages.

The end result of all that verbiage is:

  • If the receiving application is completely down, Jasper will be able to recover the outgoing messages and send them later when the receiving application is back up
  • If a node fails before it can send or process all the messages in flight, another node will be able to recover those persisted messages and process them
  • If your entire application goes down or is shut down, it will pick up the outstanding, persisted work of incoming and outgoing messages when any node is restarted
  • By using the advisory locks in the backing database, we got around having to have any kind of distributed lock mechanism (we were considering Consul) or leader election for the message recovery process, making our architecture here a lot simpler than it could have been otherwise.

 

More Work for the Future

The single biggest thing Jasper needs is early adopters and usage in real applications to know how what it already has for resiliency is working out. Beyond that though, I know we want at least a little more work in the built in transports for:

  1. Backpressure — We might need some kind of mechanism to allow the receiving applications let the senders know, “hey, I’m super busy here, could you stop sending me so many messages for a little bit” and slow down the sending.
  2. Work Stealing — We might say that its easier to implement back pressure between the listening agent and worker queues within the receiving application. In that case, if the listener sees there are too many outstanding messages waiting to be processed in the local worker queues, it would just persist the incoming messages for any other node to pick up when it can. We think this might be a cheap way to implement some form of work stealing.
  3. Diagnostics — We actually do have a working diagnostic package that adds a small website application to expose information about the running application. It could definitely use some additional work to expose metrics on the active message persistence.
  4. Sql Server backed message persistence — probably the next thing we need with Jasper at work

 

Other Related Stuff I Didn’t Get Into

  • We do have a dead letter queue mechanism where messages that just can’t be processed are shoved over to the side in the message persistence. All configurable of course
  • All the message recovery and batching thresholds are configurable. If you’re an advanced Jasper user, you could use those knobs to fine tune batch sizes and failure thresholds
  • It is possible to tell Jasper that a message expires at a certain time to prevent sending messages that are just too old