Jasper’s “Outbox” Pattern Support

Jasper supports the “outbox pattern,”  a way to achieve consistency between the outgoing messages that you send out as part of a logical unit of work without having to resort to two phase, distributed transactions between your application’s backing database and whatever queueing technology you might be using. Why do you care? Because consistency is good, and distributed transactions suck, that’s why.

Before you read this, and especially if you’re a coworker freaking out because you think I’m trying to force you to use Postgresql, Jasper is not directly coupled to Postgresql and we will shortly add similar support to what’s shown here for Sql Server message persistence with Dapper and possibly Entity Framework.

Let’s say you have an ASP.Net Core MVC controller action like this in a system that is using Marten for document persistence:

public async Task<IActionResult> CreateItem(
    [FromBody] CreateItemCommand command,
    [FromServices] IDocumentStore martenStore,
    [FromServices] IMessageContext context)
{
    var item = createItem(command);

    using (var session = martenStore.LightweightSession())
    {
        session.Store(item);
        await session.SaveChangesAsync();
    }
    
    var outgoing = new ItemCreated{Id = item.Id};
    
    await context.Send(outgoing);

    return Ok();
}

It’s all contrived, but it’s a relatively common pattern. The HTTP action:

  1. Receives a CreateItemCommand message from the client
  2. Creates a new Item document and persists that with a Marten document session
  3. Broadcasts an ItemCreated event to any known subscribers through Jasper’s IMessageContext service. For the sake of the example, let’s say that under the covers Jasper is publishing the message through RabbitMQ (because I just happened to push Jasper’s RabbitMQ support today).

Let’s say that in this case we need both the document persistence and the message being sent out to either succeed together or both fail together to keep your system and any downstream subscribers consistent. Now, let’s think about all the ways things can go wrong:

  1. If we keep the code the way that it is, what if the transaction succeeds, but the call to context.Send() fails, so we’re inconsistent
  2. If we sent the message before we persisted the document, but the call to session.SaveChangesAsync() failed, we’re inconsistent
  3. The system magically fails and shuts down in between the document getting saved and the outgoing message being completely enqueued — and that’s not that crazy if the system handles a lot of messages

We’ve got a couple options. We can try to use a distributed transaction between the underlying RabbitMQ queue and the Postgresql database, but those can be problematic and are definitely not super performant. We could also use some kind of compensating transaction to reestablish consistency, but that’s just more code to write.

Instead, let’s use Jasper’s support for the “outbox” pattern with Marten:

public async Task<IActionResult> CreateItem(
    [FromBody] CreateItemCommand command,
    [FromServices] IDocumentStore martenStore,
    [FromServices] IMessageContext context)
{
    var item = createItem(command);
    
    using (var session = martenStore.LightweightSession())
    {
        // Directs the message context to hold onto
        // outgoing messages, and persist them 
        // as part of the given Marten document
        // session when it is committed
        await context.EnlistInTransaction(session);
        
        var outgoing = new ItemCreated{Id = item.Id};
        await context.Send(outgoing);
        
        session.Store(item);
        
        await session.SaveChangesAsync();
    }

    return Ok();
}

The key things to know here are:

  • The outgoing messages are persisted in the same Postgresql database as the Item document with a native database transaction.
  • The outgoing messages are not sent to RabbitMQ until the underlying database transaction in the call to session.SaveChangesAsync() succeeds
  • For the sake of performance, the message persistence goes up to Postgresql with all the document operations in one network round trip to the database for just a wee bit of performance optimization.

For more context, here’s a sequence diagram explaining how it works under the covers using Marten’s IDocumentSessionListener:

Handling a Message w_ Unit of Work Middleware (1)

So now, let’s talk about all the things that can go wrong and how the outbox usage makes it better:

  • The transaction fails. No messages will be sent out, so there’s no inconsistency.
  • The transaction succeeds, but the RabbitMQ broker is unreachable. It’s still okay. Jasper has the outgoing messages persisted, and the durable messaging support will continue to retry the outgoing messages when the broker is available again.
  • The transaction succeeds, but the application process is killed before the outgoing message is completely sent to RabbitMQ. Same as the bullet point above.

 

Outbox Pattern inside of Message Handlers

The outbox usage within a message handler for the same CreateItemCommand in its most explicit form might look like this:

public static async Task Handle(
    CreateItemCommand command, 
    IDocumentStore store, 
    IMessageContext context)
{
    var item = createItem(command);


    using (var session = store.LightweightSession())
    {
        await context.EnlistInTransaction(session);

        var outgoing = new ItemCreated{Id = item.Id};
        await context.Send(outgoing);

        session.Store(item);

        await session.SaveChangesAsync();
    }
}

Hopefully, that’s not terrible, but we can drastically simplify this code if you don’t mind some degree of “magic” using Jasper’s cascading message support and Marten transaction middleware:

[MartenTransaction]
public static ItemCreated Handle(
    CreateItemCommand command,
    IDocumentSession session)
{
    var item = createItem(command);

    session.Store(item);
    return new ItemCreated{Id = item.Id};
}

The usage of the [MartenTransaction] attribute directs Jasper to apply a transaction against the IDocumentSession usage and automatically enlists the IMessageContext for the message in that session. The outgoing ItemCreated message returned from the action is sent out through the same IMessageContext object.

 

Advertisements

3 thoughts on “Jasper’s “Outbox” Pattern Support

    1. jeremydmiller Post author

      The outbox has the outgoing messages in memory. When it’s flushed, the envelopes go into a TPL ActionBlock that does the actual outbound sending loop. From there, it’s complicated. There are oodles of try/catch/retry logic locally trying to send the message outbound, and the message is not deleted from the DB until it’s successfully acknowledged as “sent” by the outbound transport. If the outbound circuit breaker has tripped off, the messages will get marked in the DB as “any node can grab this one” where background processes in each node would be able to recover the message and kick it along when and if the circuit is restored — with yet more logic and distributed locks to assure that only one node owns the outbound message at any one time. And at every stage, the database persistence actions have their own retry loop. The whole thing falls over if the app database is unreachable of course.

      The proof will be in the “put it under heavy, chaotic production usage,” but at *this point* I can’t think of a plausible scenario where messages can fall through the cracks when you’re using durable messaging.

      Reply
  1. paul van bladel (@paulbladel)

    Jeremy,
    Very nice.
    I tried this with the SqlServer. So, a client sending msg to Rabbitmq, a server listening on Rabbitmq backed with the outbox pattern approach.
    When both parties are connected to rabbitmq, I get the happy flow scenario. Msg goes from client over msg queue to server and response is send back.
    When I disable RabbitMq (while both client and server are still running) and send then the msg from client, the msg is stored in sql server (in the jasper_outgoing_envelopes table). Very nice !
    Nonetheless, when I restart RabbitMq, nothing happens. The message from the jasper_outgoing_envelopes are only sent when I restart the client and server app. So, how can I activate the durable message support?

    Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s