Jasper’s “Outbox” Pattern Support

Jasper supports the “outbox pattern,”  a way to achieve consistency between the outgoing messages that you send out as part of a logical unit of work without having to resort to two phase, distributed transactions between your application’s backing database and whatever queueing technology you might be using. Why do you care? Because consistency is good, and distributed transactions suck, that’s why.

Before you read this, and especially if you’re a coworker freaking out because you think I’m trying to force you to use Postgresql, Jasper is not directly coupled to Postgresql and we will shortly add similar support to what’s shown here for Sql Server message persistence with Dapper and possibly Entity Framework.

Let’s say you have an ASP.Net Core MVC controller action like this in a system that is using Marten for document persistence:

public async Task<IActionResult> CreateItem(
    [FromBody] CreateItemCommand command,
    [FromServices] IDocumentStore martenStore,
    [FromServices] IMessageContext context)
{
    var item = createItem(command);

    using (var session = martenStore.LightweightSession())
    {
        session.Store(item);
        await session.SaveChangesAsync();
    }
    
    var outgoing = new ItemCreated{Id = item.Id};
    
    await context.Send(outgoing);

    return Ok();
}

It’s all contrived, but it’s a relatively common pattern. The HTTP action:

1. Receives a CreateItemCommand message from the client
2. Creates a new Item document and persists that with a Marten document session
3. Broadcasts an ItemCreated event to any known subscribers through Jasper’s IMessageContext service. For the sake of the example, let’s say that under the covers Jasper is publishing the message through RabbitMQ (because I just happened to push Jasper’s RabbitMQ support today).

Let’s say that in this case we need both the document persistence and the message being sent out to either succeed together or both fail together to keep your system and any downstream subscribers consistent. Now, let’s think about all the ways things can go wrong:

1. If we keep the code the way that it is, what if the transaction succeeds, but the call to context.Send() fails, so we’re inconsistent
2. If we sent the message before we persisted the document, but the call to session.SaveChangesAsync() failed, we’re inconsistent
3. The system magically fails and shuts down in between the document getting saved and the outgoing message being completely enqueued — and that’s not that crazy if the system handles a lot of messages

We’ve got a couple options. We can try to use a distributed transaction between the underlying RabbitMQ queue and the Postgresql database, but those can be problematic and are definitely not super performant. We could also use some kind of compensating transaction to reestablish consistency, but that’s just more code to write.

Instead, let’s use Jasper’s support for the “outbox” pattern with Marten:

public async Task<IActionResult> CreateItem(
    [FromBody] CreateItemCommand command,
    [FromServices] IDocumentStore martenStore,
    [FromServices] IMessageContext context)
{
    var item = createItem(command);
    
    using (var session = martenStore.LightweightSession())
    {
        // Directs the message context to hold onto
        // outgoing messages, and persist them 
        // as part of the given Marten document
        // session when it is committed
        await context.EnlistInTransaction(session);
        
        var outgoing = new ItemCreated{Id = item.Id};
        await context.Send(outgoing);
        
        session.Store(item);
        
        await session.SaveChangesAsync();
    }

    return Ok();
}

The key things to know here are:

• The outgoing messages are persisted in the same Postgresql database as the Item document with a native database transaction.
• The outgoing messages are not sent to RabbitMQ until the underlying database transaction in the call to session.SaveChangesAsync() succeeds
• For the sake of performance, the message persistence goes up to Postgresql with all the document operations in one network round trip to the database for just a wee bit of performance optimization.

For more context, here’s a sequence diagram explaining how it works under the covers using Marten’s IDocumentSessionListener:

So now, let’s talk about all the things that can go wrong and how the outbox usage makes it better:

• The transaction fails. No messages will be sent out, so there’s no inconsistency.
• The transaction succeeds, but the RabbitMQ broker is unreachable. It’s still okay. Jasper has the outgoing messages persisted, and the durable messaging support will continue to retry the outgoing messages when the broker is available again.
• The transaction succeeds, but the application process is killed before the outgoing message is completely sent to RabbitMQ. Same as the bullet point above.

 

Outbox Pattern inside of Message Handlers

The outbox usage within a message handler for the same CreateItemCommand in its most explicit form might look like this:

public static async Task Handle(
    CreateItemCommand command, 
    IDocumentStore store, 
    IMessageContext context)
{
    var item = createItem(command);


    using (var session = store.LightweightSession())
    {
        await context.EnlistInTransaction(session);

        var outgoing = new ItemCreated{Id = item.Id};
        await context.Send(outgoing);

        session.Store(item);

        await session.SaveChangesAsync();
    }
}

Hopefully, that’s not terrible, but we can drastically simplify this code if you don’t mind some degree of “magic” using Jasper’s cascading message support and Marten transaction middleware:

[MartenTransaction]
public static ItemCreated Handle(
    CreateItemCommand command,
    IDocumentSession session)
{
    var item = createItem(command);

    session.Store(item);
    return new ItemCreated{Id = item.Id};
}

The usage of the [MartenTransaction] attribute directs Jasper to apply a transaction against the IDocumentSession usage and automatically enlists the IMessageContext for the message in that session. The outgoing ItemCreated message returned from the action is sent out through the same IMessageContext object.