It’s the halfway point of 2025 some how, and we’ve now gotten past the big Marten 8.0 and Wolverine 4.0 releases. Right before I go on vacation next week, I thought it would be a good time to jot down some thoughts about where the Critter Stack might go for the rest of 2025 and probably into 2026.

Critter Watch

The big ticket item is our ongoing work on “Critter Watch”, which will be a commercial management and observability add on for Wolverine, Marten, and any future new Critter tools. The top line pitch for Critter Watch is that it well help you know what your applications are, how they interact with each other, whether they’re healthy in production, and provide features to help heal the inevitable production problems when they happen.

The general idea is to have a standalone application deployed that acts as a management console for 1 or more Wolverine applications in our user’s environments:

Upfront for the Critter Watch MVP (and requests from a client), we’re focused on:

• Visualizing the systems being monitored, their Wolverine and Marten configuration, and the capabilities of the systems. We’re currently researching AsyncAPI publishing and visualization as well. The whole point of this is to help teams understand how the messages in your system are handled, published, and routed.
• Event Sourcing management, but this is mostly about managing the execution of asynchronous projections and subscriptions at runtime and being able to understand the ongoing performance or any ongoing problems
• Dead letter queue management for Wolverine

I have less clarity over development time tooling, but we’re at least interested in having some of Critter Watch usable as an embedded tool during development.

After years of talking about this and quite a bit of envisioning, development started in earnest over the past 6 weeks with a stretch goal of having a pilot usage by the end of July for a JasperFx Software client.

I do not yet have any hard pricing numbers yet, but we are very interested in talking to anyone who would be interested in Critter Watch.

Concurrency, Concurrency, Concurrency!

I think that systems built with Event Sourcing are a little more sensitive to concurrent data reads and writes, or maybe it’s just that those problems are there all the time but more readily observable with Event Sourcing and Event Driven Architectures. In my work with JasperFx Software clients, concurrency is probably the most common subject of questions.

Mostly today you deal with this either by building in selective retry capabilities based on version conflict detection, or get fancier with queueing and message routing to eliminate the concurrent access as much as possible. Or both of course.

A great way to side step the concurrent access while not sacrificing throughput through parallelization is to use Wolverine’s support for Azure Service Bus Session Identifiers and FIFO Queues.

Which is great, but what if you’re not using Azure Service Bus? What if you’re only using local queueing? And wouldn’t it be nice if the existing Azure Service Bus FIFO support was a little less cumbersome to use in your code?

I don’t have a ton of detail, but there’s a range of internal proposals to create some new recipes for Wolverine usage to enable teams to more easily “shard” logical work between queues and within the local workers listening to queues to improve Wolverine’s handling of concurrent access without sacrificing parallel work and throughput or requiring repetitive code. Some of this is being done in collaboration with JasperFx clients.

Improving Wolverine’s Declarative Data Access

For lack of a better description, Wolverine has a feature set I’m heretofore calling “declarative data access” with the [Entity] attribute that triggers code generation within message handlers or HTTP endpoints to load requested data from Marten, EF Core, or RavenDb. And of course, there’s also what we call the “aggregate handler workflow” recipe for using the Decider pattern with Wolverine and Marten that I think is the simplest way to express business logic when using Event Sourcing in the .NET ecosystem.

To take these productivity features even farther, I think we’ll add some:

1. More control over what action to take if an entity is missing. Today, the HTTP endpoints will just return a 404 status code if required entities can’t be found. In future versions, we’ll let you customize log or ProblemDetails messages and have more control over how Wolverine generates the “if missing” path
2. At least for Marten, opt into Marten’s batch querying support if you are using more than one of any combination of the existing [Aggregate], [ReadAggregate], [Entity], or [Document] attributes to load data within a single HTTP endpoint or message handler as a way of improving performance by reducing network round trips to the database. And don’t sneeze at that, chattiness is a common performance killer in enterprise applications. Especially when the code is unnecessarily complicated by typical usages of Clean or Onion Architectural approaches.

If you follow Event Sourcing related topics online, you’ll hear quite a bit of buzz from some of the commercial tools about “Dynamic Consistency Boundaries” (DCB). We get asked about this with Marten occasionally, but the Marten core team’s position is that Marten doesn’t require this feature because you can already do “read” and “write” operations across multiple event streams with transactional integrity as is.

What the batch querying I just described will do for Marten though is make the full “Critter Stack” usage be more performant when you need to potentially work with more than one event stream at a time with all the transactional support and strong consistency that Marten (really PostgreSQL) already provides.

For Marten users, this is essentially making Marten’s FetchForWriting() API able to enroll in batch querying for more efficient data querying when working across streams. That work is actually well underway.

But if you prefer to use the fancier and more novel DCB approaches that aren’t even officially released yet, feel free to pay out some big bucks to use one of the commercial tools.

Smaller, But Still Important Work!

• Partially for Critter Watch, Wolverine should support connecting to multiple brokers in a single application for each transport type. Some of this is already done, with Kafka being next up, but we need to add this to every transport
• Improved interoperability support for Wolverine talking to non-Wolverine applications. There’s an existing pull request that goes quite a ways for this, but this might end up being more a documentation effort than anything else
• More options in Wolverine with Marten or just Marten for streaming Marten data as JSON directly to HTTP. We have some support already of course, but there are more opportunities for expanding that
• Exposing an MCP server off of Marten event data, but I have very little detail about what that would be. I would be very interested in partnering with a company who wanted to do this, and a JasperFx client might be working with us later this year on AI with Marten
• Improving throughput in Marten’s event projections and subscriptions. We’ve done a lot the past couple years, but there are still some other ideas in the backlog we haven’t played yet
• Expanding Wolverine support for more database engines, with CosmosDb the most likely contender this year. This might be contingent upon client work of course.

What about the SQL Server backed Event Store?

Yeah, I don’t know. We did a ton of work in Marten 8 to pull what will be common code out in a way that it could be reused in the SQL Server backed event store. I do not know when we might work on this as CritterWatch will take priority for now.

And finally….

And on that note I’m essentially on vacation for a week and I’ll catch up with folks in late July.

You’re about to start a new system with Event Sourcing using Marten, and you’re expecting your system to be hugely successful such that it’s going to handle a huge amount of data, but you’re already starting with pretty ambitious non-functional requirements for the system to be highly performant and all the screens or exposed APIs be snappy.

Basically, what you want to do is go as fast as Marten and PostgreSQL will allow. Fortunately, Marten has a series of switches and dials that can be configured to squeeze out more performance, but for a variety of historical reasons and possible drawbacks, are not the defaults for a barebones Marten configuration as shown below:

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddMarten(opts =>
{
    opts.Connection(builder.Configuration.GetConnectionString("marten"));
});

Cut me some slack in my car choice for the analogy here. I’m not only an American, but I’m an American from a rural area who grew up dreaming about having my own Mustang or Camaro because that’s as far out as I could possibly imagine back then.

At this point, we have is the equivalent to a street legal passenger car, maybe the equivalent to an off the shelf Mustang:

Which probably easily goes fast enough for every day usage for the mass majority of us most of the time. But we really need a fully tricked out Mustang GTD that’s absurdly optimized to just flat out go fast:

Let’s start trimming weight off our street legal Marten setup to go faster with…

Opt into Lightweight Sessions by Default

Starting from a new system so we don’t care about breaking existing code by changing behavior, let’s opt for lightweight sessions by default:

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddMarten(opts =>
{
    opts.Connection(builder.Configuration.GetConnectionString("marten"));
})
    
// Jettison some "Identity Map" weight by going lighter weight    
.UseLightweightSessions();

By default, the instances of IDocumentSession you get out of an IoC container would utilize the Identity Map feature to track loaded entities by id so that if you happened to try to load the same entity from the same session, you would get the exact same object. As I’m sure you can imagine, that means that every entity fetched by a session is stuffed into a dictionary internally (Marten uses the highly performant ImTools ImHashMap everywhere, but still), and the session also has to bounce through the dictionary before loading data as well. It’s just a little bit of overhead we can omit by opting for “Lightweight Sessions” if we don’t need that behavior by default.

We’ve always been afraid to change the default behavior here to the more efficient approach because it can absolutely lead to breaking existing code that depends on the Identity Map behavior. On the flip side, I think you should not need Identity Map mechanics if you can keep the call stacks within your code short enough that you can actually “see” where you might be trying to load the same data twice or more in the same parent operation.

On to the next thing…

Make Writes Faster with Quick Append

Next, since we again don’t have any existing code that can be broken here, let’s opt for “Quick Append” writes like so:

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddMarten(opts =>
{
    opts.Connection(builder.Configuration.GetConnectionString("marten"));

    // Make event writing faster, like 2X faster in our testing
    opts.Events.AppendMode = EventAppendMode.Quick;
})
    
// Jettison some "Identity Map" weight by going lighter weight    
.UseLightweightSessions();

This will help the system be able to append new events much faster, but at the cost of not being able to use some event metadata like event versions, sequence, or timestamp information within “Inline” projections.

Again, even though this option has been clocked as being much faster, we have not wanted to make this the default because it could break existing systems for people who depend on having the rich metadata during the Inline application of projections that forces Marten to do a kind of two step process to append events. This “Quick Append” option also helps reduce concurrent access problems writing to streams and generally makes the “Async Daemon” subsystem processing asynchronous projections and subscriptions run much smoother.

We’re not out of tricks yet by any means, so let’s go on…

Use the Identity Map for Inline Aggregates

Wait, I thought you told me not to cross the streams! Yeah, about the Identity Map thing, there’s one exception where we actually do want that behavior within CQRS command handlers like this one using Wolverine and its “Aggregate Handler Workflow” integration with Marten:

    // This tells Wolverine that the first "return value" is NOT the response
    // body
    [EmptyResponse]
    [WolverinePost("/api/incidents/{incidentId:guid}/category")]
    public static IncidentCategorised Post(
        // the actual command
        CategoriseIncident command, 
        
        // Wolverine is generating code to look up the Incident aggregate
        // data for the event stream with this id
        [Aggregate("incidentId")] Incident incident)
    {
        // This is a simple case where we're just appending a single event to
        // the stream.
        return new IncidentCategorised(incident.Id, command.Category, command.CategorisedBy);
    }

In the case above, the Incident model is a projected document that’s first used by the command handler to “decide” what new events to emit. If we’re updating the Incident model with an Inline projection that tries to update the Incident model in the database at the same time it wants to append events, then it’s an advantage for performance to “just” use the original Incident model we used initially, then forwarding the new state based on the new events and persisting the results right then and there. We can opt into this optimization even for the lightweight sessions we earlier wanted to use by adopting one more UseIdentityMapForAggregates flag:

builder.Services.AddMarten(opts =>
{
    opts.Connection(builder.Configuration.GetConnectionString("marten"));

    // Make event writing faster, like 2X faster in our testing
    opts.Events.AppendMode = EventAppendMode.Quick;

    // This can cut down on the number of database round trips
    // Marten has to do during CQRS command handler execution
    opts.Events.UseIdentityMapForAggregates = true;
})
    
// Jettison some "Identity Map" weight by going lighter weight    
.UseLightweightSessions();

Note, this optimization can easily break code for folks who use some sort of stateful “Aggregate Root” approach where the state of the projected aggregate object might be mutated during the course of executing the command. As this has traditionally been a popular approach in Event Sourcing circles, we can’t make this be a default option. If you instead either make the projected aggregates like Incident either immutable or treat them as a dumb data input to your command handlers with a more Functional Programming “Decider” function approach, you can get away with the performance optimization.

And also, I strongly prefer and recommend the FP “Decider” approach to JasperFx Software clients as is and I think that folks using the older “Aggregate Root” approach tend to have more runtime bugs.

Moving on, let’s keep our database smaller…

Event Stream Archiving

By and large, you can improve system performance in almost any situation by trying to keep your database from growing too large by archiving or retiring obsolete information. Marten has first class support for “Archiving Event Streams” where you effectively just move event streams that only represent historical information and are not really active into an archived state.

Moreover, we can divide our underlying PostgreSQL storage for events into “hot” and “cold” storage by utilizing PostgreSQL’s table partitioning support like this:

builder.Services.AddMarten(opts =>
{
    opts.Connection(builder.Configuration.GetConnectionString("marten"));

    // Make event writing faster, like 2X faster in our testing
    opts.Events.AppendMode = EventAppendMode.Quick;

    // This can cut down on the number of database round trips
    // Marten has to do during CQRS command handler execution
    opts.Events.UseIdentityMapForAggregates = true;

    // Let's leverage PostgreSQL table partitioning
    // to our advantage
    opts.Events.UseArchivedStreamPartitioning = true;
})
    
// Jettison some "Identity Map" weight by going lighter weight    
.UseLightweightSessions();

If you’re aggressive with marking event streams as Archived, the PostgreSQL table partitioning can move off archived event streams into a different table partition than our active event data. This is essentially keeping the “active” event table storage relatively stable in size, and most operations will execute against this smaller table partition while still being able to access the archived data too if explicitly opt into including that.

We added this feature in a minor point 7.* release, so it had to be opt in, and I think I was too hesitant to make this a default in 8.0, so it’s still “opt in”.

Stream Compacting

Beyond archiving event streams, maybe you just want to “compact” a longer event stream so you technically retain all the existing state, but further reduce the size of your active database storage. To that end, Marten 8.0 added Stream Compacting.

Distributing Asynchronous Projections

I had been mostly talking about using projections running Inline such that the projections are updated at the same time as the events are captured. That’s sometimes applicable or desirable, but other times you’ll want to optimize the “write” operations by moving the updating of projected data to an Async projection running in the background. But now let’s say that we have quite a few asynchronous projections and several subscriptions as well. In early versions of Marten, we had to run everything in a “Hot/Cold” mode where every known projection or subscription had to run on one single “leader” node. So even if you were running your application across a dozen or more nodes, only one could be executing all of the asynchronous projections and subscriptions.

That’s obviously a potential bottleneck, so Marten 7.0 by itself introduced some ability to spread projections and subscriptions over multiple nodes. If we introduce Wolverine into the mix though, we can do quite a bit better than that by allowing Wolverine to distribute the asynchronous Marten work across our entire cluster with its ability to distribute Marten projections and subscriptions with the UseWolverineManagedEventSubscriptionDistribution option in the WolverineFx.Marten Nuget:

builder.Services.AddMarten(opts =>
{
    opts.Connection(builder.Configuration.GetConnectionString("marten"));

    // Make event writing faster, like 2X faster in our testing
    opts.Events.AppendMode = EventAppendMode.Quick;

    // This can cut down on the number of database round trips
    // Marten has to do during CQRS command handler execution
    opts.Events.UseIdentityMapForAggregates = true;

    // Let's leverage PostgreSQL table partitioning
    // to our advantage
    opts.Events.UseArchivedStreamPartitioning = true;
})
    
// Jettison some "Identity Map" weight by going lighter weight    
.UseLightweightSessions()

.IntegrateWithWolverine(opts =>
{
    opts.UseWolverineManagedEventSubscriptionDistribution = true;
});

Is there anything else for the future?

It never ends, and yes, there are still quite a few ideas in our product backlog to potentially improve performance and scalability of Marten’s Event Sourcing. Offhand, that includes looking at alternative, higher performance serializers and more options to parallelize asynchronous projections to squeeze out more throughput by sharing some data access across projections.

Summary

There are quite a few “opt in” features in Marten that will help your system perform better, but these features are “opt in” because they can be harmful if you’re not building around the assumptions these features make about how your code works. The good news though is that you’ll be able to better utilize these features if you follow the Critter Stack’s recommended practices by striving for shorter code stacks (i.e., how many jumps between methods and classes does your code make when receiving a system input like a message or HTTP request) so your code is easier to reason about anyway, and avoiding mutating projected aggregate data outside of Marten.

Time for an update on Critter Stack release plans, and a follow up on my previous Critter Stack Work in Progress post from March. The current plan is to release Marten 8.0, Weasel 8.0, and Wolverine 4.0 on June 1st. It’s not going to be a huge release in terms of new functionality, but there are some important structural changes that will help us build some future features, and we needed to jettison older .NET versions while getting onto the latest Npgsql. “CritterWatch” is still very much planned and a little bit in progress, but we’ve got to get these big releases out first.

The key takeaways are that I want to essentially freeze Marten 7.* for everything but bug fixes right now, and probably freeze Wolverine 3.* for new feature development after a last wave of pull requests gets pulled in over the next couple days.

I’m admittedly too drowsy and lazy to write much tonight, so here’s just a dump of what I wrote up for the rest of our core team to review. I think we’re already at the point where we’re ready to just work on documentation and a few last touches, so the mass majority of this doesn’t get done in time, but here’s the full brain storm:

First though, what’s been done:

• .NET 6 & 7 were dropped
• Updated to Npgsql 9 across the board
• Dropped all synchronous APIs in Marten
• Deleted some [Obsolete] APIs in Marten
• Consolidation of supporting libraries to a single JasperFx library
• JasperFx has that new consolidated configuration option for common configuration like application assembly, code generation, and the stateful resource AutoCreate mode
• Pulled out event projections and core event store abstractions to a new JasperFx.Events library
  • Removed code generation from all projections
  • Better explicit code options for aggregations and event projections
• Wolverine 4 has better handles envelope storage & the transactional inbox/outbox for modular monoliths
• Improved “Descriptor” model to describe the static configuration of Wolverine and/or Marten applications that we’ll use for CritterWatch too
• Expanded commands for dead letter queue management in Wolverine that was meant for CritterWatch
• Multi-tenancy options in Wolverine for SQL Server or PostgreSQL w/o Marten, multi-tenancy usage with EF Core

Punchlist?

1. Marten 7.40.4 release w/ a pair of outstanding PRs
   1. Cherry pick commits to Marten “master”
2. JasperFx & JasperFx.Events 1.0
   1. Documentation website?
3. Weasel “master” branch
   1. All tests should be passing
4. Marten “master” branch
   1. All tests should be passing
   2. Documentation website should be building – that’s going to take some effort because of code samples
   3. Get Anne’s PR for tutorials in (cool new guided tour of building a system using Event Sourcing and Event Driven Architecture with first Marten, then Wolverine)
   4. Stream Compacting feature – for a JasperFx customer (this is definitely in for Marten 8, this is a big improvement for keeping a larger system running fast over time by compacting the database)
   5. Fix the optimized projection rebuild options? Or rip it out and leave it for CritterWatch?
   6. Ability to overwrite the event timestamp (relatively easy)
   7. Migration guide 
   8. Figure out what the proper behavior of “Live” aggregations when there’s some ShouldDelete() action going on
5. Wolverine
   1. One last 3.14 release with easy to grab pull requests and bug fixes
   2. Rebase on 3.14
   3. Fork off the 3.0 branch
   4. 4.0 becomes main branch
   5. All tests should be passing
   6. Documentation website should build
   7. Migration guide
6. Critter Watch preparation
   1. When integrated w/ CritterWatch, Wolverine can build the descriptor model for the entire application, including EventStoreUsage. No idea where this work stands right now. Did quite a bit earlier this year, then went off in a different direction
   2. Review all Open Telemetry usage and activity naming across Marten and especially Wolverine. Add Open Telemetry & Metrics metadata to the descriptor model sent to CritterWatch. I think this is somewhat likely to get done before Wolverine 4.0.
   3. Ability to send messages from CritterWatch to Wolverine. Might push through some kind of message routing and/or message handler extensibility

Nice to do?

1. Consider consolidating the stateful resource / AutoCreate configuration so there are fewer thing to configure. See Managing Auto Creation of Database or Message Broker Resources in the Critter Stack vNext
2. Programmatic message routing in Wolverine that varies based on the message contents? This is around options to route a message to one of a set of destinations based on the message core. Thinking about concurrency here. Could be done later.
3. More open issues in the Marten 8 milestone, but it’s about time to drop any issue that isn’t a breaking change
4. More open issues in the Wolverine 4 milestone or Wolverine in general
5. Ermine/Polecat readiness? (Marten ported to SQL Server)
   1. Spike it out? 
   2. Look for opportunities to pull shared items into Weasel?

As I wrote last week, message or request concurrency is probably the single most common source of client questions in JasperFx Software consulting and support work around the Critter Stack. Wolverine is a powerful tool for command and event message processing, and it comes with a lot of built in options for wide range of usage scenarios that provider the answers for a lot of the questions we routinely field from clients and other users. More specifically, Wolverine provides a lot of adjustable knobs to limit or expand:

• Message processing parallelism. I.e., how many messages can be executed simultaneously
• Message ordering when you need messages to be processed in sequence, or a lack thereof when you don’t
• Delivery guarantees ranging from fire and forget, at least once, exactly once, and at most once

For better or worse, Wolverine has built up quite a few options over the years, and that can be admittedly confusing. Also, there are real performance or correctness tradeoffs with the choices you make around message ordering and processing parallelism. To that end, let’s go through a little whirlwind tour of Wolverine’s options for concurrency, parallelism, and delivery guarantees.

Listener Endpoints

Note that Wolverine standardizes the fluent interface options for endpoint type, message ordering, and parallel execution are consistent across all of its messaging transport types (Rabbit MQ, Azure Service Bus, Kafka, Pulsar, etc.), though not every option is available for every transport.

All messages handled in a Wolverine application come from a constantly running listener “Endpoint” that then delegates the incoming messages to the right message handler. A Wolverine “Endpoint” could be a local, in process queue, a Rabbit MQ queue, a Kafka topic, or an Azure Service Bus subscription (see Wolverine’s documentation on asynchronous messaging for the entire list of messaging options).

This does vary a bit by messaging broker or transport, but there are three modes for Wolverine endpoints, starting with Inline endpoints:

// Configuring a Wolverine application to listen to
// an Azure Service Bus queue with the "Inline" mode
opts.ListenToAzureServiceBusQueue(queueName, q => q.Options.AutoDeleteOnIdle = 5.Minutes()).ProcessInline();

With an Inline endpoint, messages are pulled off the receiving queue or topic one message at a time, and “ack-ed” back to the original queue or topic only on the successful completion of the message handler. This mode completely eschews any kind of durable, transactional inbox, but does still give you an at-least-once delivery guarantee as it’s possible that the “ack” process could fail after the message is successfully handled, potentially resulting in the message being resent from the external messaging broker. Know though that this is rare, and Wolverine puts some error retries around the “ack-ing” process.

As you would assume, using the Inline mode gives you sequential processing of messages within a single node, but limits parallel handling. You can opt into running parallel listeners for any given listening endpoint:

opts.ListenToRabbitQueue("inline")
    // Process inline, default is with one listener
    .ProcessInline()

    // But, you can use multiple, parallel listeners
    .ListenerCount(5);

The second endpoint mode is Buffered where messages are pulled off the external messaging queue or topic as quickly as they can be, and immediately put into an in memory queue and “ack-ed” to any external broker.

// I overrode the buffering limits just to show
// that they exist for "back pressure"
opts.ListenToAzureServiceBusQueue("incoming")
    .BufferedInMemory(new BufferingLimits(1000, 200));

In the sample above, I’m showing how you can override the defaults for how many messages can be buffered in memory for this listening endpoint before the endpoint is paused. Wolverine has some support for back pressure within its Buffered or Durable endpoints to prevent memory from being overrun.

With Buffered or the Durable endpoints I’ll describe next, you can specify the maximum number of parallel messages that can be processed at one time within a listener endpoint on a single node like this:

opts.LocalQueueFor<Message1>()
    .MaximumParallelMessages(6, ProcessingOrder.UnOrdered);

Or you can choose to run messages in a strict sequential order, one at a time like this:

// Make any kind of Wolverine configuration
options
    .PublishMessage<Module1Message>()
    .ToLocalQueue("module1-high-priority")
    .Sequential();

The last endpoint type is Durable, which behaves identical to the Buffered approach except that messages received from external message brokers are persisted to a backing database first before processing, then deleted when the messages are successfully processed or discarded or moved to dead letter queues by error handling policies:

opts.ListenToAzureServiceBusQueue("incoming")
    .UseDurableInbox(new BufferingLimits(1000, 200));

Using the Durable mode enrolls the listening endpoint into Wolverine’s transactional inbox. This is the single most robust option for delivery guarantees with Wolverine, and even adds some protection for idempotent receipt of messages such that Wolverine will quietly reject the same message being received multiple times. Durable endpoints are more robust in terms of delivery guarantees and resilient in the face of system hiccups than the Buffered mode, but does incur a little bit of extra overhead making calls to a database — but I should mention that Wolverine is trying really hard to batch up calls to the database whenever it can for better runtime efficiency, and there are retry loops in all the internals for resiliency as well.

If you really read this post you should hopefully be badly abused of the flippant advice floating around .NET circles right now after the MassTransit commercialization announcement that you can “just” write your own abstractions over messaging brokers instead of using a robust, off the shelf toolset that will have far more engineering for resiliency and observability than most folks realize.

Scenarios

Alright, let’s talk about some common messaging scenarios and look at possible Wolverine options. It’s important to note that there is some real tension between throughput (how many messages can you process over time), message ordering requirements, and delivery guarantees and I’ll try to call those compromises as we go.

You have a constant flood of small messages coming in that are relatively cheap to process…

In this case I would choose a Buffered endpoint and allow it to run messages in parallel:

opts.LocalQueueFor<Message1>()
    .BufferedInMemory()
    .MaximumParallelMessages(6, ProcessingOrder.UnOrdered);

Letting messages run without any strict ordering will allow the endpoint to process messages faster. Using the Buffered approach will allow the endpoint to utilize any kind of message batching that external message brokers might support, and does a lot to remove the messaging broker as a bottle neck for message processing. The Buffered approach isn’t durable of course, but if you care about throughput more than guarantees or message ordering, it’s the best option.

Note that any Buffered or Durable endpoint automatically allows for parallel message processing capped by the number of processor cores for the host process.

A message is expensive to process…

If you have a message type that turns out to require a lot of resources to process, you probably want to limit the parallelization to restrict how many resources the system uses for this message type. I would say to either use an Inline endpoint:

opts.ListenToRabbitQueue("expensive")
    // Process inline, default is with one listener
    .ProcessInline()

    // Cap it to no more than two messages in parallel at any
    // one time
    .ListenerCount(2);

or a Buffered or Durable endpoint, but cap the parallelization.

Messages should be processed in order, at least on each node…

Use either a ProcessInline endpoint, or use the Sequential() option on any other kind of endpoint to limit the local processing to single file:

opts.ListenToAzureServiceBusQueue("incoming")
    .Sequential();

A certain type of message should be processed in order across the entire application…

Sometimes there’s a need to say that a certain set of messages within your system need to be handled in strict order across the entire application. While some specific messaging brokers have some specific functionality for this scenario, Wolverine has this option to ensure that a listening endpoint for a certain location only runs on a single node within the application at any one time, and always processes in strict sequential order:

var host = await Host.CreateDefaultBuilder().UseWolverine(opts =>
{
    opts.UseRabbitMq().EnableWolverineControlQueues();
    opts.PersistMessagesWithPostgresql(Servers.PostgresConnectionString, "listeners");

    opts.ListenToRabbitQueue("ordered")

        // This option is available on all types of Wolverine
        // endpoints that can be configured to be a listener
        .ListenWithStrictOrdering();
}).StartAsync();

Watch out of course, because this throttles the processing of messages to single file on exactly one node. That’s perfect for cases where you’re not too concerned about throughput, but sequencing is very important. A JasperFx Software client is using this for messages to a stateful Saga that coordinates work across their application.

Do note that Wolverine will both ensure a listener with this option is only running on one node, and will redistribute any strict ordering listeners to better distribute work across a cluster. Wolverine will also be able to detect when it needs to switch the listening over to a different node if a node is taken down.

Messages should be processed in order within a logical group, but we need better throughput otherwise…

Let’s say that you have a case where you know the system would work much more efficiently if Wolverine could process messages related to a single business entity of some sort (an Invoice? a Purchase Order? an Incident?) in strict order. You still need more throughput than you can achieve through a strictly ordered listener that only runs on one node, but you do need the messages to be handled in order or maybe just one at a time for a single business entity to arrive at consistent state or to prevent errors due to concurrent access.

If you happened to be using Azure Service Bus as your messaging transport, you can utilize Session Identifiers and FIFO Queues with Wolverine to do exactly this:

_host = await Host.CreateDefaultBuilder()
    .UseWolverine(opts =>
    {
        opts.UseAzureServiceBusTesting()
            .AutoProvision().AutoPurgeOnStartup();

        opts.ListenToAzureServiceBusQueue("send_and_receive");
        opts.PublishMessage<AsbMessage1>().ToAzureServiceBusQueue("send_and_receive");

        opts.ListenToAzureServiceBusQueue("fifo1")

            // Require session identifiers with this queue
            .RequireSessions()

            // This controls the Wolverine handling to force it to process
            // messages sequentially
            .Sequential();

        opts.PublishMessage<AsbMessage2>()
            .ToAzureServiceBusQueue("fifo1");

        opts.PublishMessage<AsbMessage3>().ToAzureServiceBusTopic("asb3");
        opts.ListenToAzureServiceBusSubscription("asb3")
            .FromTopic("asb3")

            // Require sessions on this subscription
            .RequireSessions(1)

            .ProcessInline();
    }).StartAsync();

But, there’s a little bit more to publishing because you’ll need to tell Wolverine what the GroupId value is for your message:

I think we’ll try to make this a little more automatic in the near future with Wolverine.

// bus is an IMessageBus
await bus.SendAsync(new AsbMessage3("Red"), new DeliveryOptions { GroupId = "2" });
await bus.SendAsync(new AsbMessage3("Green"), new DeliveryOptions { GroupId = "2" });
await bus.SendAsync(new AsbMessage3("Refactor"), new DeliveryOptions { GroupId = "2" });

Of course, if you don’t have Azure Service Bus, you still have some other options. I think I’m going to save this for a later post, hopefully after building out some formal support for this, but another option is to:

1. Plan on having several different listeners for a subset of messages that all have the strictly ordered semantics as shown in the previous section. Each listener can at least process information independently
2. Use some kind of logic that can look at a message being published by Wolverine and use some kind of deterministic rule that will assign that message to one of the strictly ordered messaging destinations

Like I said, more to come on this in the hopefully near future, and this might be part of a JasperFx Software engagement soon.

What about handling events in Wolverine that are captured to Marten (or future Critter Event Stores)?

I’m Gen X, so the idea of Marten & Wolverine assembling to create the ultimate Event Driven Architecture stack makes me think of Transformers cartoons:)

It’s been a few years, but what is now Wolverine was originally called “Jasper” and was admittedly a failed project until we decided to reorient it to being a complement to Event Sourcing with Marten and renamed it “Wolverine” to continue the “Critter Stack” theme. A huge part of that strategy was having first class mechanisms to either publish or handle events captured by Marten’s Event Sourcing through Wolverine’s robust message execution and message publishing capabilities.

You have two basic mechanisms for this. The first, and original option is “Event Forwarding” where events captured by Marten are published to Wolverine upon the successful completion of the Marten transaction:

builder.Services.AddMarten(opts =>
    {
        var connString = builder
            .Configuration
            .GetConnectionString("marten");

        opts.Connection(connString);

        // There will be more here later...

        opts.Projections
            .Add<AppointmentDurationProjection>(ProjectionLifecycle.Async);

        // OR ???

        // opts.Projections
        //     .Add<AppointmentDurationProjection>(ProjectionLifecycle.Inline);

        opts.Projections.Add<AppointmentProjection>(ProjectionLifecycle.Inline);
        opts.Projections
            .Snapshot<ProviderShift>(SnapshotLifecycle.Async);
    })

    // This adds a hosted service to run
    // asynchronous projections in a background process
    .AddAsyncDaemon(DaemonMode.HotCold)

    // I added this to enroll Marten in the Wolverine outbox
    .IntegrateWithWolverine()

    // I also added this to opt into events being forward to
    // the Wolverine outbox during SaveChangesAsync()
    .EventForwardingToWolverine();

Event forwarding gives you no ordering guarantees of any kind, but will push events as messages to Wolverine immediately. Event forwarding may give you significantly better throughput then the subscription model we’ll look at next because there’s less latency between persisting the event to Marten and the event being published to Wolverine. Moreover, using “Event Forwarding” means that the event publishing happens throughout any application cluster.

However, if you need strictly ordered handling of the events being persisted to Marten, you instead need to use the Event Subscriptions model where Wolverine is handling or relaying Marten events as messages in the strict order in which they are appended to Marten, and on a single running node. This is analogous to the strictly ordered listener option explained above.

What about my scenario you didn’t discuss here?

See the Wolverine documentation, or come ask us on Discord.

Summary

There’s a real tradeoff between message ordering, processing throughput, and message delivery guarantees. Fortunately, Wolverine gives you plenty of options to meet a variety of different project requirements.

And one last time, you’re just not going to want to sign up for the level of robust options and infrastructure that’s under the covers of a tool like Wolverine can “just roll your own messaging abstractions” because you’re angry and think that community OSS tools can’t be trusted. And also, Wolverine is also a moving target that constantly improves based on the problems, needs, suggestions, and code contributions from our core team, community, and JasperFx Software customers. Your homegrown tooling will never receive that level of feedback, and probably won’t ever match Wolverine’s quality of documentation either.

If you’d prefer to start with more context, skip to the section named “Why is this important?”.

To set up the problem I’m hoping to address in this post, there are several settings across both Marten and Wolverine that need to be configured for the most optimal possible functioning between development, testing, and deployment time — but yet, some of these settings are done different ways today or have to be done independently for both Marten and Wolverine.

Below is a proposed configuration approach for Marten, Wolverine, and future “Critter” tools with the Marten 8 / Wolverine 4 “Critter Stack 2025” wave of releases:

        var builder = Host.CreateApplicationBuilder();
        
        // This would apply to both Marten, Wolverine, and future critters....
        builder.Services.AddJasperFx(x =>
        {
            // This expands in importance to be the master "AutoCreate"
            // over every resource at runtime and not just databases
            // So this would maybe take the place of AutoProvision() in Wolverine world too
            x.Production.AutoCreate = AutoCreate.None;
            x.Production.GeneratedCodeMode = TypeLoadMode.Static;
            x.Production.AssertAllPreGeneratedTypesExist = true;
            
            // Just for completeness sake, but these are the defaults
            x.Development.AutoCreate = AutoCreate.CreateOrUpdate;
            x.Development.GeneratedCodeMode = TypeLoadMode.Dynamic;

            // Unify the Marten/Wolverine/future critter application assembly
            // Default will always be the entry assembly
            x.ApplicationAssembly = typeof(Message1).Assembly;
        });
        
        // keep bootstrapping...

If you’ve used either Marten or Wolverine for production usages, you know that you probably want to turn off the dynamic code generation at production time, and you might choose to also turn off the automatic database migrations for both Marten and Wolverine in production (or not, I’ve been surprised how many folks are happy to just let the tools manage database schemas).

The killer problem for us today, is that the settings above have to be configured independently for both Marten and Wolverine — and as a bad coincidence, I just chatted with someone on Discord who got burned by this as I was starting this post. Grr.

Even worse, the syntactical options for disabling automatic database management for Wolverine’s envelope storage tables is a little different syntax altogether. And then just to make things more fun — and please cut the Critter Stack community and I some slack because all of this evolved over years — the “auto create / migrate / evolve” functionality for like Rabbit MQ queues/exchanges/bindings or Kafka topics is “opt in” instead of “opt out” like the automatic database migrations are with a completely different syntax and naming than either the Marten or Wolverine tables as shown with the AutoProvision() option below:

using var host = await Host.CreateDefaultBuilder()
    .UseWolverine(opts =>
    {
        opts.UseRabbitMq(rabbit => { rabbit.HostName = "localhost"; })
            // I'm declaring an exchange, a queue, and the binding
            // key that we're referencing below.
            // This is NOT MANDATORY, but rather just allows Wolverine to
            // control the Rabbit MQ object lifecycle
            .DeclareExchange("exchange1", ex => { ex.BindQueue("queue1", "key1"); })

            // This will direct Wolverine to create any missing Rabbit MQ exchanges,
            // queues, or binding keys declared in the application at application
            // start up time
            .AutoProvision();

        opts.PublishAllMessages().ToRabbitExchange("exchange1");
    }).StartAsync();

I’m not married to the syntax per se, but my proposal is that:

• Every possible type of “stateful resource” (database configurations or message brokers or whatever we might introduce in the future) by default follows the AutoCreate settings in one place, which for right now is in the AddJasperFx() method (should this be named something else? ConfigureJasperFx(), ConfigureCritterStack() ????
• You can override this at either the Marten or Wolverine levels, or within Wolverine, maybe you use the default behavior for the application for all database management, but turn down Azure Service Bus to AutoCreate.None.
• We’ll use the AutoCreate enumeration that originated in Marten, but will now move down to a lower level shared library to define the level for each resource
• All resource types will have a default setting of AutoCreate.CreateOrUpdate, even message brokers. This is to move the tools into more of a “it just works” out of the box developer experience. This will make the usage of AutoProvision() in Wolverine unnecessary unless you want to override the AutoCreate settings
• We deprecate the OptimizeArtifactWorkflow() mechanisms that never really caught on, and instead let folks just set potentially different settings for “Development” vs “Production” time, and let the tools apply the right settings based on the IHostEnvironment.Environment name so you don’t have to clutter up your code with too many ugly if (builder.Environment.IsDevelopment() ... calls.

Just for some context, the AutoCreate values are below:

public enum AutoCreate
{
    /// <summary>
    ///     Will drop and recreate tables that do not match the Marten configuration or create new ones
    /// </summary>
    All,

    /// <summary>
    ///     Will never destroy existing tables. Attempts to add missing columns or missing tables
    /// </summary>
    CreateOrUpdate,

    /// <summary>
    ///     Will create missing schema objects at runtime, but will not update or remove existing schema objects
    /// </summary>
    CreateOnly,

    /// <summary>
    ///     Do not recreate, destroy, or update schema objects at runtime. Will throw exceptions if
    ///     the schema does not match the Marten configuration
    /// </summary>
    None
}

For longstanding Critter Stack users, we’ll absolutely keep:

• The existing “stateful resource” model, including the resources command line helper for setting up or tearing down resource dependencies
• The existing db-* command line tooling
• The IServiceCollection.AddResourceSetupOnStartup() method for forcing all resources (databases and broker objects) to be correctly built out on application startup
• The existing Marten and Wolverine settings for configuring the AutoCreate levels, but these will be marked as [Obsolete]
• The existing Marten and Wolverine settings for configuring the code generation TypeLoadMode, but the default values will come from the AddJasperFx() options and the Marten or Wolverine options will be marked as [Obsolete]

Why is this important?

An important part of building, deploying, and maintaining an enterprise system with server side tooling like the “Critter Stack” (Marten, Wolverine, and their smaller sibling Weasel that factors quite a bit into this blog post) is dealing with creating or migrating database schema objects or message broker resources so that your application can function as expected against its infrastructure dependencies.

As any of you know who have ever walked into the development of an existing enterprise system, it’s often challenging to get your local development environment configured for that system — and that can frequently cause you days and I’ve even seen weeks of delay. What if instead you could simply start fresh with a clean clone of the code repository and be up and running very quickly?

If you pick up Marten for the first time today, spin up a brand new PostgreSQL database where you have full admin rights, and write this code it would happily work without you doing any explicit work to migrate the new PostgreSQL database:

public class Customer
{
    public Guid Id { get; set; }

    // We'll use this later for some "logic" about how incidents
    // can be automatically prioritized
    public Dictionary<IncidentCategory, IncidentPriority> Priorities { get; set; }
        = new();

    public string? Region { get; set; }

    public ContractDuration Duration { get; set; }
}

public record ContractDuration(DateOnly Start, DateOnly End);

public enum IncidentCategory
{
    Software,
    Hardware,
    Network,
    Database
}

public enum IncidentPriority
{
    Critical,
    High,
    Medium,
    Low
}

await using var store = DocumentStore
    .For("Host=localhost;Port=5432;Database=marten_testing;Username=postgres;password=postgres");

var customer = new Customer
{
    Duration = new ContractDuration(new DateOnly(2023, 12, 1), new DateOnly(2024, 12, 1)),
    Region = "West Coast",
    Priorities = new Dictionary<IncidentCategory, IncidentPriority>
    {
        { IncidentCategory.Database, IncidentPriority.High }
    }
};

// IDocumentSession is Marten's unit of work 
await using var session = store.LightweightSession();
session.Store(customer);
await session.SaveChangesAsync();

// Marten assigned an identity for us on Store(), so 
// we'll use that to load another copy of what was 
// just saved
var customer2 = await session.LoadAsync<Customer>(customer.Id);

// Just making a pretty JSON printout
Console.WriteLine(JsonConvert.SerializeObject(customer2, Formatting.Indented));

Instead, with its default settings, Marten is able to quietly check if its underlying database has all the necessary database tables, functions, sequences, and schemas for whatever it needs roughly when it needs it for the first time. The whole point of this functionality is to ensure that a new developer coming into your project for the very first time can quickly clone your repository, and be up and running either the whole system or even just integration tests that hit the database immediately because Marten is able to “auto-migrate” database changes for you so you can just focus on getting work done.

Great, right? Except that sometimes you certainly wouldn’t want this “auto-migration” business going. Maybe because the system doesn’t have permissions, or maybe just to make the system spin up faster without the overhead of calculating the necessity of a migration step (it’s not cheap, especially for something like a Serverless usage where you depend on fast cold starts). Either way, you’d like to be able to turn that off at production time with the assumption that you’re applying database changes beforehand (which the Critter Stack has worlds of tools to help with as well), so you’ll turn off the default behavior something like the following with Marten 7 and before:

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddMarten(opts =>
    {
        // Other configuration...

        // In production, let's turn off all the automatic database
        // migration stuff
        if (builder.Environment.IsProduction())
        {
            opts.AutoCreateSchemaObjects = AutoCreate.None;
        }
    })
    // Add background projection processing
    .AddAsyncDaemon(DaemonMode.HotCold)
    // This is a mild optimization
    .UseLightweightSessions();

Wolverine uses the same underlying Weasel helper library to make automatic database migrations that Marten does, and works similarly, but disabling the automatic database setup is different for reasons I don’t remember:

using var host = await Host.CreateDefaultBuilder()
    .UseWolverine(opts =>
    {
        // Disable automatic database migrations for message
        // storage
        opts.AutoBuildMessageStorageOnStartup = false;
    }).StartAsync();

Wolverine can do similar automatic management of Rabbit MQ, Azure Service Bus, AWS SQS, Kafka, Pulsar, or Google Pubsub objects at runtime, but in this case you have to explicitly “opt in” to that automatic management through the fluent interface registration of a message broker like this sample using Google Pubsub:

        var host = await Host.CreateDefaultBuilder()
            .UseWolverine(opts =>
            {
                opts.UsePubsub("your-project-id")

                    // Let Wolverine create missing topics and subscriptions as necessary
                    .AutoProvision()

                    // Optionally purge all subscriptions on application startup.
                    // Warning though, this is potentially slow
                    .AutoPurgeOnStartup();
            }).StartAsync();

So, yes, Wolverine overlaps quite a bit with both MediatR and MassTransit. If you’re a MediatR user, Wolverine just does a helluva lot more and we have an existing guide for converting from MediatR to Wolverine. For MassTransit (or NServiceBus) users, Wolverine covers a lot of the same asynchronous messaging framework use cases, but does much, much more to simplify your application code than any other .NET messaging framework and should not be compared as an apples to apples messaging feature comparison. And no other tool in the entire .NET ecosystem can come even remotely close to the Critter Stack’s support for Event Sourcing from soup to nuts.

It’s kind of a big day in .NET OSS news with both MediatR and MassTransit respectively announcing moves to commercial licensing models. I’d like to start by wishing the best of luck to my friends Jimmy Bogard and Chris Patterson respectively with their new ventures.

As any long term participant in or observer of the .NET ecosystem knows, there’s about to be a flood of negativity from various people in our community about these moves. There will also be an outcry from a sizable cohort in the .NET community who seem to believe that all development tools should be provided by Microsoft and that only Microsoft can ever be a reliable supplier of these types of tools while somehow suffering from amnesia about how Microsoft has frequently abandoned high profile tools like Silverlight or WCF.

As for Marten, Wolverine, and other future Critter Stack tools, the current JasperFx Software strategy remains following the “open core” model where the existing capabilities in the MIT-licensed tools (note below) remain under an OSS license and JasperFx Software focuses on services, support plans, and the forthcoming commercial CritterWatch tool for monitoring, management, and some advanced features for data privacy, multi-tenancy, and extreme scalability. While we certainly respect MassTransit’s decision, we’re going to try a different path and stay down the “open core” model and Marten 8 / Wolverine 4 will be released under the MIT OSS license. I will admit that you may see some increasing reluctance to be providing as much free support through Discord as we have to users in the past though.

To be technical, there is one existing feature in Marten 7.* for optimized projection rebuilds that I think we’ll redesign and move to the commercial add on tooling in the Marten 8 timeframe, but in this case the existing feature is barely usable anyway so ¯\_(ツ)_/¯