It’s the halfway point of 2025 some how, and we’ve now gotten past the big Marten 8.0 and Wolverine 4.0 releases. Right before I go on vacation next week, I thought it would be a good time to jot down some thoughts about where the Critter Stack might go for the rest of 2025 and probably into 2026.

Critter Watch

The big ticket item is our ongoing work on “Critter Watch”, which will be a commercial management and observability add on for Wolverine, Marten, and any future new Critter tools. The top line pitch for Critter Watch is that it well help you know what your applications are, how they interact with each other, whether they’re healthy in production, and provide features to help heal the inevitable production problems when they happen.

The general idea is to have a standalone application deployed that acts as a management console for 1 or more Wolverine applications in our user’s environments:

Upfront for the Critter Watch MVP (and requests from a client), we’re focused on:

• Visualizing the systems being monitored, their Wolverine and Marten configuration, and the capabilities of the systems. We’re currently researching AsyncAPI publishing and visualization as well. The whole point of this is to help teams understand how the messages in your system are handled, published, and routed.
• Event Sourcing management, but this is mostly about managing the execution of asynchronous projections and subscriptions at runtime and being able to understand the ongoing performance or any ongoing problems
• Dead letter queue management for Wolverine

I have less clarity over development time tooling, but we’re at least interested in having some of Critter Watch usable as an embedded tool during development.

After years of talking about this and quite a bit of envisioning, development started in earnest over the past 6 weeks with a stretch goal of having a pilot usage by the end of July for a JasperFx Software client.

I do not yet have any hard pricing numbers yet, but we are very interested in talking to anyone who would be interested in Critter Watch.

Concurrency, Concurrency, Concurrency!

I think that systems built with Event Sourcing are a little more sensitive to concurrent data reads and writes, or maybe it’s just that those problems are there all the time but more readily observable with Event Sourcing and Event Driven Architectures. In my work with JasperFx Software clients, concurrency is probably the most common subject of questions.

Mostly today you deal with this either by building in selective retry capabilities based on version conflict detection, or get fancier with queueing and message routing to eliminate the concurrent access as much as possible. Or both of course.

A great way to side step the concurrent access while not sacrificing throughput through parallelization is to use Wolverine’s support for Azure Service Bus Session Identifiers and FIFO Queues.

Which is great, but what if you’re not using Azure Service Bus? What if you’re only using local queueing? And wouldn’t it be nice if the existing Azure Service Bus FIFO support was a little less cumbersome to use in your code?

I don’t have a ton of detail, but there’s a range of internal proposals to create some new recipes for Wolverine usage to enable teams to more easily “shard” logical work between queues and within the local workers listening to queues to improve Wolverine’s handling of concurrent access without sacrificing parallel work and throughput or requiring repetitive code. Some of this is being done in collaboration with JasperFx clients.

Improving Wolverine’s Declarative Data Access

For lack of a better description, Wolverine has a feature set I’m heretofore calling “declarative data access” with the [Entity] attribute that triggers code generation within message handlers or HTTP endpoints to load requested data from Marten, EF Core, or RavenDb. And of course, there’s also what we call the “aggregate handler workflow” recipe for using the Decider pattern with Wolverine and Marten that I think is the simplest way to express business logic when using Event Sourcing in the .NET ecosystem.

To take these productivity features even farther, I think we’ll add some:

1. More control over what action to take if an entity is missing. Today, the HTTP endpoints will just return a 404 status code if required entities can’t be found. In future versions, we’ll let you customize log or ProblemDetails messages and have more control over how Wolverine generates the “if missing” path
2. At least for Marten, opt into Marten’s batch querying support if you are using more than one of any combination of the existing [Aggregate], [ReadAggregate], [Entity], or [Document] attributes to load data within a single HTTP endpoint or message handler as a way of improving performance by reducing network round trips to the database. And don’t sneeze at that, chattiness is a common performance killer in enterprise applications. Especially when the code is unnecessarily complicated by typical usages of Clean or Onion Architectural approaches.

If you follow Event Sourcing related topics online, you’ll hear quite a bit of buzz from some of the commercial tools about “Dynamic Consistency Boundaries” (DCB). We get asked about this with Marten occasionally, but the Marten core team’s position is that Marten doesn’t require this feature because you can already do “read” and “write” operations across multiple event streams with transactional integrity as is.

What the batch querying I just described will do for Marten though is make the full “Critter Stack” usage be more performant when you need to potentially work with more than one event stream at a time with all the transactional support and strong consistency that Marten (really PostgreSQL) already provides.

For Marten users, this is essentially making Marten’s FetchForWriting() API able to enroll in batch querying for more efficient data querying when working across streams. That work is actually well underway.

But if you prefer to use the fancier and more novel DCB approaches that aren’t even officially released yet, feel free to pay out some big bucks to use one of the commercial tools.

Smaller, But Still Important Work!

• Partially for Critter Watch, Wolverine should support connecting to multiple brokers in a single application for each transport type. Some of this is already done, with Kafka being next up, but we need to add this to every transport
• Improved interoperability support for Wolverine talking to non-Wolverine applications. There’s an existing pull request that goes quite a ways for this, but this might end up being more a documentation effort than anything else
• More options in Wolverine with Marten or just Marten for streaming Marten data as JSON directly to HTTP. We have some support already of course, but there are more opportunities for expanding that
• Exposing an MCP server off of Marten event data, but I have very little detail about what that would be. I would be very interested in partnering with a company who wanted to do this, and a JasperFx client might be working with us later this year on AI with Marten
• Improving throughput in Marten’s event projections and subscriptions. We’ve done a lot the past couple years, but there are still some other ideas in the backlog we haven’t played yet
• Expanding Wolverine support for more database engines, with CosmosDb the most likely contender this year. This might be contingent upon client work of course.

What about the SQL Server backed Event Store?

Yeah, I don’t know. We did a ton of work in Marten 8 to pull what will be common code out in a way that it could be reused in the SQL Server backed event store. I do not know when we might work on this as CritterWatch will take priority for now.

And finally….

And on that note I’m essentially on vacation for a week and I’ll catch up with folks in late July.

You’re about to start a new system with Event Sourcing using Marten, and you’re expecting your system to be hugely successful such that it’s going to handle a huge amount of data, but you’re already starting with pretty ambitious non-functional requirements for the system to be highly performant and all the screens or exposed APIs be snappy.

Basically, what you want to do is go as fast as Marten and PostgreSQL will allow. Fortunately, Marten has a series of switches and dials that can be configured to squeeze out more performance, but for a variety of historical reasons and possible drawbacks, are not the defaults for a barebones Marten configuration as shown below:

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddMarten(opts =>
{
    opts.Connection(builder.Configuration.GetConnectionString("marten"));
});

Cut me some slack in my car choice for the analogy here. I’m not only an American, but I’m an American from a rural area who grew up dreaming about having my own Mustang or Camaro because that’s as far out as I could possibly imagine back then.

At this point, we have is the equivalent to a street legal passenger car, maybe the equivalent to an off the shelf Mustang:

Which probably easily goes fast enough for every day usage for the mass majority of us most of the time. But we really need a fully tricked out Mustang GTD that’s absurdly optimized to just flat out go fast:

Let’s start trimming weight off our street legal Marten setup to go faster with…

Opt into Lightweight Sessions by Default

Starting from a new system so we don’t care about breaking existing code by changing behavior, let’s opt for lightweight sessions by default:

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddMarten(opts =>
{
    opts.Connection(builder.Configuration.GetConnectionString("marten"));
})
    
// Jettison some "Identity Map" weight by going lighter weight    
.UseLightweightSessions();

By default, the instances of IDocumentSession you get out of an IoC container would utilize the Identity Map feature to track loaded entities by id so that if you happened to try to load the same entity from the same session, you would get the exact same object. As I’m sure you can imagine, that means that every entity fetched by a session is stuffed into a dictionary internally (Marten uses the highly performant ImTools ImHashMap everywhere, but still), and the session also has to bounce through the dictionary before loading data as well. It’s just a little bit of overhead we can omit by opting for “Lightweight Sessions” if we don’t need that behavior by default.

We’ve always been afraid to change the default behavior here to the more efficient approach because it can absolutely lead to breaking existing code that depends on the Identity Map behavior. On the flip side, I think you should not need Identity Map mechanics if you can keep the call stacks within your code short enough that you can actually “see” where you might be trying to load the same data twice or more in the same parent operation.

On to the next thing…

Make Writes Faster with Quick Append

Next, since we again don’t have any existing code that can be broken here, let’s opt for “Quick Append” writes like so:

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddMarten(opts =>
{
    opts.Connection(builder.Configuration.GetConnectionString("marten"));

    // Make event writing faster, like 2X faster in our testing
    opts.Events.AppendMode = EventAppendMode.Quick;
})
    
// Jettison some "Identity Map" weight by going lighter weight    
.UseLightweightSessions();

This will help the system be able to append new events much faster, but at the cost of not being able to use some event metadata like event versions, sequence, or timestamp information within “Inline” projections.

Again, even though this option has been clocked as being much faster, we have not wanted to make this the default because it could break existing systems for people who depend on having the rich metadata during the Inline application of projections that forces Marten to do a kind of two step process to append events. This “Quick Append” option also helps reduce concurrent access problems writing to streams and generally makes the “Async Daemon” subsystem processing asynchronous projections and subscriptions run much smoother.

We’re not out of tricks yet by any means, so let’s go on…

Use the Identity Map for Inline Aggregates

Wait, I thought you told me not to cross the streams! Yeah, about the Identity Map thing, there’s one exception where we actually do want that behavior within CQRS command handlers like this one using Wolverine and its “Aggregate Handler Workflow” integration with Marten:

    // This tells Wolverine that the first "return value" is NOT the response
    // body
    [EmptyResponse]
    [WolverinePost("/api/incidents/{incidentId:guid}/category")]
    public static IncidentCategorised Post(
        // the actual command
        CategoriseIncident command, 
        
        // Wolverine is generating code to look up the Incident aggregate
        // data for the event stream with this id
        [Aggregate("incidentId")] Incident incident)
    {
        // This is a simple case where we're just appending a single event to
        // the stream.
        return new IncidentCategorised(incident.Id, command.Category, command.CategorisedBy);
    }

In the case above, the Incident model is a projected document that’s first used by the command handler to “decide” what new events to emit. If we’re updating the Incident model with an Inline projection that tries to update the Incident model in the database at the same time it wants to append events, then it’s an advantage for performance to “just” use the original Incident model we used initially, then forwarding the new state based on the new events and persisting the results right then and there. We can opt into this optimization even for the lightweight sessions we earlier wanted to use by adopting one more UseIdentityMapForAggregates flag:

builder.Services.AddMarten(opts =>
{
    opts.Connection(builder.Configuration.GetConnectionString("marten"));

    // Make event writing faster, like 2X faster in our testing
    opts.Events.AppendMode = EventAppendMode.Quick;

    // This can cut down on the number of database round trips
    // Marten has to do during CQRS command handler execution
    opts.Events.UseIdentityMapForAggregates = true;
})
    
// Jettison some "Identity Map" weight by going lighter weight    
.UseLightweightSessions();

Note, this optimization can easily break code for folks who use some sort of stateful “Aggregate Root” approach where the state of the projected aggregate object might be mutated during the course of executing the command. As this has traditionally been a popular approach in Event Sourcing circles, we can’t make this be a default option. If you instead either make the projected aggregates like Incident either immutable or treat them as a dumb data input to your command handlers with a more Functional Programming “Decider” function approach, you can get away with the performance optimization.

And also, I strongly prefer and recommend the FP “Decider” approach to JasperFx Software clients as is and I think that folks using the older “Aggregate Root” approach tend to have more runtime bugs.

Moving on, let’s keep our database smaller…

Event Stream Archiving

By and large, you can improve system performance in almost any situation by trying to keep your database from growing too large by archiving or retiring obsolete information. Marten has first class support for “Archiving Event Streams” where you effectively just move event streams that only represent historical information and are not really active into an archived state.

Moreover, we can divide our underlying PostgreSQL storage for events into “hot” and “cold” storage by utilizing PostgreSQL’s table partitioning support like this:

builder.Services.AddMarten(opts =>
{
    opts.Connection(builder.Configuration.GetConnectionString("marten"));

    // Make event writing faster, like 2X faster in our testing
    opts.Events.AppendMode = EventAppendMode.Quick;

    // This can cut down on the number of database round trips
    // Marten has to do during CQRS command handler execution
    opts.Events.UseIdentityMapForAggregates = true;

    // Let's leverage PostgreSQL table partitioning
    // to our advantage
    opts.Events.UseArchivedStreamPartitioning = true;
})
    
// Jettison some "Identity Map" weight by going lighter weight    
.UseLightweightSessions();

If you’re aggressive with marking event streams as Archived, the PostgreSQL table partitioning can move off archived event streams into a different table partition than our active event data. This is essentially keeping the “active” event table storage relatively stable in size, and most operations will execute against this smaller table partition while still being able to access the archived data too if explicitly opt into including that.

We added this feature in a minor point 7.* release, so it had to be opt in, and I think I was too hesitant to make this a default in 8.0, so it’s still “opt in”.

Stream Compacting

Beyond archiving event streams, maybe you just want to “compact” a longer event stream so you technically retain all the existing state, but further reduce the size of your active database storage. To that end, Marten 8.0 added Stream Compacting.

Distributing Asynchronous Projections

I had been mostly talking about using projections running Inline such that the projections are updated at the same time as the events are captured. That’s sometimes applicable or desirable, but other times you’ll want to optimize the “write” operations by moving the updating of projected data to an Async projection running in the background. But now let’s say that we have quite a few asynchronous projections and several subscriptions as well. In early versions of Marten, we had to run everything in a “Hot/Cold” mode where every known projection or subscription had to run on one single “leader” node. So even if you were running your application across a dozen or more nodes, only one could be executing all of the asynchronous projections and subscriptions.

That’s obviously a potential bottleneck, so Marten 7.0 by itself introduced some ability to spread projections and subscriptions over multiple nodes. If we introduce Wolverine into the mix though, we can do quite a bit better than that by allowing Wolverine to distribute the asynchronous Marten work across our entire cluster with its ability to distribute Marten projections and subscriptions with the UseWolverineManagedEventSubscriptionDistribution option in the WolverineFx.Marten Nuget:

builder.Services.AddMarten(opts =>
{
    opts.Connection(builder.Configuration.GetConnectionString("marten"));

    // Make event writing faster, like 2X faster in our testing
    opts.Events.AppendMode = EventAppendMode.Quick;

    // This can cut down on the number of database round trips
    // Marten has to do during CQRS command handler execution
    opts.Events.UseIdentityMapForAggregates = true;

    // Let's leverage PostgreSQL table partitioning
    // to our advantage
    opts.Events.UseArchivedStreamPartitioning = true;
})
    
// Jettison some "Identity Map" weight by going lighter weight    
.UseLightweightSessions()

.IntegrateWithWolverine(opts =>
{
    opts.UseWolverineManagedEventSubscriptionDistribution = true;
});

Is there anything else for the future?

It never ends, and yes, there are still quite a few ideas in our product backlog to potentially improve performance and scalability of Marten’s Event Sourcing. Offhand, that includes looking at alternative, higher performance serializers and more options to parallelize asynchronous projections to squeeze out more throughput by sharing some data access across projections.

Summary

There are quite a few “opt in” features in Marten that will help your system perform better, but these features are “opt in” because they can be harmful if you’re not building around the assumptions these features make about how your code works. The good news though is that you’ll be able to better utilize these features if you follow the Critter Stack’s recommended practices by striving for shorter code stacks (i.e., how many jumps between methods and classes does your code make when receiving a system input like a message or HTTP request) so your code is easier to reason about anyway, and avoiding mutating projected aggregate data outside of Marten.

Time for an update on Critter Stack release plans, and a follow up on my previous Critter Stack Work in Progress post from March. The current plan is to release Marten 8.0, Weasel 8.0, and Wolverine 4.0 on June 1st. It’s not going to be a huge release in terms of new functionality, but there are some important structural changes that will help us build some future features, and we needed to jettison older .NET versions while getting onto the latest Npgsql. “CritterWatch” is still very much planned and a little bit in progress, but we’ve got to get these big releases out first.

The key takeaways are that I want to essentially freeze Marten 7.* for everything but bug fixes right now, and probably freeze Wolverine 3.* for new feature development after a last wave of pull requests gets pulled in over the next couple days.

I’m admittedly too drowsy and lazy to write much tonight, so here’s just a dump of what I wrote up for the rest of our core team to review. I think we’re already at the point where we’re ready to just work on documentation and a few last touches, so the mass majority of this doesn’t get done in time, but here’s the full brain storm:

First though, what’s been done:

• .NET 6 & 7 were dropped
• Updated to Npgsql 9 across the board
• Dropped all synchronous APIs in Marten
• Deleted some [Obsolete] APIs in Marten
• Consolidation of supporting libraries to a single JasperFx library
• JasperFx has that new consolidated configuration option for common configuration like application assembly, code generation, and the stateful resource AutoCreate mode
• Pulled out event projections and core event store abstractions to a new JasperFx.Events library
  • Removed code generation from all projections
  • Better explicit code options for aggregations and event projections
• Wolverine 4 has better handles envelope storage & the transactional inbox/outbox for modular monoliths
• Improved “Descriptor” model to describe the static configuration of Wolverine and/or Marten applications that we’ll use for CritterWatch too
• Expanded commands for dead letter queue management in Wolverine that was meant for CritterWatch
• Multi-tenancy options in Wolverine for SQL Server or PostgreSQL w/o Marten, multi-tenancy usage with EF Core

Punchlist?

1. Marten 7.40.4 release w/ a pair of outstanding PRs
   1. Cherry pick commits to Marten “master”
2. JasperFx & JasperFx.Events 1.0
   1. Documentation website?
3. Weasel “master” branch
   1. All tests should be passing
4. Marten “master” branch
   1. All tests should be passing
   2. Documentation website should be building – that’s going to take some effort because of code samples
   3. Get Anne’s PR for tutorials in (cool new guided tour of building a system using Event Sourcing and Event Driven Architecture with first Marten, then Wolverine)
   4. Stream Compacting feature – for a JasperFx customer (this is definitely in for Marten 8, this is a big improvement for keeping a larger system running fast over time by compacting the database)
   5. Fix the optimized projection rebuild options? Or rip it out and leave it for CritterWatch?
   6. Ability to overwrite the event timestamp (relatively easy)
   7. Migration guide 
   8. Figure out what the proper behavior of “Live” aggregations when there’s some ShouldDelete() action going on
5. Wolverine
   1. One last 3.14 release with easy to grab pull requests and bug fixes
   2. Rebase on 3.14
   3. Fork off the 3.0 branch
   4. 4.0 becomes main branch
   5. All tests should be passing
   6. Documentation website should build
   7. Migration guide
6. Critter Watch preparation
   1. When integrated w/ CritterWatch, Wolverine can build the descriptor model for the entire application, including EventStoreUsage. No idea where this work stands right now. Did quite a bit earlier this year, then went off in a different direction
   2. Review all Open Telemetry usage and activity naming across Marten and especially Wolverine. Add Open Telemetry & Metrics metadata to the descriptor model sent to CritterWatch. I think this is somewhat likely to get done before Wolverine 4.0.
   3. Ability to send messages from CritterWatch to Wolverine. Might push through some kind of message routing and/or message handler extensibility

Nice to do?

1. Consider consolidating the stateful resource / AutoCreate configuration so there are fewer thing to configure. See Managing Auto Creation of Database or Message Broker Resources in the Critter Stack vNext
2. Programmatic message routing in Wolverine that varies based on the message contents? This is around options to route a message to one of a set of destinations based on the message core. Thinking about concurrency here. Could be done later.
3. More open issues in the Marten 8 milestone, but it’s about time to drop any issue that isn’t a breaking change
4. More open issues in the Wolverine 4 milestone or Wolverine in general
5. Ermine/Polecat readiness? (Marten ported to SQL Server)
   1. Spike it out? 
   2. Look for opportunities to pull shared items into Weasel?