Update on Wolverine 5.0 Work

A couple weeks back I posted about some upcoming feature work in Wolverine that might push us to call it “5.0” even though Wolverine 4.0 is only three months old. Despite the obvious issues with quickly cranking out yet another major point release, the core team & I mostly came down on the side of proceeding with 5.0, but there will be very few breaking API or even behavioral changes that very few people will even notice moving from 4.* to 5.0 or hopefully even from 3.* to 5.0 (I’d say “none”, but we all know that’s an impossibility).

Here’s a rundown on the progress and some thoughts about how it finishes based on the same header structure as before.

TPL DataFlow to Channels

We have branches of both Marten & Wolverine that successfully replace our previous dependency on the TPL Dataflow library with System.Threading.Channels. I think I’ll blog about that later this week. I’d like to hammer on this a bit with performance and load testing before it goes out, but right now it’s full speed ahead and I’m happy with how smoothly that went after the typical stubbed toes at first.

“Concurrency Resistant Parallelism”

We’re still workshopping whatever the final name for this feature. “Partitioned Sequential Messaging” maybe? The basic idea here is that Wolverine will be able to segment work based on some kind of business domain identifier (a tenant id? the stream id or key from Marten event streams? Saga identity?) such that all messages for a particular domain identifier run sequentially so there’s very little concurrent access problems, but work across domain identifiers are executed in parallel. Wolverine is going to be able to do this either just the local running process (with local messaging queues) or throughout the entire running cluster of nodes.

This work was one of the main drivers for the Channels conversion, and I’m very happy with how it’s gone so far. At this point, the basic functionality is in place and it just needs documentation and maybe some polished usability.

I think this is going to be a killer feature for Critter Stack users as it can almost entirely eliminate encounters with the dreaded ConcurrencyException from Event Sourcing.

Interoperability

This work was unpleasant, and still needs better documentation, but Wolverine 5.0 now has more consistent mechanisms for creating custom interoperability recipes across all external messaging transports. Moreover, we will now have MassTransit and NServiceBus interoperability via:

Rabbit MQ (this has been in place since Wolverine 1.0)
AWS SQS (this guy is the big outlier for almost everything)
Azure Service Bus

In addition, Wolverine 5.0 has opt in support for the CloudEvents specification for:

Rabbit MQ
AWS SQS
Azure Service Bus
GCP Pubsub
Kafka
Pulsar

Again, I think this feature set hopefully makes it easier to adopt Wolverine in new efforts within existing NServiceBus, MassTransit, or Dapr shops. Plus making Wolverine more interoperable with all the completely different things out there.

Integrating with Marten’s Batch Querying / Optimizing Multi-Event Stream Operations

Nothing to report on yet, but this work will definitely be in Wolverine 5.0. My thinking is that this will be an important part of the Critter Stack’s answer to the “Dynamic Consistency Boundary” concept coming out of some of the commercial Event Sourcing tools. And folks, I’m 100% petty and competitive enough that we’ll have this out before AxonIQ’s official 5.0 release.

IoC Usage

Wolverine is very much an outlier for .NET application frameworks in how it uses an IoC tool internally, and even though that definitely comes with real advantages, there’s some potential bumps in the road for new users. The Wolverine 5.0 branch already has the proposed new diagnostics and policies to keep users from unintentionally using non-Wolverine friendly IoC configuration. Wolverine.HTTP 5.0 can also be told to play nicely with the HttpContext.RequestServices container in HTTP-scoped operations. I personally don’t recommend doing that in greenfield applications, but it’s an imperfect world and folks had plenty of reasons for wanting this.

TL;DR: Wolverine does not like runtime IoC magic at all.

Wolverine.HTTP Improvements

I don’t have any update on this one, and all of this could easily get bumped back to 5.1 if the release lingers too long.

SignalR Integration

I’m hoping to spend quite a bit of time this week after Labor Day working on the Dead Letter Queue management features in “CritterWatch”, and I’m planning on building a new SignalR transport as part of that work. Right now, my theory is that we’ll use the new CloudEvents mapping code we wrote for interoperability for the SignalR integration such that messages back and forth will be wrapped something like:

{ type: “message_type_identifier”, data: { { }

I’m very happy for any feedback or requests about the SignalR integration with Wolverine. That’s come up a couple times over the years, and I’ve always said I didn’t want to build that outside of real use cases, but now CritterWatch gives us something real in terms of requirements.

Cold Start Optimization

No updates yet, but a couple different JasperFx clients are interested in this, and that makes it a priority as time allows.

What else?

I think there’s going to need to be some minor changes in observability or diagnostics just to feed CritterWatch, and I’d like for us to get as far as possible with CritterWatch before cutting 5.0 just so there are no more breaking API changes.

I’d love to do some hard core performance testing and optimization on some of the fine grained mechanics of Wolverine and Marten as part of this work. There are a few places where we might have opportunities to optimize memory usage and data shuffling.

What about Marten?

Honestly, I think in the short term that Marten development is going to be limited to possible performance improvements for a JasperFx client and whatever ends up being necessary for CritterWatch integration.

Faster & More Reliable Integration Testing Against Marten Projections or Subscriptions

I just pulled the trigger on Marten 8.8 and Wolverine 4.10 earlier today. Neither is particularly large, but there’s some new toys and an important improvement for test automation support that are worth calling out.

My goodness, that title is a mouthful. I’ve been helping a couple different JasperFx Software clients and community users on Discord with their test automation harnesses. In all cases, there was some complexity involved because of the usage of some mix of asynchronous projections or event subscriptions in Marten or asynchronous messaging with Wolverine. As part of that work to support a client today, Marten has this new trick (with a cameo from the related JasperFx Alba tool for HTTP service testing):

// This is bootstrapping the actual application using
// its implied Program.Main() set up
Host = await AlbaHost.For<Program>(b =>
{
    b.ConfigureServices((context, services) =>
    {
        // Important! You can make your test harness work a little faster (important on its own)
        // and probably be more reliable by overriding your Marten configuration to run all
        // async daemons in "Solo" mode so they spin up faster and there's no issues from
        // PostgreSQL having trouble with advisory locks when projections are rapidly started and stopped

        // This was added in V8.8
        services.MartenDaemonModeIsSolo();

        services.Configure<MartenSettings>(s =>
        {
            s.SchemaName = SchemaName;
        });
    });
});

Specifically note the new `IServiceCollection.MartenDaemonModeIsSolo()`. That is overriding any Marten async daemons that normally run with the “Hot/Cold” load distribution that is appropriate for production with Marten’s “Solo” load distribution so that your test harness can spin up much faster. In addition, this mode will enable Marten to more quickly shut down, then restart all asynchronous projections or subscriptions in tests when you use this existing testing helper to reset state:

        // OR if you use the async daemon in your tests, use this
        // instead to do the above, but also cleanly stop all projections,
        // reset the data, then start all async projections and subscriptions up again
        await Host.ResetAllMartenDataAsync();

In the above usage, that ResetAllMartenDataAsync() is smart enough to first disable all asynchronous projections and subscriptions, reset the Marten data store to your configured baseline state (effectively by wiping out all data, then reapplying all your “initial data”), then restarting all asynchronous projections and subscriptions from the new baseline.

Having the “Solo” load distribution will make the constant teardown and restart of the asynchronous projections faster than it would with a “Hot/Cold” configuration where Marten still assumes there might be other nodes running.

If you or your shop would want some assistance with test automation using the Critter Stack or otherwise, drop me a note at jeremy@jasperfx.net and I can chat about what we could do to help you out.

I’ll be discussing this new feature and quite a bit more in a live stream tomorrow (August 20th) at 2:00PM US Central time:

I’m also starting a collection of requirements for what might turn into an off the shelf test automation harness library for the Critter Stack.

Operations that Span Multiple Event Streams with the Critter Stack

Let’s just say that Marten incurs some serious benefits to being on top of PostgreSQL and its very strong support for transactional integrity as opposed to some of the high profile commercial Event Sourcing tools who are spending a lot of time and energy on their “Dynamic Consistency Boundary” concept because they lack the ACID compliant transactions that Marten gets for free by riding on top of PostgreSQL.

Marten has long had the ability to support both reading and appending to multiple event streams at one time with guarantees about data consistency and even the ability to achieve strongly consistent transactional writes across multiple streams at one time. Wolverine just added some syntactic sugar to make cross-stream command handlers be more declarative with its “aggregate handler workflow” integration with Marten.

Using the canonical example of a use case where you move money from one account to another account and need both changes to be persisted in one atomic transaction. Let’s start with a simplified domain model of events and a “self-aggregating” Account type like this:

public record AccountCreated(double InitialAmount);
public record Debited(double Amount);
public record Withdrawn(double Amount);

public class Account
{
    public Guid Id { get; set; }
    public double Amount { get; set; }

    public static Account Create(IEvent<AccountCreated> e)
        => new Account { Id = e.StreamId, Amount = e.Data.InitialAmount};

    public void Apply(Debited e) => Amount += e.Amount;
    public void Apply(Withdrawn e) => Amount -= e.Amount;
}

Moving on, here’s what a command handler could be that handles a TransferMoney command that impacts two different accounts:

public record TransferMoney(Guid FromId, Guid ToId, double Amount);

public static class TransferMoneyEndpoint
{
    [WolverinePost("/accounts/transfer")]
    public static void Post(
        TransferMoney command,

        [Aggregate(nameof(TransferMoney.FromId))] IEventStream<Account> fromAccount,
        
        [Aggregate(nameof(TransferMoney.ToId))] IEventStream<Account> toAccount)
    {
        // Would already 404 if either referenced account does not exist
        if (fromAccount.Aggregate.Amount >= command.Amount)
        {
            fromAccount.AppendOne(new Withdrawn(command.Amount));
            toAccount.AppendOne(new Debited(command.Amount));
        }
    }
}

The IEventStream<T> abstraction comes from Marten’s FetchForWriting() API that is our recommended way to interact with Marten streams in typical command handlers. This API is used underneath Wolverine’s “aggregate handler workflow”, but normally hidden from user written code if you’re only working with one stream at a time. In this case though, we’ll need to work with the raw IEventStream<T> objects that both wrap the projected aggregation of each Account as well as providing a point where we can explicitly append events separately to each event stream. FetchForWriting() guarantees that you get the most up to date information for the Account view of each event stream regardless of how you have configured Marten’s ProjectionLifecycle for Account (kind of an important detail here!).

The typical Marten transactional middleware within Wolverine is calling SaveChangesAsync() for us on the Marten unit of work IDocumentSession for the command. If there’s enough funds in the “From” account, this command will append a Withdrawn event to the “From” account and a Debited event to the “To” account. If either account has been written to between fetching the original information, Marten will reject the changes and throw its ConcurrencyException as an optimistic concurrency check.

In unit testing, we could write a unit test for the “happy path” where you have enough funds to cover the transfer like this:

public class when_transfering_money
{
    [Fact]
    public void happy_path_have_enough_funds()
    {
        // StubEventStream<T> is a type that was recently added to Marten
        // specifically to facilitate testing logic like this
        var fromAccount = new StubEventStream<Account>(new Account { Amount = 1000 }){Id = Guid.NewGuid()};
        var toAccount = new StubEventStream<Account>(new Account { Amount = 100}){Id = Guid.NewGuid()});
        
        TransferMoneyEndpoint.Post(new TransferMoney(fromAccount.Id, toAccount.Id, 100), fromAccount, toAccount);

        // Now check the events we expected to be appended
        fromAccount.Events.Single().ShouldBeOfType<Withdrawn>().Amount.ShouldBe(100);
        toAccount.Events.Single().ShouldBeOfType<Debited>().Amount.ShouldBe(100);
    }
}

Alright, so there’s a few remaining items we still need to improve over time:

Today there’s no way to pass in the expected starting version of each individual stream
There’s some ongoing work to allow Wolverine to intelligently parallelize work between business entities or event streams while doing work sequentially within a business entity or event stream to side step concurrency problems
We’re working toward making Wolverine utilize Marten’s batch querying support any time you use Wolverine’s declarative persistence helpers against Marten and request more than one item from Marten. You can use Marten’s batch querying with its FetchForWriting() API today if you just drop down to the lower level and work directly against Marten, but wouldn’t it be nice if Wolverine would just do that automatically for you in cases like the TransferMoney command handler above? We think this will be a significant performance improvement because network round trips are evil.

I covered this example at the end of a live stream we did last week on Event Sourcing with the Critter Stack:

Need Some Feedback on Near Term Wolverine Work

That’s supposed to be a play on a Wolverine as Winnie the Pooh in his “thinking spot”

I’m wrestling a little bit with whether the new features and changes coming into Wolverine very soon are worthy of a 5.0 release even though 4.0 was just a couple months ago. I’d love any and all feedback about this. I’d also like to ask for help from the community to kick the tires on any alpha/beta/RC releases we might make with these changes.

Wolverine development is unusually busy right now as new feature requests are streaming in from JasperFx customers and users as Wolverine usage has increased quite a bit this year. We’re only a couple months out from the Wolverine 4.0 release (and Marten 8.0 that was a lot bigger). I wrote about Critter Stack futures just a month ago, but things have already changed since then, so let’s do this again.

Right now, here are the major initiatives happening or planned for the near future for Wolverine in what I think is probably the priority order:

TPL DataFlow to Channels

I’m actively working on replacing both Marten & Wolverine’s dependency on the TPL Dataflow library with System.Threading.Channels. This is something I wanted to do for 4.0, but there wasn’t enough time. Because of some issues with TPL DataFlow a JasperFx client hit under load and the planned “concurrency resistant parallelism” feature work I’ll discuss next, I wanted to start using Channels now. I’m a little concerned that this change by itself justifies a Wolverine 5.0 release even though the public APIs aren’t changing. I would expect some improvement in performance from this change, but I don’t have hard numbers yet. What do you think? I’ll have this done in a local branch by the end of the day.

“Concurrency Resistant Parallelism”

For lack of a better name, we’re planning some “concurrency resistant parallelism” features for Wolverine. Roughly, this is teaching Wolverine about how to better parallelize *or* order messages in a system so that you can maximize throughput (parallelism) without incurring concurrent writes to resources or entities that are sensitive to concurrent write problems (*cough* Marten event streams *cough*). I’d ask you to just look at the GitHub issue I linked. This is to maximize throughput for an important JasperFx client who frequently gets bursts of messages related to the same event stream, but also, this has been a frequent issue for quite a few users and we hope this would be a hugely strategic addition to Wolverine

Interoperability

Improving the interoperability options for Wolverine. and non-Wolverine applications. There’s already some work underway, but I think this might be a substantial effort out of sheer permutations. At a minimum, I’m hoping we have OOTB compatibility against both NServiceBus & MassTransit for all supported message transports in Wolverine and not just Rabbit MQ like we do today. Largely based on a pull request from the community, we’ll also make it easier to build out custom interoperability with non-Wolverine applications. And then lastly, there’s enough interest in CloudEvents to push through that as well.

Integrating with Marten’s Batch Querying / Optimizing Multi-Event Stream Operations

Make the “Critter Stack” tool the best Event Store / Event Driven Architecture platform on the freaking planet for working with multiple event streams at the same time. Mostly because it would just be flat out sexy, I’m interested in enhancing Wolverine’s integration with Marten to be able to opt into Marten’s batch querying API under the covers when you use the declarative persistence options or the aggregate handler workflow in Wolverine. This would be beneficial by:

Improving performance because network chattiness is very commonly an absolute performance killer in enterprise-y systems — especially for teams that get a little too academic with Clean/Onion Architecture approachs
Be what we hope will be a superior alternative for working with multiple event streams at one time in terms of usability, testability, and performance than the complex “Dynamic Consistency Boundary” idea coming out of some of the commercial event store tool companies right now
Further Wolverine’s ability to craft much simpler Post-Clean Architecture codebases for better productivity and longer term maintenance. Seriously, I really do believe that Clean/Onion Architecture approaches absolutely strangle systems in the longer term because the code easily becomes too difficult to reason about.

IoC Usage

Improve Wolverine’s integration with IoC containers, especially for HTTP usage. I think I’d like to consider introducing an “opt out” setting where Wolverine asserts and fails on bootstrapping if any message handler or HTTP endpoint can’t use Wolverine’s inlined code generation and has to revert to service location unless users explicitly say they will allow it.

Wolverine.HTTP Improvements

Expanded support in Wolverine.HTTP for [AsParameters] usage, probably some rudimentary “content negotiation,” multi-part uploads. Really just filling some current holes in Wolverine.HTTP‘s current support as more people use that library.

SignalR

A formal SignalR integration for Wolverine, which will most likely drop out of our ongoing “Critter Watch” development. Think about having a first class transport option for Wolverine that will let you quickly integrate messages to and from a web application via SignalR

Cold Start Optimization

Optimizing the Wolverine “Cold Start Time.” I think that’s self explanatory. This work might span into Marten and even Lamar as well. I’m not going to commit to AOT compatibility in the Critter Stack this year because I like actually getting to see my family sometimes, but this work might get us closer to that for next year.

Improved Declarative Persistence in Wolverine

To continue a consistent theme about how Wolverine is becoming the antidote to high ceremony Clean/Onion Architecture approaches, Wolverine 4.8 added some significant improvements to its declarative persistence support (partially after seeing how a recent JasperFx Software client was encountering a little bit of repetitive code).

A pattern I try to encourage — and many Wolverine users do like — is to make the main method of a message handler or an HTTP endpoint be the “happy path” after validation and even data lookups so that that method can be a pure method that’s mostly concerned with business or workflow logic. Wolverine can do this for you through its “compound handler” support that gets you to a low ceremony flavor of Railway Programming.

With all that out of the way, I saw a client frequently writing code something like this endpoint that would need to process a command that referenced one or more entities or event streams in their system:

public record ApproveIncident(Guid Id);

public class ApproveIncidentEndpoint
{
    // Try to load the referenced incident
    public static async Task<(Incident, ProblemDetails)> LoadAsync(
        
        // Say this is the request body, which we can *also* use in
        // LoadAsync()
        ApproveIncident command, 
        
        // Pulling in Marten
        IDocumentSession session,
        CancellationToken cancellationToken)
    {
        var incident = await session.LoadAsync<Incident>(command.Id, cancellationToken);
        if (incident == null)
        {
            return (null, new ProblemDetails { Detail = $"Incident {command.Id} cannot be found", Status = 400 });
        }

        return (incident, WolverineContinue.NoProblems);
    }

    [WolverinePost("/api/incidents/approve")]
    public SomeResponse Post(ApproveIncident command, Incident incident)
    {
        // actually do stuff knowing that the Incident is valid
    }
}

I’d ask you to mostly pay attention to the LoadAsync() method, and imagine copy & pasting dozens of times in a system. And sure, you could go back to returning IResult as a continuation from the HTTP endpoint method above, but that moves clutter back into your HTTP method and would add more manual work to mark up the method with attributes for OpenAPI metadata. Or we could improve the OpenAPI metadata generation by returning something like Task<Results<Ok<SomeResponse>, ProblemHttpResult>>, but c’mon, that’s an absolute eye sore that detracts from the readability of the code.

Instead, let’s use the newly enhanced version of Wolverine’s [Entity] attribute to simplify the code above and still get OpenAPI metadata generation that reflects both the 200 SomeResponse happy path and 400 ProblemDetails with the correct content type. That would look like this:

    [WolverinePost("/api/incidents/approve")]
    public static SomeResponse Post(
        // The request body. Wolverine doesn't require [FromBody], but it wouldn't hurt
        ApproveIncident command, 
        
        [Entity(OnMissing = OnMissing.ProblemDetailsWith400, MissingMessage = "Incident {0} cannot be found")]
        Incident incident)
    {
        // actually do stuff knowing that the Incident is valid
        return new SomeResponse();
    }

Behaviorally, at runtime that endpoint will try to load the Incident entity from whatever persistence tooling is configured for the application (Marten in the tests) using the “Id” property of the ApproveIncident object deserialized from the HTTP request body. If the data cannot be found, the HTTP requests ends with a 400 status code and a ProblemDetails response with the configured message up above. If the Incident can be found, it’s happily passed along to the main endpoint.

Not that every endpoint or message handler is really this simple, but plenty of times you would be changing a property on the incident and persisting it. We can *still* be mostly a pure function with the existing persistence helpers in Wolverine like so:

    [WolverinePost("/api/incidents/approve")]
    public static (SomeResponse, IStorageAction<Incident>) Post(
        // The request body. Wolverine doesn't require [FromBody], but it wouldn't hurt
        ApproveIncident command, 
        
        [Entity(OnMissing = OnMissing.ProblemDetailsWith400, MissingMessage = "Incident {0} cannot be found")]
        Incident incident)
    {
        incident.Approved = true;
        
        // actually do stuff knowing that the Incident is valid
        return (new SomeResponse(), Storage.Update(incident));
    }

Here’s some things I’d like you to know about that [Entity] attribute up above and how that is going to work out in real usage:

There is some default conventional magic going on to “decide” how to get the identity value for the entity being loaded (“IncidentId” or “Id” on the command type or request body type, then the same value in routing values for HTTP endpoints or declared query string values). This can be explicitly configured on the attribute something like [Entity(nameof(ApproveIncident.Id)]
Every attribute type that I’m mentioning in this post that can be applied to method parameters supports the same identity logic as I explained in the previous bullet
Before Wolverine 4.8, the “on missing” behavior was to simply set a 404 status code in HTTP or log that required data was missing in message handlers and quit. Wolverine 4.8 adds the ability to control the “on missing” behavior
This new “on missing” behavior is available on the older [Document] attribute in Wolverine.Http.Marten, and [Document] is now a direct subclass of [Entity] that can be used with either message handlers or HTTP endpoints now
The existing [AggregateHandler] and [Aggregate] attributes that are part of the Wolverine + Marten “aggregate handler workflow” (the “C” in CQRS) now support this “on missing” behavior, but it’s “opt in,” meaning that you would have to use [Aggregate(Required = true)] to get the gating logic. We had to make that required test opt in to avoid breaking existing behavior when folks upgraded.
The lighter weight [ReadAggregate] in the Marten integration also standardizes on this “OnMissing” behavior
Because of the confusion I was seeing from some users between [Aggregate]which is meant for writing events and is a little heavier runtime wise than [ReadAggregate], there’s a new [WriteAggregate] attribute with identical behavior to [Aggregate] and now available for message handlers as well. I think that [Aggregate] might get deprecated soon-ish to sidestep the potential confusion
[Entity] attribute usage is 100% supported for EF Core and RavenDb as well as Marten. Wolverine is even smart enough to select the correct DbContext type for the declared entity
If you coded with any of that [Entity] or Storage stuff and switched persistence tooling, your code should not have to change at all
There’s no runtime Reflection going on here. The usage of [Entity] is impacting Wolverine’s code generation around your message handler or HTTP endpoint methods.

The options so far for “OnMissing” behavior is this:

public enum OnMissing
{
    /// <summary>
    /// Default behavior. In a message handler, the execution will just stop after logging that the data was missing. In an HTTP
    /// endpoint the request will stop w/ an empty body and 404 status code
    /// </summary>
    Simple404,
    
    /// <summary>
    /// In a message handler, the execution will log that the required data is missing and stop execution. In an HTTP
    /// endpoint the request will stop w/ a 400 response and a ProblemDetails body describing the missing data
    /// </summary>
    ProblemDetailsWith400,
    
    /// <summary>
    /// In a message handler, the execution will log that the required data is missing and stop execution. In an HTTP
    /// endpoint the request will stop w/ a 404 status code response and a ProblemDetails body describing the missing data
    /// </summary>
    ProblemDetailsWith404,
    
    /// <summary>
    /// Throws a RequiredDataMissingException using the MissingMessage
    /// </summary>
    ThrowException
}

The Future

This new improvement to the declarative data access is meant to be part of a bigger effort to address some bigger use cases. Not every command or query is going to involve just one single entity lookup or one single Marten event stream, so what do you do when there are multiple declarations for data lookups?

I’m not sure what everyone else’s experience is, but a leading cause of performance problems in the systems I’ve helped with over the past decade has been too much chattiness between the application servers and the database. The next step with the declarative data access is to have at least the Marten integration opt into using Marten’s batch querying mechanism to improve performance by batching up requests in fewer network round trips any time there are multiple data lookups in a single HTTP endpoint or message handler.

The step after that is to also enroll our Marten integration for command handlers so that you can craft message handlers or HTTP endpoints that work against 2 or more event streams with strong consistency and transactional support while also leveraging the Marten batch querying for all the efficiency we can wring out of the tooling. I mostly want to see this behavior because I’ve seen clients who could actually use what I was just describing as a way to make their systems more efficient and remove some repetitive code.

I’ll also admit that I think this capability to have an alternative “aggregate handler workflow” that allows you to work efficiently with more than one event stream and/or projected aggregate at one time would put the Critter Stack ahead of the commercial tools pursuing “Dynamic Consistency Boundaries” with what I’ll be arguing is an easier to use alternative.

It’s already possible to work transactionally with multiple event streams at one time with strong consistency and both optimistic and exclusive version protections, but there’s opportunity for performance optimization here.

Summary

Pride goeth before destruction, and an haughty spirit before a fall.

Proverbs 16:18 in the King James version

With the quote above out of the way, let’s jump into some cocky salesmanship! My hope and vision for the Critter Stack is that it becomes the most effective tooling for building typical server side software systems. My personal vision and philosophy for making software development more productive and effective over time is to ruthlessly reduce repetitive code and eliminate code ceremony wherever possible. Our community’s take is that we can achieve improved results compared to more typical Clean/Onion/Hexagonal Architecture codebases by compressing and compacting code down without ever sacrificing performance, resiliency, or testability.

The declarative persistence helpers in this article are, I believe, a nice example of the evolving “Critter Stack Way.”

Metadata Tracking Improvements in Marten

We just released a new batch of improvements in the Marten 8.4 release that improved Marten‘s already strong support for tracking metadata on event persistence.

Override Event Metadata on Individual Events

This work was done at the behest of a JasperFx Software client. They only needed to vary header values between events, but while the hood was popped up on event metadata, we finally addressed the long awaited ability to explicitly set event timestamps.

First, we finally have the ability to allow users to modify metadata on an event by event basis including the event timestamp. This has been a long standing request from many folks to either facilitate testing scenarios or to enable easier data importing from other databases or event stores. And especially now that Marten is arguably the best event sourcing solution for .NET, folks really should have a viable path to import data from external sources.

You can do that either by grabbing the IEvent wrapper and modifying the timestamp, causation, correlation, event id (valuable for tracing event data back to external systems), or headers like this sample:

public static async Task override_metadata(IDocumentSession session)
{
    var started = new QuestStarted { Name = "Find the Orb" };

    var joined = new MembersJoined
    {
        Day = 2, Location = "Faldor's Farm", Members = new string[] { "Garion", "Polgara", "Belgarath" }
    };

    var slayed1 = new MonsterSlayed { Name = "Troll" };
    var slayed2 = new MonsterSlayed { Name = "Dragon" };

    var joined2 = new MembersJoined { Day = 5, Location = "Sendaria", Members = new string[] { "Silk", "Barak" } };

    var action = session.Events
        .StartStream<QuestParty>(started, joined, slayed1, slayed2, joined2);

    // I'm grabbing the IEvent wrapper for the first event in the action
    var wrapper = action.Events[0];
    wrapper.Timestamp = DateTimeOffset.UtcNow.Subtract(1.Hours());
    wrapper.SetHeader("category", "important");
    wrapper.Id = Guid.NewGuid(); // Just showing that you *can* override this value
    wrapper.CausationId = wrapper.CorrelationId = Activity.Current?.Id;

    await session.SaveChangesAsync();
}

Or by appending an already wrapped IEvent as I’m showing here, along with some new convenience wrapper extension methods to make the mechanics a little more declarative:

public static async Task override_metadata2(IDocumentSession session)
{
    var started = new QuestStarted { Name = "Find the Orb" };

    var joined = new MembersJoined
    {
        Day = 2, Location = "Faldor's Farm", Members = new string[] { "Garion", "Polgara", "Belgarath" }
    };

    var slayed1 = new MonsterSlayed { Name = "Troll" };
    var slayed2 = new MonsterSlayed { Name = "Dragon" };

    var joined2 = new MembersJoined { Day = 5, Location = "Sendaria", Members = new string[] { "Silk", "Barak" } };

    // The result of this is an IEvent wrapper around the
    // started data with an overridden timestamp
    // and a value for the "color" header
    var wrapper = started.AsEvent()
        .AtTimestamp(DateTimeOffset.UtcNow.Subtract(1.Hours()))
        .WithHeader("color", "blue");

    session.Events
        .StartStream<QuestParty>(wrapper, joined, slayed1, slayed2, joined2);

    await session.SaveChangesAsync();
}

The second approach is going to be necessary if you are appending events with the FetchForWriting() API (and you should be within any kind of CQRS “write” handler).

There is of course a catch. If you use the “QuickAppend” option in Marten and want to be able to override the event timestamps, you’ll need this slightly different option instead:

var builder = Host.CreateApplicationBuilder();
builder.Services.AddMarten(opts =>
{
    opts.Connection(builder.Configuration.GetConnectionString("marten"));

    // This is important!
    opts.Events.AppendMode = EventAppendMode.QuickWithServerTimestamps;
});

To avoid causing database breaking changes when upgrading, the ability to override timestamps with the “QuickAppend” option required this new “opt in” setting because this forces Marten to generate both “glue” code and a database function a little differently.

Capturing the User Name on Persisted Events

These kinds of features have to be “opt in” so that we don’t cause database changes in a minor release when people upgrade. Having to worry about “opt in” or “opt out” mechanics and backwards compatibility is both the curse and enabler of long running software tool projects like Marten.

Another request from the back log was to have a first class tracking of the user name (or process name) in events based on the current user of whatever operation appended the events. Following along with the “opt in” support for tracking correlation and causation ids, we’ll first need to opt into storing the user name with events:

var builder = Host.CreateApplicationBuilder();
builder.Services.AddMarten(opts =>
{
    opts.Connection(builder.Configuration.GetConnectionString("marten"));

    opts.Events.MetadataConfig.UserNameEnabled = true;
});

And now we can apply the user name to persisted events something like this:

public static async Task Handle(StartInvoice command, IDocumentSession session, ClaimsPrincipal principal)
{
    // Marking the session as being modified by this active user
    session.LastModifiedBy = principal.Identity.Name;
    
    // Any events persisted by this session will be tagged with the current user
    // in the database
    session.Events.StartStream(new InvoiceStarted(command.Name, command.Amount));
    await session.SaveChangesAsync();
}

And while this should probably only be used for diagnostics mostly, you can now query against the raw event data with LINQ for the user name (assuming that it’s captured of course!) like this sample from our tests:

    [Theory]
    [InlineData(JasperFx.Events.EventAppendMode.Rich)]
    [InlineData(JasperFx.Events.EventAppendMode.Quick)]
    public async Task capture_user_name_information(EventAppendMode mode)
    {
        EventAppendMode = mode;
        var streamId = Guid.NewGuid();

        theSession.LastModifiedBy = "Larry Bird";

        // Just need a time that will be easy to assert on that is in the past
        var timestamp = (DateTimeOffset)DateTime.Today.Subtract(1.Hours()).ToUniversalTime();

        var action = theSession.Events.StartStream(streamId, new AEvent(), new BEvent(), new CEvent());
        action.Events[0].UserName = "Kevin McHale";

        await theSession.SaveChangesAsync();

        using var query = theStore.QuerySession();

        var events = await query.Events.FetchStreamAsync(streamId);

        events[0].UserName.ShouldBe("Kevin McHale");
        events[1].UserName.ShouldBe("Larry Bird");
        events[2].UserName.ShouldBe("Larry Bird");

        // Should write another test, but I'm doing it here!
        var celtics = await query.Events.QueryAllRawEvents().Where(x => x.UserName == "Larry Bird").ToListAsync();
        celtics.Count.ShouldBeGreaterThanOrEqualTo(2);
    }

Summary

Projects like Marten are never, ever completed and we have no intentions of abandoning Marten anytime soon. The features above have been requested for quite awhile, but didn’t make the cut for Marten 8.0. I’m happy to see them hit now, and this could be the basis of a long waited formal support for efficient data imports to Marten from other event stores.

And of course, if there’s something that Marten or Wolverine doesn’t do today that you need, please reach out to JasperFx Software and we can talk about an engagement to build out your features.

Example of Using Alba for HTTP Testing

Before Marten took off and we pivoted to using the “Critter Stack” naming motif, the original naming theme for the JasperFx OSS tool suite were some of the small towns near where I grew up in Southwest Missouri. Alba, MO is somewhat famous as the hometown of the Boyer brothers.

I’m taking a little time this week to build out some improvements to Wolverine’s declarative data access support based on some recent client work. As this work is largely targeted at Wolverine’s HTTP support, I’m heavily leveraging Alba to help test the HTTP behavior and I thought this work would make a great example of how Alba can help you more efficiently test HTTP API code in .NET.

Now, back to Wolverine and the current work I’m in the midst of testing today. To remove a lot of the repetitive code out of this client’s HTTP API, Wolverine is going to improve the [Entity] attribute mechanics to easily customize “on missing” handling something like this simple example from tests:

    // Should 400 w/ ProblemDetails on missing
    [WolverineGet("/required/todo400/{id}")]
    public static Todo2 Get2([Entity(OnMissing = OnMissing.ProblemDetailsWith400)] Todo2 todo) 
        => todo;

With Wolverine message handlers or HTTP endpoints, the [Entity] attribute is a little bit of declarative data access that just directs Wolverine to generate some code around your method to load data for that parameter based on its type from whatever your attached data access tooling is for that application, currently supported for Marten (of course), EF Core, and RavenDb. In its current form, if Marten/EF Core/RavenDb cannot find a Todo2 entity in the database with the identity from the route argument “id”, Wolverine will just set the HTTP status code to 404 and exit.

And while I’d argue that’s a perfectly fine default behavior, a recent client wants instead to write out a ProblemDetails response describing what data referenced in the request was unavailable and return a 400 status code instead. They’re handling that with Wolverine’s Railway Programming support just fine, but I think that’s causing my client more repetitive code than I personally prefer, and Wolverine is based on the philosophy that repetitive code should be minimized as much as possible. Hence, the enhancement work hinted at above with a new OnMissing property that lets you specify exactly how an HTTP endpoint should handle the case where a requested entity is missing.

So let’s finally introduce Alba with this test harness using xUnit:

public class reacting_to_entity_attributes : IAsyncLifetime
{
    private readonly ITestOutputHelper _output;
    private IAlbaHost theHost;

    public reacting_to_entity_attributes(ITestOutputHelper output)
    {
        _output = output;
    }

    public async Task InitializeAsync()
    {
        // This probably isn't your typical Alba usage, but
        // I'm spinning up a little AspNetCore application
        // for endpoint types in the current testing assembly
        var builder = WebApplication.CreateBuilder([]);

        // Adding Marten as the target persistence provider,
        // but the attribute does work w/ EF Core too
        builder.Services.AddMarten(opts =>
        {
            // Establish the connection string to your Marten database
            opts.Connection(Servers.PostgresConnectionString);
            opts.DatabaseSchemaName = "onmissing";
        }).IntegrateWithWolverine().UseLightweightSessions();

        builder.Host.UseWolverine(opts => opts.Discovery.IncludeAssembly(GetType().Assembly));

        builder.Services.AddWolverineHttp();

        // This is using Alba, which uses WebApplicationFactory under the covers
        theHost = await AlbaHost.For(builder, app =>
        {
            app.MapWolverineEndpoints();
        });
    }

    async Task IAsyncLifetime.DisposeAsync()
    {
        if (theHost != null)
        {
            await theHost.StopAsync();
        }
    }

    // Other tests...

    [Fact]
    public async Task problem_details_400_on_missing()
    {
        var results = await theHost.Scenario(x =>
        {
            x.Get.Url("/required/todo400/nonexistent");

            x.StatusCodeShouldBe(400);
            x.ContentTypeShouldBe("application/problem+json");
        });

        var details = results.ReadAsJson<ProblemDetails>();
        details.Detail.ShouldBe("Unknown Todo2 with identity nonexistent");
    }
    
}

Just a few things to call out about the test above:

Alba is using WebApplicationFactory and TestServer from AspNetCore under the covers to bootstrap an AspNetCore IHost without having to use Kestral
The Alba Scenario() method is running an HTTP request all the way through the application in process
Alba has declarative helpers to assert on the expected HTTP status code and content-type headers in the response, and I used those above
The ReadAsJson<T>() helper just helps us deserialize the response body into a .NET type using whatever the JSON serialization configuration is within our application — and by no means should you minimize that because that’s a humongous potential source of false test results for the unwary if folks use mismatched JSON serialization settings between their application and test harness code!

For the record, that test is passing in my local branch right now after a couple iterations. Alba just happened to make the functionality pretty easy to test through both the declarative assertions and the JSON serialization helpers.

How I Prioritize OSS Bugs

I just got back from a week long vacation with the family, and I was as rested and relaxed as I can ever be, at least before I picked up a head cold on the last day. Today though, it’s time to start catching up on OSS bug reports that have come in in the past 10 days or so. I thought it might be fun to dash off my personal prioritization for the bugs that come into the Marten, Wolverine, or related projects.

Roughly, here’s an unscientific ranking of the factors that get bugs fixed sooner rather than later:

Any issue that is blocking or harming a JasperFx Software client’s system
Issues that already have user supplied pull requests to fix the issue. You never want to leave a pull request open too long if someone has taken the time to contribute. It still happens for a variety of reasons, but you do still try.
Bugs that I find embarrassing
Any problem that would likely give a new user a poor first impression of the tools
Problems that I think would potentially impact many users
Any other issue for a JasperFx client
Issues reported by significant contributors, and I’m pretty loose with what I think of as “significant”
Easy fixes just to help keep the open GitHub issue counts as low as possible because that’s something I do care about
Open issues in whatever project I happen to be preparing a release for while that project has my attention
Any issue that will require breaking API changes in the tool, but this subset will sometimes be prioritized to the top whenever we’re making a full point release
Any issue that is going to require significant changes to the internals, but this is somewhat similar to the previous line
Issues that aren’t likely to impact many users
Issues reported by people being kind of a jerk that aren’t likely to impact many users

For older versions of any of the tools, like Marten 7.*, the list is much shorter:

For a JasperFx Software client who cannot upgrade soon, we’ll of course make fixes to the older branch and forward that fix to the current version
For everybody else, eh, probably not unless it’s really bad or they’ve just asked very nicely

Critter Stack Futures for the rest of 2025

It’s the halfway point of 2025 some how, and we’ve now gotten past the big Marten 8.0 and Wolverine 4.0 releases. Right before I go on vacation next week, I thought it would be a good time to jot down some thoughts about where the Critter Stack might go for the rest of 2025 and probably into 2026.

Critter Watch

The big ticket item is our ongoing work on “Critter Watch”, which will be a commercial management and observability add on for Wolverine, Marten, and any future new Critter tools. The top line pitch for Critter Watch is that it well help you know what your applications are, how they interact with each other, whether they’re healthy in production, and provide features to help heal the inevitable production problems when they happen.

The general idea is to have a standalone application deployed that acts as a management console for 1 or more Wolverine applications in our user’s environments:

Upfront for the Critter Watch MVP (and requests from a client), we’re focused on:

Visualizing the systems being monitored, their Wolverine and Marten configuration, and the capabilities of the systems. We’re currently researching AsyncAPI publishing and visualization as well. The whole point of this is to help teams understand how the messages in your system are handled, published, and routed.
Event Sourcing management, but this is mostly about managing the execution of asynchronous projections and subscriptions at runtime and being able to understand the ongoing performance or any ongoing problems
Dead letter queue management for Wolverine

I have less clarity over development time tooling, but we’re at least interested in having some of Critter Watch usable as an embedded tool during development.

After years of talking about this and quite a bit of envisioning, development started in earnest over the past 6 weeks with a stretch goal of having a pilot usage by the end of July for a JasperFx Software client.

I do not yet have any hard pricing numbers yet, but we are very interested in talking to anyone who would be interested in Critter Watch.

Concurrency, Concurrency, Concurrency!

I think that systems built with Event Sourcing are a little more sensitive to concurrent data reads and writes, or maybe it’s just that those problems are there all the time but more readily observable with Event Sourcing and Event Driven Architectures. In my work with JasperFx Software clients, concurrency is probably the most common subject of questions.

Mostly today you deal with this either by building in selective retry capabilities based on version conflict detection, or get fancier with queueing and message routing to eliminate the concurrent access as much as possible. Or both of course.

A great way to side step the concurrent access while not sacrificing throughput through parallelization is to use Wolverine’s support for Azure Service Bus Session Identifiers and FIFO Queues.

Which is great, but what if you’re not using Azure Service Bus? What if you’re only using local queueing? And wouldn’t it be nice if the existing Azure Service Bus FIFO support was a little less cumbersome to use in your code?

I don’t have a ton of detail, but there’s a range of internal proposals to create some new recipes for Wolverine usage to enable teams to more easily “shard” logical work between queues and within the local workers listening to queues to improve Wolverine’s handling of concurrent access without sacrificing parallel work and throughput or requiring repetitive code. Some of this is being done in collaboration with JasperFx clients.

Improving Wolverine’s Declarative Data Access

For lack of a better description, Wolverine has a feature set I’m heretofore calling “declarative data access” with the [Entity] attribute that triggers code generation within message handlers or HTTP endpoints to load requested data from Marten, EF Core, or RavenDb. And of course, there’s also what we call the “aggregate handler workflow” recipe for using the Decider pattern with Wolverine and Marten that I think is the simplest way to express business logic when using Event Sourcing in the .NET ecosystem.

To take these productivity features even farther, I think we’ll add some:

More control over what action to take if an entity is missing. Today, the HTTP endpoints will just return a 404 status code if required entities can’t be found. In future versions, we’ll let you customize log or ProblemDetails messages and have more control over how Wolverine generates the “if missing” path
At least for Marten, opt into Marten’s batch querying support if you are using more than one of any combination of the existing [Aggregate], [ReadAggregate], [Entity], or [Document] attributes to load data within a single HTTP endpoint or message handler as a way of improving performance by reducing network round trips to the database. And don’t sneeze at that, chattiness is a common performance killer in enterprise applications. Especially when the code is unnecessarily complicated by typical usages of Clean or Onion Architectural approaches.

If you follow Event Sourcing related topics online, you’ll hear quite a bit of buzz from some of the commercial tools about “Dynamic Consistency Boundaries” (DCB). We get asked about this with Marten occasionally, but the Marten core team’s position is that Marten doesn’t require this feature because you can already do “read” and “write” operations across multiple event streams with transactional integrity as is.

What the batch querying I just described will do for Marten though is make the full “Critter Stack” usage be more performant when you need to potentially work with more than one event stream at a time with all the transactional support and strong consistency that Marten (really PostgreSQL) already provides.

For Marten users, this is essentially making Marten’s FetchForWriting() API able to enroll in batch querying for more efficient data querying when working across streams. That work is actually well underway.

But if you prefer to use the fancier and more novel DCB approaches that aren’t even officially released yet, feel free to pay out some big bucks to use one of the commercial tools.

Smaller, But Still Important Work!

Partially for Critter Watch, Wolverine should support connecting to multiple brokers in a single application for each transport type. Some of this is already done, with Kafka being next up, but we need to add this to every transport
Improved interoperability support for Wolverine talking to non-Wolverine applications. There’s an existing pull request that goes quite a ways for this, but this might end up being more a documentation effort than anything else
More options in Wolverine with Marten or just Marten for streaming Marten data as JSON directly to HTTP. We have some support already of course, but there are more opportunities for expanding that
Exposing an MCP server off of Marten event data, but I have very little detail about what that would be. I would be very interested in partnering with a company who wanted to do this, and a JasperFx client might be working with us later this year on AI with Marten
Improving throughput in Marten’s event projections and subscriptions. We’ve done a lot the past couple years, but there are still some other ideas in the backlog we haven’t played yet
Expanding Wolverine support for more database engines, with CosmosDb the most likely contender this year. This might be contingent upon client work of course.

What about the SQL Server backed Event Store?

Yeah, I don’t know. We did a ton of work in Marten 8 to pull what will be common code out in a way that it could be reused in the SQL Server backed event store. I do not know when we might work on this as CritterWatch will take priority for now.

And finally….

And on that note I’m essentially on vacation for a week and I’ll catch up with folks in late July.

OSS Project Lessons Learned with David Giard

I got to talk to David Giard on his podcast last week about some of the lessons I’ve learned the hard way across several large OSS projects. For a little background, I got to follow through on a 15 to 20 year dream of mine to found a company called JasperFx Software LLC to build a services and product offerings around the “Critter Stack” family of open source tools (Marten and Wolverine) in the .NET ecosystem. The two main tools are doing well right now, with Marten being the most used Event Sourcing tool for .NET projects and Wolverine gaining traction as an alternative messaging tool and HTTP endpoint framework with its focus on reduced code ceremony and testable code.

The relative success of these tools came after I was the technical leader of a very large, ambitious project called FubuMVC (and FubuTransportation) that fizzled out after I probably sunk 2-3 man years of effort into it over a half decade. As David did helpfully point out, some of the now success of Marten and Wolverine was absolutely predicated on some lessons learned both positive (mostly technical) and negative (community engagement, documentation, samples) from the earlier FubuMVC experience.

Without further ado, here’s David & I: