My internal code name for one of the new features I’m describing is “multi-stage tracked sessions” which somehow got me thinking of the ZZ Top song “Stages” and their Afterburner album because the sound track for getting this work done this week. Not ZZ Top’s best stuff, but there’s still some bangers on it, or at least *I* loved how it sounded on my Dad’s old phonograph player when I was a kid. For what it’s worth, my favorite ZZ Top albums cover to cover are Degüello and their La Futura comeback album.

I was heavily influenced by Extreme Programming in my early career and that’s made me have a very deep appreciation for the quality of “Testability” in the development tools I use and especially for the tools like Marten and Wolverine that I work on. I would say that one of the differentiators for Wolverine over other .NET messaging libraries and application frameworks is its heavy focus and support for automated testing of your application code.

The Critter Stack community released Marten 8.14 and Wolverine 5.1 today with some significant improvements to our testing support. These new features mostly originated from my work with JasperFx Software clients that give me a first hand look into what kinds of challenges our users hit automating tests that involve multiple layers of asynchronous behavior.

Stubbed Message Handlers in Wolverine

The first improvement is Wolverine getting the ability to let you temporarily apply stubbed message handlers to a bootstrapped application in tests. The key driver for this feature is teams that take advantage of Wolverine’s request/reply capabilities through messaging.

Jumping into an example, let’s say that your system interacts with another service that estimates delivery costs for ordering items. At some point in the system you might reach out through a request/reply call in Wolverine to estimate an item delivery before making a purchase like this code:

// This query message is normally sent to an external system through Wolverine
// messaging
public record EstimateDelivery(int ItemId, DateOnly Date, string PostalCode);

// This message type is a response from an external system
public record DeliveryInformation(TimeOnly DeliveryTime, decimal Cost);

public record MaybePurchaseItem(int ItemId, Guid LocationId, DateOnly Date, string PostalCode, decimal BudgetedCost);
public record MakePurchase(int ItemId, Guid LocationId, DateOnly Date);
public record PurchaseRejected(int ItemId, Guid LocationId, DateOnly Date);

public static class MaybePurchaseHandler
{
    public static Task<DeliveryInformation> LoadAsync(
        MaybePurchaseItem command, 
        IMessageBus bus, 
        CancellationToken cancellation)
    {
        var (itemId, _, date, postalCode, budget) = command;
        var estimateDelivery = new EstimateDelivery(itemId, date, postalCode);
        
        // Let's say this is doing a remote request and reply to another system
        // through Wolverine messaging
        return bus.InvokeAsync<DeliveryInformation>(estimateDelivery, cancellation);
    }
    
    public static object Handle(
        MaybePurchaseItem command, 
        DeliveryInformation estimate)
    {

        if (estimate.Cost <= command.BudgetedCost)
        {
            return new MakePurchase(command.ItemId, command.LocationId, command.Date);
        }

        return new PurchaseRejected(command.ItemId, command.LocationId, command.Date);
    }
}

And for a little more context, the EstimateDelivery message will always be sent to an external system in this configuration:

var builder = Host.CreateApplicationBuilder();
builder.UseWolverine(opts =>
{
    opts
        .UseRabbitMq(builder.Configuration.GetConnectionString("rabbit"))
        .AutoProvision();

    // Just showing that EstimateDelivery is handled by
    // whatever system is on the other end of the "estimates" queue
    opts.PublishMessage<EstimateDelivery>()
        .ToRabbitQueue("estimates");
});

In testing scenarios, maybe the external system isn’t available at all, or it’s just much more challenging to run tests that also include the external system, or maybe you’d just like to write more isolated tests against your service’s behavior before even trying to integrate with the other system (my personal preference anyway). To that end we can now stub the remote handling like this:

public static async Task try_application(IHost host)
{
    host.StubWolverineMessageHandling<EstimateDelivery, DeliveryInformation>(
        query => new DeliveryInformation(new TimeOnly(17, 0), 1000));

    var locationId = Guid.NewGuid();
    var itemId = 111;
    var expectedDate = new DateOnly(2025, 12, 1);
    var postalCode = "78750";

    var maybePurchaseItem = new MaybePurchaseItem(itemId, locationId, expectedDate, postalCode,
        500);
    
    var tracked =
        await host.InvokeMessageAndWaitAsync(maybePurchaseItem);
    
    // The estimated cost from the stub was more than we budgeted
    // so this message should have been published
    
    // This line is an assertion too that there was a single message
    // of this type published as part of the message handling above
    var rejected = tracked.Sent.SingleMessage<PurchaseRejected>();
    rejected.ItemId.ShouldBe(itemId);
    rejected.LocationId.ShouldBe(locationId);
}

After calling making this call:

        host.StubWolverineMessageHandling<EstimateDelivery, DeliveryInformation>(
            query => new DeliveryInformation(new TimeOnly(17, 0), 1000));

Calling this from our Wolverine application:

        // Let's say this is doing a remote request and reply to another system
        // through Wolverine messaging
        return bus.InvokeAsync<DeliveryInformation>(estimateDelivery, cancellation);

Will use the stubbed logic we registered. This is enabling you to use fake behavior for difficult to use external services.

For the next test, we can completely remove the stub behavior and revert back to the original configuration like this:

public static void revert_stub(IHost host)
{
    // Selectively clear out the stub behavior for only one message
    // type
    host.WolverineStubs(stubs =>
    {
        stubs.Clear<EstimateDelivery>();
    });
    
    // Or just clear out all active Wolverine message handler
    // stubs
    host.ClearAllWolverineStubs();
}

There’s a bit more to the feature you can read about in our documentation, but hopefully you can see right away how this can be useful for effectively stubbing out the behavior of external systems through Wolverine in tests.

And yes, some older .NET messaging frameworks already had *this* feature and it’s been occasionally requested from Wolverine, so I’m happy to say we have this important and useful capability.

Forcing Marten’s Asynchronous Daemon to “Catch Up”

Marten has had the IDocumentStore.WaitForNonStaleProjectionDataAsync(timeout) API (see the documentation for an example) for quite awhile that lets you pause a test while any running asynchronous projections or subscriptions run and catch up to wherever the event store “high water mark” was when you originally called the method. Hopefully, this lets ongoing background work proceed until the point where it’s now safe for you to proceed to the “Assert” part of your automated tests. As a convenience, this API is also available through extension methods on both IHost and IServiceProvider.

We’ve recently invested time into this API to make it provide much more contextual information about what’s happening asynchronously if the “waiting” does not complete. Specifically, we’ve made the API throw an exception that embeds a table of where every asynchronous projection or subscription ended up at compared to the event store’s “high water mark” (the highest sequential identifier assigned to a persisted event in the database). In this last release we made sure that that textual table also shows any projections or subscriptions that never recorded any process with a sequence of “0” so you can see what did or didn’t happen. We have also changed the API to record any exceptions thrown by the asynchronous daemon (serialization errors? application errors from *your* projection code? database errors?) and have those exceptions piped out in the failure messages when the “WaitFor” API does not successfully complete.

Okay, with all of that out of the way, we also added a completely new, slightly alternative for the asynchronous daemon that just forces the daemon to quickly process all outstanding events through every asynchronous projection or subscription right this second and throw up any exceptions that it encounters. We call this the “catch up” API:

        using var daemon = await theStore.BuildProjectionDaemonAsync();
        await daemon.CatchUpAsync(CancellationToken.None);

This mode is faster and hopefully more reliable than WaitFor***** because it’s happening inline and shortcuts a lot of the normal asynchronous polling and messaging within the normal daemon processing.

There’s also an IHost.CatchUpAsync() or IServiceProvider.CatchUpAsync() convenience method for test usage as well.

Multi Stage Tracked Sessions

I’m obviously biased, but I’d say that Wolverine’s tracked session capability is a killer feature that makes Wolverine stand apart from other messaging tools in the .NET ecosystem and it goes a long way toward making integration testing through Wolverine asynchronous messaging be productive and effective.

But, what if you have a testing scenario where you:

1. Carry out some kind of action (an HTTP request invoked through Alba? publishing a message internally within your application?) that leads to messages being published in Wolverine that might in turn lead to even more messages getting published within your Wolverine system or other tracked systems
2. Along the way, handling one or more commands leads to events being appended to a Marten event store
3. An asynchronously executing projection might append other events or publish messages in Marten’s RaiseSideEffects() capability or an event subscription might in turn publish other Wolverine messages that start up an all new cycle of “when is the system really done with all the work it has started.”

That might sound a little bit contrived, but it reflects real world scenarios I’ve discussed with multiple JasperFx clients in just the past couple weeks. With their help and some input from the community, we came up with this new extension to Wolverine’s “tracked sessions” to also track and wait for work spawned by Marten. Consider this bit of code from the tests for this feature:

var tracked = await _host.TrackActivity()
    
    // This new helper just resets the main Marten store
    // Equivalent to calling IHost.ResetAllMartenDataAsync()
    .ResetAllMartenDataFirst()
    
    .PauseThenCatchUpOnMartenDaemonActivity(CatchUpMode.AndResumeNormally)
    .InvokeMessageAndWaitAsync(new AppendLetters(id, ["AAAACCCCBDEEE", "ABCDECCC", "BBBA", "DDDAE"]));

To add some context, handling the AppendLetters command message appends events to a Marten stream and possibly cascades another Wolverine message that also appends events. At the same time, there are asynchronous projections and event subscriptions that will publish messages through Wolverine as they run. We can now make this kind of testing scenario much more feasible and hopefully reliable (async heavy tests are super prone to being blinking tests) through the usage of the PauseThenCatchUpOnMartenDaemonActivity() extension method from the Wolverine.Marten library.

In the bit of test code above, that API is:

1. Registering a “before” action to pause all async daemon activity before executing the “Act” part of the tracked session which in this case is calling IMessageBus.InvokeAsync() against an AppendLetters command
2. Registering a 2nd stage of the tracked session

When this tracked session is executed, the following sequence happens:

1. The tracked session calls Marten’s ResetAllMartenDataAsync() in the main DocumentStore for the application to effectively rewind the database state down to your defined initial state
2. IMessageBus.InvokeAsync(AppendLetters) is called as the actual “execution” of the tracked session
3. The tracked session is watching everything going on with Wolverine messaging and waits until all “cascaded” messages are complete — and that is recursive. Basically, the tracked session waits until all subsequent messaging activity in the Wolverine application is complete
4. The 2nd stage we registered to “CatchUp” means the tracked session calls Marten’s new “CatchUp” API to force all asynchronous projections and event subscriptions in the system to immediately process all persisted events. This also restarts the tracked session monitoring of any Wolverine messaging activity so that this stage will only complete when all detected Wolverine messaging activity is completed.

By using this new capability inside of the older tracked session feature, we’re able to effectively test from the original message input through any subsequent messages triggered by the original message through asynchronous Marten behavior caused by the original messages which might in turn publish yet more messages through Wolverine.

Long story short, this gives us a reliable way to know when the “Act” part of a test is actually complete and proceed to the “Assert” portion of a test. Moreover, this new feature also tries really hard to bring out some visibility into the asynchronous Marten behavior and the second stage messaging behavior in the case of test failures.

Summary

None of this is particularly easy conceptually, and it’s admittedly here because of relatively hard problems in test automation that you might eventually run into. Selfishly, I needed to get these new features into the hands of a client tomorrow and ran out of time to better document these new features, so you get this braindump blog post.

If it helps, I’m going to talk through these new capabilities a bit more in our next Critter Stack live stream tomorrow (Nov. 6th):

Just to set myself up with some pressure to perform, let me hype up a live stream on Wolverine I’m doing later this week!

I’m doing a live stream on Thursday afternoon (U.S. friendly this time) entitled Vertical Slices the Critter Stack Way based on a fun, meandering talk I did for Houston DNUG and an abbreviated version at Commit Your Code last month.

So, yes, it’s technically about the “Vertical Slice Architecture” in general and specifically with Marten and Wolverine, but more importantly, the special sauce in Wolverine that does more — in my opinion of course — than any other server side .NET application framework to simplify your code and improve testability. In the live stream, I’m going to discuss:

• A little bit about how I think modern layered architecture approaches and “Ports and Adapters” style approaches can sometimes lead to poor results over time
• The qualities of a code base that I think are most important (the ability to reason about the behavior of the code, testability of all sorts, ease of iteration, and modularity)
• How Wolverine’s low code ceremony improves outcomes and the qualities I listed above by reducing layering and shrinking your code into a much tighter vertical slice approach so you can actually see what your system does later on
• Adopting Wolverine’s idiomatic “A-Frame Architecture” approach and “imperative shell, functional core” thinking to improve testability
• A sampling of the ways that Wolverine can hugely simplify data access in simpler scenarios and how it can help you keep more complicated data access much closer to behavioral code so you can actually reason about the cause and effects between those two things. And all of that while happily letting you leverage every bit of power in whatever your database or data access tooling happens to be. Seriously, layering approaches and abstractions that obfuscate the database technologies and queries within your system are a very common source of poor system performance in Onion/Clean Architecture approaches.
• Using Wolverine.HTTP as an alternative AspNetCore Endpoint model and why that’s simpler in the end than any kind of “Mediator” tooling inside of MVC Core or Minimal API
• Wolverine’s adaptive approach to middleware
• The full “Critter Stack” combination with Marten and how that leads to arguably the simplest and cleanest code for CQRS command handlers on the planet
• Wolverine’s goodies for the majority of .NET devs using the venerable EF Core tooling as well

If you’ve never heard of Wolverine or haven’t really paid much attention to it yet, I’m most certainly inviting you to the live stream to give it a chance. If you’ve blown Wolverine off in the past as “yet another messaging tool in .NET,” come find out why that is most certainly not the full story because Wolverine will do much more for you within your application code than other, mere messaging frameworks in .NET or even any of the numerous “Mediator” tools floating around.

In the announcement for the Wolverine 5.0 release last week, I left out a pretty big set of improvements for modular monolith support, specifically in how Wolverine can now work with multiple databases from one service process.

Wolverine works closely with databases for:

• Transactional middleware, which also includes:
• A transactional inbox, outbox, and scheduled messaging (in addition to using native message broker scheduling in other cases)
• End to end multi-tenancy with both Marten “conjoined” tenancy and a database per tenant for both Marten and EF Core
• A database backed “dead letter queue” with “replay” capabilities
• Some miscellaneous storage that helps with Wolverine’s leadership election and agent assignment
• Saga storage

And all of those features are supported for Marten, EF Core with either PostgreSQL or SQL Server, and RavenDb.

Back to the “modular monolith” approach and what I’m seeing folks do or want to do is some combination of:

• Use multiple EF Core DbContext types that target the same database, but maybe with different schemas
• Use Marten’s “ancillary or separated store” feature to divide the storage up for different modules against the same database

Wolverine 3/4 supported the previous two bullet points, but now Wolverine 5 will be able to support any combination of every possible option in the same process. That even includes the ability to:

• Use multiple DbContext types that target completely different databases altogether
• Mix and match with Marten ancillary stores that target completely different database
• Use RavenDb for some modules, even if others use PostgreSQL or SQL Server
• Utilize either Marten’s built in multi-tenancy through a database per tenant or Wolverine’s managed EF Core multi-tenancy through a database per tenant

And now do that in one process while being able to support Wolverine’s transactional inbox, outbox, scheduled messages, and saga support for every single database that the application utilizes. And oh, yeah, from the perspective of the future CritterWatch, you’ll be able to use Wolverine’s dead letter management services against every possible database in the service.

Okay, this is the point where I do have to admit that the RavenDb support for the dead letter administration is lagging a little bit, but we’ll get that hole filled in soon.

Here’s an example from the tests:

        var builder = Host.CreateApplicationBuilder();
        var sqlserver1 = builder.Configuration.GetConnectionString("sqlserver1");
        var sqlserver2 = builder.Configuration.GetConnectionString("sqlserver2");
        var postgresql = builder.Configuration.GetConnectionString("postgresql");

        builder.UseWolverine(opts =>
        {
            // This helps Wolverine "know" how to share inbox/outbox
            // storage across logical module databases where they're
            // sharing the same physical database but with different schemas
            opts.Durability.MessageStorageSchemaName = "wolverine";

            // This will be the "main" store that Wolverine will use
            // for node storage
            opts.Services.AddMarten(m =>
            {
                m.Connection(postgresql);
            }).IntegrateWithWolverine();

            // "An" EF Core module using Wolverine based inbox/outbox storage
            opts.UseEntityFrameworkCoreTransactions();
            opts.Services.AddDbContextWithWolverineIntegration<SampleDbContext>(x => x.UseSqlServer(sqlserver1));
            
            // This is helping Wolverine out by telling it what database to use for inbox/outbox integration
            // when using this DbContext type in handlers or HTTP endpoints
            opts.PersistMessagesWithSqlServer(sqlserver1, role:MessageStoreRole.Ancillary).Enroll<SampleDbContext>();
            
            // Another EF Core module
            opts.Services.AddDbContextWithWolverineIntegration<ItemsDbContext>(x => x.UseSqlServer(sqlserver2));
            opts.PersistMessagesWithSqlServer(sqlserver2, role:MessageStoreRole.Ancillary).Enroll<ItemsDbContext>();

            // Yet another Marten backed module
            opts.Services.AddMartenStore<IFirstStore>(m =>
            {
                m.Connection(postgresql);
                m.DatabaseSchemaName = "first";
            });
        });

I’m certainly not saying that you *should* run out and build a system that has that many different persistence options in a single deployable service, but now you *can* with Wolverine. And folks have definitely wanted to build Wolverine systems that target multiple databases for different modules and still get every bit of Wolverine functionality for each database.

Summary

Part of the Wolverine 5.0 work was also Jeffry Gonzalez and I pushing on JasperFx’s forthcoming “CritterWatch” tool and looking for any kind of breaking changes in the Wolverine “publinternals” that might be necessary to support CritterWatch. The “let’s let you use all the database options at one time!” improvements I tried to show in the post were suggested by the work we are doing for dead letter message management in CritterWatch.

I shudder to think how creative folks are going to be with this mix and match ability, but it’s cool to have some bragging rights over these capabilities because I don’t think that any other .NET tool can match this.

I was the guest speaker today on the .NET Data Community Standup doing a talk on how the “Critter Stack” (Marten, Wolverine, and Weasel) support a style of database migrations and even configuration for messaging brokers that greatly reduces development time friction for more productive teams.

The general theme is “it should just work” so developers and testers can get their work done and even iterate on different approaches without having to spend much time fiddling with database or other infrastructure configuration.

And I also shared some hard lessons learned from previous OSS project failures that made the Critter Stack community so adamant that the default configurations “should just work.”

We’re targeting October 1st for the release of Wolverine 5.0. At this point, I think I’d like to say that we’re not going to be adding any new features to Wolverine 4.* except for JasperFx Software client needs. And also, not that I have any pride about this, I don’t think we’re going to address bugs in 4.* if those bugs do not impact many people.

To catch you up if you want:

• The original vision for Wolverine 5.0
• A big update on ongoing work
• A video preview of some of the big new features, including the “Partitioned Sequential Messaging” and the quite extensive improvements to interoperability
• Sneak Peek at the SignalR Integration in Wolverine 5.0

In addition to the new features we originally envisioned, add in:

• A messaging transport for Redis from the community — which might also lead to a Redis backed saga model some day too, but not for 5.0
• Ongoing work to improve Wolverine’s capability to allow folks to mix and match persistent message stores in “modular monolith” applications
• Working over some of the baked in Dead Letter Queue administration, which is being done in conjunction with ongoing “CritterWatch” work

I think we’re really close to the point where it’s time to play major release triage and push back any enhancements that wouldn’t require any breaking changes to the public API, so anything not yet done or at least started probably slides to a future 5.* minor release. The one exception might be trying to tackle the “cold start optimization.” The wild card in this is that I’m desperately trying to work through as much of the CritterWatch backend plumbing as possible right now as that work is 100% causing some changes and improvements to Wolverine 5.0

What about CritterWatch?

If you understand why the image above appears in this section, I would hope you’d feel some sympathy for me here:-)

I’ve been able to devote some serious time to CritterWatch the past couple weeks, and it’s starting to be “real” after all this time. Jeffry Gonzalez and I will be marrying up the backend and a real frontend in the next couple weeks and who knows, we might be able to demo something to early adopters in about a month or so. After Wolverine 5.0 is out, CritterWatch will be my and JasperFx’s primary technical focus the rest of the year.

Just to rehash, the MVP for CritterWatch is looking like:

1. The basic shell and visualization of what your monitored Critter Stack applications are, including messaging
2. Every possible thing you need to manage Dead Letter Queue messages in Wolverine — but I’d warn you that it’s focused on Wolverine’s database backed DLQ
3. Monitoring and a control panel over Marten event projections and subscriptions and everything you need to keep those running smoothly in production