My internal code name for one of the new features I’m describing is “multi-stage tracked sessions” which somehow got me thinking of the ZZ Top song “Stages” and their Afterburner album because the sound track for getting this work done this week. Not ZZ Top’s best stuff, but there’s still some bangers on it, or at least *I* loved how it sounded on my Dad’s old phonograph player when I was a kid. For what it’s worth, my favorite ZZ Top albums cover to cover are Degüello and their La Futura comeback album.
I was heavily influenced by Extreme Programming in my early career and that’s made me have a very deep appreciation for the quality of “Testability” in the development tools I use and especially for the tools like Marten and Wolverine that I work on. I would say that one of the differentiators for Wolverine over other .NET messaging libraries and application frameworks is its heavy focus and support for automated testing of your application code.
The Critter Stack community released Marten 8.14 and Wolverine 5.1 today with some significant improvements to our testing support. These new features mostly originated from my work with JasperFx Software clients that give me a first hand look into what kinds of challenges our users hit automating tests that involve multiple layers of asynchronous behavior.
Stubbed Message Handlers in Wolverine
The first improvement is Wolverine getting the ability to let you temporarily apply stubbed message handlers to a bootstrapped application in tests. The key driver for this feature is teams that take advantage of Wolverine’s request/reply capabilities through messaging.
Jumping into an example, let’s say that your system interacts with another service that estimates delivery costs for ordering items. At some point in the system you might reach out through a request/reply call in Wolverine to estimate an item delivery before making a purchase like this code:
// This query message is normally sent to an external system through Wolverine
// messaging
public record EstimateDelivery(int ItemId, DateOnly Date, string PostalCode);
// This message type is a response from an external system
public record DeliveryInformation(TimeOnly DeliveryTime, decimal Cost);
public record MaybePurchaseItem(int ItemId, Guid LocationId, DateOnly Date, string PostalCode, decimal BudgetedCost);
public record MakePurchase(int ItemId, Guid LocationId, DateOnly Date);
public record PurchaseRejected(int ItemId, Guid LocationId, DateOnly Date);
public static class MaybePurchaseHandler
{
public static Task<DeliveryInformation> LoadAsync(
MaybePurchaseItem command,
IMessageBus bus,
CancellationToken cancellation)
{
var (itemId, _, date, postalCode, budget) = command;
var estimateDelivery = new EstimateDelivery(itemId, date, postalCode);
// Let's say this is doing a remote request and reply to another system
// through Wolverine messaging
return bus.InvokeAsync<DeliveryInformation>(estimateDelivery, cancellation);
}
public static object Handle(
MaybePurchaseItem command,
DeliveryInformation estimate)
{
if (estimate.Cost <= command.BudgetedCost)
{
return new MakePurchase(command.ItemId, command.LocationId, command.Date);
}
return new PurchaseRejected(command.ItemId, command.LocationId, command.Date);
}
}
And for a little more context, the EstimateDelivery message will always be sent to an external system in this configuration:
var builder = Host.CreateApplicationBuilder();
builder.UseWolverine(opts =>
{
opts
.UseRabbitMq(builder.Configuration.GetConnectionString("rabbit"))
.AutoProvision();
// Just showing that EstimateDelivery is handled by
// whatever system is on the other end of the "estimates" queue
opts.PublishMessage<EstimateDelivery>()
.ToRabbitQueue("estimates");
});
In testing scenarios, maybe the external system isn’t available at all, or it’s just much more challenging to run tests that also include the external system, or maybe you’d just like to write more isolated tests against your service’s behavior before even trying to integrate with the other system (my personal preference anyway). To that end we can now stub the remote handling like this:
public static async Task try_application(IHost host)
{
host.StubWolverineMessageHandling<EstimateDelivery, DeliveryInformation>(
query => new DeliveryInformation(new TimeOnly(17, 0), 1000));
var locationId = Guid.NewGuid();
var itemId = 111;
var expectedDate = new DateOnly(2025, 12, 1);
var postalCode = "78750";
var maybePurchaseItem = new MaybePurchaseItem(itemId, locationId, expectedDate, postalCode,
500);
var tracked =
await host.InvokeMessageAndWaitAsync(maybePurchaseItem);
// The estimated cost from the stub was more than we budgeted
// so this message should have been published
// This line is an assertion too that there was a single message
// of this type published as part of the message handling above
var rejected = tracked.Sent.SingleMessage<PurchaseRejected>();
rejected.ItemId.ShouldBe(itemId);
rejected.LocationId.ShouldBe(locationId);
}
After calling making this call:
host.StubWolverineMessageHandling<EstimateDelivery, DeliveryInformation>(
query => new DeliveryInformation(new TimeOnly(17, 0), 1000));
Calling this from our Wolverine application:
// Let's say this is doing a remote request and reply to another system
// through Wolverine messaging
return bus.InvokeAsync<DeliveryInformation>(estimateDelivery, cancellation);
Will use the stubbed logic we registered. This is enabling you to use fake behavior for difficult to use external services.
For the next test, we can completely remove the stub behavior and revert back to the original configuration like this:
public static void revert_stub(IHost host)
{
// Selectively clear out the stub behavior for only one message
// type
host.WolverineStubs(stubs =>
{
stubs.Clear<EstimateDelivery>();
});
// Or just clear out all active Wolverine message handler
// stubs
host.ClearAllWolverineStubs();
}
There’s a bit more to the feature you can read about in our documentation, but hopefully you can see right away how this can be useful for effectively stubbing out the behavior of external systems through Wolverine in tests.
And yes, some older .NET messaging frameworks already had *this* feature and it’s been occasionally requested from Wolverine, so I’m happy to say we have this important and useful capability.
Forcing Marten’s Asynchronous Daemon to “Catch Up”
Marten has had the IDocumentStore.WaitForNonStaleProjectionDataAsync(timeout) API (see the documentation for an example) for quite awhile that lets you pause a test while any running asynchronous projections or subscriptions run and catch up to wherever the event store “high water mark” was when you originally called the method. Hopefully, this lets ongoing background work proceed until the point where it’s now safe for you to proceed to the “Assert” part of your automated tests. As a convenience, this API is also available through extension methods on both IHost and IServiceProvider.
We’ve recently invested time into this API to make it provide much more contextual information about what’s happening asynchronously if the “waiting” does not complete. Specifically, we’ve made the API throw an exception that embeds a table of where every asynchronous projection or subscription ended up at compared to the event store’s “high water mark” (the highest sequential identifier assigned to a persisted event in the database). In this last release we made sure that that textual table also shows any projections or subscriptions that never recorded any process with a sequence of “0” so you can see what did or didn’t happen. We have also changed the API to record any exceptions thrown by the asynchronous daemon (serialization errors? application errors from *your* projection code? database errors?) and have those exceptions piped out in the failure messages when the “WaitFor” API does not successfully complete.
Okay, with all of that out of the way, we also added a completely new, slightly alternative for the asynchronous daemon that just forces the daemon to quickly process all outstanding events through every asynchronous projection or subscription right this second and throw up any exceptions that it encounters. We call this the “catch up” API:
using var daemon = await theStore.BuildProjectionDaemonAsync();
await daemon.CatchUpAsync(CancellationToken.None);
This mode is faster and hopefully more reliable than WaitFor***** because it’s happening inline and shortcuts a lot of the normal asynchronous polling and messaging within the normal daemon processing.
There’s also an IHost.CatchUpAsync() or IServiceProvider.CatchUpAsync() convenience method for test usage as well.
Multi Stage Tracked Sessions
I’m obviously biased, but I’d say that Wolverine’s tracked session capability is a killer feature that makes Wolverine stand apart from other messaging tools in the .NET ecosystem and it goes a long way toward making integration testing through Wolverine asynchronous messaging be productive and effective.
But, what if you have a testing scenario where you:
- Carry out some kind of action (an HTTP request invoked through Alba? publishing a message internally within your application?) that leads to messages being published in Wolverine that might in turn lead to even more messages getting published within your Wolverine system or other tracked systems
- Along the way, handling one or more commands leads to events being appended to a Marten event store
- An asynchronously executing projection might append other events or publish messages in Marten’s RaiseSideEffects() capability or an event subscription might in turn publish other Wolverine messages that start up an all new cycle of “when is the system really done with all the work it has started.”
That might sound a little bit contrived, but it reflects real world scenarios I’ve discussed with multiple JasperFx clients in just the past couple weeks. With their help and some input from the community, we came up with this new extension to Wolverine’s “tracked sessions” to also track and wait for work spawned by Marten. Consider this bit of code from the tests for this feature:
var tracked = await _host.TrackActivity()
// This new helper just resets the main Marten store
// Equivalent to calling IHost.ResetAllMartenDataAsync()
.ResetAllMartenDataFirst()
.PauseThenCatchUpOnMartenDaemonActivity(CatchUpMode.AndResumeNormally)
.InvokeMessageAndWaitAsync(new AppendLetters(id, ["AAAACCCCBDEEE", "ABCDECCC", "BBBA", "DDDAE"]));
To add some context, handling the AppendLetters command message appends events to a Marten stream and possibly cascades another Wolverine message that also appends events. At the same time, there are asynchronous projections and event subscriptions that will publish messages through Wolverine as they run. We can now make this kind of testing scenario much more feasible and hopefully reliable (async heavy tests are super prone to being blinking tests) through the usage of the PauseThenCatchUpOnMartenDaemonActivity() extension method from the Wolverine.Marten library.
In the bit of test code above, that API is:
- Registering a “before” action to pause all async daemon activity before executing the “Act” part of the tracked session which in this case is calling
IMessageBus.InvokeAsync()against anAppendLetterscommand - Registering a 2nd stage of the tracked session
When this tracked session is executed, the following sequence happens:
- The tracked session calls Marten’s
ResetAllMartenDataAsync()in the mainDocumentStorefor the application to effectively rewind the database state down to your defined initial state IMessageBus.InvokeAsync(AppendLetters)is called as the actual “execution” of the tracked session- The tracked session is watching everything going on with Wolverine messaging and waits until all “cascaded” messages are complete — and that is recursive. Basically, the tracked session waits until all subsequent messaging activity in the Wolverine application is complete
- The 2nd stage we registered to “CatchUp” means the tracked session calls Marten’s new “CatchUp” API to force all asynchronous projections and event subscriptions in the system to immediately process all persisted events. This also restarts the tracked session monitoring of any Wolverine messaging activity so that this stage will only complete when all detected Wolverine messaging activity is completed.
By using this new capability inside of the older tracked session feature, we’re able to effectively test from the original message input through any subsequent messages triggered by the original message through asynchronous Marten behavior caused by the original messages which might in turn publish yet more messages through Wolverine.
Long story short, this gives us a reliable way to know when the “Act” part of a test is actually complete and proceed to the “Assert” portion of a test. Moreover, this new feature also tries really hard to bring out some visibility into the asynchronous Marten behavior and the second stage messaging behavior in the case of test failures.
Summary
None of this is particularly easy conceptually, and it’s admittedly here because of relatively hard problems in test automation that you might eventually run into. Selfishly, I needed to get these new features into the hands of a client tomorrow and ran out of time to better document these new features, so you get this braindump blog post.
If it helps, I’m going to talk through these new capabilities a bit more in our next Critter Stack live stream tomorrow (Nov. 6th):
The Shade Tree Developer 