That’s of course supposed to be a 1992 Ford Mustang GT with the 5.0L V8 that high school age me thought was the coolest car I could imagine ever owning (I most certainly never did of course). Queue “Ice, Ice Baby” and sing “rolling, in my 5.0” in your head because here we go…

Wolverine 5.0 went live on Nuget earlier today after about three months of pretty intensive development from *20* different contributors with easily that many more folks having contributed to discussions and GitHub issues that helped get us here. I’m just not going to be able to list everyone, so let me just thank the very supportive Wolverine community, the 19 other contributors, and the JasperFx clients who contributed to this release.

This release came closely on the heels of Wolverine 4.0 earlier this year, with the primary reasons for a new major version release being:

• A big change in the internals as we replaced the venerable TPL DataFlow library with the System.Threading.Channels library in every place that Wolverine uses in memory queueing. We did this as a precursor to a hugely important new feature commissioned by a JasperFx Software client (who really needs to get that feature in for their “scale out” so it was definitely about time I got this out today).
• Some breaking API changes in the “publinternals” of Wolverine to support “CritterWatch”, our long planned and I promise finally in real development add on tooling for Critter Stack observability and management

With that being said, the top line new changes to Wolverine that I’ll be trying to blog about next week are:

• The new Partitioned Sequential Messaging feature is a potentially huge step forward for building a Wolverine system that can efficiently and resiliently handle concurrent access to sensitive resources.
• A new SignalR messaging transport for Wolverine (we needed this for CritterWatch, but it’s also been requested a bit over the years).
• A new Redis backed messaging transport using Redis streams from the community
• New interoperability capabilities for Wolverine including more options for MassTransit, NServiceBus, and CloudEvents usage on the other side
• The GCP Pubsub transport is more robust and suitable now for production usage

For a partial list of significant, smaller improvements:

• Wolverine can utilize Marten batch querying for the declarative data access, and that includes working with multiple Marten event streams in one logical operation. This is part of the Critter Stack’s response to the “Dynamic Consistency Boundary” idea from some of the commercial event sourcing tools
• You can finally use strong typed identifiers with the “aggregate handler workflow”
• An overhaul of the dead letter queue administration services that was part of our ongoing work for CritterWatch
• A new tutorial for dealing with concurrency when building against Wolverine
• Optimistic concurrency support for EF Core backed Sagas from the community
• Ability to target multiple Azure Service Bus namespaces from a single application and improvements to using Azure Service Bus namespace per tenant
• Improvements to Rabbit MQ for advanced usage

What’s Next?

As happens basically every time, several features that were planned for 5.0 and some significant open issues didn’t make the 5.0 cut. The bigger effort to optimize the cold start time for both Marten and Wolverine will hopefully happen later this year. I think the next minor point release will target some open issues around Wolverine.HTTP (multi-part uploads, actual content negotiation) and the Kafka transport. I would like to take a longer look sometime at how the CritterStack combination can better support operations that cross stream boundaries.

But in the meantime, I’m shifting to open Marten issues before hopefully spending a couple weeks trying to jump start CritterWatch development again.

I usually end these kinds of major release announcements with a link to Don’t Steal My Sunshine as an exhortation to hold off on reporting problems or asking for whatever didn’t make the release. After referring to “Ice, Ice Baby” in the preface to this and probably getting that bass line stuck in your head, here’s the song you want to hear now anyway — which I feel much less of after getting this damn release out:

Little update since the last check in on Wolverine 5.0. I think right now that Wolverine 5.0 hits by next Monday (October 6th). To be honest, besides documentation updates, the biggest work is just pushing more on the CritterWatch backend this week to see if that forces any breaking changes in the Wolverine internals.

What’s definitely “in” right now:

• The “Partitioned Sequential Messaging” feature work that we previewed in a live stream
• Big improvements and expansion to Wolverine’s interoperability story against NServiceBus, MassTransit, CloudEvents, and whatever custom interoperability folks need to do
• A first class SignalR transport which is getting used heavily in our own CritterWatch work
• A first class Redis messaging transport from the community
• Modernization and upgrades to the GCP Pubsub transport
• The ability to mix and match database storage with Wolverine for modular monoliths
• A bit batch of optimization for the Marten integration including improvements for multi-stream operations as our response to the “Dynamic Boundary Consistency” idea from other tools
• The utilization of System.Threading.Channels in place of the TPL DataFlow library

What’s unfortunately out:

• Any effort to optimize the cold start times for Marten and Wolverine. Just a bandwidth problem, plus I think this can get done without breaking changes

And we’ll see:

• Random improvements for Azure Service Bus and Kafka usage
• HTTP improvements for content negotiation and multi-part uploads
• Yet more improvements to the “aggregate handler workflow” with Marten to allow for yet more strong typed identifier usage

The items in the 3rd list don’t require any breaking changes, so could slide to Wolverine 5.1 if necessary.

All in all, I’d argue this turned out to be a big batch of improvements with very few breaking API changes and almost nothing that would impact the average user.

Wolverine has long had some ability to schedule some messages to be executed at a later time or schedule messages to be delivered at a later time. The Wolverine 4.12 release last night added some probably overdue test automation helpers to better deal with scheduled messaging within integration testing against Wolverine applications — and that makes now a perfectly good time to talk about this capability within Wolverine!

First, let’s say that we’re just using Wolverine locally within the current system with a setup like this:

var builder = Host.CreateApplicationBuilder();
builder.Services.AddWolverine(opts =>
{
    // The only thing that matters here is that you have *some* kind of 
    // envelope persistence for Wolverine configured for your application
    var connectionString = builder.Configuration.GetConnectionString("postgres");
    opts.PersistMessagesWithPostgresql(connectionString);
});

The only point being that we have some kind of message persistence set up in our Wolverine application because the message or execution scheduling depends on persisted envelope storage.

Wolverine actually does support in memory scheduling without any persistence, but that’s really only useful for scheduled error handling or fire and forget type semantics because you’d lose everything if the process is stopped.

So now let’s move on to simply telling Wolverine to execute a message locally at a later time with the IMessageBus service:

public static async Task use_message_bus(IMessageBus bus)
{
    // Send a message to be sent or executed at a specific time
    await bus.SendAsync(new DebitAccount(1111, 100),
        new(){ ScheduledTime = DateTimeOffset.UtcNow.AddDays(1) });
    
    // Same mechanics w/ some syntactical sugar
    await bus.ScheduleAsync(new DebitAccount(1111, 100), DateTimeOffset.UtcNow.AddDays(1));

    // Or do the same, but this time express the time as a delay
    await bus.SendAsync(new DebitAccount(1111, 225), new() { ScheduleDelay = 1.Days() });
    
    // And the same with the syntactic sugar
    await bus.ScheduleAsync(new DebitAccount(1111, 225), 1.Days());
}

In the system above, all messages are being handled locally. To actually process the scheduled messages, Wolverine is as you’ve probably guessed, polling the message storage (PostgreSQL in the case above), and looking for any messages that are ready to be played. Here’s a few notes on the mechanics:

• Every node within a cluster is trying to pull in scheduled messages, but there’s some randomness in the timing to keep every node from stomping on each other
• Any one node will only pull in a limited “page” of scheduled jobs at a time so that if you happen to be going bonkers scheduling thousands of messages at one time, Wolverine can share the load across nodes and keep any one node from blowing up
• The scheduled messages are in Wolverine’s transactional inbox storage with a Scheduled status. When Wolverine decides to “play” the messages, they move to an Incoming status before finally getting marked as Handled when they are successful
• When scheduled messages for local execution are “played” in a Wolverine node, they are put into the local queue for that message, so all the normal rules for ordering or parallelization for that queue still apply.

Now, let’s move on to scheduling message delivery to external brokers. Just say you have any external routing rules like this:

using var host = await Host.CreateDefaultBuilder()
    .UseWolverine(opts =>
    {
        opts.UseRabbitMq()
            // Opt into conventional Rabbit MQ routing
            .UseConventionalRouting();
    }).StartAsync();

And go back to the same syntax for sending messages, but this time the message will get routed to a Rabbit MQ exchange:

    await bus.ScheduleAsync(new DebitAccount(1111, 100), DateTimeOffset.UtcNow.AddDays(1));

This time, Wolverine is still using its transactional inbox, but with a twist. When Wolverine knows that it is scheduling message delivery to an outside messaging mechanism, it actually schedules a local ScheduledEnvelope message that when executed, sends the original message to the outbound delivery point. In this way, Wolverine is able to support scheduled message delivery to every single messaging transport that Wolverine supports with a common mechanism.

With idiomatic Wolverine usage, you do want to try to keep most of your handler methods as “pure functions” for easier testing and frankly less code noise due to async/await mechanics. To that end, there’s a couple helpers to schedule messages in Wolverine using its cascading messages syntax:

public IEnumerable<object> Consume(MyMessage message)
{
    // Go West in an hour
    yield return new GoWest().DelayedFor(1.Hours());

    // Go East at midnight local time
    yield return new GoEast().ScheduledAt(DateTime.Today.AddDays(1));
}

The extension methods above would give you the raw message wrapped in a Wolverine DeliveryMessage<T> object where T is the wrapped message type. You can still use that type to write assertions in your unit tests.

There’s also another helper called “timeout messages” that help you create scheduled messages by subclassing a Wolverine base class. This is largely associated with sagas just because it’s commonly a need for timing out saga workflows.

Error Handling

The scheduled message support is also useful in error handling. Consider this code:

using var host = await Host.CreateDefaultBuilder()
    .UseWolverine(opts =>
    {
        opts.Policies.OnException<TimeoutException>().ScheduleRetry(5.Seconds());
        opts.Policies.OnException<SecurityException>().MoveToErrorQueue();

        // You can also apply an additional filter on the
        // exception type for finer grained policies
        opts.Policies
            .OnException<SocketException>(ex => ex.Message.Contains("not responding"))
            .ScheduleRetry(5.Seconds());
    }).StartAsync();

In the case above, Wolverine uses the message scheduling to take a message that just failed, move it out of the current receiving endpoint so other messages can proceed, then retries it no sooner than 5 seconds later (it won’t be real time perfect on the timing). This is an important difference than the RetryWithCooldown() mechanism that is effectively just doing an await Task.Delay(timespan) inline to purposely slow down the application.

As an example of how this might be useful, I’ve had to work with 3rd party systems where users can create a pessimistic lock on a bank account, so any commands against that account would always fail because of that lock. If you can tell that the command failure is because of a pessimistic lock in the exception message, you might tell Wolverine to retry that message an hour later when hopefully the lock is released, but clear out the current receiving endpoint and/or queue for other work that can proceed.

Testing with Scheduled Messaging

We’re having some trouble with the documentation publishing for some reason that we haven’t figured out yet, but there will be docs soon on this new feature.

Finally, on to some new functionality! Wolverine 4.12 just added some improvements to Wolverine’s tracked session testing feature specifically to help you with scheduled messages.

First, for some background, let’s say you have these simple handlers:

public static DeliveryMessage<ScheduledMessage> Handle(TriggerScheduledMessage message)
{
    // This causes a message to be scheduled for delivery in 5 minutes from now
    return new ScheduledMessage(message.Text).DelayedFor(5.Minutes());
}

public static void Handle(ScheduledMessage message) => Debug.WriteLine("Got scheduled message");

And now this test using the tracked session which shows the new first class support for scheduled messaging:

[Fact]
public async Task deal_with_locally_scheduled_execution()
{
    // In this case we're just executing everything in memory
    using var host = await Host.CreateDefaultBuilder()
        .UseWolverine(opts =>
        {
            opts.PersistMessagesWithPostgresql(Servers.PostgresConnectionString, "wolverine");
            opts.Policies.UseDurableInboxOnAllListeners();
        }).StartAsync();

    // Should finish cleanly, even though there's going to be a message that is scheduled
    // and doesn't complete
    var tracked = await host.SendMessageAndWaitAsync(new TriggerScheduledMessage("Chiefs"));
    
    // Here's how you can query against the messages that were detected to be scheduled
    tracked.Scheduled.SingleMessage<ScheduledMessage>()
        .Text.ShouldBe("Chiefs");

    // This API will try to immediately play any scheduled messages immediately
    var replayed = await tracked.PlayScheduledMessagesAsync(10.Seconds());
    replayed.Executed.SingleMessage<ScheduledMessage>().Text.ShouldBe("Chiefs");
}

And a similar test, but this time where the scheduled messages are being routed externally:

var port1 = PortFinder.GetAvailablePort();
var port2 = PortFinder.GetAvailablePort();

using var sender = await Host.CreateDefaultBuilder()
    .UseWolverine(opts =>
    {
        opts.PublishMessage<ScheduledMessage>().ToPort(port2);
        opts.ListenAtPort(port1);
    }).StartAsync();

using var receiver = await Host.CreateDefaultBuilder()
    .UseWolverine(opts =>
    {
        opts.ListenAtPort(port2);
    }).StartAsync();

// Should finish cleanly
var tracked = await sender
    .TrackActivity()
    .IncludeExternalTransports()
    .AlsoTrack(receiver)
    .InvokeMessageAndWaitAsync(new TriggerScheduledMessage("Broncos"));

tracked.Scheduled.SingleMessage<ScheduledMessage>()
    .Text.ShouldBe("Broncos");

var replayed = await tracked.PlayScheduledMessagesAsync(10.Seconds());
replayed.Executed.SingleMessage<ScheduledMessage>().Text.ShouldBe("Broncos");

Here’s what’s new in the code above:

1. ITrackedSession.Scheduled is a bit special collection of all activity that happened during the tracked activity that led to messages being scheduled. You can use this just to interrogate what scheduled messages resulted from the original activity.
2. ITrackedSession.PlayScheduledMessagesAsync() will “play” all scheduled messages right now and return a new ITrackedSession for those messages. This method will immediately execute any messages that were scheduled for local execution and tries to immediately send any messages that were scheduled for later delivery to external transports.

The new support in the existing tracked session feature further extends Wolverine’s already extensive test automation story. This new work was done at the behest of a JasperFx Software client who is quite aggressive in their test automation. Certainly reach out to us at sales@jasperfx.net for any help you might want with your own efforts!