That’s of course supposed to be a 1992 Ford Mustang GT with the 5.0L V8 that high school age me thought was the coolest car I could imagine ever owning (I most certainly never did of course). Queue “Ice, Ice Baby” and sing “rolling, in my 5.0” in your head because here we go…

Wolverine 5.0 went live on Nuget earlier today after about three months of pretty intensive development from *20* different contributors with easily that many more folks having contributed to discussions and GitHub issues that helped get us here. I’m just not going to be able to list everyone, so let me just thank the very supportive Wolverine community, the 19 other contributors, and the JasperFx clients who contributed to this release.

This release came closely on the heels of Wolverine 4.0 earlier this year, with the primary reasons for a new major version release being:

• A big change in the internals as we replaced the venerable TPL DataFlow library with the System.Threading.Channels library in every place that Wolverine uses in memory queueing. We did this as a precursor to a hugely important new feature commissioned by a JasperFx Software client (who really needs to get that feature in for their “scale out” so it was definitely about time I got this out today).
• Some breaking API changes in the “publinternals” of Wolverine to support “CritterWatch”, our long planned and I promise finally in real development add on tooling for Critter Stack observability and management

With that being said, the top line new changes to Wolverine that I’ll be trying to blog about next week are:

• The new Partitioned Sequential Messaging feature is a potentially huge step forward for building a Wolverine system that can efficiently and resiliently handle concurrent access to sensitive resources.
• A new SignalR messaging transport for Wolverine (we needed this for CritterWatch, but it’s also been requested a bit over the years).
• A new Redis backed messaging transport using Redis streams from the community
• New interoperability capabilities for Wolverine including more options for MassTransit, NServiceBus, and CloudEvents usage on the other side
• The GCP Pubsub transport is more robust and suitable now for production usage

For a partial list of significant, smaller improvements:

• Wolverine can utilize Marten batch querying for the declarative data access, and that includes working with multiple Marten event streams in one logical operation. This is part of the Critter Stack’s response to the “Dynamic Consistency Boundary” idea from some of the commercial event sourcing tools
• You can finally use strong typed identifiers with the “aggregate handler workflow”
• An overhaul of the dead letter queue administration services that was part of our ongoing work for CritterWatch
• A new tutorial for dealing with concurrency when building against Wolverine
• Optimistic concurrency support for EF Core backed Sagas from the community
• Ability to target multiple Azure Service Bus namespaces from a single application and improvements to using Azure Service Bus namespace per tenant
• Improvements to Rabbit MQ for advanced usage

What’s Next?

As happens basically every time, several features that were planned for 5.0 and some significant open issues didn’t make the 5.0 cut. The bigger effort to optimize the cold start time for both Marten and Wolverine will hopefully happen later this year. I think the next minor point release will target some open issues around Wolverine.HTTP (multi-part uploads, actual content negotiation) and the Kafka transport. I would like to take a longer look sometime at how the CritterStack combination can better support operations that cross stream boundaries.

But in the meantime, I’m shifting to open Marten issues before hopefully spending a couple weeks trying to jump start CritterWatch development again.

I usually end these kinds of major release announcements with a link to Don’t Steal My Sunshine as an exhortation to hold off on reporting problems or asking for whatever didn’t make the release. After referring to “Ice, Ice Baby” in the preface to this and probably getting that bass line stuck in your head, here’s the song you want to hear now anyway — which I feel much less of after getting this damn release out:

Little update since the last check in on Wolverine 5.0. I think right now that Wolverine 5.0 hits by next Monday (October 6th). To be honest, besides documentation updates, the biggest work is just pushing more on the CritterWatch backend this week to see if that forces any breaking changes in the Wolverine internals.

What’s definitely “in” right now:

• The “Partitioned Sequential Messaging” feature work that we previewed in a live stream
• Big improvements and expansion to Wolverine’s interoperability story against NServiceBus, MassTransit, CloudEvents, and whatever custom interoperability folks need to do
• A first class SignalR transport which is getting used heavily in our own CritterWatch work
• A first class Redis messaging transport from the community
• Modernization and upgrades to the GCP Pubsub transport
• The ability to mix and match database storage with Wolverine for modular monoliths
• A bit batch of optimization for the Marten integration including improvements for multi-stream operations as our response to the “Dynamic Boundary Consistency” idea from other tools
• The utilization of System.Threading.Channels in place of the TPL DataFlow library

What’s unfortunately out:

• Any effort to optimize the cold start times for Marten and Wolverine. Just a bandwidth problem, plus I think this can get done without breaking changes

And we’ll see:

• Random improvements for Azure Service Bus and Kafka usage
• HTTP improvements for content negotiation and multi-part uploads
• Yet more improvements to the “aggregate handler workflow” with Marten to allow for yet more strong typed identifier usage

The items in the 3rd list don’t require any breaking changes, so could slide to Wolverine 5.1 if necessary.

All in all, I’d argue this turned out to be a big batch of improvements with very few breaking API changes and almost nothing that would impact the average user.

Wolverine has long had some ability to schedule some messages to be executed at a later time or schedule messages to be delivered at a later time. The Wolverine 4.12 release last night added some probably overdue test automation helpers to better deal with scheduled messaging within integration testing against Wolverine applications — and that makes now a perfectly good time to talk about this capability within Wolverine!

First, let’s say that we’re just using Wolverine locally within the current system with a setup like this:

var builder = Host.CreateApplicationBuilder();
builder.Services.AddWolverine(opts =>
{
    // The only thing that matters here is that you have *some* kind of 
    // envelope persistence for Wolverine configured for your application
    var connectionString = builder.Configuration.GetConnectionString("postgres");
    opts.PersistMessagesWithPostgresql(connectionString);
});

The only point being that we have some kind of message persistence set up in our Wolverine application because the message or execution scheduling depends on persisted envelope storage.

Wolverine actually does support in memory scheduling without any persistence, but that’s really only useful for scheduled error handling or fire and forget type semantics because you’d lose everything if the process is stopped.

So now let’s move on to simply telling Wolverine to execute a message locally at a later time with the IMessageBus service:

public static async Task use_message_bus(IMessageBus bus)
{
    // Send a message to be sent or executed at a specific time
    await bus.SendAsync(new DebitAccount(1111, 100),
        new(){ ScheduledTime = DateTimeOffset.UtcNow.AddDays(1) });
    
    // Same mechanics w/ some syntactical sugar
    await bus.ScheduleAsync(new DebitAccount(1111, 100), DateTimeOffset.UtcNow.AddDays(1));

    // Or do the same, but this time express the time as a delay
    await bus.SendAsync(new DebitAccount(1111, 225), new() { ScheduleDelay = 1.Days() });
    
    // And the same with the syntactic sugar
    await bus.ScheduleAsync(new DebitAccount(1111, 225), 1.Days());
}

In the system above, all messages are being handled locally. To actually process the scheduled messages, Wolverine is as you’ve probably guessed, polling the message storage (PostgreSQL in the case above), and looking for any messages that are ready to be played. Here’s a few notes on the mechanics:

• Every node within a cluster is trying to pull in scheduled messages, but there’s some randomness in the timing to keep every node from stomping on each other
• Any one node will only pull in a limited “page” of scheduled jobs at a time so that if you happen to be going bonkers scheduling thousands of messages at one time, Wolverine can share the load across nodes and keep any one node from blowing up
• The scheduled messages are in Wolverine’s transactional inbox storage with a Scheduled status. When Wolverine decides to “play” the messages, they move to an Incoming status before finally getting marked as Handled when they are successful
• When scheduled messages for local execution are “played” in a Wolverine node, they are put into the local queue for that message, so all the normal rules for ordering or parallelization for that queue still apply.

Now, let’s move on to scheduling message delivery to external brokers. Just say you have any external routing rules like this:

using var host = await Host.CreateDefaultBuilder()
    .UseWolverine(opts =>
    {
        opts.UseRabbitMq()
            // Opt into conventional Rabbit MQ routing
            .UseConventionalRouting();
    }).StartAsync();

And go back to the same syntax for sending messages, but this time the message will get routed to a Rabbit MQ exchange:

    await bus.ScheduleAsync(new DebitAccount(1111, 100), DateTimeOffset.UtcNow.AddDays(1));

This time, Wolverine is still using its transactional inbox, but with a twist. When Wolverine knows that it is scheduling message delivery to an outside messaging mechanism, it actually schedules a local ScheduledEnvelope message that when executed, sends the original message to the outbound delivery point. In this way, Wolverine is able to support scheduled message delivery to every single messaging transport that Wolverine supports with a common mechanism.

With idiomatic Wolverine usage, you do want to try to keep most of your handler methods as “pure functions” for easier testing and frankly less code noise due to async/await mechanics. To that end, there’s a couple helpers to schedule messages in Wolverine using its cascading messages syntax:

public IEnumerable<object> Consume(MyMessage message)
{
    // Go West in an hour
    yield return new GoWest().DelayedFor(1.Hours());

    // Go East at midnight local time
    yield return new GoEast().ScheduledAt(DateTime.Today.AddDays(1));
}

The extension methods above would give you the raw message wrapped in a Wolverine DeliveryMessage<T> object where T is the wrapped message type. You can still use that type to write assertions in your unit tests.

There’s also another helper called “timeout messages” that help you create scheduled messages by subclassing a Wolverine base class. This is largely associated with sagas just because it’s commonly a need for timing out saga workflows.

Error Handling

The scheduled message support is also useful in error handling. Consider this code:

using var host = await Host.CreateDefaultBuilder()
    .UseWolverine(opts =>
    {
        opts.Policies.OnException<TimeoutException>().ScheduleRetry(5.Seconds());
        opts.Policies.OnException<SecurityException>().MoveToErrorQueue();

        // You can also apply an additional filter on the
        // exception type for finer grained policies
        opts.Policies
            .OnException<SocketException>(ex => ex.Message.Contains("not responding"))
            .ScheduleRetry(5.Seconds());
    }).StartAsync();

In the case above, Wolverine uses the message scheduling to take a message that just failed, move it out of the current receiving endpoint so other messages can proceed, then retries it no sooner than 5 seconds later (it won’t be real time perfect on the timing). This is an important difference than the RetryWithCooldown() mechanism that is effectively just doing an await Task.Delay(timespan) inline to purposely slow down the application.

As an example of how this might be useful, I’ve had to work with 3rd party systems where users can create a pessimistic lock on a bank account, so any commands against that account would always fail because of that lock. If you can tell that the command failure is because of a pessimistic lock in the exception message, you might tell Wolverine to retry that message an hour later when hopefully the lock is released, but clear out the current receiving endpoint and/or queue for other work that can proceed.

Testing with Scheduled Messaging

We’re having some trouble with the documentation publishing for some reason that we haven’t figured out yet, but there will be docs soon on this new feature.

Finally, on to some new functionality! Wolverine 4.12 just added some improvements to Wolverine’s tracked session testing feature specifically to help you with scheduled messages.

First, for some background, let’s say you have these simple handlers:

public static DeliveryMessage<ScheduledMessage> Handle(TriggerScheduledMessage message)
{
    // This causes a message to be scheduled for delivery in 5 minutes from now
    return new ScheduledMessage(message.Text).DelayedFor(5.Minutes());
}

public static void Handle(ScheduledMessage message) => Debug.WriteLine("Got scheduled message");

And now this test using the tracked session which shows the new first class support for scheduled messaging:

[Fact]
public async Task deal_with_locally_scheduled_execution()
{
    // In this case we're just executing everything in memory
    using var host = await Host.CreateDefaultBuilder()
        .UseWolverine(opts =>
        {
            opts.PersistMessagesWithPostgresql(Servers.PostgresConnectionString, "wolverine");
            opts.Policies.UseDurableInboxOnAllListeners();
        }).StartAsync();

    // Should finish cleanly, even though there's going to be a message that is scheduled
    // and doesn't complete
    var tracked = await host.SendMessageAndWaitAsync(new TriggerScheduledMessage("Chiefs"));
    
    // Here's how you can query against the messages that were detected to be scheduled
    tracked.Scheduled.SingleMessage<ScheduledMessage>()
        .Text.ShouldBe("Chiefs");

    // This API will try to immediately play any scheduled messages immediately
    var replayed = await tracked.PlayScheduledMessagesAsync(10.Seconds());
    replayed.Executed.SingleMessage<ScheduledMessage>().Text.ShouldBe("Chiefs");
}

And a similar test, but this time where the scheduled messages are being routed externally:

var port1 = PortFinder.GetAvailablePort();
var port2 = PortFinder.GetAvailablePort();

using var sender = await Host.CreateDefaultBuilder()
    .UseWolverine(opts =>
    {
        opts.PublishMessage<ScheduledMessage>().ToPort(port2);
        opts.ListenAtPort(port1);
    }).StartAsync();

using var receiver = await Host.CreateDefaultBuilder()
    .UseWolverine(opts =>
    {
        opts.ListenAtPort(port2);
    }).StartAsync();

// Should finish cleanly
var tracked = await sender
    .TrackActivity()
    .IncludeExternalTransports()
    .AlsoTrack(receiver)
    .InvokeMessageAndWaitAsync(new TriggerScheduledMessage("Broncos"));

tracked.Scheduled.SingleMessage<ScheduledMessage>()
    .Text.ShouldBe("Broncos");

var replayed = await tracked.PlayScheduledMessagesAsync(10.Seconds());
replayed.Executed.SingleMessage<ScheduledMessage>().Text.ShouldBe("Broncos");

Here’s what’s new in the code above:

1. ITrackedSession.Scheduled is a bit special collection of all activity that happened during the tracked activity that led to messages being scheduled. You can use this just to interrogate what scheduled messages resulted from the original activity.
2. ITrackedSession.PlayScheduledMessagesAsync() will “play” all scheduled messages right now and return a new ITrackedSession for those messages. This method will immediately execute any messages that were scheduled for local execution and tries to immediately send any messages that were scheduled for later delivery to external transports.

The new support in the existing tracked session feature further extends Wolverine’s already extensive test automation story. This new work was done at the behest of a JasperFx Software client who is quite aggressive in their test automation. Certainly reach out to us at sales@jasperfx.net for any help you might want with your own efforts!

Earlier this week I did a live stream on the upcoming Wolverine 5.0 release where I just lightly touched on the concept for our planned SignalR integration with Wolverine. While there wasn’t that much to show yesterday, a big pull request just landed and I think the APIs and the approach has gelled enough that it’s worth a sneak peak.

First though, the new SignalR transport in Wolverine is being built now to support our planned “CritterWatch” tool shown below:

As it’s planned out right now, the “CritterWatch” server application communicating via SignalR to constantly push updated information to any open browser dashboards about system performance. On the other side of things, CritterWatch users will be able to submit quite a number of commands or queries from the browser to CritterWatch, when will then have to relay commands and queries to the various “Critter Stack” applications being monitored through asynchronous messaging. And of course, we expect the responses or status updates to be constantly flowing from the monitored services to CritterWatch which will then relay information or updates to the browsers, again by SignalR.

Long story short, there’s going to be a lot of asynchronous messaging back and forth between the three logical applications above, and this is where a new SignalR transport for Wolverine comes into play. Having the SignalR transport gives us a standardized way to send a number of different logical messages from the browser to the server and take advantage of everything that the normal Wolverine execution pipeline gives us, including relatively clean handler code compared to other messaging or “mediator” tools, baked in observability and traceability, and Wolverine’s error resiliency. Going back the other way, the SignalR transport gives us a standardized way to publish information right back to the client from our server.

Enough of that, let’s jump into some code. From the integration testing code, let’s say we’ve got a small web app configured like this:

var builder = WebApplication.CreateBuilder();

builder.WebHost.ConfigureKestrel(opts =>
{
    opts.ListenLocalhost(Port);
});

// Note to self: take care of this in the call
// to UseSignalR() below
builder.Services.AddSignalR();
builder.Host.UseWolverine(opts =>
{
    opts.ServiceName = "Server";
    
    // Hooking up the SignalR messaging transport
    // in Wolverine
    opts.UseSignalR();

    // These are just some messages I was using
    // to do end to end testing
    opts.PublishMessage<FromFirst>().ToSignalR();
    opts.PublishMessage<FromSecond>().ToSignalR();
    opts.PublishMessage<Information>().ToSignalR();
});

var app = builder.Build();

// Syntactic sure, really just doing:
// app.MapHub<WolverineHub>("/messages");
app.MapWolverineSignalRHub();

await app.StartAsync();

// Remember this, because I'm going to use it in test code
// later
theWebApp = app;

With that configuration, when you call IMessageBus.PublishAsync(new Information("here's something you should know")); in your system, Wolverine will be routing that message through SignalR, where it will be received in a client with the default “ReceiveMessage” operation. The JSON delivered to the client will be wrapped with the CloudEvents specification like this:

Likewise, Wolverine will expect messages posted to the server from the browser to be embedded in that lightweight CloudEvents compliant wrapper.

We are not coincidentally adding CloudEvents support for extended interoperability in Wolverine 5.0 as well.

For testing, the new WolverineFx.SignalR Nuget will also have a separate messaging transport using the SignalR Client just to facilitate testing, and you can see that usage in some of the testing code:

// This starts up a new host to act as a client to the SignalR
// server for testing
public async Task<IHost> StartClientHost(string serviceName = "Client")
{
    var host = await Host.CreateDefaultBuilder()
        .UseWolverine(opts =>
        {
            opts.ServiceName = serviceName;
            
            // Just pointing at the port where Kestrel is
            // hosting our server app that is running
            // SignalR
            opts.UseClientToSignalR(Port);
            
            opts.PublishMessage<ToFirst>().ToSignalRWithClient(Port);
            
            opts.PublishMessage<RequiresResponse>().ToSignalRWithClient(Port);
            
            opts.Publish(x =>
            {
                x.MessagesImplementing<WebSocketMessage>();
                x.ToSignalRWithClient(Port);
            });
        }).StartAsync();
    
    _clientHosts.Add(host);

    return host;
}

And now to show a little Wolverine-esque spin, let’s say that you have a handler being invoked by a browser sending a message through SignalR to a Wolverine server application, and as part of that handler, you need to send a response message right back to the original calling SignalR connection to the right browser instance.

Conveniently enough, you have this helper to do exactly that in a pretty declarative way:

    public static ResponseToCallingWebSocket<WebSocketResponse> Handle(RequiresResponse msg) 
        => new WebSocketResponse(msg.Name).RespondToCallingWebSocket();

And just for fun, here’s the test that proves the above code works:

[Fact]
public async Task send_to_the_originating_connection()
{
    var green = await StartClientHost("green");
    var red = await StartClientHost("red");
    var blue = await StartClientHost("blue");

    var tracked = await red.TrackActivity()
        .IncludeExternalTransports()
        .AlsoTrack(theWebApp)
        .SendMessageAndWaitAsync(new RequiresResponse("Leo Chenal"));

    var record = tracked.Executed.SingleRecord<WebSocketResponse>();
    
    // Verify that the response went to the original calling client
    record.ServiceName.ShouldBe("red");
    record.Message.ShouldBeOfType<WebSocketResponse>().Name.ShouldBe("Leo Chenal");
}

And for one least trick, let’s say you want to work with grouped connections in SignalR so you can send messages to a subset of connected clients. In this case, I went down the Wolverine “Side Effect” route, as you can see in these example handlers:

// Declaring that you need the connection that originated
// this message to be added to the named SignalR client group
public static AddConnectionToGroup Handle(EnrollMe msg) 
    => new(msg.GroupName);

// Declaring that you need the connection that originated this
// message to be removed from the named SignalR client group
public static RemoveConnectionToGroup Handle(KickMeOut msg) 
    => new(msg.GroupName);

// The message wrapper here sends the raw message to
// the named SignalR client group
public static object Handle(BroadCastToGroup msg) 
    => new Information(msg.Message)
        .ToWebSocketGroup(msg.GroupName);

I should say that all of the code samples are taken from our test coverage. At this point the next step is to pull this into our CritterWatch codebase to prove out the functionality. The first thing up with that is building out the server side of what will be CritterWatch’s “Dead Letter Queue Console” for viewing, querying, and managing the DLQ records for any of the Wolverine applications being monitored by CritterWatch.

For more context, here’s the live stream on Wolverine 5:

I’ll be doing a live stream tomorrow (Thursday) August 4th to preview some of the new improvements coming soon with Wolverine 5.0. The highlights are:

• The new “Partitioned Sequential Messaging” feature and why you’re going to love this feature that’s going to help make Wolverine based systems much more able to sidestep problems with concurrency
• Improvements to the code generation and IoC usage within Wolverine.HTTP
• The new SignalR transport and integration, and how we think this is going to make it easier to build asynchronous workflows between web clients and your backend services
• More powerful interoperability w/ non-Wolverine services
• How the Marten integration with Wolverine is going to be more performant by reducing network chattiness
• Some thoughts about improving the code start times for Wolverine and Marten

And of course anything else folks want to discuss on the live stream as well.

Check it out here, and the recording will be up later tomorrow anyway: