Little update since the last check in on Wolverine 5.0. I think right now that Wolverine 5.0 hits by next Monday (October 6th). To be honest, besides documentation updates, the biggest work is just pushing more on the CritterWatch backend this week to see if that forces any breaking changes in the Wolverine internals.

What’s definitely “in” right now:

• The “Partitioned Sequential Messaging” feature work that we previewed in a live stream
• Big improvements and expansion to Wolverine’s interoperability story against NServiceBus, MassTransit, CloudEvents, and whatever custom interoperability folks need to do
• A first class SignalR transport which is getting used heavily in our own CritterWatch work
• A first class Redis messaging transport from the community
• Modernization and upgrades to the GCP Pubsub transport
• The ability to mix and match database storage with Wolverine for modular monoliths
• A bit batch of optimization for the Marten integration including improvements for multi-stream operations as our response to the “Dynamic Boundary Consistency” idea from other tools
• The utilization of System.Threading.Channels in place of the TPL DataFlow library

What’s unfortunately out:

• Any effort to optimize the cold start times for Marten and Wolverine. Just a bandwidth problem, plus I think this can get done without breaking changes

And we’ll see:

• Random improvements for Azure Service Bus and Kafka usage
• HTTP improvements for content negotiation and multi-part uploads
• Yet more improvements to the “aggregate handler workflow” with Marten to allow for yet more strong typed identifier usage

The items in the 3rd list don’t require any breaking changes, so could slide to Wolverine 5.1 if necessary.

All in all, I’d argue this turned out to be a big batch of improvements with very few breaking API changes and almost nothing that would impact the average user.

Wolverine has long had some ability to schedule some messages to be executed at a later time or schedule messages to be delivered at a later time. The Wolverine 4.12 release last night added some probably overdue test automation helpers to better deal with scheduled messaging within integration testing against Wolverine applications — and that makes now a perfectly good time to talk about this capability within Wolverine!

First, let’s say that we’re just using Wolverine locally within the current system with a setup like this:

var builder = Host.CreateApplicationBuilder();
builder.Services.AddWolverine(opts =>
{
    // The only thing that matters here is that you have *some* kind of 
    // envelope persistence for Wolverine configured for your application
    var connectionString = builder.Configuration.GetConnectionString("postgres");
    opts.PersistMessagesWithPostgresql(connectionString);
});

The only point being that we have some kind of message persistence set up in our Wolverine application because the message or execution scheduling depends on persisted envelope storage.

Wolverine actually does support in memory scheduling without any persistence, but that’s really only useful for scheduled error handling or fire and forget type semantics because you’d lose everything if the process is stopped.

So now let’s move on to simply telling Wolverine to execute a message locally at a later time with the IMessageBus service:

public static async Task use_message_bus(IMessageBus bus)
{
    // Send a message to be sent or executed at a specific time
    await bus.SendAsync(new DebitAccount(1111, 100),
        new(){ ScheduledTime = DateTimeOffset.UtcNow.AddDays(1) });
    
    // Same mechanics w/ some syntactical sugar
    await bus.ScheduleAsync(new DebitAccount(1111, 100), DateTimeOffset.UtcNow.AddDays(1));

    // Or do the same, but this time express the time as a delay
    await bus.SendAsync(new DebitAccount(1111, 225), new() { ScheduleDelay = 1.Days() });
    
    // And the same with the syntactic sugar
    await bus.ScheduleAsync(new DebitAccount(1111, 225), 1.Days());
}

In the system above, all messages are being handled locally. To actually process the scheduled messages, Wolverine is as you’ve probably guessed, polling the message storage (PostgreSQL in the case above), and looking for any messages that are ready to be played. Here’s a few notes on the mechanics:

• Every node within a cluster is trying to pull in scheduled messages, but there’s some randomness in the timing to keep every node from stomping on each other
• Any one node will only pull in a limited “page” of scheduled jobs at a time so that if you happen to be going bonkers scheduling thousands of messages at one time, Wolverine can share the load across nodes and keep any one node from blowing up
• The scheduled messages are in Wolverine’s transactional inbox storage with a Scheduled status. When Wolverine decides to “play” the messages, they move to an Incoming status before finally getting marked as Handled when they are successful
• When scheduled messages for local execution are “played” in a Wolverine node, they are put into the local queue for that message, so all the normal rules for ordering or parallelization for that queue still apply.

Now, let’s move on to scheduling message delivery to external brokers. Just say you have any external routing rules like this:

using var host = await Host.CreateDefaultBuilder()
    .UseWolverine(opts =>
    {
        opts.UseRabbitMq()
            // Opt into conventional Rabbit MQ routing
            .UseConventionalRouting();
    }).StartAsync();

And go back to the same syntax for sending messages, but this time the message will get routed to a Rabbit MQ exchange:

    await bus.ScheduleAsync(new DebitAccount(1111, 100), DateTimeOffset.UtcNow.AddDays(1));

This time, Wolverine is still using its transactional inbox, but with a twist. When Wolverine knows that it is scheduling message delivery to an outside messaging mechanism, it actually schedules a local ScheduledEnvelope message that when executed, sends the original message to the outbound delivery point. In this way, Wolverine is able to support scheduled message delivery to every single messaging transport that Wolverine supports with a common mechanism.

With idiomatic Wolverine usage, you do want to try to keep most of your handler methods as “pure functions” for easier testing and frankly less code noise due to async/await mechanics. To that end, there’s a couple helpers to schedule messages in Wolverine using its cascading messages syntax:

public IEnumerable<object> Consume(MyMessage message)
{
    // Go West in an hour
    yield return new GoWest().DelayedFor(1.Hours());

    // Go East at midnight local time
    yield return new GoEast().ScheduledAt(DateTime.Today.AddDays(1));
}

The extension methods above would give you the raw message wrapped in a Wolverine DeliveryMessage<T> object where T is the wrapped message type. You can still use that type to write assertions in your unit tests.

There’s also another helper called “timeout messages” that help you create scheduled messages by subclassing a Wolverine base class. This is largely associated with sagas just because it’s commonly a need for timing out saga workflows.

Error Handling

The scheduled message support is also useful in error handling. Consider this code:

using var host = await Host.CreateDefaultBuilder()
    .UseWolverine(opts =>
    {
        opts.Policies.OnException<TimeoutException>().ScheduleRetry(5.Seconds());
        opts.Policies.OnException<SecurityException>().MoveToErrorQueue();

        // You can also apply an additional filter on the
        // exception type for finer grained policies
        opts.Policies
            .OnException<SocketException>(ex => ex.Message.Contains("not responding"))
            .ScheduleRetry(5.Seconds());
    }).StartAsync();

In the case above, Wolverine uses the message scheduling to take a message that just failed, move it out of the current receiving endpoint so other messages can proceed, then retries it no sooner than 5 seconds later (it won’t be real time perfect on the timing). This is an important difference than the RetryWithCooldown() mechanism that is effectively just doing an await Task.Delay(timespan) inline to purposely slow down the application.

As an example of how this might be useful, I’ve had to work with 3rd party systems where users can create a pessimistic lock on a bank account, so any commands against that account would always fail because of that lock. If you can tell that the command failure is because of a pessimistic lock in the exception message, you might tell Wolverine to retry that message an hour later when hopefully the lock is released, but clear out the current receiving endpoint and/or queue for other work that can proceed.

Testing with Scheduled Messaging

We’re having some trouble with the documentation publishing for some reason that we haven’t figured out yet, but there will be docs soon on this new feature.

Finally, on to some new functionality! Wolverine 4.12 just added some improvements to Wolverine’s tracked session testing feature specifically to help you with scheduled messages.

First, for some background, let’s say you have these simple handlers:

public static DeliveryMessage<ScheduledMessage> Handle(TriggerScheduledMessage message)
{
    // This causes a message to be scheduled for delivery in 5 minutes from now
    return new ScheduledMessage(message.Text).DelayedFor(5.Minutes());
}

public static void Handle(ScheduledMessage message) => Debug.WriteLine("Got scheduled message");

And now this test using the tracked session which shows the new first class support for scheduled messaging:

[Fact]
public async Task deal_with_locally_scheduled_execution()
{
    // In this case we're just executing everything in memory
    using var host = await Host.CreateDefaultBuilder()
        .UseWolverine(opts =>
        {
            opts.PersistMessagesWithPostgresql(Servers.PostgresConnectionString, "wolverine");
            opts.Policies.UseDurableInboxOnAllListeners();
        }).StartAsync();

    // Should finish cleanly, even though there's going to be a message that is scheduled
    // and doesn't complete
    var tracked = await host.SendMessageAndWaitAsync(new TriggerScheduledMessage("Chiefs"));
    
    // Here's how you can query against the messages that were detected to be scheduled
    tracked.Scheduled.SingleMessage<ScheduledMessage>()
        .Text.ShouldBe("Chiefs");

    // This API will try to immediately play any scheduled messages immediately
    var replayed = await tracked.PlayScheduledMessagesAsync(10.Seconds());
    replayed.Executed.SingleMessage<ScheduledMessage>().Text.ShouldBe("Chiefs");
}

And a similar test, but this time where the scheduled messages are being routed externally:

var port1 = PortFinder.GetAvailablePort();
var port2 = PortFinder.GetAvailablePort();

using var sender = await Host.CreateDefaultBuilder()
    .UseWolverine(opts =>
    {
        opts.PublishMessage<ScheduledMessage>().ToPort(port2);
        opts.ListenAtPort(port1);
    }).StartAsync();

using var receiver = await Host.CreateDefaultBuilder()
    .UseWolverine(opts =>
    {
        opts.ListenAtPort(port2);
    }).StartAsync();

// Should finish cleanly
var tracked = await sender
    .TrackActivity()
    .IncludeExternalTransports()
    .AlsoTrack(receiver)
    .InvokeMessageAndWaitAsync(new TriggerScheduledMessage("Broncos"));

tracked.Scheduled.SingleMessage<ScheduledMessage>()
    .Text.ShouldBe("Broncos");

var replayed = await tracked.PlayScheduledMessagesAsync(10.Seconds());
replayed.Executed.SingleMessage<ScheduledMessage>().Text.ShouldBe("Broncos");

Here’s what’s new in the code above:

1. ITrackedSession.Scheduled is a bit special collection of all activity that happened during the tracked activity that led to messages being scheduled. You can use this just to interrogate what scheduled messages resulted from the original activity.
2. ITrackedSession.PlayScheduledMessagesAsync() will “play” all scheduled messages right now and return a new ITrackedSession for those messages. This method will immediately execute any messages that were scheduled for local execution and tries to immediately send any messages that were scheduled for later delivery to external transports.

The new support in the existing tracked session feature further extends Wolverine’s already extensive test automation story. This new work was done at the behest of a JasperFx Software client who is quite aggressive in their test automation. Certainly reach out to us at sales@jasperfx.net for any help you might want with your own efforts!

Earlier this week I did a live stream on the upcoming Wolverine 5.0 release where I just lightly touched on the concept for our planned SignalR integration with Wolverine. While there wasn’t that much to show yesterday, a big pull request just landed and I think the APIs and the approach has gelled enough that it’s worth a sneak peak.

First though, the new SignalR transport in Wolverine is being built now to support our planned “CritterWatch” tool shown below:

As it’s planned out right now, the “CritterWatch” server application communicating via SignalR to constantly push updated information to any open browser dashboards about system performance. On the other side of things, CritterWatch users will be able to submit quite a number of commands or queries from the browser to CritterWatch, when will then have to relay commands and queries to the various “Critter Stack” applications being monitored through asynchronous messaging. And of course, we expect the responses or status updates to be constantly flowing from the monitored services to CritterWatch which will then relay information or updates to the browsers, again by SignalR.

Long story short, there’s going to be a lot of asynchronous messaging back and forth between the three logical applications above, and this is where a new SignalR transport for Wolverine comes into play. Having the SignalR transport gives us a standardized way to send a number of different logical messages from the browser to the server and take advantage of everything that the normal Wolverine execution pipeline gives us, including relatively clean handler code compared to other messaging or “mediator” tools, baked in observability and traceability, and Wolverine’s error resiliency. Going back the other way, the SignalR transport gives us a standardized way to publish information right back to the client from our server.

Enough of that, let’s jump into some code. From the integration testing code, let’s say we’ve got a small web app configured like this:

var builder = WebApplication.CreateBuilder();

builder.WebHost.ConfigureKestrel(opts =>
{
    opts.ListenLocalhost(Port);
});

// Note to self: take care of this in the call
// to UseSignalR() below
builder.Services.AddSignalR();
builder.Host.UseWolverine(opts =>
{
    opts.ServiceName = "Server";
    
    // Hooking up the SignalR messaging transport
    // in Wolverine
    opts.UseSignalR();

    // These are just some messages I was using
    // to do end to end testing
    opts.PublishMessage<FromFirst>().ToSignalR();
    opts.PublishMessage<FromSecond>().ToSignalR();
    opts.PublishMessage<Information>().ToSignalR();
});

var app = builder.Build();

// Syntactic sure, really just doing:
// app.MapHub<WolverineHub>("/messages");
app.MapWolverineSignalRHub();

await app.StartAsync();

// Remember this, because I'm going to use it in test code
// later
theWebApp = app;

With that configuration, when you call IMessageBus.PublishAsync(new Information("here's something you should know")); in your system, Wolverine will be routing that message through SignalR, where it will be received in a client with the default “ReceiveMessage” operation. The JSON delivered to the client will be wrapped with the CloudEvents specification like this:

Likewise, Wolverine will expect messages posted to the server from the browser to be embedded in that lightweight CloudEvents compliant wrapper.

We are not coincidentally adding CloudEvents support for extended interoperability in Wolverine 5.0 as well.

For testing, the new WolverineFx.SignalR Nuget will also have a separate messaging transport using the SignalR Client just to facilitate testing, and you can see that usage in some of the testing code:

// This starts up a new host to act as a client to the SignalR
// server for testing
public async Task<IHost> StartClientHost(string serviceName = "Client")
{
    var host = await Host.CreateDefaultBuilder()
        .UseWolverine(opts =>
        {
            opts.ServiceName = serviceName;
            
            // Just pointing at the port where Kestrel is
            // hosting our server app that is running
            // SignalR
            opts.UseClientToSignalR(Port);
            
            opts.PublishMessage<ToFirst>().ToSignalRWithClient(Port);
            
            opts.PublishMessage<RequiresResponse>().ToSignalRWithClient(Port);
            
            opts.Publish(x =>
            {
                x.MessagesImplementing<WebSocketMessage>();
                x.ToSignalRWithClient(Port);
            });
        }).StartAsync();
    
    _clientHosts.Add(host);

    return host;
}

And now to show a little Wolverine-esque spin, let’s say that you have a handler being invoked by a browser sending a message through SignalR to a Wolverine server application, and as part of that handler, you need to send a response message right back to the original calling SignalR connection to the right browser instance.

Conveniently enough, you have this helper to do exactly that in a pretty declarative way:

    public static ResponseToCallingWebSocket<WebSocketResponse> Handle(RequiresResponse msg) 
        => new WebSocketResponse(msg.Name).RespondToCallingWebSocket();

And just for fun, here’s the test that proves the above code works:

[Fact]
public async Task send_to_the_originating_connection()
{
    var green = await StartClientHost("green");
    var red = await StartClientHost("red");
    var blue = await StartClientHost("blue");

    var tracked = await red.TrackActivity()
        .IncludeExternalTransports()
        .AlsoTrack(theWebApp)
        .SendMessageAndWaitAsync(new RequiresResponse("Leo Chenal"));

    var record = tracked.Executed.SingleRecord<WebSocketResponse>();
    
    // Verify that the response went to the original calling client
    record.ServiceName.ShouldBe("red");
    record.Message.ShouldBeOfType<WebSocketResponse>().Name.ShouldBe("Leo Chenal");
}

And for one least trick, let’s say you want to work with grouped connections in SignalR so you can send messages to a subset of connected clients. In this case, I went down the Wolverine “Side Effect” route, as you can see in these example handlers:

// Declaring that you need the connection that originated
// this message to be added to the named SignalR client group
public static AddConnectionToGroup Handle(EnrollMe msg) 
    => new(msg.GroupName);

// Declaring that you need the connection that originated this
// message to be removed from the named SignalR client group
public static RemoveConnectionToGroup Handle(KickMeOut msg) 
    => new(msg.GroupName);

// The message wrapper here sends the raw message to
// the named SignalR client group
public static object Handle(BroadCastToGroup msg) 
    => new Information(msg.Message)
        .ToWebSocketGroup(msg.GroupName);

I should say that all of the code samples are taken from our test coverage. At this point the next step is to pull this into our CritterWatch codebase to prove out the functionality. The first thing up with that is building out the server side of what will be CritterWatch’s “Dead Letter Queue Console” for viewing, querying, and managing the DLQ records for any of the Wolverine applications being monitored by CritterWatch.

For more context, here’s the live stream on Wolverine 5:

I’ll be doing a live stream tomorrow (Thursday) August 4th to preview some of the new improvements coming soon with Wolverine 5.0. The highlights are:

• The new “Partitioned Sequential Messaging” feature and why you’re going to love this feature that’s going to help make Wolverine based systems much more able to sidestep problems with concurrency
• Improvements to the code generation and IoC usage within Wolverine.HTTP
• The new SignalR transport and integration, and how we think this is going to make it easier to build asynchronous workflows between web clients and your backend services
• More powerful interoperability w/ non-Wolverine services
• How the Marten integration with Wolverine is going to be more performant by reducing network chattiness
• Some thoughts about improving the code start times for Wolverine and Marten

And of course anything else folks want to discuss on the live stream as well.

Check it out here, and the recording will be up later tomorrow anyway:

That’s supposed to be a play on a Wolverine as Winnie the Pooh in his “thinking spot”

I’m wrestling a little bit with whether the new features and changes coming into Wolverine very soon are worthy of a 5.0 release even though 4.0 was just a couple months ago. I’d love any and all feedback about this. I’d also like to ask for help from the community to kick the tires on any alpha/beta/RC releases we might make with these changes.

Wolverine development is unusually busy right now as new feature requests are streaming in from JasperFx customers and users as Wolverine usage has increased quite a bit this year. We’re only a couple months out from the Wolverine 4.0 release (and Marten 8.0 that was a lot bigger). I wrote about Critter Stack futures just a month ago, but things have already changed since then, so let’s do this again.

Right now, here are the major initiatives happening or planned for the near future for Wolverine in what I think is probably the priority order:

TPL DataFlow to Channels

I’m actively working on replacing both Marten & Wolverine’s dependency on the TPL Dataflow library with System.Threading.Channels. This is something I wanted to do for 4.0, but there wasn’t enough time. Because of some issues with TPL DataFlow a JasperFx client hit under load and the planned “concurrency resistant parallelism” feature work I’ll discuss next, I wanted to start using Channels now. I’m a little concerned that this change by itself justifies a Wolverine 5.0 release even though the public APIs aren’t changing. I would expect some improvement in performance from this change, but I don’t have hard numbers yet. What do you think? I’ll have this done in a local branch by the end of the day.

“Concurrency Resistant Parallelism”

For lack of a better name, we’re planning some “concurrency resistant parallelism” features for Wolverine. Roughly, this is teaching Wolverine about how to better parallelize *or* order messages in a system so that you can maximize throughput (parallelism) without incurring concurrent writes to resources or entities that are sensitive to concurrent write problems (*cough* Marten event streams *cough*). I’d ask you to just look at the GitHub issue I linked. This is to maximize throughput for an important JasperFx client who frequently gets bursts of messages related to the same event stream, but also, this has been a frequent issue for quite a few users and we hope this would be a hugely strategic addition to Wolverine

Interoperability

Improving the interoperability options for Wolverine. and non-Wolverine applications. There’s already some work underway, but I think this might be a substantial effort out of sheer permutations. At a minimum, I’m hoping we have OOTB compatibility against both NServiceBus & MassTransit for all supported message transports in Wolverine and not just Rabbit MQ like we do today. Largely based on a pull request from the community, we’ll also make it easier to build out custom interoperability with non-Wolverine applications. And then lastly, there’s enough interest in CloudEvents to push through that as well.

Integrating with Marten’s Batch Querying / Optimizing Multi-Event Stream Operations

Make the “Critter Stack” tool the best Event Store / Event Driven Architecture platform on the freaking planet for working with multiple event streams at the same time. Mostly because it would just be flat out sexy, I’m interested in enhancing Wolverine’s integration with Marten to be able to opt into Marten’s batch querying API under the covers when you use the declarative persistence options or the aggregate handler workflow in Wolverine. This would be beneficial by:

• Improving performance because network chattiness is very commonly an absolute performance killer in enterprise-y systems — especially for teams that get a little too academic with Clean/Onion Architecture approachs
• Be what we hope will be a superior alternative for working with multiple event streams at one time in terms of usability, testability, and performance than the complex “Dynamic Consistency Boundary” idea coming out of some of the commercial event store tool companies right now
• Further Wolverine’s ability to craft much simpler Post-Clean Architecture codebases for better productivity and longer term maintenance. Seriously, I really do believe that Clean/Onion Architecture approaches absolutely strangle systems in the longer term because the code easily becomes too difficult to reason about.

IoC Usage

Improve Wolverine’s integration with IoC containers, especially for HTTP usage. I think I’d like to consider introducing an “opt out” setting where Wolverine asserts and fails on bootstrapping if any message handler or HTTP endpoint can’t use Wolverine’s inlined code generation and has to revert to service location unless users explicitly say they will allow it.

Wolverine.HTTP Improvements

Expanded support in Wolverine.HTTP for [AsParameters] usage, probably some rudimentary “content negotiation,” multi-part uploads. Really just filling some current holes in Wolverine.HTTP‘s current support as more people use that library.

SignalR

A formal SignalR integration for Wolverine, which will most likely drop out of our ongoing “Critter Watch” development. Think about having a first class transport option for Wolverine that will let you quickly integrate messages to and from a web application via SignalR

Cold Start Optimization

Optimizing the Wolverine “Cold Start Time.” I think that’s self explanatory. This work might span into Marten and even Lamar as well. I’m not going to commit to AOT compatibility in the Critter Stack this year because I like actually getting to see my family sometimes, but this work might get us closer to that for next year.