This is the kind of post I write for myself and just share on a Friday or weekend when not many folks are paying any attention.

I’ve taken a couple days at the end of this week after a month long crush to just think about the strategic technical vision for the Critter Stack and the commercial add on products that we’re building under the JasperFx Software rubric. As part of my “deep think, but don’t work too hard” day, I had Claude help me do a gap analysis between Wolverine.HTTP and ASP.Net Core Minimal API & MVC Core and even FastEndpoints. I also did the same for Wolverine’s messaging feature set and all the widely used .NET messaging frameworks (I think .NET has more strong options for this than any other platform and it still irritates me that Microsoft seriously tried to butt into that) and several options in the Java ecosystem.

Before I share the results and what I thought was and wasn’t important, let me share one big insight. Different tools in the same problem space frequently solve the same problems, but with very different technical solutions, concepts, and abstractions. Sometimes different tools even have very similar solutions to common problems, but use very different nomenclature . All this is to say that this effort helped me identify several places where we will try to improve documentation to map features from other tools to the options in Wolverine as Claude “identified” almost two dozen functional “gaps” where I felt like Wolverine already happily solved the same problems that features in MassTransit, NServiceBus, Mulesoft, or other tools did.

There’s also a lesson for folks who switch tools to understand the different concepts in the new tool instead of automatically trying to map your mental model from tool A to tool B without first learning what’s really different.

And lastly, a lesson for anybody who ever does any kind of support of development tools: remember to ask a user who is struggling what their end goals are or their real use case is instead of just focusing on the sometimes oddball implementation or API questions they’re asking you. And that goes double when a user is quite possibly trying to force fit their mental model of a completely different tool into your tool.

Anyway, here’s what I ended up adding to our backlog as well as things that I didn’t think were valuable at this time.

On the HTTP front, I came up with several things, with the big items being:

1. I originally thought an equivalent to MVC’s IExceptionFilter, but we might just use that as is. That’s come up plenty of times before
2. Anti-forgery support. I originally thought that Wolverine.HTTP would mostly be used for API development, so didn’t really bother much upfront with too much for supporting HTTP forms, but I think there’s a significant overlap between Wolverine.HTTP usage and htmx where forms are used more heavily, so here we go.
3. Routing prefixes. It’s come up occasionally, and been just barely on my radar
4. Endpoint rate limiting middleware for HTTP. This will build on our new rate limiting middleware for message handlers
5. Server Sent Events support. Why not? For whatever reason, SSE seems to be getting rediscovered by folks. FubuMVC (Wolverine’s predecessor in the early 2010’s) actually had a first class SSE support all those years ago
6. Output Caching. This has been in my thinking for quite awhile. I think this is going to be two pronged, with direct support for ASP.Net Core caching middleware and maybe some more directed “per entity” caching around our existing “declarative persistence” helpers. I think the second actually lives inside of message handlers as well
7. API versioning of some sort. It’s easy enough to just add “1.0” into your routes, but we’ll look at more alternatives as well
8. A little bit of content negotiation support, but that’s been on the periphery of my attention from the beginning. My thought all along was to not bother with that until people explicitly asked for it, but now I just want to close the gaps. FubuMVC had that 15 years ago, so I’ve already dealt with that successfully before — but that was in the ReST craze and “conneg” just isn’t nearly as common in usage as far as I can tell.

And the gap analysis helped point out several areas where we had opportunities to improve the documentation (and future AI skills) to help map Minimal API or MVC Core concepts to existing features in Wolverine.HTTP.

Now, on to the messaging support which turned up almost nothing that I was actually interested in adding to Wolverine except for these:

1. Formal support for the EIP “Claim Check” pattern. I’ve never pursued that before because I’ve felt like it’s just not that much explicit code, but I still added that to the backlog for “completeness”
2. Build in EIP “Wire Tap” support to persist messages but that was already in our backlog as that comes up from users and also because we have plans to expose that through MCP and command line AI support tools. I’m not enthusiastic thought about bothering with the “command sourcing” concept from Greg Young, but we’ll see if anybody ever wants it.

Claude came up with about 35 different things to consider, but other than those two things above, those items fell into either functionality we already had with different names or different conceptual solutions, features I just have no interest in supporting or I don’t see being used or requested by our users, or a third group of features that are happily planned and already underway with our forthcoming CritterWatch commercial add on.

Just for completeness, the features I’m saying we won’t even plan to support right now were:

• The EIP “Routing Slip” concept. I know that MassTransit supports it, but I’m deeply unenthusiastic about both the concept and any attempt to support that in Wolverine. They can have that one.
• Distributed transaction support. I don’t even know why I would need to explain why not!
• “Change Data Capture” integration with something like Debezium. I just don’t see a demand for that with Wolverine
• Any kind of visual process designer. Even on the Marten/Polecat side, I’m wanting us to focus on Markdown or Gherkin specifications or just flat out making our code as simple as possible to write instead of blowing energy on visual tools that generate XML that in turn get generated into Java code. Not that I’m necessarily giving some side eye to any other tool out there *cough* liar! *cough*
• Batch processing support that really touched on ETL concerns
• A long lived job model. Maybe down the road, but I’d push folks to just break that up into smaller actions whenever possible anyway. It’s trivial in Wolverine to have message handlers cascade out a request for the next step. Actually, this one is probably the one I’m most likely to have to change my mind about, but we’ll see
• NServiceBus has their “messaging bridge” that I think would be trivial to build out later if that’s ever valuable for someone, but nobody is asking for that today and Wolverine happily lets you mix and match all the transports and even multiple brokers in one application

And of course, there was some random quirky features of some of the other tools I just didn’t think were worth any consideration outside of client requests or common user community requests.

I’m working pretty hard this week and early next to deliver the CritterWatch MVP (our new management and observability console for the Critter Stack) to a JasperFx Software client. One of the things we need to do for testing is to fake out several failure conditions in message handlers to be able to test CritterWatch’s “Dead Letter Queue” management and alerting features. To that end, we have some fake systems that constantly process messages, and we’ve rigged up what I’m going to call the world’s crudest Chaos Monkey in Wolverine middleware:

    public static async Task Before(ChaosMonkeySettings chaos)
    {
        // Configurable slow handler for testing back pressure
        if (chaos.SlowHandlerMs > 0)
        {
            await Task.Delay(chaos.SlowHandlerMs);
        }

        if (chaos.FailureRate <= 0) return;

        // Chaos monkey — distribute failure rate equally across 5 exception types
        var perType = chaos.FailureRate / 5.0;
        var next = Random.Shared.NextDouble();

        if (next < perType)
        {
            throw new TripServiceTooBusyException("Just feeling tired at " + DateTime.Now);
        }

        if (next < perType * 2)
        {
            throw new TrackingUnavailableException("Tracking is down at " + DateTime.Now);
        }

        if (next < perType * 3)
        {
            throw new DatabaseIsTiredException("The database wants a break at " + DateTime.Now);
        }

        if (next < perType * 4)
        {
            throw new TransientException("Slow down, you move too fast.");
        }

        if (next < perType * 5)
        {
            throw new OtherTransientException("Slow down, you move too fast.");
        }
    }

And this to control it remotely in tests or just when doing exploratory manual testing:

    private static void MapChaosMonkeyEndpoints(WebApplication app)
    {
        var group = app.MapGroup("/api/chaos")
            .WithTags("Chaos Monkey");

        group.MapGet("/", (ChaosMonkeySettings settings) => Results.Ok(settings))
            .WithSummary("Get current chaos monkey settings");

        group.MapPost("/enable", (ChaosMonkeySettings settings) =>
        {
            settings.FailureRate = 0.20;
            return Results.Ok(new { message = "Chaos monkey enabled at 20% failure rate", settings });
        }).WithSummary("Enable chaos monkey with default 20% failure rate");

        group.MapPost("/disable", (ChaosMonkeySettings settings) =>
        {
            settings.FailureRate = 0;
            return Results.Ok(new { message = "Chaos monkey disabled", settings });
        }).WithSummary("Disable chaos monkey (0% failure rate)");

        group.MapPost("/failure-rate/{rate:double}", (double rate, ChaosMonkeySettings settings) =>
        {
            rate = Math.Clamp(rate, 0, 1);
            settings.FailureRate = rate;
            return Results.Ok(new { message = $"Failure rate set to {rate:P0}", settings });
        }).WithSummary("Set chaos monkey failure rate (0.0 to 1.0)");

        group.MapPost("/slow-handler/{ms:int}", (int ms, ChaosMonkeySettings settings) =>
        {
            ms = Math.Max(0, ms);
            settings.SlowHandlerMs = ms;
            return Results.Ok(new { message = $"Handler delay set to {ms}ms", settings });
        }).WithSummary("Set artificial handler delay in milliseconds (for back pressure testing)");

        group.MapPost("/projection-failure-rate/{rate:double}", (double rate, ChaosMonkeySettings settings) =>
        {
            rate = Math.Clamp(rate, 0, 1);
            settings.ProjectionFailureRate = rate;
            return Results.Ok(new { message = $"Projection failure rate set to {rate:P0}", settings });
        }).WithSummary("Set projection failure rate (0.0 to 1.0)");
    }

In this case, the Before middleware is just baked into the message handlers, but in your development the “chaos monkey” middleware could be applied only in testing with a Wolverine extension.

And I was probably listening to Simon & Garfunkel when I did the first cut at the chaos monkey:

Last week I helped a JasperFx Software client with a use case where they get a steady stream of related events from an upstream system into a downstream system where order of processing is important, but the messages might arrive out of order.

Once again referring to the venerable Enterprise Integration Patterns book, that scenario requires a Resequencer:

How can we get a stream of related but out-of-sequence messages back into the correct order?

EIP ReSequencer

To solve the message ordering challenge, we introduced the new Resequencer Saga feature into Wolverine, and combined that with the existing “Partitioned Sequential Messaging” feature.

For the new built in re-sequencing, we do need you to implement this interface on any message types in that related stream so that Wolverine “knows” what order the message is inside of a related stream:

public interface SequencedMessage
{
    int? Order { get; }
}

The next step is to use a special kind of new Wolverine Saga called ResequencerSaga<T>, where the T is just some sort of common interface for all the message types that are part of this ordered stream and also implements the SequencedMessage shown above. Here’s a simple example I used for the testing:

public record StartMyWorkflow(Guid Id);
public record MySequencedCommand(Guid SagaId, int? Order) : SequencedMessage;
public class MyWorkflowSaga : ResequencerSaga<MySequencedCommand>
{
    public Guid Id { get; set; }
    public static MyWorkflowSaga Start(StartMyWorkflow cmd)
    {
        return new MyWorkflowSaga { Id = cmd.Id };
    }
    public void Handle(MySequencedCommand cmd)
    {
        // This will only be called when messages arrive in the correct order,
        // or when out-of-order messages are replayed after gaps are filled
    }
}

At runtime, when Wolverine gets a message that is handled by that MyWorkflowSaga, there is some middleware that first compares the declared order of that message against the recorded state of the saga so far. In more concrete terms, if…

• It’s the first message in the sequence, Wolverine just processes it as normal and records in the saga state what the last processed message order was so that it “knows” what message sequence should be next
• It’s a later message in the sequence compared to the last message sequence processed, the saga state will just store the current message, persist the saga state, and otherwise skip the normal message processing
• The message is the next in the sequence according to what the saga state says should be processed next, it processes normally. If there are any previously out of order messages that the saga state already knows about that are sequentially next after the current message, Wolverine will re-publish those messages locally — but with the normal Wolverine message sequencing these cascading messages will not go anywhere until the initiating message completes

With this mechanism, Wolverine is able to put the messages arriving from the outside world back into the correct sequential order in its own processing.

Of course though, this processing is very stateful and somewhat likely to be vulnerable to concurrent access problems. Most of the saga storage mechanisms in Wolverine happily support optimistic concurrency around saving saga state, so you could just use some selective retries on concurrency violations. Or better yet, Wolverine users can just about completely side step issues with concurrency by utilizing our newest improvement to partitioned messaging we’re calling “Global Partitioning.”

Let’s say that you have a great deal of operations in your system that have to modify a resource of some sort like an entity, a file, a saga in this case, or an event stream that might be a little bit sensitive to concurrent access. Let’s also say that you have a mix of messages that impact these sensitive resources that come from both external, upstream systems and from cascaded messages within your own system.

The syntax for this next feature was added just today in Wolverine 5.21 as I realized the previous syntax was basically unusable in the course of trying to write this blog post. So it goes.

A “global partitioning” allows you to create a guarantee that messages impacting those resources can be processed sequentially within a message group while allowing for parallel processing between message groups throughout the entire cluster.

Imagine it like this (but know I drew this diagram for someone using Kafka even though the next example is using Rabbit MQ queues):

And with this configuration:

using var host = await Host.CreateDefaultBuilder()
    .UseWolverine(opts =>
    {
        // You'd *also* supply credentials here of course!
        opts.UseRabbitMq();
        // Do something to add Saga storage too!
        opts
            .MessagePartitioning
            // This tells Wolverine to "just" use implied
            // message grouping based on Saga identity among other things
            .UseInferredMessageGrouping()
            .GlobalPartitioned(topology =>
            {
                // Creates 5 sharded RabbitMQ queues named "sequenced1" through "sequenced5"
                // with matching companion local queues for sequential processing
                topology.UseShardedRabbitQueues("sequenced", 5);
                topology.MessagesImplementing<MySequencedCommand>();
            });
    }).StartAsync();

What this does is spread the work out for handling MySequencedCommand messages through five different Rabbit MQ + Local queue pairs, with each pair active on only one single node within your application. Even inside each local queue in this partitioning scheme, Wolverine is parallelizing between message groups.

Now, let’s talk about receiving any message that can be cast to MySequencedCommand. If the message is received at a completely different listener than the “sequenced1/2/3/4/5” queues defined above, like from an external system that knows absolutely nothing about your message partitioning, Wolverine is going to immediately determine the message group identity by inferring that from the saga message handler rules (that’s what the UseInferredMessageGrouping() option does for us), then forwards that message to the proper node that is currently handling that group id. If the current node happens to be assigned that message group id, Wolverine forwards the message directly to the right local queue.

Likewise, if you publish a cascading message inside one of your handlers, Wolverine will determine the message group id for that message type, then try to either route that message locally if that group happens to be assigned to the current node (and it probably would be if you were cascading from your own handlers) or sends it remotely to the right messaging endpoint (Rabbit MQ queue or a Kafka topic or an AWS SQS queue maybe).

The point being, this guarantees that related messages are processed sequentially across the entire application cluster while allowing parallel processing between unrelated messages.

Summary

These are hopefully two powerful new features that will benefit Wolverine users in the near future. Both of these features were built at the behest of JasperFx Software clients to directly support their current work. I’m very happy to just quietly fold in reasonably sized new features for JasperFx support clients without extra cost when those features likely benefit the community as a whole. Contact us at sales@jasperfx.net to find out what we can do to help your software development efforts be more successful.

And just for bragging rights tonight, I did some poking around (okay, I asked Claude to do it for me) to see if any other asynchronous messaging tools offer anything similar to what our global partitioning option does for Wolverine users. While you can certainly achieve the same goals through actor frameworks like AkkaDotNet or Orleans (I consider actor frameworks to be such a different paradigm that I don’t really think of them as direct competitors to Wolverine), it doesn’t appear that there are any equivalents out there to this feature in the .NET space. MassTransit and NServiceBus both have more limited versions of this capability, but nothing that is as easy or flexible as what Wolverine has at this point. Now, granted, we’re at this point because Marten event stream appends can be sensitive to concurrent access so we’ve had to take concurrency maybe a little more seriously than the pure play asynchronous messaging tools that don’t really have an event sourcing component.