I worked a little bit this weekend on a small new feature in Wolverine that we’ll need as part of our forthcoming “CritterWatch” tooling. What I was doing isn’t that interesting, but the killer problem was that it required me to write an integration test that would:
Spin up multiple IHost instances for the same testing application
Verify that Wolverine was correctly assigning running tasks to only the leader node
Stop the leader node, see leadership and that same task shift to the newly elected leader
Make sure that task was really only ever running on the single leader node
Needless to say, it’s a long running test and it turned out to be non trivial to get both the test harness and the necessary code exactly right. Honestly, I didn’t get this done until I stopped and integrated application logging directly into the xUnit.Net test harness (plus integrating a Wolverine specific event observer too) so I could see what the heck was going on inside all of these application instances.
So without further ado, here’s the recipe we’re using (and copy/pasting around) in Wolverine to do that. First off, we need an ILogger and ILoggerProvider implementation that will pipe logging to xUnit’s ITestOutputHelper like so:
public class XUnitLogger : ILogger { private readonly string _categoryName; private readonly List<string> _ignoredStrings = new() { "Declared", "Successfully processed message" }; private readonly ITestOutputHelper _testOutputHelper; public XUnitLogger(ITestOutputHelper testOutputHelper, string categoryName) { _testOutputHelper = testOutputHelper; _categoryName = categoryName; } public bool IsEnabled(LogLevel logLevel) { return logLevel != LogLevel.None; } public IDisposable BeginScope<TState>(TState state) { return new Disposable(); } public void Log<TState>(LogLevel logLevel, EventId eventId, TState state, Exception exception, Func<TState, Exception, string> formatter) { if (exception is DivideByZeroException) { return; } if (exception is BadImageFormatException) { return; } // Obviously this is crude and you would do something different here... if (_categoryName == "Wolverine.Transports.Sending.BufferedSendingAgent" && logLevel == LogLevel.Information) return; if (_categoryName == "Wolverine.Runtime.WolverineRuntime" && logLevel == LogLevel.Information) return; if (_categoryName == "Microsoft.Hosting.Lifetime" && logLevel == LogLevel.Information) return; if (_categoryName == "Wolverine.Transports.ListeningAgent" && logLevel == LogLevel.Information) return; if (_categoryName == "JasperFx.Resources.ResourceSetupHostService" && logLevel == LogLevel.Information) return; if (_categoryName == "Wolverine.Configuration.HandlerDiscovery" && logLevel == LogLevel.Information) return; var text = formatter(state, exception); if (_ignoredStrings.Any(x => text.Contains(x))) return; _testOutputHelper.WriteLine($"{_categoryName}/{logLevel}: {text}"); if (exception != null) { _testOutputHelper.WriteLine(exception.ToString()); } } public class Disposable : IDisposable { public void Dispose() { } } } public class OutputLoggerProvider : ILoggerProvider { private readonly ITestOutputHelper _output; public OutputLoggerProvider(ITestOutputHelper output) { _output = output; } public void Dispose() { } public ILogger CreateLogger(string categoryName) { return new XUnitLogger(_output, categoryName); } }
And register it inside the test harness like so:
public class leader_pinned_listener : IAsyncDisposable
{
private readonly ITestOutputHelper _output;
public leader_pinned_listener(ITestOutputHelper output)
{
_output = output;
}
private async Task<IHost> startHost()
{
await dropSchemaAsync();
var host = await Host.CreateDefaultBuilder()
.UseWolverine(opts =>
{
// This is where I'm adding in the custom ILoggerProvider
opts.Services.AddSingleton<ILoggerProvider>(new OutputLoggerProvider(_output));
// More configuration that isn't germane...
return host;
}
Hey, it’s crude, but the point here was that this kind of gnarly integration testing, and especially with a lot of asynchronous behavior, is a lot easier to get through when you have more insight into how the code you’re testing is actually behaving.
Railway Programming is an idea that came out of the F# community as a way to develop for “sad path” exception cases without having to resort to throwing .NET Exceptions as a way of doing flow control. Railway Programming works by chaining together functions with a standardized response in such a way that it’s relatively easy to abort workflows as preliminary steps are found to be invalid while still passing the results of the preceding function as the input into the next function.
Wolverine has some direct support for a quasi-Railway Programming approach by moving validation or data loading steps prior to the main message handler or HTTP endpoint logic. Let’s jump into a quick sample that works with either message handlers or HTTP endpoints using the built in HandlerContinuation enum:
public static class ShipOrderHandler
{
// This would be called first
public static async Task<(HandlerContinuation, Order?, Customer?)> LoadAsync(ShipOrder command, IDocumentSession session)
{
var order = await session.LoadAsync<Order>(command.OrderId);
if (order == null)
{
return (HandlerContinuation.Stop, null, null);
}
var customer = await session.LoadAsync<Customer>(command.CustomerId);
return (HandlerContinuation.Continue, order, customer);
}
// The main method becomes the "happy path", which also helps simplify it
public static IEnumerable<object> Handle(ShipOrder command, Order order, Customer customer)
{
// use the command data, plus the related Order & Customer data to
// "decide" what action to take next
yield return new MailOvernight(order.Id);
}
}
By naming convention (but you can override the method naming with attributes as you see fit), Wolverine will try to generate code that will call methods named Before/Validate/Load(Async) before the main message handler method or the HTTP endpoint method. You can use this compound handler approach to do set up work like loading data required by business logic in the main method or in this case, as validation logic that can stop further processing based on failed validation or data requirements or system state. Some Wolverine users like using these method to keep the main methods relatively simple and focused on the “happy path” and business logic in pure functions that are easier to unit test in isolation.
By returning a HandlerContinuation value either by itself or as part of a tuple returned by a Before, Validate, or LoadAsync method, you can direct Wolverine to stop all other processing.
You have more specialized ways of doing that in HTTP endpoints by using the ProblemDetails specification to stop processing like this example that uses a Validate() method to potentially stop processing with a descriptive 400 and error message:
public record CategoriseIncident(
IncidentCategory Category,
Guid CategorisedBy,
int Version
);
public static class CategoriseIncidentEndpoint
{
// This is Wolverine's form of "Railway Programming"
// Wolverine will execute this before the main endpoint,
// and stop all processing if the ProblemDetails is *not*
// "NoProblems"
public static ProblemDetails Validate(Incident incident)
{
return incident.Status == IncidentStatus.Closed
? new ProblemDetails { Detail = "Incident is already closed" }
// All good, keep going!
: WolverineContinue.NoProblems;
}
// This tells Wolverine that the first "return value" is NOT the response
// body
[EmptyResponse]
[WolverinePost("/api/incidents/{incidentId:guid}/category")]
public static IncidentCategorised Post(
// the actual command
CategoriseIncident command,
// Wolverine is generating code to look up the Incident aggregate
// data for the event stream with this id
[Aggregate("incidentId")] Incident incident)
{
// This is a simple case where we're just appending a single event to
// the stream.
return new IncidentCategorised(incident.Id, command.Category, command.CategorisedBy);
}
}
The value WolverineContinue.NoProblems tells Wolverine that everything is good, full speed ahead. Anything else will write the ProblemDetails value out to the response, return a 400 status code (or whatever you decide to use), and stop processing. Returning a ProblemDetails object hopefully makes these filter methods easy to unit test themselves.
You can also use the AspNetCore IResult as another formally supported “result” type in these filter methods like this shown below:
public static class ExamineFirstHandler
{
public static bool DidContinue { get; set; }
public static IResult Before([Entity] Todo2 todo)
{
return todo != null ? WolverineContinue.Result() : Results.Empty;
}
[WolverinePost("/api/todo/examinefirst")]
public static void Handle(ExamineFirst command) => DidContinue = true;
}
In this case, the “special” value WolverineContinue.Result() tells Wolverine to keep going, otherwise, Wolverine will execute the IResult returned from one of these filter methods and stop all other processing for the HTTP request.
It’s maybe a shameful approach for folks who are more inline with a Functional Programming philosophy, but you could also use a signature like:
[WolverineBefore]
public static UnauthorizedHttpResult? Authorize(SomeCommand command, ClaimsPrincipal user)
In the case above, Wolverine will do nothing if the return value is null, but will execute the UnauthorizedHttpResult response if there is, and stop any further processing. There is *some* minor value to expressing the actual IResult type above because that can be used to help generate OpenAPI metadata.
Lastly, let’s think about the very common need to write an HTTP endpoint where you want to return a 404 status code if the requested data doesn’t exist. In many cases the API user is supplying the identity value for an entity, and your HTTP endpoint will first query for that data, and if it doesn’t exist, abort the processing with the 404 status code. Wolverine has some built in help for this tedious task through its unique persistence helpers as shown in this sample HTTP endpoint below:
[WolverineGet("/orders/{id}")]
public static Order GetOrder([Entity] Order order) => order;
Note the presence of the [Entity] attribute for the Order argument to this HTTP endpoint route. That’s telling Wolverine that that data should be loaded using the “id” route argument as the Order key from whatever persistence mechanism in your application deals with the Order entity, which could be Marten of course, an EF Core DbContext that has a mapping for Order, or Wolverine’s RavenDb integration. Unless we purposely mark [Entity(Required = false)], Wolverine.HTTP will return a 404 status code if the Order entity does not exist. The simplistic sample from Wolverine’s test suite above doesn’t do any kind of mapping from the raw Order to a view model, but the mechanics of the [Entity] loading would work equally if you also mapped the raw Order to some kind of OrderViewModel maybe.
Last Thoughts
I’m pushing Wolverine users and JasperFx clients to utilize Wolverine’s quasi Railway Programming capabilities as guard clauses to better separate out validation or error condition handling into easily spotted, atomic operations while reducing the core HTTP request or message handler to being a “happy path” operation. Especially in HTTP services where the ProblemDetails specification and integration with Wolverine fits well with this pattern and where I’d expect many HTTP client tools to already know how to work with problem details responses.
There have been a few attempts to adapt Railway Programming to C# that I’m aware of, inevitably using some kind of custom Result type that denotes success or failure with the actual results for the next function. I’ve seen some folks and OSS tools try to chain functions together with nested lambda functions within a fluent interface. I’m not a fan of any of this because I think the custom Result types just add code noise and extra mechanical work, then the fluent Interface approach can easily be nasty to debug and detracts from readability by the extra code noise. But anyway, read a lot more about this in Andrew Lock’s Series: Working with the result pattern and make up your own mind.
I’ve also seen an approach where folks used MediatR handlers for each individual step in the “railway” where each handler had to return a custom Result type with the inputs for the next handler in the series. I beg you, please don’t do this in your own system because that leads to way too much complexity, code that’s much harder to reason about because of the extra hoops and indirection, and potentially poor system performance because again, you can’t see what the code is doing and you can easily end up making unnecessarily duplicate database round trips or just being way too “chatty” to the database. And no, replacing MediatR handlers with Wolverine handlers is not going to help because the pattern was the problem and not MediatR itself.
As always, the Wolverine philosophy is that the path to long term success in enterprise-y software systems is by relentlessly eliminating code ceremony so that developers can better reason about how the system’s logic and behavior works. To a large degree, Wolverine is a reaction to the very high ceremony Clean/Onion Architecture/iDesign architectural approaches of the past 15-20 years and how hard those systems can be to deal with over time.
And as happens with just about any halfway good thing in programming, some folks overused the Railway Programming idea and there’s a little bit of pushback or backlash to the technique. I can’t find the quote to give it the real attribution, but something I’ve heard Martin Fowler say is that “we don’t know how useful an idea really can be until we push it too far, then pull back a little bit.”
You’re about to start a new system with Event Sourcing using Marten, and you’re expecting your system to be hugely successful such that it’s going to handle a huge amount of data, but you’re already starting with pretty ambitious non-functional requirements for the system to be highly performant and all the screens or exposed APIs be snappy.
Basically, what you want to do is go as fast as Marten and PostgreSQL will allow. Fortunately, Marten has a series of switches and dials that can be configured to squeeze out more performance, but for a variety of historical reasons and possible drawbacks, are not the defaults for a barebones Marten configuration as shown below:
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddMarten(opts =>
{
opts.Connection(builder.Configuration.GetConnectionString("marten"));
});
Cut me some slack in my car choice for the analogy here. I’m not only an American, but I’m an American from a rural area who grew up dreaming about having my own Mustang or Camaro because that’s as far out as I could possibly imagine back then.
At this point, we have is the equivalent to a street legal passenger car, maybe the equivalent to an off the shelf Mustang:
Which probably easily goes fast enough for every day usage for the mass majority of us most of the time. But we really need a fully tricked out Mustang GTD that’s absurdly optimized to just flat out go fast:
Let’s start trimming weight off our street legal Marten setup to go faster with…
Opt into Lightweight Sessions by Default
Starting from a new system so we don’t care about breaking existing code by changing behavior, let’s opt for lightweight sessions by default:
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddMarten(opts =>
{
opts.Connection(builder.Configuration.GetConnectionString("marten"));
})
// Jettison some "Identity Map" weight by going lighter weight
.UseLightweightSessions();
By default, the instances of IDocumentSession you get out of an IoC container would utilize the Identity Map feature to track loaded entities by id so that if you happened to try to load the same entity from the same session, you would get the exact same object. As I’m sure you can imagine, that means that every entity fetched by a session is stuffed into a dictionary internally (Marten uses the highly performant ImTools ImHashMap everywhere, but still), and the session also has to bounce through the dictionary before loading data as well. It’s just a little bit of overhead we can omit by opting for “Lightweight Sessions” if we don’t need that behavior by default.
We’ve always been afraid to change the default behavior here to the more efficient approach because it can absolutely lead to breaking existing code that depends on the Identity Map behavior. On the flip side, I think you should not need Identity Map mechanics if you can keep the call stacks within your code short enough that you can actually “see” where you might be trying to load the same data twice or more in the same parent operation.
On to the next thing…
Make Writes Faster with Quick Append
Next, since we again don’t have any existing code that can be broken here, let’s opt for “Quick Append” writes like so:
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddMarten(opts =>
{
opts.Connection(builder.Configuration.GetConnectionString("marten"));
// Make event writing faster, like 2X faster in our testing
opts.Events.AppendMode = EventAppendMode.Quick;
})
// Jettison some "Identity Map" weight by going lighter weight
.UseLightweightSessions();
This will help the system be able to append new events much faster, but at the cost of not being able to use some event metadata like event versions, sequence, or timestamp information within “Inline” projections.
Again, even though this option has been clocked as being much faster, we have not wanted to make this the default because it could break existing systems for people who depend on having the rich metadata during the Inline application of projections that forces Marten to do a kind of two step process to append events. This “Quick Append” option also helps reduce concurrent access problems writing to streams and generally makes the “Async Daemon” subsystem processing asynchronous projections and subscriptions run much smoother.
We’re not out of tricks yet by any means, so let’s go on…
// This tells Wolverine that the first "return value" is NOT the response
// body
[EmptyResponse]
[WolverinePost("/api/incidents/{incidentId:guid}/category")]
public static IncidentCategorised Post(
// the actual command
CategoriseIncident command,
// Wolverine is generating code to look up the Incident aggregate
// data for the event stream with this id
[Aggregate("incidentId")] Incident incident)
{
// This is a simple case where we're just appending a single event to
// the stream.
return new IncidentCategorised(incident.Id, command.Category, command.CategorisedBy);
}
In the case above, the Incident model is a projected document that’s first used by the command handler to “decide” what new events to emit. If we’re updating the Incident model with an Inline projection that tries to update the Incident model in the database at the same time it wants to append events, then it’s an advantage for performance to “just” use the original Incident model we used initially, then forwarding the new state based on the new events and persisting the results right then and there. We can opt into this optimization even for the lightweight sessions we earlier wanted to use by adopting one more UseIdentityMapForAggregates flag:
builder.Services.AddMarten(opts =>
{
opts.Connection(builder.Configuration.GetConnectionString("marten"));
// Make event writing faster, like 2X faster in our testing
opts.Events.AppendMode = EventAppendMode.Quick;
// This can cut down on the number of database round trips
// Marten has to do during CQRS command handler execution
opts.Events.UseIdentityMapForAggregates = true;
})
// Jettison some "Identity Map" weight by going lighter weight
.UseLightweightSessions();
Note, this optimization can easily break code for folks who use some sort of stateful “Aggregate Root” approach where the state of the projected aggregate object might be mutated during the course of executing the command. As this has traditionally been a popular approach in Event Sourcing circles, we can’t make this be a default option. If you instead either make the projected aggregates like Incident either immutable or treat them as a dumb data input to your command handlers with a more Functional Programming “Decider” function approach, you can get away with the performance optimization.
And also, I strongly prefer and recommend the FP “Decider” approach to JasperFx Software clients as is and I think that folks using the older “Aggregate Root” approach tend to have more runtime bugs.
Moving on, let’s keep our database smaller…
Event Stream Archiving
By and large, you can improve system performance in almost any situation by trying to keep your database from growing too large by archiving or retiring obsolete information. Marten has first class support for “Archiving Event Streams” where you effectively just move event streams that only represent historical information and are not really active into an archived state.
Moreover, we can divide our underlying PostgreSQL storage for events into “hot” and “cold” storage by utilizing PostgreSQL’s table partitioning support like this:
builder.Services.AddMarten(opts =>
{
opts.Connection(builder.Configuration.GetConnectionString("marten"));
// Make event writing faster, like 2X faster in our testing
opts.Events.AppendMode = EventAppendMode.Quick;
// This can cut down on the number of database round trips
// Marten has to do during CQRS command handler execution
opts.Events.UseIdentityMapForAggregates = true;
// Let's leverage PostgreSQL table partitioning
// to our advantage
opts.Events.UseArchivedStreamPartitioning = true;
})
// Jettison some "Identity Map" weight by going lighter weight
.UseLightweightSessions();
If you’re aggressive with marking event streams as Archived, the PostgreSQL table partitioning can move off archived event streams into a different table partition than our active event data. This is essentially keeping the “active” event table storage relatively stable in size, and most operations will execute against this smaller table partition while still being able to access the archived data too if explicitly opt into including that.
We added this feature in a minor point 7.* release, so it had to be opt in, and I think I was too hesitant to make this a default in 8.0, so it’s still “opt in”.
Stream Compacting
Beyond archiving event streams, maybe you just want to “compact” a longer event stream so you technically retain all the existing state, but further reduce the size of your active database storage. To that end, Marten 8.0 added Stream Compacting.
Distributing Asynchronous Projections
I had been mostly talking about using projections running Inline such that the projections are updated at the same time as the events are captured. That’s sometimes applicable or desirable, but other times you’ll want to optimize the “write” operations by moving the updating of projected data to an Async projection running in the background. But now let’s say that we have quite a few asynchronous projections and several subscriptions as well. In early versions of Marten, we had to run everything in a “Hot/Cold” mode where every known projection or subscription had to run on one single “leader” node. So even if you were running your application across a dozen or more nodes, only one could be executing all of the asynchronous projections and subscriptions.
That’s obviously a potential bottleneck, so Marten 7.0 by itself introduced some ability to spread projections and subscriptions over multiple nodes. If we introduce Wolverine into the mix though, we can do quite a bit better than that by allowing Wolverine to distribute the asynchronous Marten work across our entire cluster with its ability to distribute Marten projections and subscriptions with the UseWolverineManagedEventSubscriptionDistribution option in the WolverineFx.Marten Nuget:
builder.Services.AddMarten(opts =>
{
opts.Connection(builder.Configuration.GetConnectionString("marten"));
// Make event writing faster, like 2X faster in our testing
opts.Events.AppendMode = EventAppendMode.Quick;
// This can cut down on the number of database round trips
// Marten has to do during CQRS command handler execution
opts.Events.UseIdentityMapForAggregates = true;
// Let's leverage PostgreSQL table partitioning
// to our advantage
opts.Events.UseArchivedStreamPartitioning = true;
})
// Jettison some "Identity Map" weight by going lighter weight
.UseLightweightSessions()
.IntegrateWithWolverine(opts =>
{
opts.UseWolverineManagedEventSubscriptionDistribution = true;
});
Is there anything else for the future?
It never ends, and yes, there are still quite a few ideas in our product backlog to potentially improve performance and scalability of Marten’s Event Sourcing. Offhand, that includes looking at alternative, higher performance serializers and more options to parallelize asynchronous projections to squeeze out more throughput by sharing some data access across projections.
Summary
There are quite a few “opt in” features in Marten that will help your system perform better, but these features are “opt in” because they can be harmful if you’re not building around the assumptions these features make about how your code works. The good news though is that you’ll be able to better utilize these features if you follow the Critter Stack’s recommended practices by striving for shorter code stacks (i.e., how many jumps between methods and classes does your code make when receiving a system input like a message or HTTP request) so your code is easier to reason about anyway, and avoiding mutating projected aggregate data outside of Marten.
Lamar, the spiritual successor to StructureMap, had a corresponding 15.0 release
And underneath those tools, the new JasperFx & JasperFx.Events library went 1.0 and the supporting Weasel library that provides some low level functionality went 8.0
Before getting into the highlights, let me start by thanking the Critter Stack Core team for all their support, contributions to both the code and documentation, and for being a constant sounding board for me and source of ideas and advice:
Next, I’d like to thank our Critter Stack community for all the interest and the continuous help we get with suggestions, pull requests that improve the tools, and especially for the folks who take the time to create actionable bug reports because that’s half the battle of getting problems fixed. And while there are plenty of days when I wish there wasn’t a veritable pack of raptors prowling around the projects probing for weaknesses in the projects, I cannot overstate the importance for an OSS project to have user and community feedback.
Alright, on to some highlights.
The big changes are that we consolidated several smaller shared libraries into one bigger shared JasperFx library and also combined some smaller libraries like Marten.CommandLine, Weasel.CommandLine, and Lamar.Diagnostics into Marten, Weasel, and Lamar respectfully. That’s hopefully going to help folks get to command line utilities quicker and easier, and the Critter Stack tools do get some value out of those command line utilities.
We’ve now got a shared model to configure behavioral differences at “Development” vs “Production” time for both Marten and Wolverine all at one time like this:
// These settings would apply to *both* Marten and Wolverine
// if you happen to be using both
builder.Services.CritterStackDefaults(x =>
{
x.ServiceName = "MyService";
x.TenantIdStyle = TenantIdStyle.ForceLowerCase;
// You probably won't have to configure this often,
// but if you do, this applies to both tools
x.ApplicationAssembly = typeof(Program).Assembly;
x.Production.GeneratedCodeMode = TypeLoadMode.Static;
x.Production.ResourceAutoCreate = AutoCreate.None;
// These are defaults, but showing for completeness
x.Development.GeneratedCodeMode = TypeLoadMode.Dynamic;
x.Development.ResourceAutoCreate = AutoCreate.CreateOrUpdate;
});
It might be awhile before this pays off for us, but everything from the last couple paragraphs is also meant to speed up the development of additional Event Sourcing “Critter” tools to expand beyond PostgreSQL — not that we’re even slightly backing off our investment in the do everything PostgreSQL database!
For Wolverine 4.0, we’ve improved Wolverine’s ability to support modular monolith architectures that might utilize multiple Marten stores or EF Core DbContext services targeting the same database or even different databases. More on this soon.
Both Wolverine and Marten got some streamlined Open Telemetry span naming changes that were suggested by Pascal Senn of ChiliCream who collaborates with JasperFx for a mutual client.
For both Wolverine and Lamar 15, we added a little more full support for the [FromKeyedService] and “keyed services” in the .NET Core DI abstractions like this for a Wolverine handler:
// From a test, just showing that you *can* do this
// *Not* saying you *should* do that very often
public static void Handle(UseMultipleThings command,
[FromKeyedServices("Green")] IThing green,
[FromKeyedServices("Red")] IThing red)
{
green.ShouldBeOfType<GreenThing>();
red.ShouldBeOfType<RedThing>();
}
And inside of Lamar itself, any dependency from a constructor function that has this:
// Lamar will inject the IThing w/ the key "Red" here
public record ThingUser([FromKeyedServices("Red")] IThing Thing);
Granted, Lamar already had its own version of keyed services and even an equivalent to the [FromKeyedService] attribute long before this was added to the .NET DI abstractions and ServiceProvider conforming container, but .NET is Microsoft’s world and lowly OSS projects pretty well have to conform to their abstractions sometimes.
Just for the record, StructureMap had an equivalent to keyed services in its first production release way back in 2004 back when David Fowler was probably in middle school making googly eyes at Rihanna.
What’s Next for the Critter Stack?
Honestly, I had to cut some corners on documentation to get the releases out for a JasperFx Software client, so I’ll be focused on that for most of this week. And of course, plenty of open issues and some outstanding pull requests didn’t make the release, so those hopefully get addressed in the next couple minor releases.
For the bigger picture, I think the rest of this year is:
“CritterWatch”, our long planned, not moving fast enough for my taste, management and observability console for both Marten and Wolverine.
Improvements to Marten’s performance and scalability for Event Sourcing. We did a lot in that regard last year throughout Marten 7.*, but there’s another series of ideas to increase the throughput even farther.
Wolverine is getting a lot of user contributions right now, and I expect that especially the asynchronous messaging support will continue to grow. I would like to see us add CosmosDb support to Wolverine by the end of the year. By and large, I would like to increase Wolverine’s community usage over all by trying to grow the tool beyond just folks already using Marten — but the Marten + Wolverine combination will hopefully continue to improve.
More Critters? We’re still talking about a SQL Server backed Event Store, with CosmosDb being a later alternative
Wrapping Up
As for the wisdom of ever again making a release cycle where the entire Critter Stack has a major release at the exact same time, this:
Finally, a lot of things didn’t make the release that folks wanted, heck that I wanted, but at some point it becomes expensive for a project to have a long running branch for “vNext” and you have to make the release. I’m hopeful that even though these major releases didn’t add a ton of new functionality that they set us up with the right foundation for where the tools go next.
I also know that folks will have plenty of questions and probably even inevitably run into problems or confusion with the new releases — especially until we can catch up on documentation — but I stole time from the family to get this stuff out this weekend and I’ll probably not be able to respond to anyone but JasperFx customers on Monday. Finally, in the meantime, right after every big push, I promise to start responding to whatever problems folks will have, but:
We were able to publish the Wolverine 3.13.4 release this morning with a handful of important fixes for error retries in a modular monolith architecture, recovering from Rabbit MQ connection interruptions, and Azure Service Bus, Kafka, and Amazon SQS fixes.
The awesome part of this release was how much of it, including a huge fix from Hamed Sabzian, came from the community (e.g. “not me”). Even one of the issues I addressed only came with some significant help from users building reproduction projects. Another issue was reported by a JasperFx Software customer who we’re working with for some new multi-tenancy functionality.
Beyond just the symbolic show of community engagement and involvement with Wolverine, this release hopefully marks the end of new development with Wolverine 3.*. There’s now a maintenance branch for 3.0, but Wolverine’s main branch is now the forthcoming 4.0 release that should hit by Monday next week.
Thank you to all the contributors to this release and recent releases, and that absolutely includes folks who took the time to open actionable issues and create reproduction steps for those issues.
One of the earliest lessons I learned designing software systems is that reigning in unchecked growth of databases through judicious pruning and archiving can do wonders for system performance over time. As yet another tool in the toolbox for scaling Marten and in collaboration with a JasperFx Software customer, we’re adding an important feature in Marten 8.0 called “Stream Compacting” that can be used to judiciously shrink Marten’s event storage to keep the database a little more limber as old data is no longer relevant.
Let’s say that you failed to be omniscient in your event stream modeling and ended up with a longer stream of events than you’d ideally like and that is bloating your database size and maybe impacting performance. Maybe you’re going to be in a spot where you don’t really care about all the old events, but really just want to maintain the current projected state and more recent events. And maybe you’d like to throw the old events in some kind of “cold” storage like an S3 bucket or [something to be determined later].
Enter the new “Stream Compacting” feature that will come with Marten 8.0 next week like so:
public static async Task compact(IDocumentSession session, Guid equipmentId, IEventsArchiver archiver)
{
// Maybe we have ceased to care about old movements of a piece of equipment
// But we want to retain an accurate positioning over the past year
// Yes, maybe we should have done a "closing the books" pattern, but we didn't
// So instead, let's just "compact" the stream
await session.Events.CompactStreamAsync<Equipment>(equipmentId, x =>
{
// We could say "compact" all events for this stream
// from version 1000 and below
x.Version = 1000;
// Or instead say, "compact all events older than 30 days ago":
x.Timestamp = DateTimeOffset.UtcNow.Subtract(30.Days());
// Carry out some kind of user defined archiving process to
// "move" the about to be archived events to something like an S3 bucket
// or an Azure Blob or even just to another table
x.Archiver = archiver;
// Pass in a cancellation token because this might take a bit...
x.CancellationToken = CancellationToken.None;
});
}
What this “compacting” does is effectively create a snapshot of the stream state (the Equipment type in the example above) and replaces the existing events that are archived in the database with a single Compacted<Equipment> event with this shape:
// Right now we're just "compacting" in place, but there's some
// thought to extending this to what one of our contributors
// calls "re-streaming" in their system where they write out an
// all new stream that just starts with a summary
public record Compacted<T>(T Snapshot, Guid PreviousStreamId, string PreviousStreamKey)
The latest, greatest Marten projection bits are always able to restart any SingleStreamProjection with the Snapshot data of a Compacted<T> event, with no additional coding on your part.
And now, to answer a few questions that my client (Carsten, this one’s for you, sorry I was slow today:)) asked me about this today:
Is there going to be a default archiver? Not yet, but I’m all ears on what that could or should be. It’ll always be pluggable of course because I’d expect a wide range of usages
How about async projections? This will not impact asynchronous projections that are already in flight. The underlying mechanism is not using any persisted, projected document state but is instead fetching the raw events and effectively doing a live aggregation to come back to the compacted version of the projected document.
Can you compact a single stream multiple times? Yes. I’m thinking folks could use a projection “side effect” to emit a request message to compact a stream every 1,000 events or some other number.
What happens in case the async daemon moves beyond (e.g. new events were saved while the compacting is ongoing) – will the compacting aggregation overwrite the projection updates done by the async daemon – basically the same for inline projections? The compacting will be done underneath the async daemon, but will not impact the daemon functionality. The projections are “smart enough” to restart the snapshot state from any Compacted<T> event found in the middle of the current events anyway.
How does rewind and replay work if a stream is compacted? Um, you would only be able to replay at or after the point of compacting. But we can talk about making this able to recover old events from archiving in a next phase!
Any other limitations? Yeah, same problem we ran into with the “optimized rebuild” feature from Marten 7.0. This will not play well if there are more than one single stream projection views for the same type of stream. Not insurmountable, but definitely not convenient. I think you’d have to explicitly handle a Compacted<T1> event in the projection for T2 if both T1 and T2 are separate views of the same stream type.
Why do I care? You probably don’t upfront, but this might easily be a way to improve the performance and scalability of a busy system over time as the database grows.
Is this a replacement or alternative to the event archival partitioning from Marten 7? You know, I’m not entirely sure, and I think your usage may vary. But if your database is likely to grow massively large over time and you can benefit from shrinking the size of the “hot” part of the database of events you no longer care about, do at least one or both of these options!
Summary
The widespread advice from event sourcing experts is to “keep your streams short”, but I also partially suspect this is driven by technical limitations of some of the commonly used, early commercial event store tools. I also believe that Marten is less impacted by long stream sizes than many other event store tools, but still, smaller databases will probably outperform bigger ones in most cases.
Time for an update on Critter Stack release plans, and a follow up on my previous Critter Stack Work in Progress post from March. The current plan is to release Marten 8.0, Weasel 8.0, and Wolverine 4.0 on June 1st. It’s not going to be a huge release in terms of new functionality, but there are some important structural changes that will help us build some future features, and we needed to jettison older .NET versions while getting onto the latest Npgsql. “CritterWatch” is still very much planned and a little bit in progress, but we’ve got to get these big releases out first.
The key takeaways are that I want to essentially freeze Marten 7.* for everything but bug fixes right now, and probably freeze Wolverine 3.* for new feature development after a last wave of pull requests gets pulled in over the next couple days.
I’m admittedly too drowsy and lazy to write much tonight, so here’s just a dump of what I wrote up for the rest of our core team to review. I think we’re already at the point where we’re ready to just work on documentation and a few last touches, so the mass majority of this doesn’t get done in time, but here’s the full brain storm:
First though, what’s been done:
.NET 6 & 7 were dropped
Updated to Npgsql 9 across the board
Dropped all synchronous APIs in Marten
Deleted some [Obsolete] APIs in Marten
Consolidation of supporting libraries to a single JasperFx library
JasperFx has that new consolidated configuration option for common configuration like application assembly, code generation, and the stateful resource AutoCreate mode
Pulled out event projections and core event store abstractions to a new JasperFx.Events library
Removed code generation from all projections
Better explicit code options for aggregations and event projections
Wolverine 4 has better handles envelope storage & the transactional inbox/outbox for modular monoliths
Improved “Descriptor” model to describe the static configuration of Wolverine and/or Marten applications that we’ll use for CritterWatch too
Expanded commands for dead letter queue management in Wolverine that was meant for CritterWatch
Multi-tenancy options in Wolverine for SQL Server or PostgreSQL w/o Marten, multi-tenancy usage with EF Core
Punchlist?
Marten 7.40.4 release w/ a pair of outstanding PRs
Cherry pick commits to Marten “master”
JasperFx & JasperFx.Events 1.0
Documentation website?
Weasel “master” branch
All tests should be passing
Marten “master” branch
All tests should be passing
Documentation website should be building – that’s going to take some effort because of code samples
Get Anne’s PR for tutorials in (cool new guided tour of building a system using Event Sourcing and Event Driven Architecture with first Marten, then Wolverine)
Stream Compacting feature – for a JasperFx customer (this is definitely in for Marten 8, this is a big improvement for keeping a larger system running fast over time by compacting the database)
Fix the optimized projection rebuild options? Or rip it out and leave it for CritterWatch?
Figure out what the proper behavior of “Live” aggregations when there’s some ShouldDelete() action going on
Wolverine
One last 3.14 release with easy to grab pull requests and bug fixes
Rebase on 3.14
Fork off the 3.0 branch
4.0 becomes main branch
All tests should be passing
Documentation website should build
Migration guide
Critter Watch preparation
When integrated w/ CritterWatch, Wolverine can build the descriptor model for the entire application, including EventStoreUsage. No idea where this work stands right now. Did quite a bit earlier this year, then went off in a different direction
Review all Open Telemetry usage and activity naming across Marten and especially Wolverine. Add Open Telemetry & Metrics metadata to the descriptor model sent to CritterWatch. I think this is somewhat likely to get done before Wolverine 4.0.
Ability to send messages from CritterWatch to Wolverine. Might push through some kind of message routing and/or message handler extensibility
Programmatic message routing in Wolverine that varies based on the message contents? This is around options to route a message to one of a set of destinations based on the message core. Thinking about concurrency here. Could be done later.
More open issues in the Marten 8 milestone, but it’s about time to drop any issue that isn’t a breaking change
As I wrote last week, message or request concurrency is probably the single most common source of client questions in JasperFx Software consulting and support work around the Critter Stack. Wolverine is a powerful tool for command and event message processing, and it comes with a lot of built in options for wide range of usage scenarios that provider the answers for a lot of the questions we routinely field from clients and other users. More specifically, Wolverine provides a lot of adjustable knobs to limit or expand:
Message processing parallelism. I.e., how many messages can be executed simultaneously
Message ordering when you need messages to be processed in sequence, or a lack thereof when you don’t
For better or worse, Wolverine has built up quite a few options over the years, and that can be admittedly confusing. Also, there are real performance or correctness tradeoffs with the choices you make around message ordering and processing parallelism. To that end, let’s go through a little whirlwind tour of Wolverine’s options for concurrency, parallelism, and delivery guarantees.
Listener Endpoints
Note that Wolverine standardizes the fluent interface options for endpoint type, message ordering, and parallel execution are consistent across all of its messaging transport types (Rabbit MQ, Azure Service Bus, Kafka, Pulsar, etc.), though not every option is available for every transport.
All messages handled in a Wolverine application come from a constantly running listener “Endpoint” that then delegates the incoming messages to the right message handler. A Wolverine “Endpoint” could be a local, in process queue, a Rabbit MQ queue, a Kafka topic, or an Azure Service Bus subscription (see Wolverine’s documentation on asynchronous messaging for the entire list of messaging options).
This does vary a bit by messaging broker or transport, but there are three modes for Wolverine endpoints, starting with Inline endpoints:
// Configuring a Wolverine application to listen to
// an Azure Service Bus queue with the "Inline" mode
opts.ListenToAzureServiceBusQueue(queueName, q => q.Options.AutoDeleteOnIdle = 5.Minutes()).ProcessInline();
With an Inline endpoint, messages are pulled off the receiving queue or topic one message at a time, and “ack-ed” back to the original queue or topic only on the successful completion of the message handler. This mode completely eschews any kind of durable, transactional inbox, but does still give you an at-least-once delivery guarantee as it’s possible that the “ack” process could fail after the message is successfully handled, potentially resulting in the message being resent from the external messaging broker. Know though that this is rare, and Wolverine puts some error retries around the “ack-ing” process.
As you would assume, using the Inline mode gives you sequential processing of messages within a single node, but limits parallel handling. You can opt into running parallel listeners for any given listening endpoint:
opts.ListenToRabbitQueue("inline")
// Process inline, default is with one listener
.ProcessInline()
// But, you can use multiple, parallel listeners
.ListenerCount(5);
The second endpoint mode is Buffered where messages are pulled off the external messaging queue or topic as quickly as they can be, and immediately put into an in memory queue and “ack-ed” to any external broker.
// I overrode the buffering limits just to show
// that they exist for "back pressure"
opts.ListenToAzureServiceBusQueue("incoming")
.BufferedInMemory(new BufferingLimits(1000, 200));
In the sample above, I’m showing how you can override the defaults for how many messages can be buffered in memory for this listening endpoint before the endpoint is paused. Wolverine has some support for back pressure within its Buffered or Durable endpoints to prevent memory from being overrun.
With Buffered or the Durable endpoints I’ll describe next, you can specify the maximum number of parallel messages that can be processed at one time within a listener endpoint on a single node like this:
Or you can choose to run messages in a strict sequential order, one at a time like this:
// Make any kind of Wolverine configuration
options
.PublishMessage<Module1Message>()
.ToLocalQueue("module1-high-priority")
.Sequential();
The last endpoint type is Durable, which behaves identical to the Buffered approach except that messages received from external message brokers are persisted to a backing database first before processing, then deleted when the messages are successfully processed or discarded or moved to dead letter queues by error handling policies:
Using the Durable mode enrolls the listening endpoint into Wolverine’s transactional inbox. This is the single most robust option for delivery guarantees with Wolverine, and even adds some protection for idempotent receipt of messages such that Wolverine will quietly reject the same message being received multiple times. Durable endpoints are more robust in terms of delivery guarantees and resilient in the face of system hiccups than the Buffered mode, but does incur a little bit of extra overhead making calls to a database — but I should mention that Wolverine is trying really hard to batch up calls to the database whenever it can for better runtime efficiency, and there are retry loops in all the internals for resiliency as well.
If you really read this post you should hopefully be badly abused of the flippant advice floating around .NET circles right now after the MassTransit commercialization announcement that you can “just” write your own abstractions over messaging brokers instead of using a robust, off the shelf toolset that will have far more engineering for resiliency and observability than most folks realize.
Scenarios
Alright, let’s talk about some common messaging scenarios and look at possible Wolverine options. It’s important to note that there is some real tension between throughput (how many messages can you process over time), message ordering requirements, and delivery guarantees and I’ll try to call those compromises as we go.
You have a constant flood of small messages coming in that are relatively cheap to process…
In this case I would choose a Buffered endpoint and allow it to run messages in parallel:
Letting messages run without any strict ordering will allow the endpoint to process messages faster. Using the Buffered approach will allow the endpoint to utilize any kind of message batching that external message brokers might support, and does a lot to remove the messaging broker as a bottle neck for message processing. The Buffered approach isn’t durable of course, but if you care about throughput more than guarantees or message ordering, it’s the best option.
Note that any Buffered or Durable endpoint automatically allows for parallel message processing capped by the number of processor cores for the host process.
A message is expensive to process…
If you have a message type that turns out to require a lot of resources to process, you probably want to limit the parallelization to restrict how many resources the system uses for this message type. I would say to either use an Inline endpoint:
opts.ListenToRabbitQueue("expensive")
// Process inline, default is with one listener
.ProcessInline()
// Cap it to no more than two messages in parallel at any
// one time
.ListenerCount(2);
or a Buffered or Durable endpoint, but cap the parallelization.
Messages should be processed in order, at least on each node…
Use either a ProcessInline endpoint, or use the Sequential() option on any other kind of endpoint to limit the local processing to single file:
A certain type of message should be processed in order across the entire application…
Sometimes there’s a need to say that a certain set of messages within your system need to be handled in strict order across the entire application. While some specific messaging brokers have some specific functionality for this scenario, Wolverine has this option to ensure that a listening endpoint for a certain location only runs on a single node within the application at any one time, and always processes in strict sequential order:
var host = await Host.CreateDefaultBuilder().UseWolverine(opts =>
{
opts.UseRabbitMq().EnableWolverineControlQueues();
opts.PersistMessagesWithPostgresql(Servers.PostgresConnectionString, "listeners");
opts.ListenToRabbitQueue("ordered")
// This option is available on all types of Wolverine
// endpoints that can be configured to be a listener
.ListenWithStrictOrdering();
}).StartAsync();
Watch out of course, because this throttles the processing of messages to single file on exactly one node. That’s perfect for cases where you’re not too concerned about throughput, but sequencing is very important. A JasperFx Software client is using this for messages to a stateful Saga that coordinates work across their application.
Do note that Wolverine will both ensure a listener with this option is only running on one node, and will redistribute any strict ordering listeners to better distribute work across a cluster. Wolverine will also be able to detect when it needs to switch the listening over to a different node if a node is taken down.
Messages should be processed in order within a logical group, but we need better throughput otherwise…
Let’s say that you have a case where you know the system would work much more efficiently if Wolverine could process messages related to a single business entity of some sort (an Invoice? a Purchase Order? an Incident?) in strict order. You still need more throughput than you can achieve through a strictly ordered listener that only runs on one node, but you do need the messages to be handled in order or maybe just one at a time for a single business entity to arrive at consistent state or to prevent errors due to concurrent access.
_host = await Host.CreateDefaultBuilder()
.UseWolverine(opts =>
{
opts.UseAzureServiceBusTesting()
.AutoProvision().AutoPurgeOnStartup();
opts.ListenToAzureServiceBusQueue("send_and_receive");
opts.PublishMessage<AsbMessage1>().ToAzureServiceBusQueue("send_and_receive");
opts.ListenToAzureServiceBusQueue("fifo1")
// Require session identifiers with this queue
.RequireSessions()
// This controls the Wolverine handling to force it to process
// messages sequentially
.Sequential();
opts.PublishMessage<AsbMessage2>()
.ToAzureServiceBusQueue("fifo1");
opts.PublishMessage<AsbMessage3>().ToAzureServiceBusTopic("asb3");
opts.ListenToAzureServiceBusSubscription("asb3")
.FromTopic("asb3")
// Require sessions on this subscription
.RequireSessions(1)
.ProcessInline();
}).StartAsync();
But, there’s a little bit more to publishing because you’ll need to tell Wolverine what the GroupId value is for your message:
I think we’ll try to make this a little more automatic in the near future with Wolverine.
// bus is an IMessageBus
await bus.SendAsync(new AsbMessage3("Red"), new DeliveryOptions { GroupId = "2" });
await bus.SendAsync(new AsbMessage3("Green"), new DeliveryOptions { GroupId = "2" });
await bus.SendAsync(new AsbMessage3("Refactor"), new DeliveryOptions { GroupId = "2" });
Of course, if you don’t have Azure Service Bus, you still have some other options. I think I’m going to save this for a later post, hopefully after building out some formal support for this, but another option is to:
Plan on having several different listeners for a subset of messages that all have the strictly ordered semantics as shown in the previous section. Each listener can at least process information independently
Use some kind of logic that can look at a message being published by Wolverine and use some kind of deterministic rule that will assign that message to one of the strictly ordered messaging destinations
Like I said, more to come on this in the hopefully near future, and this might be part of a JasperFx Software engagement soon.
What about handling events in Wolverine that are captured to Marten (or future Critter Event Stores)?
I’m Gen X, so the idea of Marten & Wolverine assembling to create the ultimate Event Driven Architecture stack makes me think of Transformers cartoons:)
It’s been a few years, but what is now Wolverine was originally called “Jasper” and was admittedly a failed project until we decided to reorient it to being a complement to Event Sourcing with Marten and renamed it “Wolverine” to continue the “Critter Stack” theme. A huge part of that strategy was having first class mechanisms to either publish or handle events captured by Marten’s Event Sourcing through Wolverine’s robust message execution and message publishing capabilities.
You have two basic mechanisms for this. The first, and original option is “Event Forwarding” where events captured by Marten are published to Wolverine upon the successful completion of the Marten transaction:
builder.Services.AddMarten(opts =>
{
var connString = builder
.Configuration
.GetConnectionString("marten");
opts.Connection(connString);
// There will be more here later...
opts.Projections
.Add<AppointmentDurationProjection>(ProjectionLifecycle.Async);
// OR ???
// opts.Projections
// .Add<AppointmentDurationProjection>(ProjectionLifecycle.Inline);
opts.Projections.Add<AppointmentProjection>(ProjectionLifecycle.Inline);
opts.Projections
.Snapshot<ProviderShift>(SnapshotLifecycle.Async);
})
// This adds a hosted service to run
// asynchronous projections in a background process
.AddAsyncDaemon(DaemonMode.HotCold)
// I added this to enroll Marten in the Wolverine outbox
.IntegrateWithWolverine()
// I also added this to opt into events being forward to
// the Wolverine outbox during SaveChangesAsync()
.EventForwardingToWolverine();
Event forwarding gives you no ordering guarantees of any kind, but will push events as messages to Wolverine immediately. Event forwarding may give you significantly better throughput then the subscription model we’ll look at next because there’s less latency between persisting the event to Marten and the event being published to Wolverine. Moreover, using “Event Forwarding” means that the event publishing happens throughout any application cluster.
However, if you need strictly ordered handling of the events being persisted to Marten, you instead need to use the Event Subscriptions model where Wolverine is handling or relaying Marten events as messages in the strict order in which they are appended to Marten, and on a single running node. This is analogous to the strictly ordered listener option explained above.
There’s a real tradeoff between message ordering, processing throughput, and message delivery guarantees. Fortunately, Wolverine gives you plenty of options to meet a variety of different project requirements.
And one last time, you’re just not going to want to sign up for the level of robust options and infrastructure that’s under the covers of a tool like Wolverine can “just roll your own messaging abstractions” because you’re angry and think that community OSS tools can’t be trusted. And also, Wolverine is also a moving target that constantly improves based on the problems, needs, suggestions, and code contributions from our core team, community, and JasperFx Software customers. Your homegrown tooling will never receive that level of feedback, and probably won’t ever match Wolverine’s quality of documentation either.
I think that even in a crowded field of existing “mediator” tools, ASP.Net Core endpoint frameworks, and asynchronous messaging frameworks in .NET that Wolverine has a great opportunity to grow its user base this year. There’s a lot of value that Wolverine brings to the table that I don’t believe other tools do, but it’s been very focused on improving development in conjunction with Marten. With Wolverine 4.0, I’m hoping that much deeper support for EF Core will help Wolverine’s community grow by attracting more developers who aren’t necessarily using Marten (yet).
The “Critter Stack” (Marten and Wolverine) already has very strong support for multi-tenancy as you can see in some previous posts of mine:
And I think you get the point. Marten and Wolverine have a much more comprehensive feature set for multi-tenancy than any other persistence or messaging tool in the .NET ecosystem — in no small part because we seem to be the only community that cares about this somehow?
For Wolverine 4.0 (expected by June 1st) and in conjunction with a JasperFx Software customer, we’re adding first class multi-tenancy support for EF Core usage. Specifically, we’re aiming to allow users to use every bit of Wolverine’s existing multi-tenancy integration, transactional inbox/outbox support, and transactional middleware with EF Core while targeting a separate database for each tenant.
This work is in flight, but here’s a preview of the (working thank you) syntax. In bootstrapping, we need to tell Wolverine about both a main database for Wolverine storage and any tenant databases as in this sample from a web application:
builder.Host.UseWolverine(opts =>
{
var mainConnectionString = builder.Configuration.GetConnectionString("main");
opts.PersistMessagesWithSqlServer(mainConnectionString)
// If you have a fixed number of tenant databases and it won't
// change w/o downtime -- but don't worry, there are other options coming...
.RegisterStaticTenants(x =>
{
x.Register("tenant1", builder.Configuration.GetConnectionString("tenant1"));
x.Register("tenant2", builder.Configuration.GetConnectionString("tenant2"));
x.Register("tenant3", builder.Configuration.GetConnectionString("tenant]3"));
});
opts.Policies.AutoApplyTransactions();
opts.Policies.UseDurableLocalQueues();
TestingOverrides.Extension?.Configure(opts);
});
Next, we need to tell Wolverine to make one or more of our EF Core DbContext types multi-tenanted using the databases we configured above like this:
// Little sleight of hand, we're registering the DbContext with Wolverine,
// but letting Wolverine deal with the connection string at runtime
builder
.Services
.AddDbContextWithWolverineManagedMultiTenancy<ItemsDbContext>((b, connectionString) =>
// Notice the specification of AutoCreate here, more in a second...
b.UseSqlServer(connectionString), AutoCreate.CreateOrUpdate);
While Wolverine does support multi-tenancy tracking through both local message publishing and asynchronous messaging, let’s pretend our workflows start from an HTTP endpoint, so I’m going to add some basic tenant id detection using Wolverine.HTTP like so in our Program file:
var app = builder.Build();
app.MapWolverineEndpoints(opts =>
{
// Set up tenant detection
opts.TenantId.IsQueryStringValue("tenant");
opts.TenantId.DefaultIs(StorageConstants.DefaultTenantId);
});
Alrighty then, here’s that simplistic little DbContext I’m using for testing right now:
public class ItemsDbContext : DbContext
{
public ItemsDbContext(DbContextOptions<ItemsDbContext> options) : base(options)
{
}
public DbSet<Item> Items { get; set; }
protected override void OnModelCreating(ModelBuilder modelBuilder)
{
// Your normal EF Core mapping
modelBuilder.Entity<Item>(map =>
{
map.ToTable("items", "mt_items");
map.HasKey(x => x.Id);
map.Property(x => x.Id).HasColumnName("id");
map.Property(x => x.Name).HasColumnName("name");
map.Property(x => x.Approved).HasColumnName("approved");
});
}
}
And finally, here’s a message handler that’s also doubling as a simplistic HTTP endpoint to create a new Item in our system by accepting a command:
public record StartNewItem(Guid Id, string Name);
public static class StartNewItemHandler
{
[WolverinePost("/item")]
public static IStorageAction<Item> Handle(StartNewItem command)
{
return new Insert<Item>(new Item
{
Id = command.Id,
Name = command.Name
});
}
}
For a little context, you might want to check out Wolverine’s storage side effects. Or with less “magic”, we could use this completely equivalent alternative:
public static class StartNewItemHandler
{
[WolverinePost("/item")]
public static void Handle(StartNewItem command, ItemsDbContext dbContext)
{
var item = new Item
{
Id = command.Id,
Name = command.Name
};
dbContext.Items.Add(item);
}
}
Either way, Wolverine’s transactional middleware actually calls ItemsDbContext.SaveChangesAsync() for us and also deals with any necessary transactional outbox mechanics to “flush” outgoing messages after the transaction has succeeded.
Alright, let’s go to runtime, where Wolverine is going to happily handle a POST to /item by trying to find the tenant id out of a query string variable named “tenant”, then build an ItemsDbContext instance for us that’s pointed at the proper SQL Server database for the named tenant id. Our code up above doesn’t have to know anything about that process, we just have to write code to carry out the business requirements.
And that’s that! Of course there are plenty of other things to worry about and questions you might have, so let me try to anticipate those:
What about dynamically adding tenants without any downtime? Like Marten’s “Master Table Tenancy” model where you can add new tenants without any system down time and it “just works”, including Wolverine being able to spin up the necessary background agents for scheduled messages and whatnot? That’s definitely planned, and it will end up sharing quite a bit of code with the existing Marten support in this case and building on existing Wolverine capabilities
Will this work with Wolverine’s existing EF Core Saga support? Yes, or at least that’s planned and I’ll test that tomorrow
Does Wolverine’s transactional inbox/outbox support and scheduled messages extend to this new multi-tenancy model? Yes, that’s already working.
Can I use this with lightweight sagas? Yep, that too. Not working yet, but that will be in place by the end of tomorrow.
I’m spoiled by Marten and Wolverine’s ability to set up development databases on the fly, is there any chance of something like that for this EF Core integration? If you follow me on BlueSky (sorry), you might have seen me gripe about EF Core migrations as compared to (in my honest opinion) Marten’s much easier model for development time. After some griping and plenty of investigation, we will have at least a model where you can opt into having Wolverine use EF Core migrations to create any missing databases and apply any missing migrations to all the tenant databases at application startup. Having that capability is helping speed up the development of all of this.
Will the tenant database discovery be pluggable so we can use whatever existing mechanism we already have? Yes.
Can I use the Wolverine managed EF Core multi-tenancy outside of Wolverine message handlers or HTTP endpoints? That work hasn’t started yet, but that’s a particular need for JasperFx Software client work
If I’m using PostgreSQL and both Marten and EF Core, can we use both tools in the same application with multi-tenancy? Hopefully with just one set of configuration? Absolutely, yes.
Will there be a Wolverine powered equivalent to Marten’s “conjoined tenancy” model for multi-tenancy through one database? Um, not sure, probably not at first — but I’d be happy to talk to anyone who wants to volunteer for that pull request or wants to engage JasperFx to build that out!
JasperFx Software is the company I founded a little over two years ago to create an Open Core business model around the “Critter Stack” suite of open source tools (primarily Marten and Wolverine, with some smaller supporting tools). So far, our main sources of revenue (right now it’s myself with contributions from Babu Annamalai, but we’d sure like to grow soon!) have been technical consulting to help our customers get the best possible results from Marten or Wolverine, custom feature development within the tools, and ongoing support contracts.
Just by the nature of what they are for (asynchronous messaging, event sourcing, and data persistence), the “Critter Stack” tools have to be considered a mission critical part of your technical infrastructure. You can pick these tools off the shelf knowing that there is a company and community behind the tools even though they’re free to use through the permissive MIT license. To that point, a support plan from JasperFx Software gives you the piece of mind to use these tools knowing that you have ready access to the technical experts for questions or to have any problems you encounter with the tools addressed.
The support contracts include a dedicated, private Discord or Slack room for your company for relatively quick response (our SLA is 24 hours, but we generally answer much faster than that). We aren’t just there for defects, we’re (JasperFx) also there to answer questions and to advise you on best usages of the tools as you need within the bounds of the contract. I’ve frequently jumped on Zoom or Teams calls with our customers for trickier questions or just when it takes more communication to really get to a solution for our customers. I can proudly say that every single JasperFx support customer has renewed their yearly support plan when the first year was up so far.
Just to give you an idea of what kind of issues JasperFx can help you with, the most common issues have been:
Concurrency, concurrency, concurrency. Sometimes it’s helping users design queueing and messaging topologies to ameliorate concurrent access, sometimes it’s helping them to leverage Marten’s optimistic and pessimistic locking support, and sometimes it’s helping to design Wolverine resiliency strategies.
Guidance on Event Sourcing usage within the Critter Stack, with designing event projections being a particularly common source of questions
Multi-tenancy usage. Marten and Wolverine both have unusually strong support for multi-tenancy scenarios as a result of our users coming up with more and more scenarios for us!
Automated testing, both how to leverage Wolverine capabilities to write more easily testable business logic code and how to use both Marten and Wolverine’s built in support for integration testing
Plenty of issues around messaging brokers and messaging patterns
There’s been some consternation about some other widely used .NET OSS tools moving to commercial licenses that have caused many people to proclaim that they should just roll their own tools instead of paying for a commercial tool or using an OSS tool off the shelf that might become commercial down the road. I’m going to suggest a little different thinking.
Before you try to roll your own Event Sourcing tool, just know that Marten is over a decade old, it’s well documented, and it’s the most widely used Event Sourcing tool in the .NET ecosystem (by Nuget downloads, and it’s not really close at all). Moreover, you get the benefit of a tool that’s been beaten on and solves a lot of very real, and quite complex problems with Event Sourcing usage that you may not even know you’re going to have.
Before you “just write your own abstraction over messaging brokers”, know that tools like Wolverine do a lot more than just abstract away tools like Rabbit MQ or Azure Service Bus. Resiliency features — and some of that is quite more complicated than just plopping in Polly, Open Telemetry tracing, other instrumentation, dealing with serialization, stateful saga workflows, multi-tenancy, scheduled message execution, and transactional inbox/outbox features are just some of the built in capabilities that Wolverine provides. And besides all the normal features you’d expect out of a messaging tool in .NET, Wolverine potentially does much, much more within your application code to simplify your development efforts. The people who really embrace Wolverine’s different approach to application code love how it drastically reduces code ceremony compared to more common Clean/Onion Architecture layered approaches using other competitors. Having an ongoing relationship through a JasperFx Software support contract will only help you wring out the very most from your Wolverine usage.
The Shade Tree Developer 