Resiliency and Low Level Improvements in Marten 7 (this post)
I hate to tell you all this, but sometimes there’s going to be occasional hiccups with your system in production. Just sticking with using Marten and its connectivity to PostgreSQL under the covers, you could easily have:
Occasional network issues that cause transactions to fail
A database could be temporarily too busy and throw exceptions
Concurrency issues from trying to write to the same rows in the underlying database (Marten isn’t particularly prone to this, but I needed another example)
These typical transient errors happen, but that doesn’t mean that the operation that just failed couldn’t happily succeed if you just retried it in a little bit. To that end, Marten 7 replaced our previous homegrown resiliency approach (that didn’t really work anyway) with a reliance on the popular Polly library.
The default settings for Marten are shown below:
// Default Polly setup
var strategy = new ResiliencePipelineBuilder().AddRetry(new()
{
ShouldHandle = new PredicateBuilder().Handle<NpgsqlException>().Handle<MartenCommandException>().Handle<EventLoaderException>(),
MaxRetryAttempts = 3,
Delay = TimeSpan.FromMilliseconds(50),
BackoffType = DelayBackoffType.Exponential
}).Build();
The Marten docs here will tell you how to customize these policies at your discretion.
To put this into context, if you call IDocumentSession.SaveChangesAsync() to commit a unit of work, and there’s an exception that matches the criteria shown above, Polly will re-execute the queued operations from scratch with a brand new connection so the retry can succeed after the database or network has settled down.
Moreover, we followed the Polly recommendations on performance to utilize “static Lambdas” within the Marten internals to reduced the number of object allocations within Marten 7’s execution pipeline compared to the Marten 6 and earlier pipeline. It’s kind of ugly code, but you can see where and how we used Polly in QuerySession.Execution.cs.
Another big change for Marten 7 was how Marten is going to manage database connection lifecycles inside of Marten sessions. Prior to Marten 7, Marten sessions would open a database connection and keep it open until the session was ultimately disposed. This decision was originally made to optimize the integration between Marten and Dapper when Marten’s functionality was very limited.
Now though, Marten will only open a database connection within a session immediately before any operation that involves a database connection, and close that connection immediately after the operation is over (really just returning the underlying connection to the connection pool managed by Npgsql). With this change, it is now safe to run read-only queries through IQuerySession (or lightweight IDocumentSession) objects in multiple threads. That should make Marten be more effective within Hot Chocolate integrations:
Marten sessions can actually be thread safe so that you can let the default IoC registration behavior for Marten sessions happily work within Hot Chocolate requests where it wants to parallelize queries
Marten usage will be far, far less prone to connection leaks when developers create sessions without disposing them properly
And even if you’re not using Hot Chocolate at all, Marten 7 will be more parsimonious over all with how it uses database connections which should help with scalability of your system — and definitely will help with cloud hosting options that charge by the number of database connections used!
Final Thoughts
In the past month or so I’ve had more interaction with developers than usual who are highly suspicious of open source tooling and especially of alternative open source tooling in .NET that isn’t directly supported by Microsoft. It’s a fair point to some degree, because “Google/Bing-bility” is a highly underrated quality of a development tool. Some of that fear was thinking that the responsible developers behind a non mainstream tool are just going to get tired of it and abandon it on a whim when a shinier object comes around so it’s not even worth the time to care about those alternatives from the “cool kids.”
What I would like to point out with Marten in particular is that it’s nearly a decade old as a project (the document db features in Marten actually predate CosmosDb in .NET world) and still very active and growing. The work in this post is all about making fine grained refinements to the core project and I would hope demonstrate a dedication on the part of the whole community toward continuously improving Marten for serious work. In other words, I think development shops can absolutely feel confident in placing a technical bet on Marten.
Hey, did you know that JasperFx Software offers formal support plans for Marten and Wolverine? Not only are we making the “Critter Stack” tools be viable long term options for your shop, we’re also interested in hearing your opinions about the tools and how they should change.We’re also certainly open to help you succeed with your software development projects on a consulting basis whether you’re using any part of the Critter Stack or some completely different .NET server side tooling.
Marten 7.0 was released over the weekend, and Wolverine 2.0 followed yesterday mostly to catch up with the Marten dependency. One of the major improvements in this round of “Critter Stack” releases was to address a JasperFx client’s need for dynamically adding new tenant databases at runtime without having to do any kind of system deployment.
Doing that successfully meant adding a couple capabilities throughout the “Critter Stack.” First off, Marten needed a new multi-tenancy configuration strategy that allowed users to keep a list of valid tenant id and database connection strings for those tenants in a database table. Enter Marten 7’s new Master Table Tenancy Model. You can see the usage in this sample from the documentation:
using var host = await Host.CreateDefaultBuilder()
.ConfigureServices(services =>
{
services.AddMarten(sp =>
{
var configuration = sp.GetRequiredService<IConfiguration>();
var masterConnection = configuration.GetConnectionString("master");
var options = new StoreOptions();
// This is opting into a multi-tenancy model where a database table in the
// master database holds information about all the possible tenants and their database connection
// strings
options.MultiTenantedDatabasesWithMasterDatabaseTable(x =>
{
x.ConnectionString = masterConnection;
// You can optionally configure the schema name for where the mt_tenants
// table is stored
x.SchemaName = "tenants";
// If set, this will override the database schema rules for
// only the master tenant table from the parent StoreOptions
x.AutoCreate = AutoCreate.CreateOrUpdate;
// Optionally seed rows in the master table. This may be very helpful for
// testing or local development scenarios
// This operation is an "upsert" upon application startup
x.RegisterDatabase("tenant1", configuration.GetConnectionString("tenant1"));
x.RegisterDatabase("tenant2", configuration.GetConnectionString("tenant2"));
x.RegisterDatabase("tenant3", configuration.GetConnectionString("tenant3"));
// Tags the application name to all the used connection strings as a diagnostic
// Default is the name of the entry assembly for the application or "Marten" if
// .NET cannot determine the entry assembly for some reason
x.ApplicationName = "MyApplication";
});
// Other Marten configuration
return options;
})
// All detected changes will be applied to all
// the configured tenant databases on startup
.ApplyAllDatabaseChangesOnStartup();;
}).StartAsync();
With this tenancy model, Marten is able to discover newly added tenant databases at runtime as needed during normal transactions. And don’t fret about the extra activity, Marten is able to cache that information in memory to avoid making too many unnecessary database calls.
But what about if I need to decommission and remove a customer database? What if we need to move a tenant database at runtime? Can Marten create databases on the fly for us? How will I do that?!? And sorry, my reply is that will require at least an application restart for now, and any kind of fancier management for advanced multi-tenancy will likely go into the forthcoming “Critter Stack Pro” paid model later this year.
That’s part one. The next issue was that for users who also used Marten’s asynchronous projection feature, the “async daemon” subsystem in Marten needed to be able to discover new tenant databases in the background and ensure that all the asynchronous projections for these newly discovered databases are running in the background somewhere in the application. This led to a partial rewrite of the “async daemon” subsystem for Marten 7, but you can see the positive effect of that work in this test that “proves” that Marten is able to spin up projection building agents in the background at runtime:
[Fact]
public async Task add_tenant_database_and_verify_the_daemon_projections_are_running()
{
// In this code block, I'm adding new tenant databases to the system that I
// would expect Marten to discover and start up an asynchronous projection
// daemon for all three newly discovered databases
var tenancy = (MasterTableTenancy)theStore.Options.Tenancy;
await tenancy.AddDatabaseRecordAsync("tenant1", tenant1ConnectionString);
await tenancy.AddDatabaseRecordAsync("tenant2", tenant2ConnectionString);
await tenancy.AddDatabaseRecordAsync("tenant3", tenant3ConnectionString);
// This is a new service in Marten specifically to help you interrogate or
// manipulate the state of running asynchronous projections within the current process
var coordinator = _host.Services.GetRequiredService<IProjectionCoordinator>();
var daemon1 = await coordinator.DaemonForDatabase("tenant1");
var daemon2 = await coordinator.DaemonForDatabase("tenant2");
var daemon3 = await coordinator.DaemonForDatabase("tenant3");
// Just proving that the configured projections for the 3 new databases
// are indeed spun up and running after Marten's new daemon coordinator
// "finds" the new databases
await daemon1.WaitForShardToBeRunning("TripCustomName:All", 30.Seconds());
await daemon2.WaitForShardToBeRunning("TripCustomName:All", 30.Seconds());
await daemon3.WaitForShardToBeRunning("TripCustomName:All", 30.Seconds());
}
Now, switching over to Wolverine 2.0 and its contribution to the party, the third part to make this dynamic tenant database discovery work throughout the entire “Critter Stack” was for Wolverine to be able to also discover the new tenant databases at runtime and spin up its “durability agents” for background message scheduling and its transactional inbox/outbox support. It’s admittedly not well at all documented yet, but Wolverine has an internal system for leader election and “agent assignment” between running nodes in an application cluster to distribute work. Wolverine uses this subsystem to distribute the transactional inbox/outbox work for each tenant database across the application cluster. Look for more information on this capability as JasperFx Software will be exploiting this for a different customer engagement this year.
A new feature in the big Marten 7.0 release this weekend is an alternative to add numeric revisions to a document as a way of enforcing optimistic concurrency checks.
First off, from Martin Fowler’s seminal Patterns of Enterprise Application Architecture (which is just visible to me on my book case across the room as I write this even though it’s 20 years old now), an Optimistic Offline Lock is:
Prevents conflicts between concurrent business transactions by detecting a conflict and rolling back the transaction.
David Rice
In a simple usage, let’s say we’re building some kind of system to make reservations for restaurants. Logically, we’d have a document named Reservation, and we’ve decided that we want to use the numeric revisioning on this document. That document type could look something like this:
// By implementing the IRevisioned
// interface, we're telling Marten to
// use numeric revisioning with this
// document type and keep the version number
// on the Version property
public class Reservation: IRevisioned
{
public Guid Id { get; set; }
// other properties
public int Version { get; set; }
}
Now, let’s see this in action just a little bit:
public static async Task try_revisioning(IDocumentSession session, Reservation reservation)
{
// This will create a new document with Version = 1
session.Insert(reservation);
// "Store" is an upsert, but if the revisioned document
// is all new, the Version = 1 after changes are committed
session.Store(reservation);
// If Store() is called on an existing document
// this will just assign the next revision
session.Store(reservation);
// *This* operation will enforce the optimistic concurrency
// The supplied revision number should be the *new* revision number,
// but will be rejected with a ConcurrencyException when SaveChanges() is
// called if the version
// in the database is equal or greater than the supplied revision
session.UpdateRevision(reservation, 3);
// This operation will update the document if the supplied revision
// number is greater than the known database version when
// SaveChanges() is called, but will do nothing if the known database
// version is equal to or greater than the supplied revision
session.TryUpdateRevision(reservation, 3);
// Any checks happen only here
await session.SaveChangesAsync();
}
Summary
In the end, this is another alternative to the older Guid based version tracking that Marten has supported since 1.0. I don’t know about you, but I can certainly read and understand an integer much more easily than a random string of letters, numbers, and dashes.
In reality though, this feature was specifically built as a prerequisite to some serious improvements to the asynchronous projection support in Marten. Time and ambition permitting, the next Marten 7.0 blog post will show how Marten can support the strongly consistent “write model” projections you need for command processing while also being performant and allowing for zero downtime projection rebuilds.
Marten 7.0 is released to the wild as of right now! Before getting into the highlights of what’s improved in this release, let’s go right to thanking some of the folks who made big contributions to this release either through code, testing, or feedback:
Vedran Zakanj for sponsoring Marten and also contributing ideas around schema management
And a huge thanks to Lucas Wyland for specifically sponsoring the improved LINQ provider work with an equally huge apology from me on how long that took to finish
And to many more community members who helped improve Marten throughout this very long release cycle.
Highlights
This was a huge release, if not nearly as disruptive as Marten 4 was several years ago. I do not anticipate a lot of issues for users upgrading from Marten 6 to Marten 7, but see the migration guide for more details.
The highlights of Marten 7 are:
The LINQ query support was given a large overhaul that both expanded its supported use cases and led to significantly improved performance of many common sub collection queries — which has been a large complaint and request for improvement from the Marten community for several years
A “Partial” document update capability using native PostgreSQL functionality with no JavaScript in sight! That’s been a long requested capability.
The very basic database execution pipeline underneath Marten was largely rewritten to be far more parsimonious with how it uses database connections and to take advantage of more efficient Npgsql usage. We think these changes will make Marten both more efficient overall (these changes reduced the number of object allocations by quite a bit) and help system health through using fewer database connections
We introduced Polly for resiliency to transient errors like network hiccups or a temporarily overloaded database and actually made Marten able to properly execute retries of database writes and database reads
The “async daemon” subsystem was somewhat rewritten with substantial improvements for application scalability. The asynchronous projection support also has an all new scheme for resiliency that we think will be a big improvement for our users
An option to utilize Marten’s recommended FetchForWriting() API for “write model” aggregation with asynchronous projections. This may sound like a lot of mumbo jumbo, but it’s vital because this enables the next bullet point
The ability to do zero downtime deployments of some projection changes as well as to do blue/green deployments of revisioned projections. Much more on this later this week.
A new alternative for “revisioned” documents with a numeric version as an alternative to Marten’s existing GUID based versioning scheme for optimistic concurrency
We’ll see how big of a deal this turns out to be, but Marten 7 enables the usage of Project Aspire with Marten
As time permits, I will be writing deep dive blog posts on each of the individual bullet points above over the next couple weeks — partially as a way to force the completion of some not perfectly updated documentation!
You can Place a Technical Bet on Marten
There’s frequently an understandable hesitation on the part of software shops to take a bet on an open source tool as a critical piece of their technical infrastructure — and that’s sometimes worse in the .NET ecosystem where OSS adoption isn’t as widespread. All that aside, I’m here to tell you that you can feel safe making a large technical bet on Marten because:
Marten is already a very mature project that has been in production usage since its 1.0 release in 2016
While Marten doesn’t have every single issue around production support, deployments, and schema management fixed yet, we’ve got a detailed roadmap to shore up any remaining weaknesses of the tool and we’re in this for the long haul!
PostgreSQL itself is a very successful open source project that continuously innovates and provides a very solid technical foundation for Marten itself
We’ve invested a lot of time into refining Marten’s usability over the years and we think that attention to detail shines through
JasperFx Software offers support contracts and consulting work for Marten users
In conjunction with Wolverine’s integration with Marten, the full “Critter Stack” provides a very efficient and usable stack for Event Driven Architecture using Event Sourcing and a CQRS architecture
While Marten 7.0 made some significant improvements for scalability, the forthcoming “Critter Stack Pro” commercial add on tooling will take Marten to much larger data sets and transactional throughput
Because Marten does target .NET, it’s worth pointing out that at this point, Microsoft has no technical offerings for Event Sourcing and that will absolutely contribute to Marten’s viability
What’s Next and Summary
A lot of big, important, long requested, long planned features and improvements did not make the cut for V7. I blogged last week about the current roadmap for the entire Critter Stack. Moreover, some open bugs didn’t make it into 7.0 as well. And let’s be honest, there’s going to be a slew of bug reports streaming in this week when folks try out new 7.0 features and encounter usage permutations we didn’t anticipate. I’ve finally learned my lesson and made this release after having gotten some rest to be ready for whatever the issues turn out to be in the morning.
Wolverine 2.0 will also follow shortly, but the roadmap for that is pretty well just upgrading to Marten 7, dumping .NET 6, and fixing some low hanging fruit issues and requests before a release in the next couple days.
We’ll jump on whatever those Marten 7 issues turn out to be and all the questions about “what about *my* use case I don’t see on your list!” starting tomorrow, but for right now, this was a huge release filled with all kinds of substantial improvements that for the first time included significant client sponsored requests and please don’t steal my sunshine!
From some notes that Oskar, Babu, and I banged out this past week, so keep your expectations for the quality of prose here! Notes in bold are my updates since this original document was banged out last weekend.
Marten 7.0
Try to release Marten 7.0 no later than early next week. This is admittedly based on JasperFx client deliverables.
Blue/green & zero downtime deployment. Ongoing work that just needs more testing at this point. This includes the projection version stuff. Actually all working locally, but my development branch is rebased on the daemon stuff, so I’d like that to go in first. Done.
Projection Snapshots – I’d really like to see this mostly land in Critter Stack Pro. Probably not happening until 2nd quarter 2024
First class subscriptions from the event store to Wolverine transports – might be in Wolverine 2.0 proper. Dunno. Not sure yet
Async projection optimizations – Probably not happening until 2nd quarter 2024
2nd level caching for aggregates
Rebuild single stream projections stream by stream
Allow for selective identity map usage of reference types.
Batched data lookups – so you can keep projections from doing chatty data access
Allow grouping logic to express optimization hints like “no data access required” or “requires aggregate state”. That could be used to optimize projection rebuilds
Wolverine 2.0
Discovery and activation of new tenant databases at runtime (client deliverable). Done.
Update to Marten 7
Project Aspire? Wolverine 2.1? This is a little more involved, so I’m not sure yet when this lands. Probably in Wolverine 2.1.
Marten 7.1
Open Telemetry Support – Sean Farrow is working on this. I don’t think it’s going to be a breaking change, so could float to 7.1. Very Basic
Sharding the event store tables – I’d love to do this sooner, and would love to stretch this in. I’m saying that we would tackle the is archived / not archived sharding in a first pass, then come back w/ fancier sharding possibilities later. This would have a potentially huge positive implication for Marten event store scalability.
The ability to “emit” new events in the async daemon during the course of processing asynchronous projections. I think this is going to take some spikes and analysis, so we gotta commit to this ASAP if it’s going into 7.0. This is falling to Marten 7.1
First class subscriptions. Hot, cold, replay, whatever. I just want a little more time and space. Does this require any breaking changes in the daemon we might want to deal with right now though? Very likely dropping to Marten 7.1
Custom event type naming strategy – it’s a breaking change to the API I think. I don’t think it’s huge though – little pluggable strategy. Can be additive.
Optimize inline projections in FetchForWriting()? Idea here is to force aggregates that are calculated Inline (or Async maybe) that are queried in FetchForWriting() be forced to use the identity map for just that document type. That does a lot to optimize the typical “aggregate handler workflow” by avoiding the current double fetching of the document when you are using lightweight sessions. Strong candidate to drop down to 7.1
Marten 7.Later
Downcasters – I vote to put this into Critter Stack Pro all the way
Marten 8.0???
More advanced Event Store partitioning
Wolverine 2.1
Likely a focus on the Wolverine.HTTP backlog
Options for strict ordering requirements of event or message processing
Hey, did you know that JasperFx Software is ready for formal support plans for Marten and Wolverine? Not only are we making the “Critter Stack” tools be viable long term options for your shop, we’re also interested in hearing your opinions about the tools and how they should change.We’re also certainly open to help you succeed with your software development projects on a consulting basis whether you’re using any part of the Critter Stack or some completely different .NET server side tooling.
In the continuing saga of trying to build a sustainable business model around Marten and Wolverine (the “Critter Stack”), JasperFx Software is quietly building a new set of tools code named “Critter Stack Pro” as a commercially licensed add on to the MIT-licensed OSS core tools.
While there’s some very serious progress on a potential management user interface tool for Marten & Wolverine features underway, the very first usable piece will be a new library for scaling Marten’s asynchronous projection model by much more efficiently distributing work across a clustered application than Marten by itself can today.
In the first wave of work, we’re aiming for this feature set:
When using a single Marten database, the execution of asynchronous projections will be distributed evenly across the application cluster
When using multiple Marten databases for multi-tenancy, the execution of asynchronous projections will be distributed by database and evenly across the application cluster
In blue/green deployments, “Critter Stack Pro” will be able to ensure that all known versions of each projection and database are executing in a suitable “blue” or “green” node within the application cluster
When using multiple Marten databases for multi-tenancy and also using the new dynamic tenant capability in Marten 7.0, “Critter Stack Pro” will discover the new tenant databases at runtime and redistribute projection work across the application cluster
“First class subscriptions” of Marten events with strict ordering through any of Wolverine’s supported messaging transports (locally, Rabbit MQ, Kafka, Azure Service Bus, AWS SQS, soon to be more!).
We’re certainly open to more suggestions from long term and potential users about what other features would make “Critter Stack Pro” a must have tool for your production environment. Trigger projection projection rebuilds on demand? Apply a new subscription? Pause a subscription? Force “Critter Stack Pro” to redistribute projections across the cluster? Smarter distribution algorithms based on predicted load? Adaptive distribution based on throughput?
And do know that we’re already working up a potential user interface for visualizing and monitoring Marten and Wolverine’s behavior at runtime.
This new product (knock on wood) is going to be delivered to a JasperFx customer within the next week or two for integration into their systems using Marten 7.0 and Wolverine 2.0 (also not coincidentally forthcoming at the end of the next week). I’m not going to commit to when this will be generally available, but I’d sure hope it’s sometime in the 2nd quarter this year.
Hey, did you know that JasperFx Software is ready for formal support plans for Marten and Wolverine? Not only are we making the “Critter Stack” tools be viable long term options for your shop, we’re also interested in hearing your opinions about the tools and how they should change.We’re also certainly open to help you succeed with your software development projects on a consulting basis whether you’re using any part of the Critter Stack or some completely different .NET server side tooling.
Marten 7.0.0 RC just dropped on Nuget with some new fixes and some very long awaited enhancements that I’m personally very excited about. The docs need to catch up, but the 7.0 release is shaping up for next week (come hell or high water). One of the new highlights for Marten 7 that was sponsored by a JasperFx Software client was the ability to add new tenant databases at runtime when using Marten’s “database per tenant” strategy.
It’s not documented yet (I’m working on it! I swear!), but here’s a sneak peek from an integration test:
// Setting up a Host with Multi-tenancy
_host = await Host.CreateDefaultBuilder()
.ConfigureServices(services =>
{
services.AddMarten(opts =>
{
// This is a new strategy for configuring tenant databases with Marten
// In this usage, Marten is tracking the tenant databases in a single table in the "master"
// database by tenant
opts.MultiTenantedDatabasesWithMasterDatabaseTable(ConnectionSource.ConnectionString, "tenants");
opts.RegisterDocumentType<User>();
opts.RegisterDocumentType<Target>();
opts.Projections.Add<TripProjectionWithCustomName>(ProjectionLifecycle.Async);
})
.AddAsyncDaemon(DaemonMode.Solo)
// All detected changes will be applied to all
// the configured tenant databases on startup
.ApplyAllDatabaseChangesOnStartup();
}).StartAsync();
With this model, Marten is setting up a table named mt_tenant_databases to store with just two columns:
tenant_id
connection_string
At runtime, when you ask for a new session for a specific tenant like so:
using var session = store.LightweightSession("tenant1");
This new Marten tenancy strategy will first look for a database with the “tenant1” identifier its own memory, and if it’s not found, will try to reach into the database table to “find” the connection string for this newly discovered tenant. If a record is found, the new tenancy strategy caches the information, and proceeds just like normal.
Now, let me try to anticipate a couple questions you might have here:
Can Marten track and apply database schema changes to new tenant databases at runtime? Yes, Marten does the schema check tracking on a database by database basis. This means that if you add a new tenant database to that underlying table, Marten will absolutely be able to make schema changes as needed to just that tenant database regardless of the state of other tenant databases.
Will the Marten command line tools recognize new tenant databases? Yes, same thing. If you call dotnet run -- marten-apply for example, Marten will do the schema migrations independently for each tenant database, so any outstanding changes will be performed on each tenant database.
Can Marten spin up asynchronous projections for a new tenant database without requiring downtime? Yes! Check out this big ol’ integration test proving that the new Marten V7 version of the async daemon can handle that just fine:
[Fact]
public async Task add_tenant_database_and_verify_the_daemon_projections_are_running()
{
// In this code block, I'm adding new tenant databases to the system that I
// would expect Marten to discover and start up an asynchronous projection
// daemon for all three newly discovered databases
var tenancy = (MasterTableTenancy)theStore.Options.Tenancy;
await tenancy.AddDatabaseRecordAsync("tenant1", tenant1ConnectionString);
await tenancy.AddDatabaseRecordAsync("tenant2", tenant2ConnectionString);
await tenancy.AddDatabaseRecordAsync("tenant3", tenant3ConnectionString);
// This is a new service in Marten specifically to help you interrogate or
// manipulate the state of running asynchronous projections within the current process
var coordinator = _host.Services.GetRequiredService<IProjectionCoordinator>();
var daemon1 = await coordinator.DaemonForDatabase("tenant1");
var daemon2 = await coordinator.DaemonForDatabase("tenant2");
var daemon3 = await coordinator.DaemonForDatabase("tenant3");
// Just proving that the configured projections for the 3 new databases
// are indeed spun up and running after Marten's new daemon coordinator
// "finds" the new databases
await daemon1.WaitForShardToBeRunning("TripCustomName:All", 30.Seconds());
await daemon2.WaitForShardToBeRunning("TripCustomName:All", 30.Seconds());
await daemon3.WaitForShardToBeRunning("TripCustomName:All", 30.Seconds());
}
At runtime, if the Marten V7 version of the async daemon (our sub system for building asynchronous projections constantly in a background IHostedService) is constantly doing “health checks” to make sure that *some process* is running all known asynchronous projections on all known client databases. Long story, short, Marten 7 is able to detect new tenant databases and spin up the asynchronous projection handling for these new tenants with zero downtime.
There’s a helluva lot more new stuff and big improvements to the old stuff in Marten coming in V7, but this one was a definite highlight.
Look for the official Marten 7.0 release next week!
Nick Chapsas just released a video about Wolverine with his introduction and take on the framework. Not to take anything away from the video that was mostly positive, but I thought there were quite a few misconceptions about Wolverine evident in the comments and some complaints I would like to address so I can stop fussing about this and work on much more important things.
First off, what is Wolverine? Wolverine is a full blown application framework and definitely not merely a “library,” so maybe consider that when you are judging the merits of its opinions or not. More specifically, Wolverine is a framework built around the idea of message processing where “messages” could be coming from inline invocation like MediatR or local in process queues or external message brokers through asynchronous messaging ala the much older MassTransit or NServiceBus frameworks. In addition, Wolverine’s basic runtime pipeline has also been adapted into an alternative HTTP endpoint framework that could be used in place of or as a complement to MVC Core or Minimal API.
I should also point out that Wolverine was largely rescued off the scrap heap and completely rebooted specifically to work in conjunction with Marten as a full blow event driven architecture stack. This is what we mean when we say “Critter Stack.”
In its usage, Wolverine varies quite a bit from the older messaging and mediator tools out there like NServiceBus, MassTransit, MediatR, Rebus, or Brighter.
Basically all of these existing tools one way or another force you to constrain your code within some kind of “IHandler of T” abstraction something like this:
By and large, these frameworks assume that you will be using an IoC container to fill any dependencies of the actual message handler classes through constructor injection. Part of the video I linked to was the idea that Wolverine was very opinionated, so let’s just get to that and see how Wolverine very much differs from all the older “IHandler of T” frameworks out there.
Wolverine’s guiding philosophies are to:
Reduce code ceremony and minimize coupling between application code and the surrounding framework. As much as possible — and it’s an imperfect world so the word is “minimize” and not “eliminate” — Wolverine attempts to minimize the amount of code cruft from required inheritance, marker interfaces, and attribute usage within your application code. Wolverine’s value proposition is that lower ceremony code leads to easier to read code that offsets any disadvantages that might arise from using conventional approaches
Promote testability — both by helping developers structure code in such a way that they can keep infrastructure concerns out of business logic for easy unit testing and to facilitate effective automated integration testing as well. I’ll throw this stake in the ground right now, Wolverine does much more to promote testability than any other comparable framework that I’m aware of, and I don’t mean just .NET frameworks either (Proverbs 16:18 might be relevant here, but shhh).
“It should just work” — meaning that as much as possible, Wolverine should try to set up infrastructural state (database schemas, message broker configuration, etc.) that your application depends on for an efficient developer experience
Bake in logging, auditing, and observability so that developers don’t have to think about it. This is partially driven by the desire for low code ceremony because nothing is more repetitive in systems than copy/paste log statements every which way
Be as performant as possible. Wolverine is descended and influenced by an earlier failed OSS project called FubuMVC that strived for very low code ceremony and testability, but flunked on performance and how it handled “magic” conventions. Let’s just say that failure is a harsh but effective teacher. In particular, Wolverine tries really damn hard to reduce the number of object allocations and dictionary lookups at runtime as those are usually the main culprits of poor performance in application frameworks. I fully believe that before everything is said and done, that Wolverine will be able to beat the other tools in this space because of its unique runtime architecture.
A Wolverine message handler might look something like this from one of our samples in the docs that happens to use EF Core for persistence:
public static class CreateItemCommandHandler
{
public static ItemCreated Handle(
// This would be the message
CreateItemCommand command,
// Any other arguments are assumed
// to be service dependencies
ItemsDbContext db)
{
// Create a new Item entity
var item = new Item
{
Name = command.Name
};
// Add the item to the current
// DbContext unit of work
db.Items.Add(item);
// This event being returned
// by the handler will be automatically sent
// out as a "cascading" message
return new ItemCreated
{
Id = item.Id
};
}
}
There’s a couple things I’d ask you to notice right off the bat that will probably help inform you if you’d like Wolverine’s approach or not:
There’s no required IHandler<T> type interface. Nor do we require any kind of IMessage/IEvent/ICommand interface on the message type itself
The method signatures of Wolverine message handlers are pretty flexible. Wolverine can do “method injection” like .NET developers are used to now in Minimal API or the very latest MVC Core where services from the IoC container are pushed into the handler methods via method parameters (Wolverine will happily do constructor injection just like you would in other frameworks as well). Moreover, Wolverine can even do different things with the handler responses like “know” that it’s a separate message to publish via Wolverine or a “side effect” that should be executed inline. Heck, the message handlers can even be static classes or methods to micro-optimize your code to be as low allocation as possible.
Wolverine is not doing any kind of runtime Reflection against these handler methods, because as a commenter pointed out, this would indeed be very slow. Instead, Wolverine is generating and compiling C# code at runtime that wraps around your method. Going farther, Wolverine will use your application’s DI configuration code and try to generate code that completely takes the place of your DI container at runtime. Some folks complain that Wolverine forces you to use Lamar as the DI container for your application, but doing so enabled Wolverine to do the codegen the way that it is. Nick pushed back on that by asking what if the built in DI container becomes much faster than Lamar (it’s the other way around btw)? I responded by pointing out that the fasted DI container is “no DI container” like Wolverine is able to do at runtime.
The message handlers are found by default through naming conventions. But if you hate that, no worries, there are options to use much more explicit approaches. Out of the box, Wolverine also supports discovery using marker interfaces or attributes. I don’t personally like that because I think it “junks up the code”, but if you do, you can have it your way.
The handler code above was written with the assumption that it’s using automatic transactional middleware around it all that handles the asynchronous code invocation, but if you prefer explicit code, Wolverine happily lets you eschew any of the conventional magic and write explicit code where you would be completely in charge of all the EF Core usage. The importance of being able to immediately bypass any conventions and drop into explicit code as needed was an important takeaway from my earlier FubuMVC failure.
Various Objections to Wolverine
It’s opinionated, and I don’t agree with all of Wolverine’s opinions. This one is perfectly valid. If you don’t agree with the idiomatic approach of a software development tool, you’re far better off to just pick something else instead of fighting with the tool and trying to use it differently than its idiomatic usage. That goes for every tool, not just Wolverine. If you’d be unhappy using Wolverine and likely to gripe about it online, I’d much rather you go use MassTransit.
Runtime reflection usage? As I said earlier, Wolverine does not use reflection at runtime to interact with the message handlers or HTTP endpoint methods
Lamar is required as your IoC tool. I get the objection to that, and other people have griped about that from time to time. I’d say that the integration with Lamar enables some of the very important “special sauce” that makes Wolverine different. I will also say that at some point in the future we’ll investigate being able to at least utilize Wolverine with the built in .NET DI container instead
Oakton is a hard dependency, and why is Wolverine mandating console usage? Yeah, I get that objection, but I think that’s very unlikely to ever really matter much. You don’t have to use Oakton even though it’s there, but Wolverine (and Marten) both heavily utilize Oakton for command line diagnostics that can do a lot for infrastructure management, environment checks, code generation, database migrations, and important diagnostics that help users unravel and understand Wolverine’s “magic”. We could have made that all be separate adapter package or add ons, but from painful experience, I know that the complexity of usage and development of something like Wolverine goes up quite a bit with the number of satellite packages you use and require — and that’s already an issue even so with Wolverine. I did foresee the Lamar & Oakton objections, but consciously decided that Wolverine development and adoption would be easier — especially early on — by just bundling things together. I’d be willing to reconsider this in later versions, but it’s just not up there in ye olde priority list
There are “TODO” comments scattered in the documentation website! There’s a lot of documentation up right now, and also quite a few samples. That work is never, ever done and we’ll be improving those docs as we go. The one thing I can tell you definitively about technical documentation websites is that it’s never good enough for everyone.
Hey, did you know that JasperFx Software is ready for formal support plans for Marten and Wolverine? Not only are we making the “Critter Stack” tools be viable long term options for your shop, we’re also interested in hearing your opinions about the tools and how they should change.We’re also certainly open to help you succeed with your software development projects on a consulting basis whether you’re using any part of the Critter Stack or some completely different .NET server side tooling.
Wolverine has a pair of features called cascading messages and side effects that allow users to designate “work” that happens after your main message handler or HTTP endpoint method. In our Discord room this week, there was a little bit of user confusion over what the difference was and when you should use one or the other. To that end, let’s do a little dive into what both of these features are (both are unique to Wolverine and don’t have analogues to the best of my knowledge in any other messaging or command processing framework in the .NET space).
First, let’s take a look at “side effects.” Here’s the simple example of a custom “side effect” from the Wolverine documentation to write string data to a file on the file system:
public class WriteFile : ISideEffect
{
public string Path { get; }
public string Contents { get; }
public WriteFile(string path, string contents)
{
Path = path;
Contents = contents;
}
// Wolverine will call this method.
public Task ExecuteAsync(PathSettings settings)
{
if (!Directory.Exists(settings.Directory))
{
Directory.CreateDirectory(settings.Directory);
}
return File.WriteAllTextAsync(Path, Contents);
}
}
And a message handler that uses this custom side effect:
public class RecordTextHandler
{
public WriteFile Handle(RecordText command)
{
return new WriteFile(command.Id + ".txt", command.Text);
}
}
A Wolverine “side effect” really just designates some work that should happen inline with your message or HTTP request handling, so we could eschew the “side effect” and rewrite our message handler as:
public Task Handle(RecordText command, PathSettings settings)
{
if (!Directory.Exists(settings.Directory))
{
Directory.CreateDirectory(settings.Directory);
}
return File.WriteAllTextAsync(command.Id + ".txt", command.Text);
}
The value of the “side effect” usage within Wolverine is to allow you to make a message or HTTP endpoint method be responsible for deciding what to do, without coupling that method and its logic to some kind of pesky infrastructure like the file system that becomes a pain to deal with in unit tests. The “side effect” object returned from a message handler or HTTP endpoint is running inline within the same transaction (if there is one) and retry loop for the message itself.
On the other hand, a cascading message is really just sending a subsequent message after the successful completion of the original message. Here’s an example from the “Building a Critter Stack Application” blog series:
[WolverinePost("/api/incidents/categorise"), AggregateHandler]
// Any object in the OutgoingMessages collection will
// be treated as a "cascading message" to be published by
// Wolverine after the original CategoriseIncident command
// is successfully completed
public static (Events, OutgoingMessages) Post(
CategoriseIncident command,
IncidentDetails existing,
User user)
{
var events = new Events();
var messages = new OutgoingMessages();
if (existing.Category != command.Category)
{
events += new IncidentCategorised
{
Category = command.Category,
UserId = user.Id
};
// Send a command message to try to assign the priority
messages.Add(new TryAssignPriority
{
IncidentId = existing.Id
});
}
return (events, messages);
}
}
In the example above, the TryAssignPriority message will be published by Wolverine to whatever subscribes to that message type (local queues, external transports, nowhere because nothing actually cares?). The “cascading messages” are really the equivalent to calling IMessageBus.PublishAsync() on each cascaded message. It’s important to note that cascaded messages are not executed inline with your original message. Instead, they are only published after the original message is completely handled, and will run in completely different contexts, retry loops, and database transactions.
To sum up, you would:
Use a “side effect” to select actions that need to happen within the current message context as part of an atomic transaction and as something that should succeed or fail (and be retried) along with the message handler itself and any other middleware or post processors for that message type. In other words, “side effects” are for actions that should absolutely happen right there and now!
Use a “cascaded message” for a subsequent action that should happen after the current message, should be executed within a separate transaction, or could be retried in its own retry loop after the original message handling has succeeded.
I’d urge users to consider the proper transaction boundaries and retry boundaries to decide which approach to use. And remember that in both cases, there is value in trying to use pure functions for any kind of business or workflow logic — and both side effects and cascaded messages help you do exactly that for easily unit testable code.
Hey, did you know that JasperFx Software is ready for formal support plans for Marten and Wolverine? Not only are we making the “Critter Stack” tools be viable long term options for your shop, we’re also interested in hearing your opinions about the tools and how they should change.We’re also certainly open to help you succeed with your software development projects on a consulting basis whether you’re using any part of the Critter Stack or some completely different .NET server side tooling.
David Giard was kind enough to have me onto his Technology and Friends show to give a contrarian view of “Clean Architecture” as commonly practiced.
If you’re happy with however you’re using Clean Architecture or any other hexagonal architecture, no need to be angry, I think we were giving a relatively nuanced discussion of the common problems folks run into when using architectural guidance and templates common in our industry.
From my side, I’d say the issues are:
A harmful focus on prescriptive rules that don’t actually help software teams succeed over time
A lack of adaptation caused by prescriptive rules
Organizing code first by layers, technical stereotypes, and business entity
Technical coupling within a layer actually leading to problems upgrading codebases
Too much emphasis on using abstractions and mock object libraries to create testability
And on a more positive angle, I’m in the camp that wants to pursue a vertical slice architecture, a “verb” centric code organization that attempts to keep closely related code together regardless of technical stereotypes, and being a technical leader who focuses on teaching your colleagues how to think through problems in the system as opposed to giving down rules from on high that may or may not actually reflect their reality.
The Shade Tree Developer 