Resiliency and Low Level Improvements in Marten 7

Just continuing a loose series about recent improvements in the big Marten 7.0 release:

• Revisioned Documents in Marten 7
• Marten 7 makes “Write Model” Projections Super
• Recent Critter Stack Multi-Tenancy Improvements (w/ Wolverine 2.0 too!)
• Resiliency and Low Level Improvements in Marten 7 (this post)

I hate to tell you all this, but sometimes there’s going to be occasional hiccups with your system in production. Just sticking with using Marten and its connectivity to PostgreSQL under the covers, you could easily have:

• Occasional network issues that cause transactions to fail
• A database could be temporarily too busy and throw exceptions
• Concurrency issues from trying to write to the same rows in the underlying database (Marten isn’t particularly prone to this, but I needed another example)

These typical transient errors happen, but that doesn’t mean that the operation that just failed couldn’t happily succeed if you just retried it in a little bit. To that end, Marten 7 replaced our previous homegrown resiliency approach (that didn’t really work anyway) with a reliance on the popular Polly library.

The default settings for Marten are shown below:

// Default Polly setup
var strategy = new ResiliencePipelineBuilder().AddRetry(new()
{
    ShouldHandle = new PredicateBuilder().Handle<NpgsqlException>().Handle<MartenCommandException>().Handle<EventLoaderException>(),
    MaxRetryAttempts = 3,
    Delay = TimeSpan.FromMilliseconds(50),
    BackoffType = DelayBackoffType.Exponential
}).Build();

The Marten docs here will tell you how to customize these policies at your discretion.

To put this into context, if you call IDocumentSession.SaveChangesAsync() to commit a unit of work, and there’s an exception that matches the criteria shown above, Polly will re-execute the queued operations from scratch with a brand new connection so the retry can succeed after the database or network has settled down.

Moreover, we followed the Polly recommendations on performance to utilize “static Lambdas” within the Marten internals to reduced the number of object allocations within Marten 7’s execution pipeline compared to the Marten 6 and earlier pipeline. It’s kind of ugly code, but you can see where and how we used Polly in QuerySession.Execution.cs.

Another big change for Marten 7 was how Marten is going to manage database connection lifecycles inside of Marten sessions. Prior to Marten 7, Marten sessions would open a database connection and keep it open until the session was ultimately disposed. This decision was originally made to optimize the integration between Marten and Dapper when Marten’s functionality was very limited.

Now though, Marten will only open a database connection within a session immediately before any operation that involves a database connection, and close that connection immediately after the operation is over (really just returning the underlying connection to the connection pool managed by Npgsql). With this change, it is now safe to run read-only queries through IQuerySession (or lightweight IDocumentSession) objects in multiple threads. That should make Marten be more effective within Hot Chocolate integrations:

• Marten sessions can actually be thread safe so that you can let the default IoC registration behavior for Marten sessions happily work within Hot Chocolate requests where it wants to parallelize queries
• Marten usage will be far, far less prone to connection leaks when developers create sessions without disposing them properly

And even if you’re not using Hot Chocolate at all, Marten 7 will be more parsimonious over all with how it uses database connections which should help with scalability of your system — and definitely will help with cloud hosting options that charge by the number of database connections used!

Final Thoughts

In the past month or so I’ve had more interaction with developers than usual who are highly suspicious of open source tooling and especially of alternative open source tooling in .NET that isn’t directly supported by Microsoft. It’s a fair point to some degree, because “Google/Bing-bility” is a highly underrated quality of a development tool. Some of that fear was thinking that the responsible developers behind a non mainstream tool are just going to get tired of it and abandon it on a whim when a shinier object comes around so it’s not even worth the time to care about those alternatives from the “cool kids.”

What I would like to point out with Marten in particular is that it’s nearly a decade old as a project (the document db features in Marten actually predate CosmosDb in .NET world) and still very active and growing. The work in this post is all about making fine grained refinements to the core project and I would hope demonstrate a dedication on the part of the whole community toward continuously improving Marten for serious work. In other words, I think development shops can absolutely feel confident in placing a technical bet on Marten.