Kicking off Marten v4 Development

If you’re not familiar with Marten, it’s a library that allows .Net developers to treat Postgresql as a Document Database and also as an Event Store. For a quick background on Marten, here’s my presentation on Marten from Dotnetconf 2018.

The Marten community is gearing up to work on the long awaited v4.0 release, which I think is going to be the most ambitious release of Marten since the original v1.0 release in September of 2016. Marten is blessed with a strong user community, and we’ve got a big backlog of issues, feature requests, and suggestions for improvements that have been building up for quite some time.

First, some links for anyone who wants to be a part of that conversation:

• The big Marten v4 discussion is happening right now on GitHub
• The v4 issue milestone list on GitHub
• The Marten Gitter room for ongoing discussions

Now would be a perfect time to add any kind of feedback or requests for Marten. We do have an unofficial “F# advisory board” this time around, but the more the merrier on that side. There’ll be plenty of opportunities for folks to contribute, whether that’s in coding, experimenting with early alphas, or just being heard in the discussions online.

Now, on to the release plans:

Event Sourcing Improvements

The biggest area of focus in Marten v4.0 is likely going to be the event store functionality. At the risk of being (rightly) mocked, I’d sum up our goals here as “Make the Marten Event Store be Web Scale!”

There’s a lot of content on the aforementioned Marten v4.0 discussion, and also on an older GitHub discussion on Event Store improvements for v4.0. The highlights (I think) are:

• Event metadata (a long running request)
• Partitioning Support
• Improved projection support, including the option to project event data to flat database tables
• A lot of improvements (near rewrite) to the existing support for asynchronous projections including multi-tenancy support, running across multiple application nodes, performance optimizations, and projection rebuild improvements
• New support for asynchronous projection builders using messaging as an alternative to the polling async daemon. The first cut of this is very likely to be based around Jasper (I’m using this as a way to push Jasper forward too of course) and either Rabbit MQ or Azure Service Bus. If things go well, I’d hope that that expands to at least Jasper + Kafka and maybe parallel support for NServiceBus and MassTransit.
• Snapshotting
• The ability to rebuild projections with zero downtime

Linq Support Improvements

I can’t overstate how complex building and maintaining a comprehensive Linq provider can be just out of the sheer number of usage permutations.

Marten v4.0 is going to include a pretty large overhaul of the Linq support. Along the way, we’re hopefully going to:

• Rewrite the functionality to include related documents altogether and finally support Include() on collections that’s been a big user request for 3+ years.
• Improve the efficiency of the generated SQL
• More variable data type behavior within the Linq support. You can read more about the mechanics on this issue. This also includes being quite a bit smarter about the Json serialization and how that varies for different .Net data types. As an example, I think these changes will allow Marten’s Linq support to finally deal with things like F# Discriminator Union types.
• Improve the internals by making it more modular and a little more performant in how it handles marshaling data from the database to C# objects.
• Improve Marten’s ability to query within child collections
• More flexibility in event stream identifier strateggies

 

Other Improvements

• Document Metadata improvements, both exposing Marten’s metadata and user customizable metadata for things like user id or correlation identifiers
• An option to stream JSON directly to .Net Stream objects like HTTP responses without going through serialization first or even a JSON string.
• Optimizing the Partial update support to use Postgresql native operators when possible
• Formal support for .Net Core integration through the generic HostBuilder mechanism

 

Planning

Right now it’s all about the discussions and approach, but that’s going pretty well right now. The first thing I’m personally going to jump into is a round of automated test performance just to make everything go faster later.

The hope is that a subsequent V5 release might finally take Marten into supporting other database engines besides Postgresql, with Sql Server being the very obvious next step. We might take some preliminary steps with V4 to make the multi-database engine support be more feasible later with fewer breaking API changes, but don’t hold me to that.

I don’t have any idea about timing yet, sorry.