JasperFx OSS Plans for .Net 6 (Marten et al)

I’m going to have to admit that I got caught flat footed by the .Net 6 release a couple weeks ago. I hadn’t really been paying much attention to the forthcoming changes, maybe got cocky by how easy the transition from netcoreapp3.1 to .Net 5 was, and have been unpleasantly surprised by how much work it’s going to take to move some OSS projects up to .Net 6. All at the same time that the advance users of the world are clamoring for all their dependencies to target .Net 6 yesterday.

All that being said, here’s my running list of plans to get the projects in the JasperFx GitHub organization successfully targeting .Net 6. I’ll make edits to this page as things get published to Nuget.

Baseline

Baseline is a grab bag utility library full of extension methods that I’ve relied on for years. Nobody uses it directly per se, but it’s a dependency of just about every other project in the organization, so it went first with the 3.2.2 release adding a .Net 6 target. No code changes were necessary other than adding .Net 6 to the CI testing. Easy money.

Oakton

EDIT: Oakton v4.0 is up on Nuget. WebApplication is supported, but you can’t override configuration in commands with this model like you can w/ HostBuilder only. I’ll do a follow up at some point to fill in this gap.

Oakton is a tool to add extensible command line options to .Net applications based on the HostBuilder model. Oakton is my problem child right now because it’s a dependency in several other projects and its current model does not play nicely with the new WebApplicationBuilder approach for configuring .Net 6 applications. I’d also like to get the Oakton documentation website moved to the VitePress + MarkdownSnippets model we’re using now for Marten and some of the other JasperFx projects. I think I’ll take a shortcut here and publish the Nuget and let the documentation catch up later.

Alba

Alba is an automated testing helper for ASP.Net Core. Just like Oakton, Alba worked very well with the HostBuilder model, but was thrown for a loop with the new WebApplicationBuilder configuration model that’s the mechanism for using the new Minimal API (*cough* inevitable Sinatra copy *cough*) model. Fortunately though, Hawxy came through with a big pull request to make Alba finally work with the WebApplicationFactory model that can accommodate the new WebApplicationBuilder model, so we’re back in business soon. Alba 5.1 will be published soon with that work after some documentation updates and hopefully some testing with the Oakton + WebApplicationBuilder + Alba model.

EDIT: Alba 7.0 is up with the necessary changes, but the docs will come later this week

Lamar

Lamar is an IoC/DI container and the modern successor to StructureMap. The biggest issue with Lamar on v6 was Nuget dependencies on the IServiceCollection model, plus needing some extra implementation to light up the implied service model of Minimal APIs. All the current unit tests and even integration tests with ASP.Net Core are passing on .Net 6. To finish up a new Lamar 7.0 release is:

  • One .Net 6 related bug in the diagnostics
  • Better Minimal API support
  • Upgrade Oakton & Baseline dependencies in some of the Lamar projects
  • Documentation updates for the new IAsyncDisposable support and usage with WebApplicationBuilder with or without Minimal API usage

Marten/Weasel

We just made the gigantic V4 release a couple months ago knowing that we’d have to follow up quickly with a V5 release with a few breaking changes to accommodate .Net 6 and the latest version of Npgsql. We are having to make a full point release, so that opens the door for other breaking changes that didn’t make it into V4 (don’t worry, I think shifting from V4 to V5 will be easy for most people). The other Marten core team members have been doing most of the work for this so far, but I’m going to jump into the fray later this week to do some last minute changes:

  • Review some internal changes to Npgsql that might have performance impacts on Marten
  • Consider adding an event streaming model within the new V4 async daemon. For folks that wanna use that to publish events to some kind of transport (Kafka? Some kind of queue?) with strict ordering. This won’t be much yet, but it keeps coming up so we might as well consider it.
  • Multi-tenancy through multiple databases. It keeps coming up, and potentially causes breaking API changes, so we’re at least going to explore it

I’m trying not to slow down the Marten V5 release with .Net 6 support for too long, so this is all either happening really fast or not at all. I’ll blog more later this week about multi-tenancy & Marten.

Weasel is a spin off library from Marten for database change detection and ADO.Net helpers that are reused in other projects now. It will be published simultaneously with Marten.

Jasper

Oh man, I’d love, love, love to have Jasper 2.0 done by early January so that it’ll be available for usage at my company on some upcoming work. This work is on hold while I deal with the other projects, my actual day job, and family and stuff.

Rebooting Jasper

Jasper is a long running OSS passion project of mine. As it is now, Jasper is a command processing tool similar to Brighter or MassTransit that can be used as either an in memory mediator tool (like a superset of Mediatr) or as a service bus framework for asynchronous messaging. Jasper was originally conceived as a way to recreate the “good” parts of FubuMVC like low code ceremony, minimal intrusion of the framework into application code, and effective usage of the Russian Doll model for the execution pipeline. At the same time though, I wanted Jasper to improve upon the earlier FubuMVC architecture by maximizing performance, minimizing object allocations, easier configuration and bootstrapping, and making it much easier for developers to troubleshoot runtime issues.

I actually did cut a Jasper V1 release early in the COVID-19 pandemic, but otherwise dropped it to focus on Marten and stopped paying any attention to it. With Marten V4 is in the books, I’m going back to working on Jasper for a little bit. For right now, I’m thinking that the Jasper V2 work is something like this:

  1. Upgrading all the dependencies and targeting .Net 5/6 (basically done)
  2. Benchmarking and optimizing the core runtime internals. Sometimes the best way to improve a codebase is to step away from it for quite a bit and come back in with fresh perspectives. There’s also some significant lessons from Marten V4 that might apply to Jasper
  3. Build in Open Telemetry tracing through Jasper’s pipeline. Really all about getting me up to speed on distributed tracing.
  4. Support the AsyncAPI standard (Swagger for asynchronous messaging). I’m really interested in this, but haven’t taken much time to dive into it yet
  5. Wire compatibility with NServiceBus so a Jasper app can talk bi-directionally with an NServiceBus app
  6. Same with MassTransit. If I decide to pursue Jasper seriously, I’d have to do that to have any shot at using Jasper at work
  7. More transport options. Right now there’s a Kafka & Pulsar transport option stuck in PR purgatory from another contributor. Again, a learning opportunity.
  8. Optimize the heck out of the Rabbit MQ usage.
  9. Go over the usability of the configuration. To be honest here, I’ve less than thrilled with our MassTransit usage and the hoops you have to jump through to bend it to our will and I’d like to see if I could do better with Jasper
  10. Improve the documentation website (if I’m serious about Jasper)
  11. Play with some kind of Jasper/Azure Functions integration. No idea what that would look like, but the point is to go learn more about Azure Functions
  12. Maybe, but a low priority — I have a working version of FubuMVC style HTTP endpoints in Jasper already. With everybody all excited about the new Minimal API stuff in ASP.Net Core v6, I wouldn’t mind showing a slightly different approach

Integrating Marten and Jasper

Maybe the single biggest reason for me to play more with Jasper is to explore some deeper integration with Marten for some more complicated CQRS and event sourcing architectural problems. Jasper already has an outbox/inbox pattern implementation for Marten. Going farther, I’d like to have out of the box solutions for:

  • Event streaming from Marten to message queues using Jasper
  • An alternative to Marten’s async daemon using Kafka/Pulsar topics
  • Using Jasper to host Marten’s asynchronous projections in a way that distributes work across running nodes
  • Experimenting more with CQRS architectures using Jasper + Marten

Anyway, I’m jotting this down mostly for me, but I’m absolutely happy for any kind of feedback or maybe to see if anyone else would be interested in helping with Jasper development.

Marten V4: Hard Deletes, Soft Deletes, Un-Deletes, All the Deletes You Meet

If you haven’t seen this yet, Marten V4.0 officially dropped on Nuget late last week (finally).

V4 was a huge release for Marten and there’s lots to talk about in the new release, but I want to start simply with just talking about Marten’s support for deleting documents. In all the examples, we’re going to use a User document type from our testing code.

Let’s say we have the identity of a User document that we want to delete in our Marten database. That code is pretty simple, it’s just this below:

internal Task DeleteByDocumentId(IDocumentSession session, Guid userId)
{
    // Tell Marten the type and identity of a document to
    // delete
    session.Delete<User>(userId);

    return session.SaveChangesAsync();
}

By default, Marten will do a “hard” delete where the actual database row in Postgresql is deleted and that’s all she wrote. Marten has long had support for “soft” deletes where the underlying rows are marked as deleted with a metadata column that tracks “is deleted” instead of actually deleting the row, so let’s opt into that by configuring the User document as soft-deleted like so:

// Configuring a new Marten document store
var store = DocumentStore.For(opts =>
{
    opts.Connection("some connection string");

    opts.Schema.For<User>().SoftDeleted();
});

Or if you prefer, you can also use an attribute like so:

[SoftDeleted]
public class User
{
    public Guid Id { get; set; }
    
    // Other props we don't care about here
}

And I’ll show a 3rd way that’s new in Marten V4 later on.

The API to delete a User document is exactly the same, but the mechanics of what Marten is doing to the underlying Postgresql database have changed. Instead of deleting the underlying rows, Marten just marks an mt_deleted column as true. But now, think about this Linq query below where I’m searching for all the users that have the “admin” role:

            using var session = store.QuerySession();
            var admins = await session.Query<User>()
                .Where(x => x.Roles.Contains("admin"))
                .ToListAsync();

You’ll notice that I’m not doing anything to explicitly filter out the deleted users in this query, and that’s because Marten is doing that for you — and that’s the behavior you’d want most of the time. If for some reason you’d like to query for all the documents, even the ones that are marked as deleted, you can query like this:

            var admins = await session.Query<User>()
                .Where(x => x.Roles.Contains("admin"))
                
                // This is Marten specific
                .Where(x => x.MaybeDeleted())
                .ToListAsync();

And now, if you only want to query for documents that are explicitly marked as deleted, you can do this:

            var admins = await session.Query<User>()
                .Where(x => x.Roles.Contains("admin"))
                
                // This is Marten specific
                .Where(x => x.IsDeleted())
                .ToListAsync();

So far, so good? Now what if you change your mind about your deleted documents and you want to bring them back? Marten V4 adds the IDocumentSession.UndoDeleteWhere() method to reverse the soft deletes in the database. In the usage below, we’re going to mark every admin user as “not deleted”:

            using var session = store.LightweightSession();
            
            session.UndoDeleteWhere<User>(x => x.Roles.Contains("admin"));
            await session.SaveChangesAsync();

But hold on, what if instead you really want to just wipe out the database rows with a good old fashioned “hard” delete instead? Marten V4 adds some IDocumentSession.HardDelete*****() APIs to do exactly that as shown below:

            using var session = store.LightweightSession();

            session
                .HardDeleteWhere<User>(x => x.Roles.Contains("admin") && x.IsDeleted());
            await session.SaveChangesAsync();

There are also single document versions of HardDelete() as well.

So back to the sample of querying all admin users both past and present:

            var admins = await session.Query<User>()
                .Where(x => x.Roles.Contains("admin"))

                // This is Marten specific
                .Where(x => x.MaybeDeleted())
                .ToListAsync();

If you’re going to do that, it might be nice to be able to understand which users are marked as deleted, when they were marked deleted, and which users are not deleted. And it’d also be helpful if Marten would just tag the documents themselves with that metadata.

Enter the new Marten.Metadata.ISoftDeleted interface in V4. Let’s make our User document implement that interface as shown below:

public class User : ISoftDeleted
{
    public Guid Id { get; set; }

    // These two properties reflect Marten metadata
    // and will be updated upon Delete() calls
    // or when loading through Marten
    public bool Deleted { get; set; }
    public DateTimeOffset? DeletedAt { get; set; }

    // Other props we don't care about here
}

If a document type implements the ISoftDeleted interface, the document type will automatically be treated as soft-deleted by Marten, and the Deleted and DeletedAt properties will be set at document load time by Marten.

I’m not showing it here, but you can effectively create the same behavior without the marker interface by using a fluent interface. Check out the Marten documentation for that approach.

Going into Marten I thought of the delete operation as something simple conceptually, but thousands of users means getting requests for more control over the process and that’s what we hope V4 delivers here.

Marten Takes a Giant Leap Forward with the Official V4 Release!

Starting next week I’ll be doing some more deep dives into new Marten V4 improvements and some more involved sample usages.

Today I’m very excited to announce the official release of Marten V4.0! The Nugets just went live, and we’ve published out completely revamped project website at https://martendb.io.

This has been at least a two year journey of significant development effort by the Marten core team and quite a few contributors, preceded by several years of brainstorming within the Marten community about the improvements realized by this release. There’s plenty more to do in the Marten backlog, but I think this V4 release puts Marten on a very solid technical foundation for the long term future.

This was a massive effort, and I’d like to especially thank the other core team members Oskar Dudycz for answering so many user questions and being the champion for our event sourcing feature set, and Babu Annamalai for the newly improved website and all our grown up DevOps infrastructure. Their contributions over the years and especially on this giant release have been invaluable.

I’d also like to thank:

  • JT for taking on the nullability sweep and many other things
  • Ville Häkli might have accidentally become our best tester and helped us discover and deal with several issues along the way
  • Julien Perignon and his team for their patience and help with the Marten V4 shakedown cruise
  • Barry Hagan started the ball rolling with Marten’s new, expanded metadata collection
  • Raif Atef for several helpful bug reports and some related fixes
  • Simon Cropp for several pull requests and doing some dirty work
  • Kasper Damgård for a lot of feedback on Linq queries and memory usage
  • Adam Barclay helped us improve Marten’s multi-tenancy support and its usability

and many others who raised actionable issues, gave us feedback, and even made code contributions. Keeping in mind that I personally grew up on a farm in the middle of nowhere in the U.S., it’s a little mind-blowing to me to work on a project of this magnitude that at a quick glance included contributors from at least five continents on this release.

One of my former colleagues at Calavista likes to ask prospective candidates for senior architect roles what project they’ve done that they’re the most proud of. I answered “Marten” at the time, but I think I mean that even more now.

What Changed in this Release?

To quote the immortal philosopher Ferris Bueller:

The question isn‘t ‘what are we going to do’, the question is ‘what aren’t we going to do? ‘

We did try to write up a list of breaking changes for V4 in the migration guide, but here’s some highlights:

  • We generally made a huge sweep of the Marten code internals looking for every possible opportunity to reduce object allocations and dictionary lookups for low level performance improvements. The new dynamic code generation approach in Marten helped get us to that point.
  • We think Marten is even easier to bootstrap in new projects with improvements to the IServiceCollection.AddMarten() extensions
  • Marten supports System.Text.Json — but use that with some caution of course
  • The Linq support took a big step forward with a near rewrite and filled in some missing support for better querying through child collections as a big example. The Linq support is now much more modular and we think that will help us continue to grow that support. It’s a small thing, but the Linq parsing was even optimized a little bit for performance
  • Event Sourcing in Marten got a lot of big improvements that were holding up adoption by some users, especially in regards to the asynchronous projection support. The “async daemon” was completely rewritten and is now much easier to incorporate into .Net systems.
  • As a big user request, Marten supports much more options for tracking flexible metadata like correlation ids and even user defined headers in both document and event storage
  • Multi-tenancy support was improved
  • Soft delete support got some additional usability features
  • PLv8 adoption has been a stumbling block, so all the features related to PLv8 were removed to a separate add-on library called Marten.PLv8
  • The schema management features in Marten made some significant strides and should be able to handle more scenarios with less manual intervention — we think/hope/let’s just be positive for now

What’s Next for Marten?

Full point OSS releases inevitably bring a barrage of user reported errors, questions about migrating, possibly confusing wording in new documentation, and lots of queries about some significant planned features we just couldn’t fit into this already giant release. For that matter, we’ll probably have to quickly spin out a V5 release for .Net 6 and Npgsql 6 because there’s breaking changes coming due to those dependencies. OSS projects are never finished, only abandoned, and there’ll be a world of things to take care of in the aftermath of 4.0 — but for right now, Don’t Steal My Sunshine!.

Efficient Web Services with Marten V4

We’re genuinely close to finally pulling the damn trigger on Marten V4. One of the last things I’m coding for the release is a new recipe for users to write very efficient web services backed by Marten database storage.

A Traditional .Net Approach

Before I get into the exact mechanics of that, let’s set the stage a little bit and draw some contrasts with a more traditional .Net web service stack. Let’s start by saying that in that traditional stack, you’re not using any kind of CQRS and the system state is persisted in the “one true model” approach using Entity Framework Core and an RDBMS like Sql Server. Now, in some web services you need to query data from the database and serve up a subset of that data into the outgoing HTTP response in a web service endpoint. The typical flow — with a focus on what’s happening under the covers — would be to:

  1. ASP.Net Core receives an HTTP GET, finds the proper endpoint, and calls the right handler for that route. Not that it actually matters, but let’s assume the endpoint handler is an MVC Core Controller method
  2. ASP.Net Core invokes a call to your DI container to build up the MVC controller object, and calls the right method for the route
  3. You’ve been to .Net conferences and internalized the idea that an MVC controller shouldn’t get too big or do things besides HTTP mediation, so you’re delegating to a tool like MediatR. MediatR itself is going to go through another DI container service resolution to find the right handler for the input model, then invoke that handler
  4. EF Core issues a query against your Sql Server database. If you’re needing to fetch data on more than just the root aggregate model, the query is going to be an outer join against all the child tables too
  5. EF Core loops around in the database rows and creates objects for your .Net domain model classes based on its ORM mappings
  6. You certainly don’t want to send the raw domain model on the wire because of coupling concerns, or don’t want every bit of data exposed to the client in a particular web service, so you use some kind of tool like AutoMapper to transform the internal domain model objects built up by EF Core into Data Transfer Objects (DTO) purposely designed to go over the wire.
  7. Lastly, you return the outgoing DTO model, which is serialized to JSON and sent down the HTTP response by MVC Core

Sound pretty common? That’s also a lot of overhead. A lot of memory allocations, data mappings between structures, JSON serialization, and a lot of dictionary lookups just to get data out of the database and spit it out into the HTTP response. It’s also a non-trivial amount of code, and I’d argue that some of the tools I mentioned are high ceremony.

Now do CQRS!

I initially thought of CQRS as looking like a whole lot more work to code, and that’s not an uncommon impression when folks are first introduced to it. I’ve come to realize over time that it’s not really more work so much as it’s really just doing about the same amount of work in different places and different times in the application’s workflow.

Now let’s at least introduce CQRS into our application architecture. I’m not saying that that automatically implies using event sourcing, but let’s say that you are writing a pre-built “read side” model of the state of your system directly to a database of some sort. Now from that same web service I was describing before, you just need to fetch that persisted “read side” model from the database and spit that state right out to the HTTP response.

Now then, I’ve just yada, yada’d all the complexity of the CQRS architecture that continuously updates the read side view for you, but hey, Marten does that for you too and that can be a shortly forthcoming follow up blog post.

Finally bringing Marten V4 into play, let’s say our read side model for an issue tracking system looks like this:

    public class Note
    {
        public string UserName { get; set; }
        public DateTime Timestamp { get; set; }
        public string Text { get; set; }
    }

    public class Issue
    {
        public Guid Id { get; set; }
        public string Description { get; set; }
        public bool Open { get; set; }

        public IList<Note> Notes { get; set; }
    }

Before anyone gets bent out of shape by this, it’s perfectly possible to tell Marten to serialize the persisted documents to JSON with camel or even snake casing to be more idiomatic JSON or Javascript friendly.

Now, let’s build out two controller endpoints, one that gives you an Issue payload by searching by its id, and a second endpoint that gives you all the open Issue models in a single JSON array payload. That controller — using some forthcoming functionality in a new Marten.AspNetCore Nuget — looks like this:

    public class GetIssueController: ControllerBase
    {
        private readonly IQuerySession _session;

        public GetIssueController(IQuerySession session)
        {
            _session = session;
        }

        [HttpGet("/issue/{issueId}")]
        public Task Get(Guid issueId)
        {
            // This "streams" the raw JSON to the HttpResponse
            // w/o ever having to read the full JSON string or
            // deserialize/serialize within the HTTP request
            return _session.Json
                .WriteById<Issue>(issueId, HttpContext);
        }

        [HttpGet("/issue/open")]
        public Task OpenIssues()
        {
            // This "streams" the raw JSON to the HttpResponse
            // w/o ever having to read the full JSON string or
            // deserialize/serialize within the HTTP request
            return _session.Query<Issue>()
                .Where(x => x.Open)
                .WriteArray(HttpContext);
        }

In the GET: /issue/{issueId} endpoint, you’ll notice the call to the new IQuerySession.Json.WriteById() extension method, and how I’m passing to it the current HttpContext. That method is:

  1. Executing a database query against the underlying Postgresql database. And in this case, all of the data is stored in a single column in a single row, so there’s not JOINs or sparse datasets like there would be with an ORM querying an object that has child collections.
  2. Write the raw bytes of the persisted JSON data directly to the HttpResponse.Body without ever bothering to write the whole thing to a .Net string and definitely without having to incur the overhead of JSON deserialization/serialization. That extension method also sets the HTTP content-length and content-type response headers, as well as setting the HTTP status code to 200 if the document is found, or 404 if the data is not found.

In the second HTTP endpoint for GET: /issue/open, the call to WriteArray(HttpContext) is doing something very similar, but writing the results as a JSON array.

By no means is this technique going to be applicable to every HTTP GET endpoint, but when it does, this is far, far more efficient and simpler to code than the more traditional approach that involves all the extra pieces and just has so much more memory allocations and hardware operations going on just to shoot some JSON down the wire.

For a little more context, here’s a test against the /issue/{issueId} endpoint, with a cameo from Alba to help me test the HTTP behavior:

        [Fact]
        public async Task stream_a_single_document_hit()
        {
            var issue = new Issue {Description = "It's bad", Open = true};

            var store = theHost.Services.GetRequiredService<IDocumentStore>();
            using (var session = store.LightweightSession())
            {
                session.Store(issue);
                await session.SaveChangesAsync();
            }

            var result = await theHost.Scenario(s =>
            {
                s.Get.Url($"/issue/{issue.Id}");
                s.StatusCodeShouldBeOk();
                s.ContentTypeShouldBe("application/json");
            });

            var read = result.ReadAsJson<Issue>();

            read.Description.ShouldBe(issue.Description);
        }

and one more test showing the “miss” behavior:

        [Fact]
        public async Task stream_a_single_document_miss()
        {
            await theHost.Scenario(s =>
            {
                s.Get.Url($"/issue/{Guid.NewGuid()}");
                s.StatusCodeShouldBe(404);
            });
        }

Feedback anyone?

This code isn’t even released yet even in an RC Nuget, so feel very free to make any kind of suggestion or send us feedback.

Dynamic Code Generation in Marten V4

Marten V4 extensively uses runtime code generation backed by Roslyn runtime compilation for dynamic code. This is both much more powerful than source generators in what it allows us to actually do, but can have significant memory usage and “cold start” problems (seems to depend on exact configurations, so it’s not a given that you’ll have these issues). In this post I’ll show the facility we have to “generate ahead” the code to dodge the memory and cold start issues at production time.

Before V4, Marten had used the common model of building up Expression trees and compiling them into lambda functions at runtime to “bake” some of our dynamic behavior into fast running code. That does work and a good percentage of the development tools you use every day probably use that technique internally, but I felt that we’d already outgrown the dynamic Expression generation approach as it was, and the new functionality requested in V4 was going to substantially raise the complexity of what we were going to need to do.

Instead, I (Marten is a team effort, but I get all the blame for this one) opted to use the dynamic code generation and compilation approach using LamarCodeGeneration and LamarCompiler that I’d originally built for other projects. This allowed Marten to generate much more complex code than I thought was practical with other models (we could have used IL generation too of course, but that’s an exercise in masochism). If you’re interested, I gave a talk about these tools and the approach at NDC London 2019.

I do think this has worked out in terms of performance improvements at runtime and certainly helped to be able to introduce the new document and event store metadata features in V4, but there’s a catch. Actually two:

  1. The Roslyn compiler sucks down a lot of memory sometimes and doesn’t seem to ever release it. It’s gotten better with newer releases and it’s not consistent, but still.
  2. There’s a sometimes significant lag in cold start scenarios on the first time Marten needs to generate and compile code at runtime

What we could do though, is provide what I call..

Marten’s “Generate Ahead” Strategy

To side step the problems with the Roslyn compilation, I developed a model (I did this originally in Jasper) to generate the code ahead of time and have it compiled into the entry assembly for the system. The last step is to direct Marten to use the pre-compiled types instead of generating the types at runtime.

Jumping straight into a sample console project to show off this functionality, I’m configuring Marten with the AddMarten() method you can see in this code on GitHub.

The important line of code you need to focus on here is this flag:

opts.GeneratedCodeMode = TypeLoadMode.LoadFromPreBuiltAssembly;

This flag in the Marten configuration directs Marten to first look in the entry assembly of the application for any types that it would normally try to generate at runtime, and if that type exists, load it from the entry assembly and bypass any invocation of Roslyn. I think in a real application you’d wrap that call something like this so that it only applies when the application is running in production mode:

if (Environment.IsProduction())
{

    options.GeneratedCodeMode = 
        TypeLoadMode.LoadFromPreBuiltAssembly;
}

The next thing to notice is that I have to tell Marten ahead of time what the possible document types and even about any compiled query types in this code so that Marten will “know” what code to generate in the next section. The compiled query registration is new, but you already had to let Marten know about the document types to make the schema migration functionality work anyway.

Generating and exporting the code ahead of time is done from the command line through an Oakton command. First though, add the LamarCodeGeneration.Commands Nuget to your entry project, which will also add a transitive reference to Oakton if you’re not already using it. This is all described in the Oakton getting started page, but you’ll need to change your Program.Main() method slightly to activate Oakton:

        // The return value needs to be Task<int> 
        // to communicate command success or failure
        public static Task<int> Main(string[] args)
        {
            return CreateHostBuilder(args)
                
                // This makes Oakton be your CLI
                // runner
                .RunOaktonCommands(args);
        }

If you’ll open up the command terminal of your preference at the root directory of the entry project, type this command to see the available Oakton commands:

dotnet run -- help

That’ll spit out a list of commands and the assemblies where Oakton looked for command types. You should see output similar to this:

Searching 'LamarCodeGeneration.Commands, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null' for commands
Searching 'Marten.CommandLine, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null' for commands

  -------------------------------------------------
    Available commands:
  -------------------------------------------------
        check-env -> Execute all environment checks against the application
          codegen -> Utilities for working with LamarCodeGeneration and LamarCompiler

and assuming you got that far, now type dotnet run -- help codegen to see the list of options with the codegen command.

If you just want to preview the generated code in the console, type:

dotnet run -- codegen preview

To just verify that the dynamic code can be successfully generated and compiled, use:

dotnet run -- codegen test

To actually export the generated code, use:

dotnet run — codegen write

That command will write a single C# file at /Internal/Generated/DocumentStorage.cs for any document storage types and another at `/Internal/Generated/Events.cs` for the event storage and projections.

Just as a short cut to clear out any old code, you can use:

dotnet run -- codegen delete

If you’re curious, the generated code — and remember that it’s generated code so it’s going to be butt ugly — is going to look like this for the document storage, and this code for the event storage and projections.

The way that I see this being used is something like this:

  • The LoadFromPreBuiltAssembly option is only turned on in Production mode so that developers can iterate at will during development. That should be disabled at development time.
  • As part of the CI/CD process for a project, you’ll run the dotnet run -- codegen write command as an initial step, then proceed to the normal compilation and testing cycles. That will bake in the generated code right into the compiled assemblies and enable you to also take advantage of any kind AOT compiler optimizations

Duh. Why not source generators doofus?

Why didn’t we use source generators you may ask? The Roslyn-based approach in Marten is both much better and much worse than source generators. Source generators are part of the compilation process itself and wouldn’t have any kind of cold start problem like Marten has with the runtime Roslyn compilation approach. That part is obviously much better, plus there’s no weird two step compilation process at CI/CD time. But on the downside, source generators can only use information that’s available at compilation time, and the Marten code generation relies very heavily on type reflection, conventions applied at runtime, and configuration built up through a fluent interface API. I do not believe that we could use source generators for what we’ve done in Marten because of that dependency on runtime information.

Marten, the Generic Host Builder in .Net Core, and why this could be the golden age for OSS in .Net

Before I dive into some newly improved mechanics for quickly integrating Marten into just about any possible kind of .Net 5+ application, I want to take a trip through yesteryear and appreciate just how much better the .Net platform is today than it’s every been. If you wanna skip down to the positive part of the post, just look for sub heading “The better future for .Net is already here!.”

I’ve been a .Net developer since the very ugly beginning when we were all just glad to be off VB6/COM and onto a “modern” programming language that felt like a better Java but not yet completely horrified at how disastrously bad WebForms was going to turn out to be or disappointed at how poorly all our new .Net stuff worked out when we tried to adopt Agile development practices like Test Driven Development.

I’ve also been very deeply invested in developing open source tools and frameworks on the .Net framework, starting with StructureMap, the very first production capable Inversion of Control library on .Net that first went into a production system in the summer of 2004.

As the maintainer of StructureMap throughout, I had a front row seat to the absolute explosion of adapter libraries for seemingly every possible permutation of framework, IoC tool, and logging library. And this happened because every framework of any note in the .Net world had a completely different way to abstract away the actual IoC container or allowed you to use different logging tools through the specific framework’s special adapter mechanisms.

I’m also old enough to remember when every tool took a direct dependency on log4net around the time they lost their public strong naming key and broke compatibility. That was loads of fun.

There also wasn’t a standard way to deal with application lifecycle events across different types of .Net applications or application frameworks (ASP.Net’s Global.ASAX being an exception I guess). Every framework at that time had a different way of being bootstrapped at application start up time, so developers had to do a lot of plumbing code as they pieced all this junk together with the myriad adapter libraries.

To recap the “old”, pre-Core .Net world:

  • Application framework developers had to create their own IoC and logging abstractions and support a plethora of adapter libraries as well as possibly having to create all new application lifecycle management facilities as part of their framework
  • Library developers had to support users through their usage of problematic framework integrations where the library in question was often used in non-optimal ways (my personal pet peeve, and I edited this post before publishing to eliminate some harsh criticism of certain well known .Net application frameworks).
  • Developers of actual .Net applications were rightfully intimidated by the complexity necessary to integrate open source tools, or simply struggled to do so as there was no standardization of how all those tools went about their integration into .Net applications.

I would argue that the situation in .Net as it was before .Net Core seriously impeded the development and adoption of open source tooling in our community — which also had the effect of slowing down the rate of innovation throughout the .Net community.

You might notice that I used the past tense quite a bit in the previous paragraphs, and that’s because I think…

The better future for .Net is already here!.

It’s hard to overstate this, but the generic host builder (IHostBuilder and IHost) in .Net Core / .Net 5 and the related is hugely advantageous for the .Net ecosystem. I tried to explain why I think this a bit in a recent episode of .Net Rocks, but it’s probably easier to demonstrate this by integrating Marten into a new .Net 5 web api project.

Assuming that you’re already familiar with the usage of the Startup class for configuring ASP.Net Core applications, I’m going to add Marten to the new service with the following code:

public void ConfigureServices(IServiceCollection services)
{
    // This is the absolute, simplest way to integrate Marten into your
    // .Net Core application with Marten's default configuration
    services.AddMarten(options =>
    {
        // Establish the connection string to your Marten database
        options.Connection(Configuration.GetConnectionString("Marten"));

        // If we're running in development mode, let Marten just take care
        // of all necessary schema building and patching behind the scenes
        if (Environment.IsDevelopment())
        {
            options.AutoCreateSchemaObjects = AutoCreate.All;
        }

        if (Environment.IsProduction())
        {
            // This is a new V4 feature I'll blog about soon...
            options.GeneratedCodeMode = TypeLoadMode.LoadFromPreBuiltAssembly;
        }

        // Turn on the async projection daemon in only one node
        options.Projections.AsyncMode = DaemonMode.HotCold;

        // Configure other elements of the event store or document types...
    });
}

Even if you’re a developer who is completely unfamiliar with Marten itself, the integration through AddMarten() follows the recent .Net idiom and would be conceptually similar to many other tools you probably already use. That by itself is a very good thing as there’s less novelty for new Marten developers (there is of course matching UseMarten() signatures that hang off of IHostBuilder that is useful for other kinds of .Net projects)

I purposely used a few bells and whistles in Marten to demonstrate more capabilities of the generic host in .Net Core. To go into more detail, the code above:

  • Registers all the necessary Marten services with the proper service lifecycles in the underlying system container. And that works for any IoC container that supports the standard DI abstractions in the Microsoft.Extensions.DependencyInjection namespace. There isn’t any kind of Marten.Lamar or Marten.DryIoC or Marten.WhateverIoCToolYouHappenToLike adapter libraries.
  • I’ve also turned on Marten’s asynchronous projection process which will run as a long running task in the application by utilizing the standard .Net Core IHostedService mechanism whose lifecycle will be completely managed by the generic host in .Net.
  • We were able to do a little bit of environment specific configuration to let Marten do all necessary schema management automatically at development time, but not allow Marten to change database schemas in production. I also opted into a new Marten V4 optimization for production cold starts when the application is running in “Production” mode. That’s easy to do now using .Net’s standard IHostEnvironment mechanism at bootstrapping time.
  • It’s not obvious from that code, but starting in V4.0, Marten will happily log to the standard ILogger interface from .Net Core. Marten itself doesn’t care where or how you actually capture log information, but because .Net itself now specifies a single set of logging abstractions to rule them all, Marten can happily log to any commonly used logging mechanism (NLog, Serilog, the console, Azure logging, whatever) without any other kind of adapter.
  • And oh yeah, I had to add the connection string to a Postgresql database from .Net Core’s IConfiguration interface, which itself could be pulling information any number of ways, but our code doesn’t have to care about anything else but that one standard interface.

As part of the team behind the Marten, I feel like our jobs are easier now than it was just prior to .Net Core and the generic host builder because now there are:

  • .Net standard abstractions for logging, configuration, and IoC service registration
  • The IHostedService mechanism is an easy, standard way in .Net to run background processes or even just to hook into application start up and tear down events with custom code
  • The IServiceCollection.Add[Tool Name]() and IHostBuilder.Use[Tool Name]() idioms provided us a way to integrate Marten into basically any kind of .Net Core / .Net 5/6 application through a mechanism that is likely to feel familiar to experienced .Net developers — which hopefully reduces the barrier to adoption for Marten

As some of you reading this may know, I was very heavily involved in building an alternative web development framework for .Net called FubuMVC well before .Net Core came on to the scene. When we built FubuMVC, we had to create from scratch lot of the exact same functionality that’s in the generic host today to manage application lifecycle events, bootstrapping the application’s IoC container through its own IoC abstractions, and hooks for extension with other .Net libraries. I wouldn’t say that was the primary reason that FubuMVC failed as a project, but it certainly didn’t help.

Flash forward to today, and I’ve worked off and on on a complete replacement for FubuMVC on .Net Core called Jasper for asynchronous messaging and an alternative for HTTP services. That project isn’t going anywhere at the moment, but Jasper’s internals are much, much smaller than FubuMVC’s were for similar functionality because Jasper can rely on the functionality and standard abstractions of the generic host in .Net Core. That makes tools like Jasper much easier for .Net developers to write.

Unsurprisingly, there are plenty of design and implementation decisions that Microsoft made with the generic host builder that I don’t agree with. I can point to a few things that I believe FubuMVC did much better for composition a decade ago than ASP.Net Core does today (and vice versa too of course). I deeply resent how the ASP.Net Core team wrote the IoC compliance rules that were rammed down the throats of all us IoC library maintainers. You can also point out that there are cultural problems within the .Net community in regards to OSS and that Microsoft itself severely retards open source innovation in .Net by stomping on community projects at will.

All of that may be true, but from a strictly technical perspective, I think that it is now much easier for the .Net community to build and adopt innovative open source tools than it ever has been. I largely credit the work that’s been done with the generic host, the common idioms for tool integration, and the standard interfaces that came in the .Net Core wave for potentially opening up a new golden age for .Net OSS development.

Brief Update on Marten V4

The latest, greatest Marten V4 release candidate is up on Nuget this morning. Marten V4 has turned out to be a massive undertaking for an OSS project with a small team, but we’re not that far off what’s turned into our “Duke Nukem Forever” release. We’re effectively “code complete” other than occasional bugs coming in from early users — which is fortunately a good thing as it’s helped us make some things more robust without impacting the larger community.

At this point it’s mostly a matter of completing the documentation updates for V4, and I’m almost refusing to work on new features until that’s complete. The Marten documentation website is moving to VitePress going forward, so it’ll be re-skinned with better search capabilities. We’re also taking this opportunity to reorganize the documentation with the hopes of making that site more useful for users.

Honestly, the biggest thing holding us back right now in my opinion is that I’ve been a little burned out and slow working on the documentation. Right now, I’m hoping we can pull the trigger on V4 in the next month or two.

Using Alba to Test ASP.Net Services

TL;DR: Alba is a handy library for integration tests against ASP.Net web services, and the rapid feedback that enables is very useful

One of our projects at Calavista right now is helping a client modernize and optimize a large .Net application, with the end goal being everything running on .Net 5 and an order of magnitude improvement in system throughput. As part of the effort to upgrade the web services, I took on a task to replace this system’s usage of IdentityServer3 with IdentityServer4, but still use the existing Marten-backed data storage for user membership information.

Great, but there’s just one problem. I’ve never used IdentityServer4 before and it changed somewhat between the IdentityServer3 code I was trying to reverse engineer and its current model. I ended up getting through that work just fine. A key element of doing that was using the Alba library to create a test harness so I could iterate through configuration changes quickly by rerunning tests on the new IdentityServer4 project. It didn’t start out this way, but Alba is essentially a wrapper around the ASP.Net TestServer and just acts as a utility to make it easier to write automated tests around the HTTP services in your web service projects.

I ended up starting two new .Net projects:

  1. A new web service that hosts IdentityServer4 and is configured to use user membership information from our client’s existing Marten/Postgresql database
  2. A new xUnit.Net project to hold integration tests against the new IdentityServer4 web service.

Let’s dive right into how I set up Alba and xUnit.Net as an automated test harness for our new IdentityServer4 service. If you start a new ASP.Net project with one of the built in project templates, you’ll get a Program file that’s the main entry point for the application and a Startup class that has most of the system’s bootstrapping configuration. The templates will generate this method that’s used to configure the IHostBuilder for the application:

public static IHostBuilder CreateHostBuilder(string[] args)
{
    return Host.CreateDefaultBuilder(args)
        .ConfigureWebHostDefaults(webBuilder =>
        {
            webBuilder.UseStartup<Startup>();
        });
}

For more information on what role of the IHostBuilder is within your application, see .NET Generic Host in ASP.NET Core.

That’s important, because that gives us the ability to stand up the application exactly as it’s configured in an automated test harness. Switching to the new xUnit.Net test project, referenced my new web service project that will host IdentityServer4. Because spinning up your ASP.Net system can be relatively expensive, I only want to do that once and share the IHost between tests. That’s a perfect usage for xUnit.Net’s shared context support.

First I make what will be the shared test fixture context class for the integration tests shown below:

public class AppFixture : IDisposable
{
    public AppFixture()
    {
        // This is calling into the actual application's
        // configuration to stand up the application in 
        // memory
        var builder = Program.CreateHostBuilder(new string[0]);

        // This is the Alba "SystemUnderTest" wrapper
        System = new SystemUnderTest(builder);
    }
    
    public SystemUnderTest System { get; }

    public void Dispose()
    {
        System?.Dispose();
    }
}

The Alba SystemUnderTest wrapper is responsible for building the actual IHost object for your system, and does so using the in memory TestServer in place of Kestrel.

Just as a convenience, I like to create a base class for integration tests I tend to call IntegrationContext:

    public abstract class IntegrationContext 
        // This is some xUnit.Net mechanics
        : IClassFixture<AppFixture>
    {
        public IntegrationContext(AppFixture fixture)
        {
            System = fixture.System;
            
            // Just casting the IServiceProvider of the underlying
            // IHost to a Lamar IContainer as a convenience
            Container = (IContainer)fixture.System.Services;
        }
        
        public IContainer Container { get; }
        
        // This gives the tests access to run
        // Alba "Scenarios"
        public ISystemUnderTest System { get; }
    }

We’re using Lamar as the underlying IoC container in this application, and I wanted to use Lamar-specific IoC diagnostics in the tests, so I expose the main Lamar container off of the base class as just a convenience.

To finally turn to the tests, the very first thing to try with IdentityServer4 was just to hit the descriptive discovery endpoint just to see if the application was bootstrapping correctly and IdentityServer4 was functional *at all*. I started a new test class with this declaration:

    public class EndToEndTests : IntegrationContext
    {
        private readonly ITestOutputHelper _output;
        private IDocumentStore theStore;

        public EndToEndTests(AppFixture fixture, ITestOutputHelper output) : base(fixture)
        {
            _output = output;

            // I'm grabbing the Marten document store for the app
            // to set up user information later
            theStore = Container.GetInstance<IDocumentStore>();
        }

And then a new test just to exercise the discovery endpoint:

[Fact]
public async Task can_get_the_discovery_document()
{
    var result = await System.Scenario(x =>
    {
        x.Get.Url("/.well-known/openid-configuration");
        x.StatusCodeShouldBeOk();
    });
    
    // Just checking out what's returned
    _output.WriteLine(result.ResponseBody.ReadAsText());
}

The test above is pretty crude. All it does is try to hit the `/.well-known/openid-configuration` url in the application and see that it returns a 200 OK HTTP status code.

I tend to run tests while I’m coding by using keyboard shortcuts. Most IDEs support some kind of “re-run the last test” keyboard shortcut. Using that, my preferred workflow is to run the test once, then assuming that the test is failing the first time, work in a tight cycle of making changes and constantly re-running the test(s). This turned out to be invaluable as it took me a couple iterations of code changes to correctly re-create the old IdentityServer3 configuration into the new IdentityServer4 configuration.

Moving on to doing a simple authentication, I wrote a test like this one to exercise the system with known credentials:

[Fact]
public async Task get_a_token()
{
    // All I'm doing here is wiping out the existing database state
    // with Marten and using a little helper method to add a new
    // user to the database as part of the test setup
    // This would obviously be different in your system
    theStore.Advanced.Clean.DeleteAllDocuments();
    await theStore
        .CreateUser("aquaman@justiceleague.com", "dolphin", "System Administrator");
    
    
    var body =
        "client_id=something&client_secret=something&grant_type=password&scope=ourscope%20profile%20openid&username=aquaman@justiceleague.com&password=dolphin";

    // The test would fail here if the status
    // code was NOT 200
    var result = await System.Scenario(x =>
    {
        x.Post
            .Text(body)
            .ContentType("application/x-www-form-urlencoded")
            .ToUrl("/connect/token");
    });

    // As a convenience, I made a new class called ConnectData
    // just to deserialize the IdentityServer4 response into 
    // a .Net object
    var token = result.ResponseBody.ReadAsJson<ConnectData>();
    token.access_token.Should().NotBeNull();
    token.token_type.Should().Be("Bearer");
}

public class ConnectData
{
    public string access_token { get; set; }
    public int expires_in { get; set; }
    public string token_type { get; set; }
    public string scope { get; set; }
    
}

Now, this test took me several iterations to work through until I found exactly the right way to configure IdentityServer4 and adjusted our custom Marten backing identity store (IResourceOwnerPasswordValidator and IProfileService in IdentityServer4 world) until the tests pass. I found it extremely valuable to be able to debug right into the failing tests as I worked, and even needed to take advantage of JetBrains Rider’s capability to debug through external code to understand how IdentityServer4 itself worked. I’m very sure that I was able to get through this work much faster by iterating through tests as opposed to just trying to run the application and driving it through something like Postman or through the connected user interface.

Improvements to Event Sourcing in Marten V4

Marten V4 is still in flight, but everything I’m showing in this post is in the latest alpha release (4.0.0-alpha.6) release to Nuget.

TL;DR: Marten V4 has some significant improvements to its event sourcing support that will help developers deal with potential concurrency issues. The related event projection support in Marten V4 is also significantly more robust than previous versions.

A Sample Project Management Event Store

Imagine if you will, a simplistic application that uses Marten’s event sourcing functionality to do project management tracking with a conceptual CQRS architecture. The domain will include tracking each project task within the greater project. In this case, I’m choosing to model the activity of a single project and its related tasks as a stream of events like:

  1. ProjectStarted
  2. TaskRecorded
  3. TaskStarted
  4. TaskFinished
  5. ProjectCompleted

Starting a new Event Stream

Using event sourcing as our persistence strategy, the real system state is the raw events that model state changes or transitions of our project. As an example, let’s say that our system records and initializes a “stream” of events for a new project with this command handler:

    public class NewProjectCommand
    {
        public string Name { get; set; }
        public string[] Tasks { get; set; }
        public Guid ProjectId { get; set; } = Guid.NewGuid();
    }

    public class NewProjectHandler
    {
        private readonly IDocumentSession _session;

        public NewProjectHandler(IDocumentSession session)
        {
            _session = session;
        }

        public Task Handle(NewProjectCommand command)
        {
            var timestamp = DateTimeOffset.Now;
            var started = new ProjectStarted  
            {
                Name = command.Name, 
                Timestamp = timestamp
            };

            var tasks = command.Tasks
                .Select(name => new TaskRecorded {
                    Timestamp = timestamp, 
                    Title = name
                });

            // This tells Marten that it should create a new project
            // stream
            _session.Events.StartStream(command.ProjectId, started);

            // This quietly appends events to the event stream
            // created in the line of code above
            _session.Events.Append(command.ProjectId, tasks);

            // The actual persistence to Postgresql happens here
            return _session.SaveChangesAsync();
        }
    }

The call to StartStream() up above tells Marten that it should create a new event stream with the supplied project id. As a long requested improvement in Marten for V4, StartStream() guarantees that the transaction cannot succeed if another project event stream with that project id already exists. This helps the system prevent users from accidentally duplicating projects — at least by its project id anyway.

Assuming that there is no existing stream with the project id, Marten will create a new record to track the new project stream and individual records in Postgresql to persist the raw event data along with metadata like the stream version, time stamps, and the event type.

But we need a “read side” projection of the Projects!

So great, we can persist the raw events, and there’s plenty of valuable things we can do later with those events. However, our application sooner or later is going to need to know what the current state of an ongoing project for user screens or validation logic, so we need some way to compile the events for a single project into some kind of “here’s the current state of the Project” data structure — and that’s where Marten’s support for projections comes into play.

To model a “projected” view of a project from its raw events, I’m creating a single Project class to model the full state of a single, ongoing project. To reduce the number of moving parts, I’m going to make Project be “self-aggregating” such that it’s responsible for mutating itself based on incoming events:

    public class Project
    {
        private readonly IList _tasks = new List();

        public Project(ProjectStarted started)
        {
            Version = 1;
            Name = started.Name;
            StartedTime = started.Timestamp;
        }

        // This gets set by Marten
        public string Id { get; set; }

        public long Version { get; set; }
        public DateTimeOffset StartedTime { get; private set; }
        public DateTimeOffset? CompletedTime { get; private set; }

        public string Name { get; private set; }

        public ProjectTask[] Tasks
        {
            get
            {
                return _tasks.ToArray();
            }
            set
            {
                _tasks.Clear();
                _tasks.AddRange(value);
            }
        }

        public void Apply(TaskRecorded recorded, IEvent e)
        {
            Version = e.Version;
            var task = new ProjectTask
            {
                Title = recorded.Title,
                Number = _tasks.Max(x => x.Number) + 1,
                Recorded = recorded.Timestamp
            };

            _tasks.Add(task);
        }

        public void Apply(TaskStarted started, IEvent e)
        {
            Version = e.Version;
            var task = _tasks.FirstOrDefault(x => x.Number == started.Number);

            // Remember this isn't production code:)
            if (task != null) task.Started = started.Timestamp;
        }

        public void Apply(TaskFinished finished, IEvent e)
        {
            Version = e.Version;
            var task = _tasks.FirstOrDefault(x => x.Number == finished.Number);

            // Remember this isn't production code:)
            if (task != null) task.Finished = finished.Timestamp;
        }

        public void Apply(ProjectCompleted completed, IEvent e)
        {
            Version = e.Version;
            CompletedTime = completed.Timestamp;
        }
    }

I didn’t choose to use that here, but I want to point out that you can use immutable aggregates with Marten V4. In that case you’d simply have the Apply() methods return a new Project object (as far as Marten is concerned anyway). Functional programming enthusiasts will cheer the built in support for immutability, some of you will point out that that leads to less efficient code by increasing the number of object allocations and more code ceremony, and I will happily say that Marten V4 let’s you make that decision to suit your own needs and preferences;-)

Also see Event Sourcing with Marten V4: Aggregated Projections for more information on other alternatives for expressing aggregated projections in Marten.

Working in our project management domain, I’d like the Project aggregate document to be updated every time events are captured for an event related to a project. I’ll add the Project document as a self-aggregating, inline projection in my Marten system like this:

    public class Startup
    {
        public void ConfigureServices(IServiceCollection services)
        {
            services.AddMarten(opts =>
            {
                opts.Connection("some connection");

                // Direct Marten to update the Project aggregate
                // inline as new events are captured
                opts.Events
                    .Projections
                    .SelfAggregate<Project>(ProjectionLifecycle.Inline);

            });
        }
    }

As an example of how inline projections come into play, let’s look at another sample command handler for adding a new task to an ongoing project with a new task title :

    public class CreateTaskCommand
    {
        public string ProjectId { get; set; }
        public string Title { get; set; }
    }

    public class CreateTaskHandler
    {
        private readonly IDocumentSession _session;

        public CreateTaskHandler(IDocumentSession session)
        {
            _session = session;
        }

        public Task Handle(CreateTaskCommand command)
        {
            var recorded = new TaskRecorded
            {
                Timestamp = DateTimeOffset.UtcNow, 
                Title = command.Title
            };

            _session.Events.Append(command.ProjectId, recorded);
            return _session.SaveChangesAsync();
        }
    }

When SaveChangesAsync() is called above, Marten will issue a single database commit that captures a new TaskRecorded event. In that same commit Marten will fetch the persisted Project document for that project event stream, apply the changes from the new event, and persist the Project document that reflects the information in the events.

Let’s say that we have a user interface in our project management system that allows users to review and edit projects and tasks. Independent of the event streams, the “query” side of our application can happily retrieve the latest data for the Project documents by querying with Marten’s built in document database functionality like this simple MVC controller endpoint:

    public class ProjectController: ControllerBase
    {
        private readonly IQuerySession _session;

        public ProjectController(IQuerySession session)
        {
            _session = session;
        }

        [HttpGet("/api/project/{projectId}")]
        public Task<Project> Get(string projectId)
        {
            return _session.LoadAsync<Project>(projectId);
        }
    }

For a future blog post, I’ll show you a new Marten V4 feature for ultra efficient “read side” web services by streaming JSON data directly from Postgresql straight down the HTTP response body without ever wasting any time with JSON serialization.

Concurrency Problems Everywhere!!!

Don’t name a software project “Genesis”. Hubristic project names lead to massive project failures.

The “inline” projection strategy is easy to use in Marten, but it’s vulnerable to oddball concurrency problems if you’re not careful in your system design. Let’s say that we have two coworkers named Jack and Jill using our project management system. Jack pulls up the data for the “Genesis Project” in the UI client, then gets pulled into a hallway conversation (remember when we used to have those in the before times?). While he’s distracted, Jill makes some changes to the “Genesis Project” to record some tasks she and Jack had talked about earlier and saves the changes to the server. Jack finally gets back to his machine and makes basically the same edits to “Genesis” that Jill already did and saves the data. If we build our project management system naively, we’ve now allowed Jack and Jill to duplicate work and the “Genesis Project” task management is all messed up.

One way to prevent that concurrent change issue is to detect that the project has been changed by Jill when Jack tries to persist his duplicate changes and give him the chance to update his screen with the latest data before pushing new project task changes.

Going back to the Project aggregate document, as Marten appends events to a stream, it increments a numeric version number to each event within a stream. Let’s say in our project management system that we always want to know what the latest version of the project. Marten V4 finally allows you to use event metadata like the event version number within inline projections (this has been a long running request from some of our users). You can see that in action in the method below that updates Project based on a TaskStarted event. You might notice that I also pass in the IEvent object that would let us access the event metadata:

public void Apply(TaskStarted started, IEvent e)
{
    // Update the Project document based on the event version
    Version = e.Version;
    var task = _tasks.FirstOrDefault(x => x.Number == started.Number);

    // Remember this isn't production code:)
    if (task != null) task.Started = started.Timestamp;
}

Now, having the stream version number on the Project document turns out to be very helpful for concurrency issues. Since we’re worried about a Project event stream state getting out of sync if the system receives concurrent updates you can pass the current version that’s conveniently updated on the Project read side document down to the user interface, and have the user interface send you what it thinks is the current version of the project when it tries to make updates. If the underlying project event stream has changed since the user interface fetched the original data, we can make Marten reject the additional events. This is a form of an offline optimistic concurrency check, as we’re going to assume that everything is hunky dory, and just let the infrastructure reject the changes as an exceptional case.

To put that into motion, let’s say that our project management user interface posts this command up to the server to close a single project task:

    public class CompleteTaskCommand
    {
        public Guid ProjectId { get; set; }
        public int TaskNumber { get; set; }
        
        // This is the version of the project data
        // that was being edited in the user interface
        public long ExpectedVersion { get; set; }
    }

In the command handler, we can direct Marten to reject the new events being appended if the stream has been changed in between the user interface having fetched the project data and submitting its updates to the server:

    public class CompleteTaskHandler
    {
        private readonly IDocumentSession _session;

        public CompleteTaskHandler(IDocumentSession session)
        {
            _session = session;
        }

        public Task Handle(CompleteTaskCommand command)
        {
            var @event = new TaskFinished
            {
                Number = command.TaskNumber,
                Timestamp = DateTimeOffset.UtcNow
            };

            _session.Events.Append(
                command.ProjectId,

                // Using this overload will make Marten do
                // an optimistic concurrency check against
                // the existing version of the project event
                // stream as it commits
                command.ExpectedVersion,

                @event);
            return _session.SaveChangesAsync();
        }
    }

In the CompleteTaskHandler above, the call to SaveChangesAsync() will throw a EventStreamUnexpectedMaxEventIdException exception if the project event stream has advanced past the existing version assumed by the originator of the command in command.ExpectedVersion. To make our system more resilient, we would need to catch the Marten exception and deal with the proper application workflow.

What I showed above is pretty draconian in terms of what edits it allows to go through. In other cases you may get by with a simple workflow that just tries to guarantee that a single Project aggregate is only being updated by a single process at any time. Marten V4 comes through with a couple new ways to append events with more control over concurrent updates to a single event stream.

Let’s stick to optimistic checks for now and look at the new AppendOptimistic() method for appending events to an existing event stream with a rewritten version of our CompleteTaskCommand handling:

    public class CompleteTaskHandler2
    {
        private readonly IDocumentSession _session;

        public CompleteTaskHandler2(IDocumentSession session)
        {
            _session = session;
        }

        public Task Handle(CompleteTaskCommand command)
        {
            var @event = new TaskFinished
            {
                Number = command.TaskNumber,
                Timestamp = DateTimeOffset.UtcNow
            };

            // If some other process magically zips
            // in and updates this project event stream
            // between the call to AppendOptimistic()
            // and SaveChangesAsync(), Marten will detect
            // that and reject the transaction
            _session.Events.AppendOptimistic(
                command.ProjectId,
                @event);

            return _session.SaveChangesAsync();
        }
    }

We would still need to catch the Marten exceptions and handle those somehow. I myself would typically try to handle that with a messaging or command execution framework like MassTransit or my own Jasper project that comes with robust exception handling and retry capabilities.

At this point everything I’ve shown for concurrency control involves optimistic locks that result in exceptions being thrown and transactions being rejected. For another alternative, Marten V4 leverages Postgresql’s robust row locking support with this version of our CompleteTaskCommand handler that uses the new V4 AppendExclusive() method:

    public class CompleteTaskHandler3
    {
        private readonly IDocumentSession _session;

        public CompleteTaskHandler3(IDocumentSession session)
        {
            _session = session;
        }

        public Task Handle(CompleteTaskCommand command)
        {
            var @event = new TaskFinished
            {
                Number = command.TaskNumber,
                Timestamp = DateTimeOffset.UtcNow
            };

            // This tries to acquire an exclusive
            // lock on the stream identified by
            // command.ProjectId in the database
            // so that only one process at a time
            // can update this event stream
            _session.Events.AppendExclusive(
                command.ProjectId,
                @event);
            return _session.SaveChangesAsync();
        }
    }

This is heavier in terms of its impact on the database, but simpler to implement as there is no flow control by exceptions to worry about.

I should also note that both the AppendOptimistic() and AppendExclusive() methods will verify that the stream already exists and throw an exception if the stream does not.