We’ve got an upcoming Marten 5.0 release ostensibly to support breaking changes related to .Net 6, but that also gives us an opportunity to consider work that would result in breaking API changes. A strong candidate for V5 right now is finally adding long delayed first class support for multi-tenancy through separate databases.

Let’s say that you’re building an online database-backed, web application of some sort that will be servicing multiple clients. At a minimum, you need to isolate data access so that client users can only interact with the data for the correct client or clients. Ideally, you’d like to get away with only having one deployed instance of your application that services the users of all the clients. In other words, you want to support “multi-tenancy” in your architecture.

For the rest of this post, I’m going to use the term “tenant” to refer to whatever the organizational entity is that owns separate database data. Depending on your business domain, that could be a client, a sub-organization, a geographic area, or some other organizational concept.

Fortunately, if you use Marten as your backing database store, Marten has strong support for multi-tenancy with new improvements in the recent V4 release and more potential improvements tentatively planned for V5.

There are three basic approaches to segregating tenant data in a database:

1. Single database, single schema, but use a field or property in each table to denote the tenant. This is Marten’s approach today with what we call the “Conjoined” model. The challenge here is that all queries and writes to the database need to take into account the currently used tenant — and that’s where Marten’s multi-tenancy support helps a great deal. Database schema management is easier with this approach because there’s only one set of database objects to worry about. More on this later.
2. Separate schema per tenant in a single database. Marten does not support this model, and it doesn’t play well with Marten’s current internal design. I seriously doubt that Marten will ever support this.
3. Separate database per tenant. This has been in Marten’s backlog forever, and maybe now is the time this finally gets done (plenty of folks have used Marten this way already with custom infrastructure on top of Marten, but there’s some significant overhead). I’ll speak to this much more in the last section of this post.

Basic Multi-Tenancy Support in Marten

To set up multi-tenancy in your document storage with Marten, we can set up a document store with these options:

    var store = DocumentStore.For(opts =>
    {
        opts.Connection("some connection string");

        // Let's just say that each and every document
        // type is going to be multi-tenanted
        opts.Policies.AllDocumentsAreMultiTenanted();

        // Or you can do this document type by document type
        // if some document types are not related to a tenant
        opts.Schema.For<User>().MultiTenanted();
    });

There’s a couple other ways to opt document types into multi-tenancy, but you get the point. With just this, we can start a new Marten session for a particular tenant and carry out basic operations isolated to a single tenant like so:

    // Open a session specifically for the tenant "tenant1"
    await using var session = store.LightweightSession("tenant1");

    // This would return *only* the admin users from "tenant1"
    var users = await session.Query<User>().Where(x => x.Roles.Contains("admin"))
        .ToListAsync();

    // This user would be automatically be tagged as belonging to "tenant1" 
    var user = new User {UserName = "important_guy", Roles = new string[] {"admin"}};
    session.Store(user);

    await session.SaveChangesAsync();

The key thing to note here is that other than telling Marten which tenant you want to work with as you open a new session, you don’t have to do anything else to keep the tenant data segregated as Marten is dealing with those mechanics behind the scenes on all queries, inserts, updates, and deletions from that session.

Awesome, except that some folks needed to occasionally do operations against multiple tenants at one time…

Tenant Spanning Operations in Marten V4

The big improvements in Marten V4 for multi-tenancy was in making it much easier to work with data from multiple tenants in one document session. Marten has long had the ability to query data across tenants with the AnyTenant() or ` like so:

    var allAdmins = await session.Query<User>()
        .Where(x => x.Roles.Contains("admin"))
        
        // This is a Marten specific extension to Linq
        // querying
        .Where(x => x.AnyTenant())
        
        .ToListAsync();

Which is great for what it is, but there wasn’t any way to know what tenant each document returned belonged to. We made a huge effort in V4 to expand Marten’s document metadata capabilities, and part of that is the ability to write the tenant id to a document being fetched from the database by Marten. The easiest way to do that is to have your document type implement the new ITenanted interface like so:

    public class MyTenantedDoc: ITenanted
    {
        public Guid Id { get; set; }
        
        // This property will be set by Marten itself
        // when the document is persisted or loaded
        // from the database
        public string TenantId { get; set; }
    }

So now we at least have the ability to know which documents we queried across the tenants belong to which tenant.

The next thing folks wanted from V4 was the ability to make writes against multiple tenants with one single document session in a single unit of work. To that end, Marten V4 introduced the concept of ITenantOperations to log operations against a specific tenants besides the tenant the current session was opened as. And all those operations should be committed to the underlying Postgresql database as a single transaction.

To make that concrete, here’s some sample code, but this time adding two new User document with the same user name, but to two different tenants by tenant id:

    // Same user name, but in different tenants
    var user1 = new User {UserName = "bob"};
    var user2 = new User {UserName = "bob"};

    // This exposes operations against only tenant1
    session.ForTenant("tenant1").Store(user1);

    // This exposes operations that would apply to 
    // only tenant2
    session.ForTenant("tenant2").Store(user2);
 
    // And both operations get persisted in one transaction
    await session.SaveChangesAsync();

So that’s the gist of the V4 multi-tenancy improvements. We also finally support multi-tenancy within the asynchronous projection support, but I’ll blog about that some other time.

Now though, it’s time to consider…

Database per Tenant

To be clear, I’m looking for any possible feedback about the requirements for this feature in Marten. Blast away here in comments, or here’s a link to the GitHub issue, or go to Gitter.

While you can — and many folks have successfully achieved — multi-tenancy through database per tenant by just keeping an otherwise identically configured DocumentStore per named tenant in memory with the only difference being a connection string. That certainly can work, especially with a low number of tenants. There’s a few problems with that approach though:

• You’re on your own to configure that in the DI container within your application
• DocumentStore is a relatively expensive object to create, and it potentially generates a lot of runtime objects that get held in memory. You don’t really want a bunch of those hanging around
• Going around AddMarten() negates the Marten CLI support, which is the easiest possible way to manage Marten database schema migrations. Now you’re completely on your own about how to do database migrations without using pure runtime database patching — which we do not recommend in production.

So let’s just call it a given that we do want to add some formal support for multi-tenancy through separate databases per tenant to Marten. Moreover, Database per Tenant been in our backlog forever, but pushed off every time we’ve struggled to make big Marten releases.

I think there’s some potential for this story to cause breaking API changes (I don’t have anything specific in mind, it’s just likely in my opinion), so that makes that story a very good candidate to get in place for Marten V5. From the backlog issue writeup I made back in 2017:

• Have all tenants tracked in memory, such that a single DocumentStore can share all the expensive runtime built internal objects across tenants
• A tenanting strategy that can lookup the database connection string per tenant, and create sessions per separate tenants. There’s actually an interface hook in Marten all ready to go that may serve out of the box when we do this (I meant to do this work years ago, but it just didn’t happen).
• At development time (AutoCreate != AutoCreate.None), be able to spin up a new database on the fly for a tenant if it doesn’t already exist
• “Know” what all the existing tenants are so that we could apply database migrations from the CLI or through the DocumentStore schema migration APIs
• Extend the CLI support to support multiple tenant databases
• Make the database registry mechanism be a little bit pluggable. Thinking that some folks would have a few tenants where you’d be good with just writing everything into a static configuration file. Other folks may have a *lot* of tenants (I’ve personally worked on a system that had >100 separate tenant databases in one deployed application), so they may want a “master” database

If you haven’t seen this yet, Marten V4.0 officially dropped on Nuget late last week (finally).

V4 was a huge release for Marten and there’s lots to talk about in the new release, but I want to start simply with just talking about Marten’s support for deleting documents. In all the examples, we’re going to use a User document type from our testing code.

Let’s say we have the identity of a User document that we want to delete in our Marten database. That code is pretty simple, it’s just this below:

internal Task DeleteByDocumentId(IDocumentSession session, Guid userId)
{
    // Tell Marten the type and identity of a document to
    // delete
    session.Delete<User>(userId);

    return session.SaveChangesAsync();
}

By default, Marten will do a “hard” delete where the actual database row in Postgresql is deleted and that’s all she wrote. Marten has long had support for “soft” deletes where the underlying rows are marked as deleted with a metadata column that tracks “is deleted” instead of actually deleting the row, so let’s opt into that by configuring the User document as soft-deleted like so:

// Configuring a new Marten document store
var store = DocumentStore.For(opts =>
{
    opts.Connection("some connection string");

    opts.Schema.For<User>().SoftDeleted();
});

Or if you prefer, you can also use an attribute like so:

[SoftDeleted]
public class User
{
    public Guid Id { get; set; }
    
    // Other props we don't care about here
}

And I’ll show a 3rd way that’s new in Marten V4 later on.

The API to delete a User document is exactly the same, but the mechanics of what Marten is doing to the underlying Postgresql database have changed. Instead of deleting the underlying rows, Marten just marks an mt_deleted column as true. But now, think about this Linq query below where I’m searching for all the users that have the “admin” role:

            using var session = store.QuerySession();
            var admins = await session.Query<User>()
                .Where(x => x.Roles.Contains("admin"))
                .ToListAsync();

You’ll notice that I’m not doing anything to explicitly filter out the deleted users in this query, and that’s because Marten is doing that for you — and that’s the behavior you’d want most of the time. If for some reason you’d like to query for all the documents, even the ones that are marked as deleted, you can query like this:

            var admins = await session.Query<User>()
                .Where(x => x.Roles.Contains("admin"))
                
                // This is Marten specific
                .Where(x => x.MaybeDeleted())
                .ToListAsync();

And now, if you only want to query for documents that are explicitly marked as deleted, you can do this:

            var admins = await session.Query<User>()
                .Where(x => x.Roles.Contains("admin"))
                
                // This is Marten specific
                .Where(x => x.IsDeleted())
                .ToListAsync();

So far, so good? Now what if you change your mind about your deleted documents and you want to bring them back? Marten V4 adds the IDocumentSession.UndoDeleteWhere() method to reverse the soft deletes in the database. In the usage below, we’re going to mark every admin user as “not deleted”:

            using var session = store.LightweightSession();
            
            session.UndoDeleteWhere<User>(x => x.Roles.Contains("admin"));
            await session.SaveChangesAsync();

But hold on, what if instead you really want to just wipe out the database rows with a good old fashioned “hard” delete instead? Marten V4 adds some IDocumentSession.HardDelete*****() APIs to do exactly that as shown below:

            using var session = store.LightweightSession();

            session
                .HardDeleteWhere<User>(x => x.Roles.Contains("admin") && x.IsDeleted());
            await session.SaveChangesAsync();

There are also single document versions of HardDelete() as well.

So back to the sample of querying all admin users both past and present:

            var admins = await session.Query<User>()
                .Where(x => x.Roles.Contains("admin"))

                // This is Marten specific
                .Where(x => x.MaybeDeleted())
                .ToListAsync();

If you’re going to do that, it might be nice to be able to understand which users are marked as deleted, when they were marked deleted, and which users are not deleted. And it’d also be helpful if Marten would just tag the documents themselves with that metadata.

Enter the new Marten.Metadata.ISoftDeleted interface in V4. Let’s make our User document implement that interface as shown below:

public class User : ISoftDeleted
{
    public Guid Id { get; set; }

    // These two properties reflect Marten metadata
    // and will be updated upon Delete() calls
    // or when loading through Marten
    public bool Deleted { get; set; }
    public DateTimeOffset? DeletedAt { get; set; }

    // Other props we don't care about here
}

If a document type implements the ISoftDeleted interface, the document type will automatically be treated as soft-deleted by Marten, and the Deleted and DeletedAt properties will be set at document load time by Marten.

I’m not showing it here, but you can effectively create the same behavior without the marker interface by using a fluent interface. Check out the Marten documentation for that approach.

Going into Marten I thought of the delete operation as something simple conceptually, but thousands of users means getting requests for more control over the process and that’s what we hope V4 delivers here.