Dynamic Tenant Databases in Marten

Hey, did you know that JasperFx Software is ready for formal support plans for Marten and Wolverine? Not only are we making the “Critter Stack” tools be viable long term options for your shop, we’re also interested in hearing your opinions about the tools and how they should change. We’re also certainly open to help you succeed with your software development projects on a consulting basis whether you’re using any part of the Critter Stack or some completely different .NET server side tooling.

Marten 7.0.0 RC just dropped on Nuget with some new fixes and some very long awaited enhancements that I’m personally very excited about. The docs need to catch up, but the 7.0 release is shaping up for next week (come hell or high water). One of the new highlights for Marten 7 that was sponsored by a JasperFx Software client was the ability to add new tenant databases at runtime when using Marten’s “database per tenant” strategy.

It’s not documented yet (I’m working on it! I swear!), but here’s a sneak peek from an integration test:

        // Setting up a Host with Multi-tenancy
        _host = await Host.CreateDefaultBuilder()
            .ConfigureServices(services =>
            {
                services.AddMarten(opts =>
                    {
                        // This is a new strategy for configuring tenant databases with Marten
                        // In this usage, Marten is tracking the tenant databases in a single table in the "master"
                        // database by tenant
                        opts.MultiTenantedDatabasesWithMasterDatabaseTable(ConnectionSource.ConnectionString, "tenants");

                        opts.RegisterDocumentType<User>();
                        opts.RegisterDocumentType<Target>();

                        opts.Projections.Add<TripProjectionWithCustomName>(ProjectionLifecycle.Async);
                    })
                    .AddAsyncDaemon(DaemonMode.Solo)

                    // All detected changes will be applied to all
                    // the configured tenant databases on startup
                    .ApplyAllDatabaseChangesOnStartup();

            }).StartAsync();

With this model, Marten is setting up a table named mt_tenant_databases to store with just two columns:

tenant_id
connection_string

At runtime, when you ask for a new session for a specific tenant like so:

using var session = store.LightweightSession("tenant1");

This new Marten tenancy strategy will first look for a database with the “tenant1” identifier its own memory, and if it’s not found, will try to reach into the database table to “find” the connection string for this newly discovered tenant. If a record is found, the new tenancy strategy caches the information, and proceeds just like normal.

Now, let me try to anticipate a couple questions you might have here:

Can Marten track and apply database schema changes to new tenant databases at runtime? Yes, Marten does the schema check tracking on a database by database basis. This means that if you add a new tenant database to that underlying table, Marten will absolutely be able to make schema changes as needed to just that tenant database regardless of the state of other tenant databases.
Will the Marten command line tools recognize new tenant databases? Yes, same thing. If you call dotnet run -- marten-apply for example, Marten will do the schema migrations independently for each tenant database, so any outstanding changes will be performed on each tenant database.
Can Marten spin up asynchronous projections for a new tenant database without requiring downtime? Yes! Check out this big ol’ integration test proving that the new Marten V7 version of the async daemon can handle that just fine:

    [Fact]
    public async Task add_tenant_database_and_verify_the_daemon_projections_are_running()
    {
        // In this code block, I'm adding new tenant databases to the system that I
        // would expect Marten to discover and start up an asynchronous projection
        // daemon for all three newly discovered databases
        var tenancy = (MasterTableTenancy)theStore.Options.Tenancy;
        await tenancy.AddDatabaseRecordAsync("tenant1", tenant1ConnectionString);
        await tenancy.AddDatabaseRecordAsync("tenant2", tenant2ConnectionString);
        await tenancy.AddDatabaseRecordAsync("tenant3", tenant3ConnectionString);

        // This is a new service in Marten specifically to help you interrogate or
        // manipulate the state of running asynchronous projections within the current process
        var coordinator = _host.Services.GetRequiredService<IProjectionCoordinator>();
        var daemon1 = await coordinator.DaemonForDatabase("tenant1");
        var daemon2 = await coordinator.DaemonForDatabase("tenant2");
        var daemon3 = await coordinator.DaemonForDatabase("tenant3");

        // Just proving that the configured projections for the 3 new databases
        // are indeed spun up and running after Marten's new daemon coordinator
        // "finds" the new databases
        await daemon1.WaitForShardToBeRunning("TripCustomName:All", 30.Seconds());
        await daemon2.WaitForShardToBeRunning("TripCustomName:All", 30.Seconds());
        await daemon3.WaitForShardToBeRunning("TripCustomName:All", 30.Seconds());
    }

At runtime, if the Marten V7 version of the async daemon (our sub system for building asynchronous projections constantly in a background IHostedService) is constantly doing “health checks” to make sure that *some process* is running all known asynchronous projections on all known client databases. Long story, short, Marten 7 is able to detect new tenant databases and spin up the asynchronous projection handling for these new tenants with zero downtime.

There’s a helluva lot more new stuff and big improvements to the old stuff in Marten coming in V7, but this one was a definite highlight.

Look for the official Marten 7.0 release next week!