Recent Critter Stack Multi-Tenancy Improvements

Hey, did you know that JasperFx Software offers formal support plans for Marten and Wolverine? Not only are we making the “Critter Stack” tools be viable long term options for your shop, we’re also interested in hearing your opinions about the tools and how they should change. We’re also certainly open to help you succeed with your software development projects on a consulting basis whether you’re using any part of the Critter Stack or some completely different .NET server side tooling.

Marten 7.0 was released over the weekend, and Wolverine 2.0 followed yesterday mostly to catch up with the Marten dependency. One of the major improvements in this round of “Critter Stack” releases was to address a JasperFx client’s need for dynamically adding new tenant databases at runtime without having to do any kind of system deployment.

Doing that successfully meant adding a couple capabilities throughout the “Critter Stack.” First off, Marten needed a new multi-tenancy configuration strategy that allowed users to keep a list of valid tenant id and database connection strings for those tenants in a database table. Enter Marten 7’s new Master Table Tenancy Model. You can see the usage in this sample from the documentation:

using var host = await Host.CreateDefaultBuilder()
    .ConfigureServices(services =>
    {
        services.AddMarten(sp =>
            {
                var configuration = sp.GetRequiredService<IConfiguration>();
                var masterConnection = configuration.GetConnectionString("master");
                var options = new StoreOptions();

                // This is opting into a multi-tenancy model where a database table in the
                // master database holds information about all the possible tenants and their database connection
                // strings
                options.MultiTenantedDatabasesWithMasterDatabaseTable(x =>
                {
                    x.ConnectionString = masterConnection;

                    // You can optionally configure the schema name for where the mt_tenants
                    // table is stored
                    x.SchemaName = "tenants";

                    // If set, this will override the database schema rules for
                    // only the master tenant table from the parent StoreOptions
                    x.AutoCreate = AutoCreate.CreateOrUpdate;

                    // Optionally seed rows in the master table. This may be very helpful for
                    // testing or local development scenarios
                    // This operation is an "upsert" upon application startup
                    x.RegisterDatabase("tenant1", configuration.GetConnectionString("tenant1"));
                    x.RegisterDatabase("tenant2", configuration.GetConnectionString("tenant2"));
                    x.RegisterDatabase("tenant3", configuration.GetConnectionString("tenant3"));

                    // Tags the application name to all the used connection strings as a diagnostic
                    // Default is the name of the entry assembly for the application or "Marten" if
                    // .NET cannot determine the entry assembly for some reason
                    x.ApplicationName = "MyApplication";
                });

                // Other Marten configuration

                return options;
            })
            // All detected changes will be applied to all
            // the configured tenant databases on startup
            .ApplyAllDatabaseChangesOnStartup();;
    }).StartAsync();

With this tenancy model, Marten is able to discover newly added tenant databases at runtime as needed during normal transactions. And don’t fret about the extra activity, Marten is able to cache that information in memory to avoid making too many unnecessary database calls.

But what about if I need to decommission and remove a customer database? What if we need to move a tenant database at runtime? Can Marten create databases on the fly for us? How will I do that?!? And sorry, my reply is that will require at least an application restart for now, and any kind of fancier management for advanced multi-tenancy will likely go into the forthcoming “Critter Stack Pro” paid model later this year.

That’s part one. The next issue was that for users who also used Marten’s asynchronous projection feature, the “async daemon” subsystem in Marten needed to be able to discover new tenant databases in the background and ensure that all the asynchronous projections for these newly discovered databases are running in the background somewhere in the application. This led to a partial rewrite of the “async daemon” subsystem for Marten 7, but you can see the positive effect of that work in this test that “proves” that Marten is able to spin up projection building agents in the background at runtime:

    [Fact]
    public async Task add_tenant_database_and_verify_the_daemon_projections_are_running()
    {
        // In this code block, I'm adding new tenant databases to the system that I
        // would expect Marten to discover and start up an asynchronous projection
        // daemon for all three newly discovered databases
        var tenancy = (MasterTableTenancy)theStore.Options.Tenancy;
        await tenancy.AddDatabaseRecordAsync("tenant1", tenant1ConnectionString);
        await tenancy.AddDatabaseRecordAsync("tenant2", tenant2ConnectionString);
        await tenancy.AddDatabaseRecordAsync("tenant3", tenant3ConnectionString);

        // This is a new service in Marten specifically to help you interrogate or
        // manipulate the state of running asynchronous projections within the current process
        var coordinator = _host.Services.GetRequiredService<IProjectionCoordinator>();
        var daemon1 = await coordinator.DaemonForDatabase("tenant1");
        var daemon2 = await coordinator.DaemonForDatabase("tenant2");
        var daemon3 = await coordinator.DaemonForDatabase("tenant3");

        // Just proving that the configured projections for the 3 new databases
        // are indeed spun up and running after Marten's new daemon coordinator
        // "finds" the new databases
        await daemon1.WaitForShardToBeRunning("TripCustomName:All", 30.Seconds());
        await daemon2.WaitForShardToBeRunning("TripCustomName:All", 30.Seconds());
        await daemon3.WaitForShardToBeRunning("TripCustomName:All", 30.Seconds());
    }

Now, switching over to Wolverine 2.0 and its contribution to the party, the third part to make this dynamic tenant database discovery work throughout the entire “Critter Stack” was for Wolverine to be able to also discover the new tenant databases at runtime and spin up its “durability agents” for background message scheduling and its transactional inbox/outbox support. It’s admittedly not well ~~at all~~ documented yet, but Wolverine has an internal system for leader election and “agent assignment” between running nodes in an application cluster to distribute work. Wolverine uses this subsystem to distribute the transactional inbox/outbox work for each tenant database across the application cluster. Look for more information on this capability as JasperFx Software will be exploiting this for a different customer engagement this year.