Hot Chocolate, GraphQL, and the Critter Stack

Hey folks, this is more a brain dump to collect my own thoughts than any kind of tome of accumulated wisdom and experience. Please treat this accordingly, and absolutely chime in on the Critter Stack Discord discussion going on about this right now. I’m also very willing and maybe even likely to change my mind about anything I’m going to say in this post.

There’s been some recent interest and consternation about the combination of Marten within Hot Chocolate as a GraphQL framework. At the same time, I am working with a new JasperFx client who wants to use Hot Chocolate with Marten’s event store functionality behind mutations and projected data behind GraphQL queries.

Long story short, Marten and Hot Chocolate do not mix well without some significant thought and deviation from normal, out of the box Marten usage. Likewise, I’m seeing some significant challenges in using Wolverine behind Hot Chocolate mutations. The rest of this post is a rundown of the issues, sticking points, and possible future ameliorations to make this combination more effective for our various users.

Connections are Sticky in Marten

If you use the out of the box IServiceCollection.AddMarten() mechanism to add Marten into a .NET application, you’re registering Marten’s IQuerySession and IDocumentSession as a Scoped lifetime — which is optimal for usage within short lived ASP.Net Core HTTP requests or within message bus handlers (like Wolverine!). In both of those cases, the session can be expected to have a short lifetime and generally be running in a single thread — which is good because Marten sessions are absolutely not thread safe.

However, for historical reasons (integration with Dapper was a major use case in early Marten usage, so there’s some optimization for that with Marten sessions), Marten sessions have a “sticky” connection lifecycle where an underlying Npgsql connection is retained on the first database query until the session is disposed. Again, if you’re utilizing Marten within ASP.Net Core controller methods or Minimal API calls or Wolverine message handlers or most other service bus frameworks, the underlying IoC container of your application is happily taking care of resource disposal for you at the right times in the request lifecycle.

The last sentence is one of the most important, but poorly understood advantages of using IoC containers in applications in my opinion.

Ponder the following Marten usage:

    public static async Task using_marten(IDocumentStore store)
    {
        // The Marten query session is IDisposable,
        // and that absolutely matters!
        await using var session = store.QuerySession();

        // Marten opens a database connection at the first
        // need for that connection, then holds on to it
        var doc = await session.LoadAsync<User>("jeremy");
        
        // other code runs, but the session is still open
        // just in case...
        
        // The connection is closed as the method exits
        // and the session is disposed
    }

The problem with Hot Chocolate comes in because Hot Chocolate is trying to parallelize queries when you get multiple queries in one GraphQL request — which since that query batching was pretty well the raison d’être for GraphQL in the first place, so you should assume that’s quite common!

Now, consider a naive usage of a Marten session in a Hot Chocolate query:

    public async Task<SomeEntity> GetEntity(
        [Service] IQuerySession session
        Input input)
    {
        // load data using the session
    }

Without taking some additional steps to serialize access to the IQuerySession across Hot Chocolate queries, you will absolutely hit concurrency errors when Hot Chocolate tries to parallelize data fetching. You can beat this by either forcing Hot Chocolate to serialize access like so:

    builder.Services
        .AddGraphQLServer()
        
        // Serialize access to the IQuerySession within Hot Chocolate
        .RegisterService<IQuerySession>(ServiceKind.Synchronized)

or by making the session lifetime in your container Transient by doing this:

    builder.Services
        .AddTransient<IQuerySession>(s => s.GetRequiredService<IDocumentStore>().QuerySession());

The first choice will potentially slow down your GraphQL endpoints by serializing access to the IQuerySession while fetching data. The second choice is a non-idiomatic usage of Marten that potentially fouls up usage of Marten within non-GraphQL operations as you could potentially be using separate Marten sessions when you really meant to be using a shared instance.

If you’re really concerned about needing the queries within a single GraphQL request to be parallelized, there’s another option using object pooling like Jackson Cribb wrote about in the Marten Discord room.

For Marten V7, we’re going to strongly consider some kind of query runner that does not have sticky connections for the express purpose of simplifying Hot Chocolate + Marten integration, but I can’t promise any particular timeline for that work. You can track that work here though.

Multi-Tenancy and Session Lifecycles

Multi-Tenancy throws yet another spanner into the works. Consider the following Hot Chocolate query method:

    public IQueryable<User> GetUsers(
        [Service] IDocumentStore documentStore, [GlobalState] string tenant)
    {
        using var session = documentStore.LightweightSession(tenant);

        return session.Query<User>();
    }

Assuming that you’ve got some kind of Hot Chocolate interceptor to detect the tenant id for you, and that value is communicated through Hot Chocolate’s global state mechanism, you might think to open a Marten session directly like the code above. That code above will absolutely not work under any kind of system load because it’s putting you into a damned if you do, damned if you don’t situation. If you dispose the session before this method completes,
the IQueryable execution will throw an ObjectDisposedException when Hot Chocolate tries to execute the query. If you *don’t* dispose the session, the IoC container for the request scope doesn’t know about it, so can’t dispose it for you and Marten is going to be hanging on to the open database connection until garbage collection comes for it — and under a significant load, that means your system will behave very badly when the database connection pool is exhausted!

What we need to do is to have some way that our sessions can be created for the right tenant for the current request, but have the session tracked some how so that the scoped IoC container can be used to clean up the open sessions at the end of the request. As a first pass, I’m using this crude approach first with this service that’s registered with the IoC container with a Scoped lifetime:

/// <summary>
/// This will be Scoped in the container per request, "knows" what
/// the tenant id for the request is. Also tracks the active Marten
/// session
/// </summary>
public class ActiveTenant : IDisposable
{
    public ActiveTenant(IHttpContextAccessor contextAccessor, IDocumentStore store)
    {
        if (contextAccessor.HttpContext is not null)
        {
            // Try to detect the active tenant id from
            // the current HttpContext
            var context = contextAccessor.HttpContext;
            if (context.Request.Headers.TryGetValue("tenant", out var tenant))
            {
                var tenantId = tenant.FirstOrDefault();
                if (tenantId.IsNotEmpty())
                {
                    this.Session = store.QuerySession(tenant!);
                }
            }
        }

        this.Session ??= store.QuerySession();
    }

    public IQuerySession Session { get; }

    public void Dispose()
    {
        this.Session.Dispose();
    }
}

Now, rewrite the Hot Chocolate query from way up above with:

    public IQueryable<User> GetUsers(
        [Service] ActiveTenant tenant)
    {
        return tenant.Session.Query<User>();
    }

That does still have to be paired with this Hot Chocolate configuration to dodge the concurrent access problems like so:

    builder.Services 
        .AddGraphQLServer()
        .RegisterService<ActiveTenant>(ServiceKind.Synchronized)

Hot Chocolate Mutations

I took some time this morning to research Hot Chocolate’s Mutation model (think “writes”). Since my client is using Marten as an event store and I’m me, I was looking for opportunities to:

What I’ve found so far has been a series of blockers once you zero in on the fact that Hot Chocolate is built around the possibility of having zero to many mutation messages in any one request — and that that request should be treated as a logical transaction such that every mutation should either succeed or fail together. With that being said, I see the blockers as:

  • Wolverine doesn’t yet support message batching in any kind of built in way, and is unlikely to do so before a 2.0 release that isn’t even so much as a glimmer in my eyes yet
  • Hot Chocolate depends on ambient transactions (Boo!) to manage the transaction boundaries. That by itself almost knocks out the out of the box Marten integration and forces you to use more custom session mechanics to enlist in ambient transactions.
  • The existing Wolverine transactional outbox depends on an explicit “Flush” operation after the actual database transaction is committed. That’s handled quite gracefully by Wolverine’s Marten integration in normal issue (in my humble and very biased opinion), but that can’t work across multiple mutations in one GraphQL request
  • There is a mechanism to replace. the transaction boundary management in Hot Chocolate, but it was very clearly built around ambient transactions and it has a synchronous signature to commit the transaction. Like any sane server side development framework, Wolverine performs the IO intensive database transactional mechanics and outbox flushing operations with asynchronous methods. To fit that within Hot Chocolate’s transactional boundary abstraction would require calls to turn the asynchronous Marten and Wolverine APIs into synchronous calls with GetAwaiter().GetResult(), which is tantamount to painting a bullseye on your chest and daring the Fates to not make your application crater with deadlocks under load.

I think at this point, my recommended approach is going to forego integrating Wolverine into Hot Chocolate mutations altogether with some combination of:

  • Don’t use Hot Chocolate mutations whatsoever if there’s no need for the operation batching and use old fashioned ASP.Net Core with or without Wolverine’s HTTP support
  • Or document a pattern for using the Decider pattern within Hot Chocolate as an alternative to Wolverine’s “aggregate handler” usage. The goal here is to document a way for developers to keep infrastructure out of business logic code and maximize testability
  • If using Hot Chocolate mutations, I think there’s a need for a better outbox subscription model directly against Marten’s event store. The approach Oskar outlined here would certainly be a viable start, but I’d rather have an improved version of that built directly into Wolverine’s Marten integration. The goal here is to allow for an Event Driven Architecture which Wolverine supports quite well and the application in question could definitely utilize, but do so without creating any complexity around the Hot Chocolate integration.

In the long, long term:

  • Add a message batch processing option to Wolverine that manages transactional boundaries between messages for you
  • Have a significant palaver between the Marten/Wolverine core teams and the fine folks behind Hot Chocolate to iron a bit of this out

My Recommendations For Now

Honestly, I don’t think that I would recommend using GraphQL in general in your system whatsoever unless you’re building some kind of composite user interface where GraphQL would be beneficial in reducing chattiness between your user interface and backing service by allowing unrelated components in your UI happily batch up requests to your server. Maybe also if you were using GraphQL as a service gateway to combine disparate data sources on the server side in a consistent way, but even then I wouldn’t automatically use GraphQL.

I’m not knowledgeable enough to say how much GraphQL usage would help speed up your user interface development, so take all that I said in the paragraph above with a grain of salt.

At this point I would urge folks to be cautious about using the Critter Stack with Hot Chocolate. Marten can be used if you’re aware of the potential problems I discussed above. Even when we beat the sticky connection thing and the session lifecycle problems, Marten’s basic model of storing JSON in the database is really not optimized for plucking out individual fields in Select() transforms. While Marten does support Select() transforms, it’s may not as efficient as the equivalent functionality on top of a relational database model would be. It’s possible that GraphQL might be a better fit with Marten if you were primarily using projected read models purposely designed for client consumption through GraphQL or even projecting event data to flat tables that are queried by Hot Chocolate.

Wolverine with Hot Chocolate maybe not so much if you’d have any problems with the transactional boundary issues.

I would be urge you to do load testing with any usage of Hot Chocolate as I think from peeking into its internals that it’s not the most efficient server side tooling around. Again, that doesn’t mean that you will automatically have performance problems with Hot Chocolate, but I think you you should be cautious with its usage.

In general, I’d say that GraphQL creates way too much abstraction over your underlying data storage — and my experience consistently says that abstracted data access can lead to some very poor system performance by both making your application harder to understand and by eliminating the usage of advanced features of your data storage tooling behind least common denominator abstractions.

This took much longer than I wanted it too, as always. I might write a smaller follow up on how I’d theoretically go about building an optimized GraphQL layer from scratch for Marten — which I have zero intension of ever doing, but it’s a fun thought experiment.

Unraveling the Magic in Wolverine

Just to pick a fight here, I think that folks who eschew all conventional approaches and insist on code being as explicit as possible end up writing very high ceremony code that’s completely unmaintainable in the end. I don’t see folks who insist on this style of “I never use frameworks” coding actually being all that effective in practice any time I’ve seen this other extreme. With all that being said, if you are using Wolverine you can choose to write very explicit code at any time and avoid using any of the built in conventions any time that’s necessary.

When you’re building or choosing an application framework, there’s a bit of tension between “magic” (convention over configuration) and explicitness in the code targeting that application framework. Speaking for myself, I lean very heavily toward low code ceremony tools that result in relatively uncluttered code with minimal boilerplate code for infrastructure. That bias admittedly leans toward conventional approaches, and Wolverine (and Marten to a much lesser degree) is chalk full of naming conventions.

Great! Except when it’s not. To make any kind of “magical” framework really work well for users, I think you need to:

  • First off, make the conventions be easy to understand and predictable (let’s call that a work in progress)
  • Document the conventions as well as you can
  • Hope that you don’t run into too many creative users that stretch the conventions farther than they were meant to — and relentlessly adapt as you inevitable run into those users
  • Provide the ability to bypass the conventions at will and write explicit code anytime the conventions don’t fit a use case — and that one’s a hard lesson learned from my experiences with FubuMVC/FubuTransportation back in the day:-(
  • Provide some some easily accessible mechanisms to unravel the magic and understand how the framework itself is calling into your code, routing requests, and even what middleware is being applied

For now, let’s focus on the last bullet point (while you mentally beat me up for the first). There is a Telehealth service application in the Wolverine codebase that has code for tracking and governing the workflow of “telehealth” appointments between patiens and various health professionals. Part of the system is a set of events and the following Marten aggregate for ProviderShift that models the activity and state of a health care provider (doctors, nurses, nurse practitioners, etc.) in a given day:

public class ProviderShift
{
    public Guid Id { get; set; }
    public int Version { get; set; }
    public Guid BoardId { get; private set; }
    public Guid ProviderId { get; init; }
    public ProviderStatus Status { get; private set; }
    public string Name { get; init; }
    public Guid? AppointmentId { get; set; }

    public static async Task<ProviderShift> Create(
        ProviderJoined joined,
        IQuerySession session)
    {
        var provider = await session
            .LoadAsync<Provider>(joined.ProviderId);

        return new ProviderShift
        {
            Name = $"{provider.FirstName} {provider.LastName}",
            Status = ProviderStatus.Ready,
            ProviderId = joined.ProviderId,
            BoardId = joined.BoardId
        };
    }

    public void Apply(ProviderReady ready)
    {
        AppointmentId = null;
        Status = ProviderStatus.Ready;
    }

    public void Apply(ProviderAssigned assigned)
    {
        Status = ProviderStatus.Assigned;
        AppointmentId = assigned.AppointmentId;
    }

    public void Apply(ProviderPaused paused)
    {
        Status = ProviderStatus.Paused;
        AppointmentId = null;
    }

    // This is kind of a catch all for any paperwork the
    // provider has to do after an appointment has ended
    // for the just concluded appointment
    public void Apply(ChartingStarted charting)
    {
        Status = ProviderStatus.Charting;
    }
}

The ProviderShift model above takes advantage of Marten’s “self-aggregate” functionality to teach Marten how to apply a stream of event data to update the current state of the model (the Apply() conventions are explained here).

So there’s a little bit of magic above, but let’s add some more before we get to the diagnostics. Now consider this HTTP endpoint from a Wolverine sample that’s using Marten‘s event store functionality within the sample “Telehealth” system for health providers to mark when they are done with their charting process (writing up their notes and follow up actions) after finishing an appointment:

    [WolverinePost("/shift/charting/complete")]
    [AggregateHandler]
    public (ChartingResponse, ChartingFinished) CompleteCharting(
        CompleteCharting charting,
        ProviderShift shift)
    {
        if (shift.Status != ProviderStatus.Charting)
        {
            throw new Exception("The shift is not currently charting");
        }
        
        return (
C// The HTTP response body
            new ChartingResponse(ProviderStatus.Paused),
            
            // An event to be appended to the ProviderShift aggregate event stream
            new ChartingFinished()
        );
    }

That HTTP endpoint uses Wolverine’s Aggregate Handler conventions as usage of the Decider Pattern to determine the event(s) that should be created for the given CompleteCharting and the current ProviderShift state of the provider shift referred to in the incoming command:

public record CompleteCharting(
    Guid ProviderShiftId,
    int Version
);

What’s it doing at runtime you ask? All told it’s:

  1. Deserializing the HTTP request body into the CompleteCharting command
  2. If we were applying any validation middleware, that might be happening next
  3. Loading the current state of the ProviderShift identified by the CompleteCharting command using Marten
  4. As it does #3, it’s opting into Marten’s optimistic concurrency checks for the provider shift event stream using the CompleteCharting.Version value
  5. Calling our actual endpoint method from up above
  6. Assuming there’s no validation exception, the `ChartingFinished` returned from our endpoint method is appended to the Marten event stream for the provider
  7. All pending Marten changes are persisted to the database
  8. The `ChartingResponse` object also returned from our endpoint method is serialized to the HTTP response stream

There’s a fair amount of infrastructure going on behind the scenes up above, but the goal of the Wolverine “Aggregate Handler” version of the “Decider pattern” is to allow our users to focus on writing the business logic for their application while letting Wolverine & Marten worry about all the important, but repetitive infrastructure code. Arguably, the result is that our users are mostly writing pure functions that are pretty easy to unit test.

Awesome! But there’s admittedly some room for confusion, especially for newer users. So let’s finally move on to Wolverine’s facilities to dispel the magic.

The Command Line is Sexy

No, seriously. Wolverine comes with a lot of diagnostic helpers that can be exposed from the command line of your application assuming that you’ve used Oakton for your command line runner as shown in the bottom of the Program file from the Telehealth sample:

// This is using the Oakton library for command running
await app.RunOaktonCommands(args);

First off, you can go check out everything that Wolverine is discovering or configuring from within your application with this command from the root folder of your main application project:

dotnet run -- describe

By itself, that’s going to tell you a lot about the static configuration of the application including all Wolverine HTTP endpoints with a textual display like this:

That tooling may help you right off the bat for troubleshooting handler discovery or message routing behavior in Wolverine applications, but let’s move on to understanding the actual logic of our CompleteCharting endpoint introduced earlier.

Wolverine has a significantly different runtime model than all the other HTTP endpoint models or message handling tools in .NET in that it uses runtime code generation to wrap its adapters around your code rather than forcing you to constrain your code for Wolverine. One of the upsides of all the gobbledy-gook I just spouted is that I can preview or write out Wolverine’s generated code by using the command line tooling like so:

dotnet run -- codegen write

Alright, start by preparing yourself to see some auto-generated code which inevitably means an eye sore. That command up above will write out the C# code around all the HTTP endpoints and Wolverine message handlers to the Internal/Generated/WolverineHandlers folder within your entry project. For HTTP endpoints, the generated file is named after the HTTP route, so in this case, we’re looking for the `POST_shift_charting_complete.cs` file, and here it is:

// <auto-generated/>
#pragma warning disable
using Microsoft.AspNetCore.Routing;
using System;
using System.Linq;
using Wolverine.Http;
using Wolverine.Marten.Publishing;
using Wolverine.Runtime;

namespace Internal.Generated.WolverineHandlers
{
    public class POST_shift_charting_complete : Wolverine.Http.HttpHandler
    {
        private readonly Wolverine.Http.WolverineHttpOptions _wolverineHttpOptions;
        private readonly Wolverine.Runtime.IWolverineRuntime _wolverineRuntime;
        private readonly Wolverine.Marten.Publishing.OutboxedSessionFactory _outboxedSessionFactory;

        public POST_shift_charting_complete(Wolverine.Http.WolverineHttpOptions wolverineHttpOptions, Wolverine.Runtime.IWolverineRuntime wolverineRuntime, Wolverine.Marten.Publishing.OutboxedSessionFactory outboxedSessionFactory) : base(wolverineHttpOptions)
        {
            _wolverineHttpOptions = wolverineHttpOptions;
            _wolverineRuntime = wolverineRuntime;
            _outboxedSessionFactory = outboxedSessionFactory;
        }



        public override async System.Threading.Tasks.Task Handle(Microsoft.AspNetCore.Http.HttpContext httpContext)
        {
            var messageContext = new Wolverine.Runtime.MessageContext(_wolverineRuntime);
            var providerShiftEndpoint = new TeleHealth.WebApi.ProviderShiftEndpoint();
            // Reading the request body via JSON deserialization
            var (charting, jsonContinue) = await ReadJsonAsync<TeleHealth.WebApi.CompleteCharting>(httpContext);
            if (jsonContinue == Wolverine.HandlerContinuation.Stop) return;
            await using var documentSession = _outboxedSessionFactory.OpenSession(messageContext);
            var eventStore = documentSession.Events;
            
            // Loading Marten aggregate
            var eventStream = await eventStore.FetchForWriting<TeleHealth.Common.ProviderShift>(charting.ProviderShiftId, charting.Version, httpContext.RequestAborted).ConfigureAwait(false);

            
            // The actual HTTP request handler execution
            (var chartingResponse_response, var chartingFinished) = providerShiftEndpoint.CompleteCharting(charting, eventStream.Aggregate);

            eventStream.AppendOne(chartingFinished);
            await documentSession.SaveChangesAsync(httpContext.RequestAborted).ConfigureAwait(false);
            // Writing the response body to JSON because this was the first 'return variable' in the method signature
            await WriteJsonAsync(httpContext, chartingResponse_response);
        }

    }
}

It’s fugly code, but we’re trying to invest much more into adding explanatory comments into this generated code to try to explain *why* the code is generated around the signature of your inner message or HTTP handler. To go a little farther, the same codegen write command also wrote out the Marten code for the ProviderShift aggregate from up above as well (but that code is even uglier so I’m not showing it here).

Summary

Honestly, I’m out of time and need to leave to meet a friend for lunch, so let me leave you with:

Utilize dotnet run -- codegen write to understand how Wolverine is calling your code and how any middleware is being applied!

See the Wolverine docs on Code Generation for more help too. And don’t be afraid of magic as long as you’ve got the tools to understand it!

Wolverine Interoperability with Others

Wolverine is the relative newcomer on the scene for asynchronous messaging in the .NET ecosystem. While many Wolverine users are starting in greenfield circumstances, it’s far more likely that the exact shops who would be interested in Wolverine’s messaging support already have a significant amount of existing systems communicating with other messaging infrastructure solutions. And while I absolutely believe in Wolverine, there is likely no world in which it makes sense to completely replace every bit of existing messaging infrastructure all at once. Moreover, it’s also common to have systems built on completely other platforms or 3rd party systems that communicate with message queueing.

All that said, Wolverine obviously needs to have a strong interoperability story to enable its adoption. Other than the interoperability with NServiceBus through Rabbit MQ we needed at my previous company, I quite admittedly skimped a little on that in the original push to 1.0 as I inevitably start triaging user stories to make my self imposed deadline for 1.0 this summer.

In the past couple weeks there were several folks trying to utilize Wolverine to receive messages from external systems using various transports, so it turned into the perfect time to focus on improving Wolverine’s interoperability features in the recent Wolverine 1.7 release.

First, some links for more information:

Feel very free to skip down to the samples below that.

A Little Background

For just a little background, each messaging transport has a little bit different API for shuffling data between systems, but that mostly boils down to message body data and message metadata (headers). Wolverine (and other messaging alternatives) maps the specific messaging API of Rabbit MQ, Azure Service Bus, or AWS SQS into Wolverine’s internal Envelope representation. The message body itself would be deserialized into the actual .NET message type, and the rest of that metadata helps Wolverine perform distributed tracing through correlation identifiers, “know” how to send replies back to the original sender, and to even just know what the incoming message type is. That all works seamlessly when Wolverine is on both sides of the messaging pipe, but when interoperating with a non-Wolverine system you have to override Wolverine’s mapping between its Envelope model and the incoming and outgoing API for the underlying transport.

Fortunately, this mapping is either completely pluggable on an endpoint by endpoint basis, or you can now start with the built in mapping from Wolverine and selectively override a subset of the metadata mappings.

Receive “Just” JSON via Rabbit MQ

A prospective Wolverine user reached out to us on Discord to tell us about trying to receive pure JSON messages from a service written in Python (hence the image up above). After some internal changes in Wolverine 1.7, you can now receive “just” JSON to a Rabbit MQ queue assuming that that queue will only ever receive one message type by telling Wolverine what the default message type name is like this:

using var host = await Host.CreateDefaultBuilder()
    .UseWolverine((context, opts) =>
    {
        var rabbitMqConnectionString = context.Configuration.GetConnectionString("rabbit");

        opts.UseRabbitMq(rabbitMqConnectionString);

        opts.ListenToRabbitQueue("emails")
            // Tell Wolverine to assume that all messages
            // received at this queue are the SendEmail
            // message type
            .DefaultIncomingMessage<SendEmail>();
    }).StartAsync();

Also see the documentation on Rabbit MQ interoperability.

Interop with AWS SQS

Wolverine has to work with AWS SQS in a much different way than the other transports. Via a pull request in Wolverine 1.7, you can now receive “just” JSON from external systems via AWS SQS like this:

using var host = await Host.CreateDefaultBuilder()
    .UseWolverine(opts =>
    {
        opts.UseAmazonSqsTransport();

        opts.ListenToSqsQueue("incoming").ReceiveRawJsonMessage(
            // Specify the single message type for this queue
            typeof(Message1), 
            
            // Optionally customize System.Text.Json configuration
            o =>
            {
                o.PropertyNamingPolicy = JsonNamingPolicy.CamelCase;
            });
    }).StartAsync();

To send “just” JSON to external systems, use this:

using var host = await Host.CreateDefaultBuilder()
    .UseWolverine(opts =>
    {
        opts.UseAmazonSqsTransport();

        opts.PublishAllMessages().ToSqsQueue("outgoing").SendRawJsonMessage(
            // Specify the single message type for this queue
            typeof(Message1), 
            
            // Optionally customize System.Text.Json configuration
            o =>
            {
                o.PropertyNamingPolicy = JsonNamingPolicy.CamelCase;
            });
    }).StartAsync();

For custom interoperability strategies using AWS SQS, you can create your own implementation of the `ISqsEnvelopeMapper` interface like this one:

public class CustomSqsMapper : ISqsEnvelopeMapper
{
    public string BuildMessageBody(Envelope envelope)
    {
        // Serialized data from the Wolverine message
        return Encoding.Default.GetString(envelope.Data);
    }

    // Specify header values for the SQS message from the Wolverine envelope
    public IEnumerable<KeyValuePair<string, MessageAttributeValue>> ToAttributes(Envelope envelope)
    {
        if (envelope.TenantId.IsNotEmpty())
        {
            yield return new KeyValuePair<string, MessageAttributeValue>("tenant-id", new MessageAttributeValue{StringValue = envelope.TenantId});
        }
    }

    public void ReadEnvelopeData(Envelope envelope, string messageBody, IDictionary<string, MessageAttributeValue> attributes)
    {
        envelope.Data = Encoding.Default.GetBytes(messageBody);

        if (attributes.TryGetValue("tenant-id", out var att))
        {
            envelope.TenantId = att.StringValue;
        }
    }
}

And apply that to Wolverine endpoints like this:

using var host = await Host.CreateDefaultBuilder()
    .UseWolverine(opts =>
    {
        opts.UseAmazonSqsTransport()
            .UseConventionalRouting()

            .ConfigureListeners(l => l.InteropWith(new CustomSqsMapper()))

            .ConfigureSenders(s => s.InteropWith(new CustomSqsMapper()));

    }).StartAsync();

Interop with Azure Service Bus

You can create interoperability with non-Wolverine applications by writing a custom IAzureServiceBusEnvelopeMapper as shown in the following sample:

public class CustomAzureServiceBusMapper : IAzureServiceBusEnvelopeMapper
{
    public void MapEnvelopeToOutgoing(Envelope envelope, ServiceBusMessage outgoing)
    {
        outgoing.Body = new BinaryData(envelope.Data);
        if (envelope.DeliverWithin != null)
        {
            outgoing.TimeToLive = envelope.DeliverWithin.Value;
        }
    }

    public void MapIncomingToEnvelope(Envelope envelope, ServiceBusReceivedMessage incoming)
    {
        envelope.Data = incoming.Body.ToArray();
        
        // You will have to help Wolverine out by either telling Wolverine
        // what the message type is, or by reading the actual message object,
        // or by telling Wolverine separately what the default message type
        // is for a listening endpoint
        envelope.MessageType = typeof(Message1).ToMessageTypeName();
    }

    public IEnumerable<string> AllHeaders()
    {
        yield break;
    }
}

and apply that to various endpoints like this:

using var host = await Host.CreateDefaultBuilder()
    .UseWolverine(opts =>
    {
        opts.UseAzureServiceBus("some connection string")
            .UseConventionalRouting()

            .ConfigureListeners(l => l.InteropWith(new CustomAzureServiceBusMapper()))

            .ConfigureSenders(s => s.InteropWith(new CustomAzureServiceBusMapper()));
    }).StartAsync();

Interop with NServiceBus via Rabbit MQ

The original production usage of Wolverine was replacing NServiceBus for one service within a large constellation of services that all communicated asynchronously with Rabbit MQ. Unsurprisingly, Wolverine launched with a strong, fully functional interoperability between NServiceBus systems and Wolverine systems through Rabbit MQ with this usage taken from a test project within the Wolverine codebase:

Wolverine = await Host.CreateDefaultBuilder().UseWolverine(opts =>
{
    opts.UseRabbitMq()
        .AutoProvision().AutoPurgeOnStartup()
        .BindExchange("wolverine").ToQueue("wolverine")
        .BindExchange("nsb").ToQueue("nsb")
        .BindExchange("NServiceBusRabbitMqService:ResponseMessage").ToQueue("wolverine");

    opts.PublishAllMessages().ToRabbitExchange("nsb")

        // Tell Wolverine to make this endpoint send messages out in a format
        // for NServiceBus
        .UseNServiceBusInterop();

    opts.ListenToRabbitQueue("wolverine")
        .UseNServiceBusInterop()
        

        .UseForReplies();
    
    // This facilitates messaging from NServiceBus (or MassTransit) sending as interface
    // types, whereas Wolverine only wants to deal with concrete types
    opts.Policies.RegisterInteropMessageAssembly(typeof(IInterfaceMessage).Assembly);
}).StartAsync();

Interop with MassTransit via Rabbit MQ

A little less battle tested — and much weirder under the covers — is a similar interoperability recipe for talking to MassTransit applications via Rabbit MQ:

Wolverine = await Host.CreateDefaultBuilder().UseWolverine(opts =>
{
    opts.ApplicationAssembly = GetType().Assembly;

    opts.UseRabbitMq()
        .CustomizeDeadLetterQueueing(new DeadLetterQueue("errors", DeadLetterQueueMode.InteropFriendly))
        .AutoProvision().AutoPurgeOnStartup()
        .BindExchange("wolverine").ToQueue("wolverine")
        .BindExchange("masstransit").ToQueue("masstransit");

    opts.PublishAllMessages().ToRabbitExchange("masstransit")

        // Tell Wolverine to make this endpoint send messages out in a format
        // for MassTransit
        .UseMassTransitInterop();

    opts.ListenToRabbitQueue("wolverine")

        // Tell Wolverine to make this endpoint interoperable with MassTransit
        .UseMassTransitInterop(mt =>
        {
            // optionally customize the inner JSON serialization
        })
        .DefaultIncomingMessage<ResponseMessage>().UseForReplies();
}).StartAsync();

Wolverine 1.7 is a community affair!

Look for details on official support plans for Marten and/or Wolverine and the rest of the “Critter Stack” from JasperFx Software early next week. If you’re looking at Wolverine and wondering if it’s going to be a viable choice in the long run, just know we’re trying very hard to make it so.

Wolverine had a pretty significant 1.7.0 release on Friday. What’s most encouraging to me was how many community contributions were in this one including pull requests, issues where community members took a lot of time to create actionable reproduction steps, and suggestions from our Wolverine Discord room. This always misses folks, but thank you goes to:

This issue addressed several open bugs, but beyond that the high points were:

  • Multi-tenancy support from the HTTP layer down as I blogged about yesterday
  • A much better interoperability story for Wolverine and non-Wolverine applications using Rabbit MQ, AWS SQS, or Azure Service Bus. More on this later this week
  • A lot more diagnostics and explanatory comments in the generated code to unravel the “magic” within Wolverine and the message handler / http endpoint method discovery logic. Much more on this in a later blog post this week
  • Much more control over the Open Telemetry and message logging that is published by Wolverine to tone down the unnecessary noise that might be happening to some users today. Definitely more on that later this week

I’m working with a couple clients who are using Wolverine, and I can’t say that there are zero problems, but overall I’m very happy with how Wolverine is being received and how it’s working out in real applications so far.

Wolverine Expands its Multi-Tenancy Story to HTTP

As folks read more about Wolverine and its multi-tenancy support, you may quickly notice that we’ve clearly focused much more on the Marten integration than we have with EF Core so far. You’ll also notice that Wolverine today does not yet have direct support for Finbuckle. At some point in the future I think that Wolverine will provide some Finbuckle integration into Wolverine’s runtime model and will probably use Finbuckle’s EF Core support to provide end to end multi-tenancy with EF Core. For right now though, I think that Marten actually has a much stronger story out of the box for multi-tenancy than EF Core does anyway.

Let’s say that you’re tasked with building an online SaaS solution where the same application service is going to be used by completely different sets of users from different client organizations. It’s going to be important to segregate data between these different client organizations so the various users for each client are only viewing and modifying data for their organization. Maybe you have to go all the way to creating a separate database for each client organization or “tenant,” or maybe you’re using programmatic ways to keep data segregated through your persistence tooling’s capabilities like Marten’s “conjoined tenancy” model. Having one shared system being able to serve different populations of users while keeping all their data correctly segregated is referred to generally as “multi-tenancy” in the software development world.

As an example, let’s consider the sample MultiTenantedTodoService project from the Wolverine codebase. That project is utilizing Marten with a database per tenant strategy using this project configuration for Marten:

var connectionString = "Host=localhost;Port=5433;Database=postgres;Username=postgres;password=postgres";

// Adding Marten for persistence
builder.Services.AddMarten(m =>
    {
        // With multi-tenancy through a database per tenant
        m.MultiTenantedDatabases(tenancy =>
        {
            // You would probably be pulling the connection strings out of configuration,
            // but it's late in the afternoon and I'm being lazy building out this sample!
            tenancy.AddSingleTenantDatabase("Host=localhost;Port=5433;Database=tenant1;Username=postgres;password=postgres", "tenant1");
            tenancy.AddSingleTenantDatabase("Host=localhost;Port=5433;Database=tenant2;Username=postgres;password=postgres", "tenant2");
            tenancy.AddSingleTenantDatabase("Host=localhost;Port=5433;Database=tenant3;Username=postgres;password=postgres", "tenant3");
        });
        
        m.DatabaseSchemaName = "mttodo";
    })
    .IntegrateWithWolverine(masterDatabaseConnectionString:connectionString);

Wolverine has had a strong story for multi-tenancy solutions within its message handling since its 1.0 release this summer (more on that later). I’m not showing this capability in this post, but Wolverine (with Marten) is able to create and use separate transactional outbox functionality for each and every separate tenant database. To the best of my knowledge, Wolverine is the only tool in the .NET world with that capability.

What had been missing had been direct support within Wolverine.HTTP handlers meaning that designing a simple HTTP endpoint that created a new Todo meant building an HTTP endpoint that picked out the tenant id from the route like so:

    [WolverinePost("/todoitems/{tenant}")]
    public static async Task<IResult> Create(string tenant, CreateTodo command, IMessageBus bus)
    {
        // At the 1.0 release, you would have to use Wolverine as a mediator
        // to get the full multi-tenancy feature set.
        
        // That hopefully changes in 1.1
        var created = await bus.InvokeForTenantAsync<TodoCreated>(tenant, command);

        return Results.Created($"/todoitems/{tenant}/{created.Id}", created);
    }

and immediately delegated to an inner Wolverine message handler through the InvokeForTenantAsync() usage shown up above. That handler code is simpler because Wolverine is handling all the multi-tenancy mechanics and looked like this:

    public static TodoCreated Handle(CreateTodo command, IDocumentSession session)
    {
        var todo = new Todo { Name = command.Name };
        session.Store(todo);

        return new TodoCreated(todo.Id);
    }

Alright, not too awful, and very reminiscent of systems that use the MediatR library within MVC Core controllers. However, it’s more moving parts and more code ceremony than I was hoping for out of Wolverine.

Likewise, to create a GET endpoint that returns all the completed Todo documents for the current tenant, maybe you did something like this that explicitly opened a Marten session for the tenant id detected from the request:

    [WolverineGet("/todoitems/{tenant}/complete")]
    public static Task<IReadOnlyList<Todo>> GetComplete(string tenant, IDocumentStore store)
    {
        using var session = store.QuerySession(tenant);
        return session.Query<Todo>().Where(x => x.IsComplete).ToListAsync();
    }

Again, not too much code, but there’s some repetitive code around opening the right session for the right tenant that would be easy for a developer to forget and maybe possible to sneak by testing some times. Also, I had to explicitly dispose the Marten query session — and failing to do so can easily lead to orphaned database connections and all kinds of hurting within your system at runtime. Don’t laugh or blow that off, because that’s happened to the unwary.

Enter Wolverine 1.7.0 on this past Friday. Now Wolverine.HTTP got stretched to include tenant id detection and “know” how to pass that tenant id along to the Marten sessions created within HTTP endpoints.

With Wolverine.HTTP 1.7, I opened the MultiTenantedTodoService Program file again and added this configuration:

// Let's add in Wolverine HTTP endpoints to the routing tree
app.MapWolverineEndpoints(opts =>
{
    // Letting Wolverine HTTP automatically detect the tenant id!
    opts.TenantId.IsRouteArgumentNamed("tenant");
    
    // Assert that the tenant id was successfully detected,
    // or pull the rip cord on the request and return a 
    // 400 w/ ProblemDetails
    opts.TenantId.AssertExists();
});

Using Wolverine.HTTP’s new tenant id detection capability, I’ve told Wolverine to pluck the tenant id out of an expected route argument named “tenant.” With its Marten integration, Wolverine is able to pass sessions into our HTTP endpoint methods that point to the correct tenant database.

Don’t worry, there are plenty of other options for tenant id detection than this simple mechanism I used for testing and the simple demonstration here.

Revisiting the endpoint for fetching all the completed Todo documents for a client, that code reduces down to this:

    [WolverineGet("/todoitems/{tenant}/complete")]
    public static Task<IReadOnlyList<Todo>> GetComplete(IQuerySession session) 
        => session
            .Query<Todo>()
            .Where(x => x.IsComplete)
            .ToListAsync();

Better yet, let’s revisit the endpoint for creating a new Todo and we’re now able to collapse this down to just a single endpoint method:

    [WolverinePost("/todoitems/{tenant}")]
    public static CreationResponse<TodoCreated> Create(
        // Only need this to express the location of the newly created
        // Todo object
        string tenant, 
        CreateTodo command, 
        IDocumentSession session)
    {
        var todo = new Todo { Name = command.Name };
        
        // Marten itself sets the Todo.Id identity
        // in this call
        session.Store(todo); 

        // New syntax in Wolverine.HTTP 1.7
        // Helps Wolverine 
        return CreationResponse.For(new TodoCreated(todo.Id), $"/todoitems/{tenant}/{todo.Id}");
    }

Notice that we really didn’t do anything with the tenant argument except as a helper to build the Url for a newly created Todo. Wolverine & Marten together took care of everything else for us creating the correct session for the correct tenant database.

Summary

The new Wolverine.HTTP multi-tenancy capability is going to allow users to write simpler code and less error prone code for teams needing multi-tenanted persistence.

Using Alba to Test ASP.Net Core Web Services

Hey, JasperFx Software is more than just some silly named open source frameworks. We’re also deeply experienced in test driven development, designing for testability, and making test automation work without driving into the ditch with over dependence on slow, brittle Selenium testing. Hit us up about what we could do to help you be more successful in your own test automation or TDD efforts.

I have been working furiously on getting an incremental Wolverine release out this week, with one of the new shiny features being end to end support for multi-tenancy (the work in progress GitHub issue is here) through Wolverine.Http endpoints. I hit a point today where I have to admit that I can’t finish that work today, but did see the potential for a blog post on the Alba library (also part of JasperFx’s OSS offerings) and how I was using Alba today to write integration tests for this new functionality, show how the sausage is being made, and even work in a test-first manner.

To put the desired functionality in context, let’s say that we’re building a “Todo” web service using Marten for persistence. Moreover, we’re expecting this system to have a massive number of users and want to be sure to isolate data between customers, so we plan on using Marten’s support for using a separate database for each tenant (think user organization in this case). Within that “Todo” system, let’s say that we’ve got a very simple web service endpoint to just serve up all the completed Todo documents for the current tenant like this one:

[WolverineGet("/todoitems/{tenant}/complete")]
public static Task<IReadOnlyList<Todo>> GetComplete(IQuerySession session) 
    => session
        .Query<Todo>()
        .Where(x => x.IsComplete)
        .ToListAsync();

Now, you’ll notice that there is a route argument named “tenant” that isn’t consumed at all by this web api endpoint. What I want Wolverine to do in this case is to infer that the value of that “tenant” value within the route is the current tenant id for the request, and quietly select the correct Marten tenant database for me without me having to write a lot of repetitive code.

Just a note, all of this is work in progress and I haven’t even pushed the code at the time of writing this post. Soon. Maybe tomorrow.

Stepping into the bootstrapping for this web service, I’m going to add these new lines of code to the Todo web service’s Program file to teach Wolverine.HTTP how to handle multi-tenancy detection for me:

// Let's add in Wolverine HTTP endpoints to the routing tree
app.MapWolverineEndpoints(opts =>
{
    // Letting Wolverine HTTP automatically detect the tenant id!
    opts.TenantId.IsRouteArgumentNamed("tenant");
    
    // Assert that the tenant id was successfully detected,
    // or pull the rip cord on the request and return a 
    // 400 w/ ProblemDetails
    opts.TenantId.AssertExists();
});

So that’s some of the desired, built in multi-tenancy features going into Wolverine.HTTP 1.7 some time soon. Back to the actual construction of these new features and how I used Alba this morning to drive the coding.

I started by asking around on social media about what other folks used as strategies to detect the tenant id in ASP.Net Core multi-tenancy, and came up with this list (plus a few other options):

  • Use a custom request header
  • Use a named route argument
  • Use a named query string value (I hate using the query string myself, but like cockroaches or scorpions in our Central Texas house, they always sneak in somehow)
  • Use an expected Claim on the ClaimsPrincipal
  • Mix and match the strategies above because you’re inevitably retrofitting this to an existing system
  • Use sub domain names (I’m arbitrarily skipping this one for now just because it was going to be harder to test and I’m pressed for time this week)

Once I saw a little bit of consensus on the most common strategies (and thank you to everyone who responded to me today), I jotted down some tasks in GitHub-flavored markdown (I *love* this feature) on what the configuration API would look like and my guesses for development tasks:

- [x] `WolverineHttpOptions.TenantId.IsRouteArgumentNamed("foo")` -- creates a policy
- [ ] `[TenantId("route arg")]`, or make `[TenantId]` on a route parameter for one offs. Will need to throw if not a route argument
- [x] `WolverineHttpOptions.TenantId.IsQueryStringValue("key") -- creates policy
- [x] `WolverineHttpOptions.TenantId.IsRequestHeaderValue("key") -- creates policy
- [x] `WolverineHttpOptions.TenantId.IsClaimNamed("key") -- creates policy
- [ ] New way to add custom middleware that's first inline
- [ ] Documentation on custom strategies
- [ ] Way to register the "preprocess context" middleware methods
- [x] Middleware or policy that blows it up with no tenant id detected. Use ProblemDetails
- [ ] Need an attribute to opt into tenant id is required, or tenant id is NOT required on certain endpoints

Knowing that I was going to need to quickly stand up different configurations of a test web service’s IHost, I started with this skeleton that I hoped would make the test setup relatively easy:

public class multi_tenancy_detection_and_integration : IAsyncDisposable, IDisposable
{
    private IAlbaHost theHost;

    public void Dispose()
    {
        theHost.Dispose();
    }

    // The configuration of the Wolverine.HTTP endpoints is the only variable
    // part of the test, so isolate all this test setup noise here so
    // each test can more clearly communicate the relationship between
    // Wolverine configuration and the desired behavior
    protected async Task configure(Action<WolverineHttpOptions> configure)
    {
        var builder = WebApplication.CreateBuilder(Array.Empty<string>());
        builder.Services.AddScoped<IUserService, UserService>();

        // Haven't gotten around to it yet, but there'll be some end to
        // end tests in a bit from the ASP.Net request all the way down
        // to the underlying tenant databases
        builder.Services.AddMarten(Servers.PostgresConnectionString)
            .IntegrateWithWolverine();
        
        // Defaults are good enough here
        builder.Host.UseWolverine();
        
        // Setting up Alba stubbed authentication so that we can fake
        // out ClaimsPrincipal data on requests later
        var securityStub = new AuthenticationStub()
            .With("foo", "bar")
            .With(JwtRegisteredClaimNames.Email, "guy@company.com")
            .WithName("jeremy");
        
        // Spinning up a test application using Alba 
        theHost = await AlbaHost.For(builder, app =>
        {
            app.MapWolverineEndpoints(configure);
        }, securityStub);
    }

    public async ValueTask DisposeAsync()
    {
        // Hey, this is important!
        // Make sure you clean up after your tests
        // to make the subsequent tests run cleanly
        await theHost.StopAsync();
    }

Now, the intermediate step of tenant detection even before Marten itself gets involved is to analyze the HttpContext for the current request, try to derive the tenant id, then set the MessageContext.TenantId in Wolverine for this current request — which Wolverine’s Marten integration will use a little later to create a Marten session pointing at the correct database for that tenant.

Just to measure the tenant id detection — because that’s what I want to build and test first before even trying to put everything together with a real database too — I built these two simple GET endpoints with Wolverine.HTTP:

public static class TenantedEndpoints
{
    [WolverineGet("/tenant/route/{tenant}")]
    public static string GetTenantIdFromRoute(IMessageBus bus)
    {
        return bus.TenantId;
    }

    [WolverineGet("/tenant")]
    public static string GetTenantIdFromWhatever(IMessageBus bus)
    {
        return bus.TenantId;
    }
}

That folks is the scintillating code that brings droves of readership to my blog!

Alright, so now I’ve got some support code for the “Arrange” and “Assert” part of my Arrange/Act/Assert workflow. To finally jump into a real test, I started with detecting the tenant id with a named route pattern using Alba with this code:

    [Fact]
    public async Task get_the_tenant_id_from_route_value()
    {
        // Set up a new application with the desired configuration
        await configure(opts => opts.TenantId.IsRouteArgumentNamed("tenant"));
        
        // Run a web request end to end in memory
        var result = await theHost.Scenario(x => x.Get.Url("/tenant/route/chartreuse"));
        
        // Make sure it worked!
        // ZZ Top FTW! https://www.youtube.com/watch?v=uTjgZEapJb8
        result.ReadAsText().ShouldBe("chartreuse");
    }

The code itself is a little wonky, but I had that quickly working end to end. I next proceeded to the query string strategy like this:

    [Fact]
    public async Task get_the_tenant_id_from_the_query_string()
    {
        await configure(opts => opts.TenantId.IsQueryStringValue("t"));
        
        var result = await theHost.Scenario(x => x.Get.Url("/tenant?t=bar"));
        
        result.ReadAsText().ShouldBe("bar");
    }

Hopefully you can see from the two tests above how that configure() method already helped me quickly write the next test. Sometimes — but not always so be careful with this — the best thing you can do is to first invest in a test harness that makes subsequent tests be more declarative, quicker to write mechanically, and easier to read later.

Next, let’s go to the request header strategy test:

    [Fact]
    public async Task get_the_tenant_id_from_request_header()
    {
        await configure(opts => opts.TenantId.IsRequestHeaderValue("tenant"));
        
        var result = await theHost.Scenario(x =>
        {
            x.Get.Url("/tenant");
            
            // Alba is helping set up the request header
            // for me here
            x.WithRequestHeader("tenant", "green");
        });
        
        result.ReadAsText().ShouldBe("green");
    }

Easy enough, and hopefully you see how Alba helped me get the preconditions into the request quickly in that test. Now, let’s go for a little more complicated test where I first ran into a little trouble and work with the Claim strategy:

    [Fact]
    public async Task get_the_tenant_id_from_a_claim()
    {
        await configure(opts => opts.TenantId.IsClaimTypeNamed("tenant"));
        
        var result = await theHost.Scenario(x =>
        {
            x.Get.Url("/tenant");
            
            // Add a Claim to *only* this request
            x.WithClaim(new Claim("tenant", "blue"));
        });
        
        result.ReadAsText().ShouldBe("blue");
    }

I hit a little friction at first because I didn’t have Alba set up exactly right at first, but since Alba runs your application code completely within process, it was very quick to step right into the code and figure out why the code wasn’t working at first (I’d forgotten to set up the SecurityStub shown above). Refreshing my memory on how Alba’s Security Extensions worked, I was able to get going again. Arguably, Alba’s ability to fake out or even work with your application’s security in tests is its best features.

So that’s been a lot of “happy path” tests, so now let’s break things by specifying Wolverine’s new behavior to validate that a request has a valid tenant id with these two new tests. First, a happy path:

    [Fact]
    public async Task require_tenant_id_happy_path()
    {
        await configure(opts =>
        {
            opts.TenantId.IsQueryStringValue("tenant");
            opts.TenantId.AssertExists();
        });

        // Got a 200? All good!
        await theHost.Scenario(x =>
        {
            x.Get.Url("/tenant?tenant=green");
        });
    }

Note that Alba would cause a test failure if the web request did not return a 200 status code.

And to lock down the binary behavior, here’s the “sad path” where Wolverine should be returning a 400 status code with ProblemDetails data:

    [Fact]
    public async Task require_tenant_id_sad_path()
    {
        await configure(opts =>
        {
            opts.TenantId.IsQueryStringValue("tenant");
            opts.TenantId.AssertExists();
        });

        var results = await theHost.Scenario(x =>
        {
            x.Get.Url("/tenant");
            
            // Tell Alba we expect a non-200 response
            x.StatusCodeShouldBe(400);
        });

        // Alba's helpers to deserialize JSON responses
        // to a strong typed object for easy
        // assertions
        var details = results.ReadAsJson<ProblemDetails>();
        
        // I like to refer to constants in test assertions sometimes
        // so that you can tweak error messages later w/o breaking
        // automated tests. And inevitably regret it when I 
        // don't do this
        details.Detail.ShouldBe(TenantIdDetection
            .NoMandatoryTenantIdCouldBeDetectedForThisHttpRequest);
    }

To be honest, it took me a few minutes to get the test above to pass because of some internal middleware mechanics I didn’t expect. As usual. All the same though, Alba helped me drive the code through “outside in” tests that ran quickly so I could iterate rapidly.

As always, I use Jeremy’s Only Law of Testing to decide on a mix of solitary or socialable tests in any particular scenario.

A bit about Alba

Alba itself is a descendant of some very old test helper code in FubuMVC, then was ported to OWIN (RIP, but I don’t miss you), then to early ASP.Net Core, and finally rebuilt as a helper around ASP.Net Core’s. built in TestServer and WebApplicationFactory. Alba has been continuously used for well over a decade now. If you’re looking for selling points for Alba, I’d say:

  • Alba makes your integration tests more declarative
  • There are quite a few helpers for common repetitive tasks in integration tests like reading JSON data with the application’s built in serialization
  • Simplifies test setup
  • It runs completely in memory where you can quickly spin up your application and jump right into debugging when necessary
  • Testing web services with Alba is much more efficient and faster than trying to do the same thing through inevitably slow, brittle, and laborious Selenium/Playwright/Cypress testing

Scheduled or Delayed Messages in Wolverine

Wolverine has first class support for delayed or scheduled message delivery. While I don’t think I’d recommend using Wolverine as a one for one replacement for a Hangfire or Quartz.Net, Wolverine’s functionality is great for:

  • Scheduling or delaying message retries on failures where you want the message retried, but definitely want that message out of the way of any subsequent messages in a queue
  • Enforcing “timeout” conditions for any kind of long running workflow
  • Explicit scheduling from within message handlers

Mechanically, you can publish a message with a delayed message delivery with Wolverine’s main IMessageBus entry point with this extension method:

public async Task schedule_send(IMessageContext context, Guid issueId)
{
    var timeout = new WarnIfIssueIsStale
    {
        IssueId = issueId
    };

    // Process the issue timeout logic 3 days from now
    await context.ScheduleAsync(timeout, 3.Days());

    // The code above is short hand for this:
    await context.PublishAsync(timeout, new DeliveryOptions
    {
        ScheduleDelay = 3.Days()
    });
}

Or using an absolute time with this overload of the same extension method:

public async Task schedule_send_at_5_tomorrow_afternoon(IMessageContext context, Guid issueId)
{
    var timeout = new WarnIfIssueIsStale
    {
        IssueId = issueId
    };

    var time = DateTime.Today.AddDays(1).AddHours(17);

    // Process the issue timeout at 5PM tomorrow
    // Do note that Wolverine quietly converts this
    // to universal time in storage
    await context.ScheduleAsync(timeout, time);
}

Now, Wolverine tries really hard to enable you to use pure functions for as many message handlers as possible, so there’s of course an option to schedule message delivery while still using cascading messages with the DelayedFor() and ScheduledAt() extension methods shown below:

public static IEnumerable<object> Consume(Incoming incoming)
{
    // Delay the message delivery by 10 minutes
    yield return new Message1().DelayedFor(10.Minutes());
    
    // Schedule the message delivery for a certain time
    yield return new Message2().ScheduledAt(new DateTimeOffset(DateTime.Today.AddDays(2)));
}

Lastly, there’s a special base class called TimeoutMessage that your message types can extend to add scheduling logic directly to the message itself for easy usage as a cascaded message. Here’s an example message type:

// This message will always be scheduled to be delivered after
// a one minute delay
public record OrderTimeout(string Id) : TimeoutMessage(1.Minutes());

Which is used within this sample saga implementation:

// This method would be called when a StartOrder message arrives
// to start a new Order
public static (Order, OrderTimeout) Start(StartOrder order, ILogger<Order> logger)
{
    logger.LogInformation("Got a new order with id {Id}", order.OrderId);

    // creating a timeout message for the saga
    return (new Order{Id = order.OrderId}, new OrderTimeout(order.OrderId));
}

How does it work?

The actual mechanics for how Wolverine is doing the scheduled delivery are determined by the destination endpoint for the message being published. In order of precedence:

  • If the destination endpoint has native message delivery capabilities, Wolverine uses that capability. Outbox mechanics still apply to when the outgoing message is released to the external endpoint’s sender. At the time of this post, the only transport with native scheduling support is Wolverine’s Azure Service Bus transport or the recently added Sql Server backed transport.
  • If the destination endpoint is durable, meaning that it’s enrolled in Wolverine’s transactional outbox, then Wolverine will store the scheduled messages in the outgoing envelope storage for later execution. In this case, Wolverine is polling for the ready to execute or deliver messages across all running Wolverine nodes. This option is durable in case of process exits.
  • In lieu of any other support, Wolverine has an in memory option that can do scheduled delivery or execution

Plans for Marten V7 and Beyond

As you might know, the Marten core team is in the process of building a new business around the “Critter Stack” tools (Marten + Wolverine and a couple smaller things). I think we’ll shortly be able to offer formal paid support contracts through JasperFx Software, and we’re absolutely open for business for any kind of immediate consulting on your software initiatives.

Call this a follow up from the Critter Stack roadmap from back in March. Everything takes longer than you wish it would, but at least Wolverine went 1.0 and I’m happy with how that’s been received so far and its uptake.

In the immediate future, we’re kicking out a Marten 6.1 release with a new health check integration and some bug fixes. Shortly afterward, we hope to get another 6.1.1 release out with as many bug fixes as we can address quickly to clear the way for the next big release. With that out of the way, let’s get on to the big initiatives for Marten!

Marten 7 Roadmap

Marten 6 was a lot of bug fixing and accommodating the latest version of Npgsql, our dependency for low level communication with PostgreSQL. Marten 7 is us getting on track for our strategic initiatives. Right now the goal is to move to an “open core” model for Marten where the current library and its capabilities stay open and free, but we will be building more advanced features for bigger workloads in commercial add on projects.

For the open core part of Marten, we’re aiming for:

  • Significant improvements to the LINQ provider support for better performing SQL and knocking down an uncomfortably long list of backlogged LINQ related issues and user requests. You can read more about that in Marten Linq Provider Improvements. Honestly, I think this is likely more work than the rest of the issues combined.
  • Maybe adding Strong Typed Identifiers as a logical follow on to the LINQ improvements. It’s been a frequent request, and I can see some significant opportunities for integration with Wolverine later
  • First class subscriptions in the event sourcing. This is going to be simplistic model for you to build your own persistent subscriptions to Marten events that’s a little more performant and robust than the current “IProjection for subscriptions” approach. More on this in the next section.
  • A way to parallelize the asynchronous projections in Marten for improved scalability based on Oskar’s strategy described in Scaling Marten.
  • .NET 8 integration. No idea what if anything that’s going to do to us.
  • Incorporating Npgsql 8. Also no idea if that’s going to be full of unwanted surprises or a walk in the park yet
  • A lightweight option for partial updates of Marten documents. We support this through our PLv8 add on, but that’s likely being deprecated and this comes up quite a bit from people moving to Marten from MongoDb

Critter Stack Enterprise-y Edition

Now, for the fun part, the long planned, long dreamed of commercial add ons for true enterprise Critter Stack usage. We have initial plans to build an all new library using both Marten and Wolverine that will be available under a commercial subscription.

Projection Scalability

Most of the biggest issues with scaling Marten to bigger systems are related to the asynchronous projection support. The first goal is scalability through distributing projection work across all the running nodes within your application. Wolverine already has leader election and “agent assignment” functionality that was built with this specific need in mind. To make that a little more clear, let’s say that you’re using Marten with a database per tenant multi-tenancy strategy. With “Critter Stack Enterprise” (place holder until we get a better name), the projection work might be distributed across nodes by tenant something like this:

The “leader agent” would help redistribute work as nodes come online or offline.

Improving scalability by distributing load across nodes is a big step, but there’s more tricks to play with projection throughput that would be part of this work.

Zero Downtime, Blue/Green Deployments

With the new projection daemon alternative, we will also be introducing a new “blue/green deployment” scheme where you will be able to change existing projections, introduce all new projections, or introduce new event signatures without having to have a potentially long downtime for rebuilding projections the way you might have to with Marten today. I feel like we have some solid ideas for how to finally pull this off.

More Robust Subscription Recipes

I don’t have many specifics here, but I think there’s an opportunity to also support more robust subscription offerings out of the Marten events using existing or new Wolverine capabilities. I also think we can offer stricter ordering and delivery guarantees with the Marten + Wolverine combination than we ever could with Marten alone. And frankly, I think we can do something more robust than what our obvious competitor tools do today.

Additional Event Store Throughput Improvements

Some other ideas we’re kicking around:

  • Introduce 2nd level caching into the aggregation projections
  • Elastic scalability for rebuilding projections
  • Hot/cold event store archiving that could improve both performance and scalability
  • Optional usage of higher performance serializers in the event store. That mostly knocks out LINQ querying for the event data

Other Stuff

We have many more ideas, but I think that the biggest theme is going to be ratcheting up the scalability of the event sourcing functionality and CQRS usage in general. There’s also a possibility of taking Marten’s event store functionality into cross-platform usage this year as well.

Thoughts? Requests? Wanna run jump in line to hire JasperFx Software?

Wolverine’s Improved Azure Service Bus Support

Wolverine 1.6.0 came out today, and one of the main themes was a series of improvements to the Azure Service Bus integration with Wolverine. In addition to the basic support Wolverine already had support for messaging with Azure Service Bus queues, topics, and subscriptions, you can now use native scheduled delivery, session identifiers for FIFO delivery, and expanded options for conventional routing topology.

First though, to get started with Azure Service Bus and Wolverine, install the WolverineFx.AzureServiceBus with the Nuget mechanism of your choice:

dotnet add package WolverineFx.AzureServiceBus

Next, you’ll add just a little bit to your Wolverine bootstrapping like this:

using var host = await Host.CreateDefaultBuilder()
    .UseWolverine((context, opts) =>
    {
        // One way or another, you're probably pulling the Azure Service Bus
        // connection string out of configuration
        var azureServiceBusConnectionString = context
            .Configuration
            .GetConnectionString("azure-service-bus");

        // Connect to the broker in the simplest possible way
        opts.UseAzureServiceBus(azureServiceBusConnectionString)
            .AutoProvision()
            .UseConventionalRouting();
    }).StartAsync();

Native Message Scheduling

You can now use native Azure Service Bus scheduled delivery within Wolverine without any explicit configuration beyond what you already do to connect to Azure Service Bus. Putting that into perspective, if you have a message type name ValidateInvoiceIsNotLate that is routed to an Azure Service Bus queue or subscription, you can use this feature:

public async Task SendScheduledMessage(IMessageContext bus, Guid invoiceId)
{
    var message = new public async Task SendScheduledMessage(IMessageContext bus, Guid invoiceId)
{
    var message = new ValidateInvoiceIsNotLate
    {
        InvoiceId = invoiceId
    };

    // Schedule the message to be processed in a certain amount
    // of time
    await bus.ScheduleAsync(message, 30.Days());

    // Schedule the message to be processed at a certain time
    await bus.ScheduleAsync(message, DateTimeOffset.Now.AddDays(30));
}
    {
        InvoiceId = invoiceId
    };

    // Schedule the message to be processed in a certain amount
    // of time
    await bus.ScheduleAsync(message, 30.Days());

    // Schedule the message to be processed at a certain time
    await bus.ScheduleAsync(message, DateTimeOffset.Now.AddDays(30));
}

That would also apply to scheduled retry error handling if the endpoint is also Inline:

using var host = Host.CreateDefaultBuilder()
    .UseWolverine(opts =>
    {
        opts.Policies.OnException<TimeoutException>()
            // Just retry the message again on the
            // first failure
            .RetryOnce()

            // On the 2nd failure, put the message back into the
            // incoming queue to be retried later
            .Then.Requeue()

            // On the 3rd failure, retry the message again after a configurable
            // cool-off period. This schedules the message
            .Then.ScheduleRetry(15.Seconds())

            // On the next failure, move the message to the dead letter queue
            .Then.MoveToErrorQueue();

    }).StartAsync();

Topic & Subscription Conventions

The original conventional routing with Azure Service Bus just sent and listened to queues named after the message type within the application. Wolverine 1.6 adds an additional routing convention to publish outgoing messages to topics and listen for known handled messages with topics and subscriptions. In all cases, you can customize the convention naming and any element of the Wolverine listening, sending, or any of the effected Azure Service Bus topics or subscriptions.

The syntax for this option is shown below:

opts.UseAzureServiceBusTesting()
    .UseTopicAndSubscriptionConventionalRouting(convention =>
    {
        // Optionally control every aspect of the convention and
        // its applicability to types
        // as well as overriding any listener, sender, topic, or subscription
        // options
    })

    .AutoProvision()
    .AutoPurgeOnStartup();

Session Identifiers and FIFO Delivery

You can now take advantage of sessions and first-in, first out queues in Azure Service Bus with Wolverine. To tell Wolverine that an Azure Service Bus queue or subscription should require sessions, you have this syntax shown in an internal test:

_host = await Host.CreateDefaultBuilder()
    .UseWolverine(opts =>
    {
        opts.UseAzureServiceBusTesting()
            .AutoProvision().AutoPurgeOnStartup();

        opts.ListenToAzureServiceBusQueue("send_and_receive");
        opts.PublishMessage<AsbMessage1>().ToAzureServiceBusQueue("send_and_receive");

        opts.ListenToAzureServiceBusQueue("fifo1")
            
            // Require session identifiers with this queue
            .RequireSessions()
            
            // This controls the Wolverine handling to force it to process
            // messages sequentially
            .Sequential();
        
        opts.PublishMessage<AsbMessage2>()
            .ToAzureServiceBusQueue("fifo1");

        opts.PublishMessage<AsbMessage3>().ToAzureServiceBusTopic("asb3");
        opts.ListenToAzureServiceBusSubscription("asb3")
            .FromTopic("asb3")
            
            // Require sessions on this subscription
            .RequireSessions(1)
            
            .ProcessInline();
    }).StartAsync();

Wolverine is using the “group-id” nomenclature from the AMPQ standard, but for Azure Service Bus, this is directly mapped to the SessionId property on the Azure Service Bus client internally.

To publish messages to Azure Service Bus with a session id, you will need to of course supply the session id:

// bus is an IMessageBus
await bus.SendAsync(new AsbMessage3("Red"), new DeliveryOptions { GroupId = "2" });
await bus.SendAsync(new AsbMessage3("Green"), new DeliveryOptions { GroupId = "2" });
await bus.SendAsync(new AsbMessage3("Refactor"), new DeliveryOptions { GroupId = "2" });

You can also send messages with session identifiers through cascading messages as shown in a fake message handler below:

public static IEnumerable<object> Handle(IncomingMessage message)
{
    yield return new Message1().WithGroupId("one");
    yield return new Message2().WithGroupId("one");

    yield return new Message3().ScheduleToGroup("one", 5.Minutes());

    // Long hand
    yield return new Message4().WithDeliveryOptions(new DeliveryOptions
    {
        GroupId = "one"
    });
}

Using Sql Server as a Message Queue with Wolverine

Wolverine 1.4.0 was released last week (and a smaller 1.5.0, with a medium sized 1.6.0 coming Monday). The biggest new feature was a brand new option to use Microsoft Sql Server (or Azure Sql) as a durable message transport with Wolverine.

Let’s say your system is already using Sql Server for persistence, you need some durable, asynchronous messaging, and wouldn’t it be nice to not have to introduce any new infrastructure into the mix? Assuming you’ve decided to also use Wolverine, you can get started with this approach by adding the WolverineFx.SqlServer Nuget to your application:

dotnet add package WolverineFx.SqlServer

Here’s a sample application bootstrapping that shows the inclusion and configuration of Sql Server-backed queueing:

using var host = await Host.CreateDefaultBuilder()
    .UseWolverine((context, opts) =>
    {
        var connectionString = context
            .Configuration
            .GetConnectionString("sqlserver");
        
        // This adds both Sql Server backed
        // transactional inbox/outbox support
        // and the messaging transport support
        opts
           .UseSqlServerPersistenceAndTransport(connectionString, "myapp")
            
            // Tell Wolverine to build out all necessary queue or scheduled message
            // tables on demand as needed
            .AutoProvision()
            
            // Option that may be helpful in testing, but probably bad
            // in production!
            .AutoPurgeOnStartup();

        // Use this extension method to create subscriber rules
        opts.PublishAllMessages()
            .ToSqlServerQueue("outbound");

        // Use this to set up queue listeners
        opts.ListenToSqlServerQueue("inbound")

            // Optional circuit breaker usage
            .CircuitBreaker(cb =>
            {
                // fine tune the circuit breaker
                // policies here
            })
            
            // Optionally specify how many messages to 
            // fetch into the listener at any one time
            .MaximumMessagesToReceive(50);
    }).StartAsync();

The Sql Server transport is pretty simple, it basically just supports named queues right now. Here’s a couple useful properties of the transport that will hopefully make it more useful to you:

  • Scheduled message delivery is absolutely supported with the Sql Server Transport, and some care was taken to optimize the database load and throughput when using this feature
  • Sql Server backed queues can be either “buffered in memory” (Wolverine’s message batching) or be “durable” meaning that the queues are integrated into both the transactional inbox and outbox for durable systems
  • Wolverine can build database tables as necessary for the queue much like it does today for the transactional inbox and outbox. Moreover, the configured queue tables are also part of the stateful resource model in the Critter Stack world that provide quite a bit of command line management directly into your application.
  • The Sql Server backed queues support Wolverine’s circuit breaker functionality on listeners

This feature is something that folks have asked about in the past, but I’ve always been reticent to try because databases don’t make for great, 1st class queueing mechanisms. That being said, I’m working with a JasperFx Software client who wanted a more robust local queueing mechanism that could handle much more throughput for scheduled messaging, and thus, the new Sql Server Transport was born.

There will be a full fledged PostgreSQL backed queue at some point, and it might be a little more robust even based on some preliminary work from a Wolverine contributor, but that’s probably not an immediate priority.