Hey folks, this is more a brain dump to collect my own thoughts than any kind of tome of accumulated wisdom and experience. Please treat this accordingly, and absolutely chime in on the Critter Stack Discord discussion going on about this right now. I’m also very willing and maybe even likely to change my mind about anything I’m going to say in this post.

There’s been some recent interest and consternation about the combination of Marten within Hot Chocolate as a GraphQL framework. At the same time, I am working with a new JasperFx client who wants to use Hot Chocolate with Marten’s event store functionality behind mutations and projected data behind GraphQL queries.

Long story short, Marten and Hot Chocolate do not mix well without some significant thought and deviation from normal, out of the box Marten usage. Likewise, I’m seeing some significant challenges in using Wolverine behind Hot Chocolate mutations. The rest of this post is a rundown of the issues, sticking points, and possible future ameliorations to make this combination more effective for our various users.

Connections are Sticky in Marten

If you use the out of the box IServiceCollection.AddMarten() mechanism to add Marten into a .NET application, you’re registering Marten’s IQuerySession and IDocumentSession as a Scoped lifetime — which is optimal for usage within short lived ASP.Net Core HTTP requests or within message bus handlers (like Wolverine!). In both of those cases, the session can be expected to have a short lifetime and generally be running in a single thread — which is good because Marten sessions are absolutely not thread safe.

However, for historical reasons (integration with Dapper was a major use case in early Marten usage, so there’s some optimization for that with Marten sessions), Marten sessions have a “sticky” connection lifecycle where an underlying Npgsql connection is retained on the first database query until the session is disposed. Again, if you’re utilizing Marten within ASP.Net Core controller methods or Minimal API calls or Wolverine message handlers or most other service bus frameworks, the underlying IoC container of your application is happily taking care of resource disposal for you at the right times in the request lifecycle.

The last sentence is one of the most important, but poorly understood advantages of using IoC containers in applications in my opinion.

Ponder the following Marten usage:

    public static async Task using_marten(IDocumentStore store)
    {
        // The Marten query session is IDisposable,
        // and that absolutely matters!
        await using var session = store.QuerySession();

        // Marten opens a database connection at the first
        // need for that connection, then holds on to it
        var doc = await session.LoadAsync<User>("jeremy");
        
        // other code runs, but the session is still open
        // just in case...
        
        // The connection is closed as the method exits
        // and the session is disposed
    }

The problem with Hot Chocolate comes in because Hot Chocolate is trying to parallelize queries when you get multiple queries in one GraphQL request — which since that query batching was pretty well the raison d’être for GraphQL in the first place, so you should assume that’s quite common!

Now, consider a naive usage of a Marten session in a Hot Chocolate query:

    public async Task<SomeEntity> GetEntity(
        [Service] IQuerySession session
        Input input)
    {
        // load data using the session
    }

Without taking some additional steps to serialize access to the IQuerySession across Hot Chocolate queries, you will absolutely hit concurrency errors when Hot Chocolate tries to parallelize data fetching. You can beat this by either forcing Hot Chocolate to serialize access like so:

    builder.Services
        .AddGraphQLServer()
        
        // Serialize access to the IQuerySession within Hot Chocolate
        .RegisterService<IQuerySession>(ServiceKind.Synchronized)

or by making the session lifetime in your container Transient by doing this:

    builder.Services
        .AddTransient<IQuerySession>(s => s.GetRequiredService<IDocumentStore>().QuerySession());

The first choice will potentially slow down your GraphQL endpoints by serializing access to the IQuerySession while fetching data. The second choice is a non-idiomatic usage of Marten that potentially fouls up usage of Marten within non-GraphQL operations as you could potentially be using separate Marten sessions when you really meant to be using a shared instance.

If you’re really concerned about needing the queries within a single GraphQL request to be parallelized, there’s another option using object pooling like Jackson Cribb wrote about in the Marten Discord room.

For Marten V7, we’re going to strongly consider some kind of query runner that does not have sticky connections for the express purpose of simplifying Hot Chocolate + Marten integration, but I can’t promise any particular timeline for that work. You can track that work here though.

Multi-Tenancy and Session Lifecycles

Multi-Tenancy throws yet another spanner into the works. Consider the following Hot Chocolate query method:

    public IQueryable<User> GetUsers(
        [Service] IDocumentStore documentStore, [GlobalState] string tenant)
    {
        using var session = documentStore.LightweightSession(tenant);

        return session.Query<User>();
    }

Assuming that you’ve got some kind of Hot Chocolate interceptor to detect the tenant id for you, and that value is communicated through Hot Chocolate’s global state mechanism, you might think to open a Marten session directly like the code above. That code above will absolutely not work under any kind of system load because it’s putting you into a damned if you do, damned if you don’t situation. If you dispose the session before this method completes,
the IQueryable execution will throw an ObjectDisposedException when Hot Chocolate tries to execute the query. If you *don’t* dispose the session, the IoC container for the request scope doesn’t know about it, so can’t dispose it for you and Marten is going to be hanging on to the open database connection until garbage collection comes for it — and under a significant load, that means your system will behave very badly when the database connection pool is exhausted!

What we need to do is to have some way that our sessions can be created for the right tenant for the current request, but have the session tracked some how so that the scoped IoC container can be used to clean up the open sessions at the end of the request. As a first pass, I’m using this crude approach first with this service that’s registered with the IoC container with a Scoped lifetime:

/// <summary>
/// This will be Scoped in the container per request, "knows" what
/// the tenant id for the request is. Also tracks the active Marten
/// session
/// </summary>
public class ActiveTenant : IDisposable
{
    public ActiveTenant(IHttpContextAccessor contextAccessor, IDocumentStore store)
    {
        if (contextAccessor.HttpContext is not null)
        {
            // Try to detect the active tenant id from
            // the current HttpContext
            var context = contextAccessor.HttpContext;
            if (context.Request.Headers.TryGetValue("tenant", out var tenant))
            {
                var tenantId = tenant.FirstOrDefault();
                if (tenantId.IsNotEmpty())
                {
                    this.Session = store.QuerySession(tenant!);
                }
            }
        }

        this.Session ??= store.QuerySession();
    }

    public IQuerySession Session { get; }

    public void Dispose()
    {
        this.Session.Dispose();
    }
}

Now, rewrite the Hot Chocolate query from way up above with:

    public IQueryable<User> GetUsers(
        [Service] ActiveTenant tenant)
    {
        return tenant.Session.Query<User>();
    }

That does still have to be paired with this Hot Chocolate configuration to dodge the concurrent access problems like so:

    builder.Services 
        .AddGraphQLServer()
        .RegisterService<ActiveTenant>(ServiceKind.Synchronized)

Hot Chocolate Mutations

I took some time this morning to research Hot Chocolate’s Mutation model (think “writes”). Since my client is using Marten as an event store and I’m me, I was looking for opportunities to:

• Possibly use Wolverine as a true “mediator” within Hot Chocolate mutations to take advantage of the aggregate handler workflow from the Wolverine + Marten integration
• Or find an easy to use equivalent to the aggregate handler workflow within Hot Chocolate Mutation classes
• At a minimum, look for ways to tie in the Wolverine transactional outbox within the Hot Chocolate mutation workflow

What I’ve found so far has been a series of blockers once you zero in on the fact that Hot Chocolate is built around the possibility of having zero to many mutation messages in any one request — and that that request should be treated as a logical transaction such that every mutation should either succeed or fail together. With that being said, I see the blockers as:

• Wolverine doesn’t yet support message batching in any kind of built in way, and is unlikely to do so before a 2.0 release that isn’t even so much as a glimmer in my eyes yet
• Hot Chocolate depends on ambient transactions (Boo!) to manage the transaction boundaries. That by itself almost knocks out the out of the box Marten integration and forces you to use more custom session mechanics to enlist in ambient transactions.
• The existing Wolverine transactional outbox depends on an explicit “Flush” operation after the actual database transaction is committed. That’s handled quite gracefully by Wolverine’s Marten integration in normal issue (in my humble and very biased opinion), but that can’t work across multiple mutations in one GraphQL request
• There is a mechanism to replace. the transaction boundary management in Hot Chocolate, but it was very clearly built around ambient transactions and it has a synchronous signature to commit the transaction. Like any sane server side development framework, Wolverine performs the IO intensive database transactional mechanics and outbox flushing operations with asynchronous methods. To fit that within Hot Chocolate’s transactional boundary abstraction would require calls to turn the asynchronous Marten and Wolverine APIs into synchronous calls with GetAwaiter().GetResult(), which is tantamount to painting a bullseye on your chest and daring the Fates to not make your application crater with deadlocks under load.

I think at this point, my recommended approach is going to forego integrating Wolverine into Hot Chocolate mutations altogether with some combination of:

• Don’t use Hot Chocolate mutations whatsoever if there’s no need for the operation batching and use old fashioned ASP.Net Core with or without Wolverine’s HTTP support
• Or document a pattern for using the Decider pattern within Hot Chocolate as an alternative to Wolverine’s “aggregate handler” usage. The goal here is to document a way for developers to keep infrastructure out of business logic code and maximize testability
• If using Hot Chocolate mutations, I think there’s a need for a better outbox subscription model directly against Marten’s event store. The approach Oskar outlined here would certainly be a viable start, but I’d rather have an improved version of that built directly into Wolverine’s Marten integration. The goal here is to allow for an Event Driven Architecture which Wolverine supports quite well and the application in question could definitely utilize, but do so without creating any complexity around the Hot Chocolate integration.

In the long, long term:

• Add a message batch processing option to Wolverine that manages transactional boundaries between messages for you
• Have a significant palaver between the Marten/Wolverine core teams and the fine folks behind Hot Chocolate to iron a bit of this out

My Recommendations For Now

Honestly, I don’t think that I would recommend using GraphQL in general in your system whatsoever unless you’re building some kind of composite user interface where GraphQL would be beneficial in reducing chattiness between your user interface and backing service by allowing unrelated components in your UI happily batch up requests to your server. Maybe also if you were using GraphQL as a service gateway to combine disparate data sources on the server side in a consistent way, but even then I wouldn’t automatically use GraphQL.

I’m not knowledgeable enough to say how much GraphQL usage would help speed up your user interface development, so take all that I said in the paragraph above with a grain of salt.

At this point I would urge folks to be cautious about using the Critter Stack with Hot Chocolate. Marten can be used if you’re aware of the potential problems I discussed above. Even when we beat the sticky connection thing and the session lifecycle problems, Marten’s basic model of storing JSON in the database is really not optimized for plucking out individual fields in Select() transforms. While Marten does support Select() transforms, it’s may not as efficient as the equivalent functionality on top of a relational database model would be. It’s possible that GraphQL might be a better fit with Marten if you were primarily using projected read models purposely designed for client consumption through GraphQL or even projecting event data to flat tables that are queried by Hot Chocolate.

Wolverine with Hot Chocolate maybe not so much if you’d have any problems with the transactional boundary issues.

I would be urge you to do load testing with any usage of Hot Chocolate as I think from peeking into its internals that it’s not the most efficient server side tooling around. Again, that doesn’t mean that you will automatically have performance problems with Hot Chocolate, but I think you you should be cautious with its usage.

In general, I’d say that GraphQL creates way too much abstraction over your underlying data storage — and my experience consistently says that abstracted data access can lead to some very poor system performance by both making your application harder to understand and by eliminating the usage of advanced features of your data storage tooling behind least common denominator abstractions.

This took much longer than I wanted it too, as always. I might write a smaller follow up on how I’d theoretically go about building an optimized GraphQL layer from scratch for Marten — which I have zero intension of ever doing, but it’s a fun thought experiment.

Last week I started a new series of blog posts about Wolverine capabilities with:

• Producer/Consumer Pattern with Wolverine
• Resiliency with Wolverine
• Useful Tricks with Lamar for Integration Testing (don’t laugh, it was relevant to a project at work ;-))

Today I’m going to continue with a contrived example from the “payment ingestion service,” this time on what I’m so far calling “compound handlers” in Wolverine. When building a system with any amount of business logic or workflow logic, there’s some philosophical choices that Wolverine is trying to make:

• To maximize testability, business or workflow logic — as much as possible — should be in pure functions that are easily testable in isolated unit tests. In other words, you should be able to test this code without integration tests or mock objects. Just data in, and state-based assertions.
• Of course your message handler will absolutely need to read data from our database in the course of actually handling messages. It’ll also need to write data to the underlying database. Yet we still want to push toward the pure function approach for all logic. To get there, I like Jim Shore’s A-Frame metaphor for how code should be organized to isolate business logic away from infrastructure and into nicely testable code.
• I certainly didn’t set out this way years ago when what’s now Wolverine was first theorized, but Wolverine is trending toward using more functional decomposition with fewer abstractions rather than “traditional” class centric C# usage with lots of interfaces, constructor injection, and IoC usage. You’ll see what I mean when we hit the actual code

I don’t think that mock objects are evil per se, but they’re absolutely over-used in our industry. All I’m trying to suggest in this post is to structure code such that you don’t have to depend on stubs or any other kind of fake to set up test inputs to business or workflow logic code.

Consider the case of a message handler that needs to process a command message to apply a payment to principal within an existing loan. Depending on the amount and the account in question, the handler may need to raise domain events for early principle payment penalties (or alerts or whatever you actually do in this situation). That logic is going to need to know about both the related loan and account information in order to make that decision. The handler will also make changes to the loan to reflect the payment made as well, and commit those changes back to the database.

Just to sum things up, this message handler needs to:

1. Look up loan and account data
2. Use that data to carry out the business logic
3. Potentially persist the changed state

Alright, on to the handler, which I’m going to accomplish with a single class that uses two separate methods:

public record PayPrincipal(Guid LoanId, decimal Amount, DateOnly EffectiveDate);

public static class PayPrincipalHandler
{
    // Wolverine will call this method first by naming convention.
    // If you prefer being more explicit, you can use any name you like and decorate
    // this with [Before] 
    public static async Task<(Account, LoanInformation)> LoadAsync(PayPrincipal command, IDocumentSession session,
        CancellationToken cancellation)
    {
        Account? account = null;
        var loan = await session
            .Query<LoanInformation>()
            .Include<Account>(x => x.AccountId, a => account = a)
            .Where(x => x.Id == command.LoanId)
            .FirstOrDefaultAsync(token: cancellation);

        if (loan == null) throw new UnknownLoanException(command.LoanId);
        if (account == null) throw new UnknownAccountException(loan.AccountId);
        
        return (account, loan);
    }

    // This is the main handler, but it's able to use the data built
    // up by the first method
    public static IEnumerable<object> Handle(
        // The command
        PayPrincipal command,
        
        // The information loaded from the LoadAsync() method above
        LoanInformation loan, 
        Account account,
        
        // We need this only to mark items as changed
        IDocumentSession session)
    {
        // The next post will switch this to event sourcing I think

        var status = loan.AcceptPrincipalPayment(command.Amount, command.EffectiveDate);
        switch (status)
        {
            case PrincipalStatus.BehindSchedule:
                // Maybe send an alert? Act on this in some other way?
                yield return new PrincipalBehindSchedule(loan.Id);
                break;
            
            case PrincipalStatus.EarlyPayment:
                if (!account.AllowsEarlyPayment)
                {
                    // Maybe just a notification?
                    yield return new EarlyPrincipalPaymentDetected(loan.Id);
                }

                break;
        }

        // Mark the loan as being needing to be persisted
        session.Store(loan);
    }
}

Wolverine itself is weaving in the call first to LoadAsync(), and piping the results of that method to the inputs of the inner Handle() method, which now gets to be almost a pure function with just the call to IDocumentSession.Store() being “impure” — but at least that one single method is relatively painless to mock.

The point of doing this is really just to make the main Handle() method where the actual business logic is happening be very easily testable with unit tests as you can just push in the Account and Loan information. Especially in cases where there’s likely many permutations of inputs leading to different behaviors, it’s very advantageous to be able to walk right up to just the business rules and push inputs right into that, then do assertions on the messages returned from the Handle() function and/or assert on modifications to the Loan object.

TL:DR — Repository abstractions over persistence tooling can cause more harm than good.

Also notice that I directly used a reference to the Marten IDocumentSession rather than wrapping some kind of IRepository<Loan> or IAccountRepository abstraction right around Marten. That was very purposeful. I think those abstractions — especially narrow, entity-centric abstractions around basic CRUD or load methods cause more harm than good in nontrivial enterprise systems. In the case above, I was using a touch of advanced, Marten-specific behavior to load related documents in one network round trip as a performance optimization. That’s the exact kind of powerful ability of specific persistence tools that’s thrown away by generic “IRepository of T” strategies “just in case we decide to change database technologies later” that I believe to be harmful in larger enterprise systems. Moreover, I think that kind of abstraction bloats the codebase and leads to poorly performing systems.

We’re dogfooding Wolverine at work and the Critter Stack Discord is pretty active right now. All of that means that issues and opportunities to improve Wolverine are coming in fast right now. I just pushed Wolverine 0.9.13 (the Nugets are all named “WolverineFx” something because someone is squatting on the “Wolverine” name in Nuget).

First, quick thanks to Robin Arrowsmith for finding and fixing an issue with Wolverine’s Azure Service Bus support. And a more general thank you to the nascent Wolverine community for being so helpful working with me in Discord to improve Wolverine.

A few folks are reporting various issues with Wolverine handler discovery. To help alleviate whatever those issues turn out to be, Wolverine has a new mechanism to troubleshoot “why is my handler not being found by Wolverine?!?” issues.

We’re converting a service at work that lives within a giant distributed system that’s using NServiceBus for messaging today, so weirdly enough, there’s some important improvements for Wolverine’s interoperability with NServiceBus.

This will be worth a full blog post soon, but there’s some ability to add contextual logging about your domain (account numbers, tenants, product numbers, etc.) to Wolverine’s open telemetry and/or logging support. My personal goal here is to have all the necessary and valuable correlation between system activity, performance, and logged problems without forcing the development team to write repetitive code throughout their message handler code.

And one massive bug fix for how Wolverine generates runtime code in conjunction with your IoC service registrations for objects created by Wolverine itself. That’s a huge amount of technical mumbo jumbo that amounts to “even though Jeremy really doesn’t approve, you can inject Marten IDocumentSession or EF Core DbContext objects into repository classes while still using Wolverine transactional middleware and outbox support.” See this issue for more context. It’s a hugely important fix for folks who choose to use Wolverine with a typical, .NET Onion/Clean architecture with lots of constructor injection, repository wrappers, and making the garbage collection work like crazy at runtime.