The feature was built for a current JasperFx Software client, and came with a wave of developments across both Marten and Wolverine to support a fairly complex, mission critical set of application integrations. The PostgreSQL transport new to Wolverine was part of this wave. Some time next week I’ll be blogging about the Marten event subscription capabilities that were built into Marten & Wolverine to support this client as well. The point being, JasperFx is wide open for business and we can help your shop succeed with challenging project work!
Wolverine now has the ability to support strict messaging order with its message listeners. Given any random listening endpoint in Wolverine, just add this directive below to make the message processing be strictly sequential (with the proviso that your error handling policies may impact the order on failures):
var host = await Host.CreateDefaultBuilder().UseWolverine(opts =>
{
opts.UseRabbitMq().EnableWolverineControlQueues();
opts.PersistMessagesWithPostgresql(Servers.PostgresConnectionString, "listeners");
opts.ListenToRabbitQueue("ordered")
// This option is available on all types of Wolverine
// endpoints that can be configured to be a listener
.ListenWithStrictOrdering();
}).StartAsync();
Some notes about the ListenWithStrictOrdering() directive you might have:
It’s supported with every external messaging broker that Wolverine supports, including Kafka, Azure Service Bus, AWS SQS, and Rabbit MQ. It is also supported with the two database backed transports (we have both kinds, Sql Server and PostgreSQL!)
When this directive is applied, Wolverine will only make the listener for each endpoint (in the case above, the Rabbit MQ named “ordered”) be active on a single node within your application. Today that distribution is just crudely spreading out the “exclusive listeners” evenly across the whole application cluster. Definitely note that the strict ordering comes at the cost of reduced throughput, so use this feature wisely! Did I mention that JasperFx Software is here and ready to work with your company on Critter Stack projects?
Every exclusive listener will quickly start up on a single node if WolverineOptions.Durability.Mode = DurabilityMode.Solo, and you may want to do that for local testing and development just to be a little quicker on cold starts
The ListenWithStrictOrdering will make the internal worker queue (Wolverine uses an internal TPL Dataflow ActionBlock in these cases) for “buffered” or “durable” endpoints be strictly sequential
You will have to have a durable message store configured for your application in order for Wolverine to perform the leadership election and “agent tracking” (what’s running where)
Summary
This is a powerful tool in the continually growing Wolverine tool belt. The strict ordering may also be used to alleviate some concurrency issues that some users have hit with event sourcing using Marten when a single stream may be receiving bursts of commands that impact the event stream. The leadership election and agent distribution in Wolverine, in conjunction with this “sticky” listener assignment, gives Wolverine a nascent ability for virtual actors that we will continue to exploit. More soon-ish!
Wolverine just got a new PostgreSQL-backed messaging transport (with the work sponsored by a JasperFx Software client!). The use case is just this, say you’re already using Wolverine to build a system with PostgreSQL as your backing database, and want to introduce some asynchronous, background processing in your system — which you could already do with just a database backed, local queue. Going farther though, let’s say that we’d like to have a competing consumers setup for our queueing for load balancing between active nodes and we’d like to do that without having to introduce some kind of new message broker infrastructure into our existing architecture.
That’s time to bring in Wolverine’s new option for asynchronous messaging just using our existing PostgreSQL database. To set that up by itself (without using Marten, but we’ll get to that in a second), it’s these couple lines of code:
var builder = WebApplication.CreateBuilder(args);
var connectionString = builder.Configuration.GetConnectionString("postgres");
builder.Host.UseWolverine(opts =>
{
// Setting up Postgresql-backed message storage
// This requires a reference to Wolverine.Postgresql
opts.PersistMessagesWithPostgresql(connectionString);
// Other Wolverine configuration
});
Of course, you’d want to setup PostgreSQL queues for Wolverine to send to and to listen to for messages to process. That’s shown below:
using var host = await Host.CreateDefaultBuilder()
.UseWolverine((context, opts) =>
{
var connectionString = context.Configuration.GetConnectionString("postgres");
opts.UsePostgresqlPersistenceAndTransport(connectionString, "myapp")
// Tell Wolverine to build out all necessary queue or scheduled message
// tables on demand as needed
.AutoProvision()
// Optional that may be helpful in testing, but probably bad
// in production!
.AutoPurgeOnStartup();
// Use this extension method to create subscriber rules
opts.PublishAllMessages().ToPostgresqlQueue("outbound");
// Use this to set up queue listeners
opts.ListenToPostgresqlQueue("inbound")
.CircuitBreaker(cb =>
{
// fine tune the circuit breaker
// policies here
})
// Optionally specify how many messages to
// fetch into the listener at any one time
.MaximumMessagesToReceive(50);
}).StartAsync();
And that’s that, we’re completely set up for messaging via the PostgreSQL database we already have with our Wolverine application!
Just a couple things to note before you run off and try to use this:
Like I alluded to earlier, the PostgreSQL queueing mechanism supports competing consumers, so different nodes at runtime can be pulling and processing messages from the PostgreSQL queues
There is a separate set of tables for each named queue (one for the actual inbound/outbound messages, and a separate table to segregate “scheduled” messages). Utilize that separation for better performance as needed by effectively sharding the message transfers
As that previous bullet point implies, the PostgreSQL transport is able to support scheduled message delivery
As in most cases, Wolverine is able to detect whether or not the necessary tables all exist in your database, and create any missing tables for you at runtime
There’s some optimizations and integration between these queues and the transactional inbox/outbox support in Wolverine for performance by reducing database chattiness whenever possible
Summary
I’m not sure I’d recommend this approach over dedicated messaging infrastructure for high volumes of messages, but it’s a way to get things done with less infrastructure in some cases and it’s a valuable tool in the Wolverine toolbox.
JasperFx Software is open for business and offering consulting services (like helping you craft modular monolith strategies!) and support contracts for both Marten and Wolverine so you know you can feel secure taking a big technical bet on these tools and reap all the advantages they give for productive and maintainable server side .NET development.
I’ve been thinking, discussing, and writing a bit lately about the whole “modular monolith” idea, starting with Thoughts on “Modular Monoliths” and continuing onto Actually Talking about Modular Monoliths. This time out I think I’d like to just put out some demos and thoughts about where Marten and Wolverine fit well into the modular monolith idea — and also some areas where I think there’s room for improvement.
First off, let’s talk about…
Modular Configuration
Both tools use the idea of a “configuration model” (what Marten Fowler coined a Semantic Model years ago) that is compiled and built from a combination of attributes in the code, explicit configuration, user supplied policies, and built in policies in baseline Marten or Wolverine as shown below with an indication of the order of precedence:
In code, when you as a user configure Marten and Wolverine inside of the Program file for your system like so:
builder.Services.AddMarten(opts =>
{
var connectionString = builder.Configuration.GetConnectionString("marten");
opts.Connection(connectionString);
// This will create a btree index within the JSONB data
opts.Schema.For<Customer>().Index(x => x.Region);
})
// Adds Wolverine transactional middleware for Marten
// and the Wolverine transactional outbox support as well
.IntegrateWithWolverine();
builder.Host.UseWolverine(opts =>
{
opts.CodeGeneration.TypeLoadMode = TypeLoadMode.Static;
// Let's build in some durability for transient errors
opts.OnException<NpgsqlException>().Or<MartenCommandException>()
.RetryWithCooldown(50.Milliseconds(), 100.Milliseconds(), 250.Milliseconds());
// Shut down the listener for whatever queue experienced this exception
// for 5 minutes, and put the message back on the queue
opts.OnException<MakeBelieveSubsystemIsDownException>()
.PauseThenRequeue(5.Minutes());
// Log the bad message sure, but otherwise throw away this message because
// it can never be processed
opts.OnException<InvalidInputThatCouldNeverBeProcessedException>()
.Discard();
// Apply the validation middleware *and* discover and register
// Fluent Validation validators
opts.UseFluentValidation();
// Automatic transactional middleware
opts.Policies.AutoApplyTransactions();
// Opt into the transactional inbox for local
// queues
opts.Policies.UseDurableLocalQueues();
// Opt into the transactional inbox/outbox on all messaging
// endpoints
opts.Policies.UseDurableOutboxOnAllSendingEndpoints();
// Connecting to a local Rabbit MQ broker
// at the default port
opts.UseRabbitMq();
// Adding a single Rabbit MQ messaging rule
opts.PublishMessage<RingAllTheAlarms>()
.ToRabbitExchange("notifications");
opts.LocalQueueFor<TryAssignPriority>()
// By default, local queues allow for parallel processing with a maximum
// parallel count equal to the number of processors on the executing
// machine, but you can override the queue to be sequential and single file
.Sequential()
// Or add more to the maximum parallel count!
.MaximumParallelMessages(10)
// Pause processing on this local queue for 1 minute if there's
// more than 20% failures for a period of 2 minutes
.CircuitBreaker(cb =>
{
cb.PauseTime = 1.Minutes();
cb.SamplingPeriod = 2.Minutes();
cb.FailurePercentageThreshold = 20;
// Definitely worry about this type of exception
cb.Include<TimeoutException>();
// Don't worry about this type of exception
cb.Exclude<InvalidInputThatCouldNeverBeProcessedException>();
});
// Or if so desired, you can route specific messages to
// specific local queues when ordering is important
opts.Policies.DisableConventionalLocalRouting();
opts.Publish(x =>
{
x.Message<TryAssignPriority>();
x.Message<CategoriseIncident>();
x.ToLocalQueue("commands").Sequential();
});
});
The nested lambdas in AddMarten() and UseWolverine() are configuring the MartenOptions and WolverineOptions models respectively (the “configuration model” in that diagram above).
I’m not aware of any one commonly used .NET idiom for building modular configuration, but I do commonly see folks using extension methods for IServiceCollection or IHostBuilder to segregate configuration that’s specific to a single module, and that’s what I think I’d propose. Assuming that we have a module in our modular monolith system for handling workflow around “incidents”, there might be an extension method something like this:
public static class IncidentsConfigurationExtensions
{
public static WebApplicationBuilder AddIncidentModule(this WebApplicationBuilder builder)
{
// Whatever other configuration, services, et al
// we need for just the Incidents module
// Extra Marten configuration
builder.Services.ConfigureMarten(opts =>
{
// I'm just adding an index for a document type within
// this module
opts.Schema.For<IncidentDetails>()
.Index(x => x.Priority);
// Purposely segregating all document types in this module's assembly
// to a separate database schema
opts.Policies.ForAllDocuments(m =>
{
if (m.DocumentType.Assembly == typeof(IncidentsConfigurationExtensions).Assembly)
{
m.DatabaseSchemaName = "incidents";
}
});
});
return builder;
}
}
Which would be called from the overall system’s Program file like so:
var builder = WebApplication.CreateBuilder(args);
builder.AddIncidentModule();
// Much more below...
Between them, the two main Critter Stack tools have a lot of support for modularity through:
Wolverine’s extension and modules allow you to easily import message handlers or HTTP endpoints from other assemblies than the main application assembly
Both Marten and Wolverine allow you to register special extensions in your IoC container that allow you to add additional configuration to both Marten (IConfigureMarten) and Wolverine (IWolverineExtension)
All of the facilities I described above can be used to separate specific configuration for different modules within the module code itself.
Modular Monoliths and Backing Persistence
In every single experience report you’ll ever find about a team trying to break up and modernize a large monolithic application the authors will invariably say that breaking apart the database was the single most challenging task. If we are really doing things better this time with the modular monolith approach, we’d probably better take steps ahead of time to make it easier to extract services by attempting to keep the persistence for each module at least somewhat decoupled from the persistence of other modules.
Going even farther in terms of separation, it’s not unlikely that some modules have quite different persistence needs and might be better served by using a completely different style of persistence than the other modules. Just as an example, one of our current JasperFx Software clients has a large monolithic application where some workflow-centric modules would be a good fit for an event sourcing approach, while other modules are more CRUD centric or reporting-centric where a straight up RDBMS approach is probably much more appropriate.
So let’s finally bring Marten and Wolverine into the mix and talk about the Good, the Bad, and the (sigh) Ugly of how the Critter Stack fits into modular monoliths:
wah, wah, wah…
Let’s start with a positive. Marten sits on top of the very robust PostgreSQL database. So in addition to Marten’s ability to use PostgreSQL as a document database and as an event store, PostgreSQL out of the box is a rock solid relational database. Heck, PostgreSQL even has some ability to be used as either a graph database! The point is that using the Marten + PostgreSQL combination gives you a lot of flexibility in terms of persistence style between different modules in a modular monolith without introducing a lot more infrastructure. Moreover, Wolverine can happily utilize its PostgreSQL-backed transactional outbox with both Entity Framework Core and Marten targeting the same PostgreSQL database in the same application.
Continuing with another positive, let’s say that we want to create some logical separation between our modules in the database, and one way to do so would be to simply keep Marten documents in separate database schemas for each module. Repeating a code sample from above, you can see that configuration below:
public static WebApplicationBuilder AddIncidentModule(this WebApplicationBuilder builder)
{
// Whatever other configuration, services, et al
// we need for just the Incidents module
// Extra Marten configuration
builder.Services.ConfigureMarten(opts =>
{
// Purposely segregating all document types in this module's assembly
// to a separate database schema
opts.Policies.ForAllDocuments(m =>
{
if (m.DocumentType.Assembly == typeof(IncidentsConfigurationExtensions).Assembly)
{
m.DatabaseSchemaName = "incidents";
}
});
});
return builder;
}
So, great, the storage for Marten documents could easily be segregated by schema. Especially considering there’s little or no referential integrity relationships between Marten document tables, it should be relatively easy to move these document tables to completely different databases later!
And with that, let’s move more into “Bad” or hopefully not too “Ugly” territory.
The event store data in Marten is all in one single set of tables (mt_streams and mt_events). So every module utilizing Marten’s event sourcing could be intermingling their events in just these tables through the one single, AddMarten() store for the application’s IHost. You could depend on marking event streams by their aggregate type like so:
public static async Task start_stream(IDocumentSession session)
{
// the Incident type argument is strictly a marker for
// Marten
session.Events.StartStream<Incident>(new IncidentLogged());
await session.SaveChangesAsync();
}
I think we could ameliorate this situation with a couple future changes:
A new flag in Marten that would make it mandatory to mark every new event stream with an aggregate type specifically to make it easier to separate the events later to extract a service and its event storage
Some kind of helper to move event streams from one database to another. It’s just not something we have in our tool belt at the moment
Of course, it would also help immeasurably if we had a way to split the event store storage for different types of event streams, but somehow that idea has never gotten any traction within Marten and never rises to the level of a high priority. Most of our discussions about sharding or partitioning the event store data has been geared around scalability — which is certainly an issue here too of course.
Marten also has its concept of “separate stores” that was meant to allow an application to interact with multiple Marten-ized databases from a single .NET process. This could be used with modular monoliths to segregate the event store data, even if targeting the same physical database in the end. The very large downside to this approach is that Wolverine’s Marten integration does not today do anything with the separate store model. So no Wolverine transactional middleware, event forwarding, transactional inbox/outbox integration, and no aggregate handler workflow. So basically everything about the full “Critter Stack” integration that makes that tooling the single most productive event sourcing development experience in all of .NET (in my obviously biased opinion). Ugly.
Randomly, I heard an NPR interview with Eli Wallach very late in his life who was the actor who played the “Ugly” character in the famous western, and I could only describe him as effusively jolly. So basically a 180 degree difference from his character!
Module to Module Communication
I’ve now spent much more time on this post than I had allotted, so it’s time to go fast…
In my last post I used this diagram to illustrate the risk of coupling modules through direct usage of internals (the red arrows):
Instead of the red arrows everywhere above, I think I’m in favor of trying to limit the module to module communication to using some mix of a “mediator” tool or an in memory message bus between modules. That’s obviously going to come with some overhead, but I think (hope) that overhead is a net positive.
For a current client, I’m recommending they further utilize MediatR as they move a little more in the direction of modularity in their current monolith. For greenfield codebases, I’d recommend Wolverine instead because I think it does much, much more.
First, Wolverine has a full set of functionality to be “just a Mediator” to decouple modules from too much of the internals of another module. Secondly, Wolverine has a lot of support for background processing through local, in memory queues that could be very advantageous in modular monoliths where Wolverine can de facto be an in memory message bus. Moreover, Wolverine’s main entry point usage is identical for messages processed locally versus messages published through external messaging brokers to external processes:
public static async Task using_message_bus(IMessageBus bus)
{
// Use Wolverine as a "mediator"
// This is normally executed inline, in process, but *could*
// also be invoking this command in an external process
// and waiting for the success or failure ack
await bus.InvokeAsync(new CategoriseIncident());
// Use Wolverine for asynchronous messaging. This could
// start by publishing to a local, in process queue, or
// it could be routed to an external message broker -- but
// the calling code doesn't have to know that
await bus.PublishAsync(new CategoriseIncident());
}
The point here is that Wolverine can potentially set your modular monolith architecture up so that it’s possible to extract or move functionality out into separate services later.
All that being said about messaging or mediator tools, some of the ugliest systems I’ve ever seen utilized messaging or proto-Mediatr command handlers between logical modules. Those systems had code that was almost indecipherable by introducing too many layers and far too much internal messaging. I think I’d say that some of the root cause of the poor system code was from getting the bounded context boundaries wrong so that the messaging was too chatty. Using high ceremony anti-corruption layers also adds a lot of mental overhead to follow information flow through several mapping transformations. One of these systems was using the iDesign architectural approach that I think naturally leads to very poorly factored software architectures and too much harmful code ceremony. I do not recommend.
I guess my only point here is that no matter what well intentioned advice people like me try to give, or any theory of how to make code more maintainable any of us might have, if you find yourself saying to yourself about code that “this is way harder than it should be” you should challenge the approach and look for something different — even if that just leads you right back to where you are now if the alternatives don’t look any better.
One important point here about both modular monoliths or a micro service strategy or a mix of the two: if two or more services/modules are chatty between themselves and very frequently have to be modified at the same time, they’re best described as a single bounded context and should probably be combined into a single service or module.
Summary
Anyway, that’s enough from me on this subject for now, and this took way longer than I meant to spend on it. Time to get my nose back to the grindstone. I am certainly very open to any feedback about the Critter Stack tools limitations for modular monolith construction and any suggestions or requests to improve those tools.
In that previous post though, the messages held in those in memory, local queues could conceivably be lost if the application is shut down unexpectedly (Wolverine will attempt to “drain” the local queues of outstanding work on graceful process shutdowns). That’s perfectly acceptable sometimes, but in other times you really need those queued up messages to be durable so that the in flight messages can be processed even if the service process is unexpectedly killed while work is in flight — so let’s opt into Wolverine’s ability to do exactly that!
To that end, let’s just assume that we’re a very typical .NET shop and we’re already using Sql Server as our backing database for the system. Knowing that, let’s add a new Nuget reference to our project:
dotnet add package WolverineFx.SqlServer
And let’s break into our Program file for the service where all the system configuration is, and expand the Wolverine configuration within the UseWolverine() call to this:
// This is good enough for what we're trying to do
// at the moment
builder.Host.UseWolverine(opts =>
{
// Just normal .NET stuff to get the connection string to our Sql Server database
// for this service
var connectionString = builder.Configuration.GetConnectionString("SqlServer");
// Telling Wolverine to build out message storage with Sql Server at
// this database and using the "wolverine" schema to somewhat segregate the
// wolverine tables away from the rest of the real application
opts.PersistMessagesWithSqlServer(connectionString, "wolverine");
// In one fell swoop, let's tell Wolverine to make *all* local
// queues be durable and backed up by Sql Server
opts.Policies.UseDurableLocalQueues();
});
Nothing else in our previous code needs to change. As a matter of fact, once you restart your application — assuming that your box can reach the Sql Server database in the appsettings.json file — Wolverine is going to happily see that those necessary tables are missing, and build them out for you in your database so that Wolverine “can just work” on its first usage. That automatic schema creation can of course be disabled and/or done with pure SQL through other Wolverine facilities, but for right now, we’re taking the easy road.
Before I get into the runtime mechanics, here’s a refresher about our first message handler:
public static class SendWelcomeEmailHandler
{
public static void Handle(SignUpRequest @event, ILogger logger)
{
// Just logging, a real handler would obviously do something real
// to send an email
logger.LogInformation("Send a Send a welcome email to {Name} at {Email}", @event.Name, @event.Email);
}
}
And the code that publishes a SignUpRequest message to a local Wolverine queue in a Minimal API endpoint:
After our new configuration up above to add message durability to our local queues, when a service client posts a SignUpRequest message is published to Wolverine as a result of a client posting valid data to the /signup Url, Wolverine will:
Persist all the necessary information about the new SignUpRequest message that Wolverine uses to process that message in the Sql Server database (this is using the “Envelope Wrapper” pattern from the old EIP book, which is quite originally called Envelope in the Wolverine internals).
If the message is successfully processed, Wolverine will delete that stored record for the message in Sql Server
If the message processing fails, and there’s some kind of retry policy in effect, Wolverine will increment the number of failed attempts in the Sql Server database (with an UPDATE statement because it’s trying to be as efficient as possible)
If the process somehow fails while the message is floating around in the in memory queues, Wolverine will be able to recover that local message from the database storage later when the system is restarted. Or if the system is running in a load balanced cluster, a different Wolverine node will be able to see that the messages are orphaned in the database and will steal that work into another node so that the messages eventually get processed
Summary and What’s Next?
That’s a lot of detail about what is happening in your system, but I’d argue that was very little code necessary to make the background processing with Wolverine be durable. And all without introducing any other new infrastructure other than the Sql Server database we were probably already using. Moreover, Wolverine can do a lot to make the necessary database setup for you at runtime so there’s hopefully very little friction getting up and running after a fresh git clone.
I’ll add at least a couple more entries to this series by looking at error handling strategies, controller the parallelism or strict ordering of message processing, a simple implementation of the Producer/Consumer pattern with Wolverine, and message scheduling.
Hey, did you know that JasperFx Software offers formal support plans for Marten and Wolverine? Not only are we making the “Critter Stack” tools be viable long term options for your shop, we’re also interested in hearing your opinions about the tools and how they should change.We’re also certainly open to help you succeed with your software development projects on a consulting basis whether you’re using any part of the Critter Stack or some completely different .NET server side tooling.
Hey, when you’re building grown up software systems in a responsible way, who likes effective automated testing? Me, too! Moreover, I like automated tests that are reliable — and anyone who has ever been remotely near a large automated test suite testing through the web application layer with any kind of asynchronous behavior knows exactly how painful “flake-y” tests are that suffer from timing issues.
Wolverine of course is an application framework for performing background processing and asynchronous messaging — meaning that there’s no end of the exact kind of asynchronous behavior that is notoriously hard to deal with in automated tests. At a minimum, what you need is a way to exercise the message handling within Wolverine (the “act” in the “act, arrange, assert” test pattern), but wait until all cascading activity is really complete before allowing the automated test to continue making assertions on expected outcomes. Fortunately, Wolverine has that very functionality baked into its core library. Here’s a fake saga that we recently used to fix a bug in Wolverine:
public class LongProcessSaga : Saga
{
public Guid Id { get; init; }
[Middleware(typeof(BeginProcessMiddleware))]
public static (LongProcessSaga, OutgoingMessages) Start(BeginProcess message, RecordData? sourceData = null)
{
var outgoingMessages = new OutgoingMessages();
var saga = new LongProcessSaga
{
Id = message.DataId,
};
if (sourceData is not null)
{
outgoingMessages.Add(new ContinueProcess(saga.Id, message.DataId, sourceData.Data));
}
return (
saga,
outgoingMessages
);
}
public void Handle(ContinueProcess process)
{
Continued = true;
}
public bool Continued { get; set; }
}
When the BeginProcess message is handled by Wolverine, it might also spawn a ContinueProcess message. So let’s write a test that exercises the first message, but waits until the second message that we expect to be spawned while handling the first message before allowing the test to proceed:
[Fact]
public async Task can_compile_without_issue()
{
// Arrange -- and sorry, it's a bit of "Arrange" to get an IHost
var builder = WebApplication.CreateBuilder(Array.Empty<string>());
builder.Services
.AddMarten(options =>
{
options.Connection(Servers.PostgresConnectionString);
})
.UseLightweightSessions()
.IntegrateWithWolverine();
builder.Host.UseWolverine(options =>
{
options.Discovery.IncludeAssembly(GetType().Assembly);
options.Policies.AutoApplyTransactions();
options.Policies.UseDurableLocalQueues();
options.Policies.UseDurableOutboxOnAllSendingEndpoints();
});
builder.Services.AddScoped<IDataService, DataService>();
// This is using Alba, which uses WebApplicationFactory under the covers
await using var host = await AlbaHost.For(builder, app =>
{
app.MapWolverineEndpoints();
});
// Finally, the "Act"!
var originalMessage = new BeginProcess(Guid.NewGuid());
// This is a built in extension method to Wolverine to "wait" until
// all activity triggered by this operation is completed
var tracked = await host.InvokeMessageAndWaitAsync(originalMessage);
// And now it's okay to do assertions....
// This would have failed if there was 0 or many ContinueProcess messages
var continueMessage = tracked.Executed.SingleMessage<ContinueProcess>();
continueMessage.DataId.ShouldBe(originalMessage.DataId);
}
The IHost.InvokeMessageAndWaitAsync() is part of Wolverine’s “tracked session” feature that’s descended from an earlier system some former colleagues and I developed and used at my then employer about a decade ago. The original mechanism was quite successful for our integration testing efforts of the time, and was built into Wolverine quite early. This “tracked session” feature is very heavily used within the Wolverine test suites to test Wolverine itself.
But wait, there’s more! Here’s a bigger sample from the documentation just showing you some more things that are possible:
public async Task using_tracked_sessions_advanced(IHost otherWolverineSystem)
{
// The point here is just that you somehow have
// an IHost for your application
using var host = await Host.CreateDefaultBuilder()
.UseWolverine().StartAsync();
var debitAccount = new DebitAccount(111, 300);
var session = await host
// Start defining a tracked session
.TrackActivity()
// Override the timeout period for longer tests
.Timeout(1.Minutes())
// Be careful with this one! This makes Wolverine wait on some indication
// that messages sent externally are completed
.IncludeExternalTransports()
// Make the tracked session span across an IHost for another process
// May not be super useful to the average user, but it's been crucial
// to test Wolverine itself
.AlsoTrack(otherWolverineSystem)
// This is actually helpful if you are testing for error handling
// functionality in your system
.DoNotAssertOnExceptionsDetected()
// Again, this is testing against processes, with another IHost
.WaitForMessageToBeReceivedAt<LowBalanceDetected>(otherWolverineSystem)
// There are many other options as well
.InvokeMessageAndWaitAsync(debitAccount);
var overdrawn = session.Sent.SingleMessage<AccountOverdrawn>();
overdrawn.AccountId.ShouldBe(debitAccount.AccountId);
}
As hopefully implied by the earlier example, the “tracked session” functionality also gives you:
Recursive tracking of all message activity to wait for everything to finish
Enforces timeouts in case of hanging tests that probably won’t finish successfully
The ability to probe the exact messaging activity that happened as a result of your original message
Visibility into any exceptions recorded by Wolverine during message processing that might otherwise be hidden from you. This functionality will re-throw these exceptions to fail a test unless explicitly told to ignore processing exceptions — which you may very well want to do to test error handling logic
If a test fails because of a timeout, or doesn’t reach the expected conditions, the test failure exception will show you a (hopefully) neatly formatted textual table explaining what it did observe in terms of what messages were sent, received, started, and finished executing. Again, this is to give you more visibility into test failures, because those inevitably do happen!
Last Thoughts
Supporting a complicated OSS tool like Marten or Wolverine is a little bit like being trapped in somewhere in Jurassic Park while the raptors (users, and especially creative users) are prowling around the perimeter of your tool just looking for weak spots in your tools — a genuine bug, a use case you didn’t anticipate, an awkward API, some missing documentation, or even just some wording in your documentation that isn’t clear enough. The point is, it’s exhausting and sometimes demoralizing when raptors are getting past your defenses a little too often just because you rolled out a near complete rewrite of your LINQ provider subsystem:)
Yesterday I was fielding questions from a fellow whose team was looking to move to Wolverine from one of the older .NET messaging frameworks, and he was very complimentary of the integration testing support that’s the subject of this post. My only point here is to remember to celebrate your successes to balance out the constant worry about what’s not yet great about your tool or project or codebase.
And by success, I mean a very important feature that will absolutely help teams build reliable software more productively with Wolverine that does not exist in other .NET messaging frameworks. And certainly doesn’t exist in the yet-to-be-built Microsoft eventing framework where they haven’t even considered the idea of testability.
I got a little feedback over the weekend that some folks newly encountering Wolverine for the first time think that it’s harder to use than some other tools because “it doesn’t do the message routing for you.” This fortunately isn’t true, but there’s obviously some work to do on improving documentation and samples to dispel that impression.
For the sake of this post, let’s assume that you want to use Wolverine for asynchronous messaging between processes using an external messaging broker transport (or using asynchronous messaging in a single application but queueing up work in an external message broker). And while Wolverine does indeed have support for interoperability with non-Wolverine applications, let’s assume that it’s going to be Wolverine on both sides of all the message pipes.
First though, just know that for all for external transports with Wolverine, the conventional routing is opt in, meaning that you have to explicitly turn it on when you configure Wolverine within the UseWolverine() bootstrapping. Likewise, know that you can also control exactly how Wolverine configures the listening or message sending behavior in the conventionally determined endpoints.
Now then, to just go fast and make Wolverine do all the message routing for you with predictable conventions “just” see these recipes:
using var host = await Host.CreateDefaultBuilder()
.UseWolverine(opts =>
{
opts.UseRabbitMq()
// Opt into conventional Rabbit MQ routing
.UseConventionalRouting();
}).StartAsync();
In this convention, message routing is to:
Publish all messages to a Rabbit MQ exchange named after Wolverine’s message type name that’s effectively the alias for that message type. Most of the time that’s just the full name of the concrete .NET type.
Create a Rabbit MQ queue named with Wolverine’s message type name for the message type of every known message handler within your application. Wolverine also configures a binding from a Rabbit MQ exchange with that same name to the Rabbit MQ queue for that message type.
This routing behavior was absolutely influenced by similar functionality in the older MassTransit framework. Imitation is the sincerest form of flattery, plus I just agree with what they did anyway.Wolverine adds additional value through its richer resource setup model (AutoProvision / AddResourceSetupOnStartup()) and stronger model of automatic handler discovery.
That’s the quickest possible way to get started, but you have plenty of other levers, knobs, and dials to farther control the conventions as shown below:
using var host = await Host.CreateDefaultBuilder()
.UseWolverine(opts =>
{
opts.UseRabbitMq()
// Opt into conventional Rabbit MQ routing
.UseConventionalRouting(c =>
{
// Override naming conventions
c.QueueNameForListener(type => type.Name);
c.ExchangeNameForSending(type => type.Name);
// Override the configuration for each listener
c.ConfigureListeners((l, _) =>
{
l.ProcessInline().ListenerCount(5);
});
// Override the sending configuration for each subscriber
c.ConfigureSending((s, _) =>
{
s.UseDurableOutbox();
});
})
// Let Wolverine both discover all these necessary exchanges, queues,
// and binding, then also build them as necessary on the broker if
// they are missing
.AutoProvision();
}).StartAsync();
The “extra” optional configuration I used above hopefully show you how to take more exacting control over the conventional routing, but the pure defaults in the first sample will help you get up and going fast.
Let’s move on.
Azure Service Bus
Wolverine’s Azure Service Bus integration comes with a pair of conventional routing options. The simpler is to let Wolverine create and utilize an Azure Service Bus queue named after each message type name for both sending the receiving with this syntax:
using var host = await Host.CreateDefaultBuilder()
.UseWolverine(opts =>
{
opts.UseAzureServiceBus("some connection string")
.UseConventionalRouting();
}).StartAsync();
Slightly more complicated is another option to use Azure Service Bus topics and subscriptions like so:
// opts is a WolverineOptions object
opts.UseAzureServiceBusTesting()
.UseTopicAndSubscriptionConventionalRouting(convention =>
{
// Optionally control every aspect of the convention and
// its applicability to types
// as well as overriding any listener, sender, topic, or subscription
// options
});
In this usage:
Outgoing messages are routed to an Azure Service Bus topic named after Wolverine’s message type name for that type
At application startup, Wolverine will listen for an Azure Service Bus subscription for each message type from the application’s known message handlers. Likewise, that subscription will automatically be bound to a topic named after the message type name
As was the case with the Rabbit MQ conventions shown first, you have complete control over the naming conventions, the listener configuration, and the subscriber configuration.
For Wolverine’s AWS SQS integration, you can conventionally route to SQS queues named after Wolverine’s message type name (by default) like this:
As was the case with the Rabbit MQ conventions shown first, you have complete control over the naming conventions, the listener configuration, and the subscriber configuration.
Wolverine’s Kafka integration is a little bit different animal. In this case you can set up rules to publish to different Kafka topics by message type like this:
using var host = await Host.CreateDefaultBuilder()
.UseWolverine(opts =>
{
opts.UseKafka("localhost:29092").AutoProvision();
opts.PublishAllMessages().ToKafkaTopics();
opts.Services.AddResourceSetupOnStartup();
}).StartAsync();
In this case, Wolverine will send each message type to a topic name derived from the message type that’s either Wolverine’s message type name or an explicitly configured topic name configured by attribute like:
[Topic("color.purple")]
public class PurpleMessage
{
}
What if I want a completely different routing convention?!?
There’s an extension point for programmatic message routing rules in Wolverine that’s utilized by all the capabilities shown above called IMessageRouteSource, but if you think you really want to use that, maybe just head to the Wolverine Discord room and we’ll try to help you out!
Summary
Wolverine has strong support for conventional message routing using external brokers, but you need to make at least a one line of code entry in your configuration to explicitly add this behavior to your system. In all cases, Wolverine is able to help build the necessary queues, topics, subscriptions, binding, and exchanges derived by these conventions in your message broker for an efficient developer experience. Moreover, you have complete power to fine tune the usage of this conventional routing for your application.
Hey, did you know that JasperFx Software offers formal support plans for Marten and Wolverine? Not only are we making the “Critter Stack” tools be viable long term options for your shop, we’re also interested in hearing your opinions about the tools and how they should change.We’re also certainly open to help you succeed with your software development projects on a consulting basis whether you’re using any part of the Critter Stack or some completely different .NET server side tooling.
Marten 7.0 was released over the weekend, and Wolverine 2.0 followed yesterday mostly to catch up with the Marten dependency. One of the major improvements in this round of “Critter Stack” releases was to address a JasperFx client’s need for dynamically adding new tenant databases at runtime without having to do any kind of system deployment.
Doing that successfully meant adding a couple capabilities throughout the “Critter Stack.” First off, Marten needed a new multi-tenancy configuration strategy that allowed users to keep a list of valid tenant id and database connection strings for those tenants in a database table. Enter Marten 7’s new Master Table Tenancy Model. You can see the usage in this sample from the documentation:
using var host = await Host.CreateDefaultBuilder()
.ConfigureServices(services =>
{
services.AddMarten(sp =>
{
var configuration = sp.GetRequiredService<IConfiguration>();
var masterConnection = configuration.GetConnectionString("master");
var options = new StoreOptions();
// This is opting into a multi-tenancy model where a database table in the
// master database holds information about all the possible tenants and their database connection
// strings
options.MultiTenantedDatabasesWithMasterDatabaseTable(x =>
{
x.ConnectionString = masterConnection;
// You can optionally configure the schema name for where the mt_tenants
// table is stored
x.SchemaName = "tenants";
// If set, this will override the database schema rules for
// only the master tenant table from the parent StoreOptions
x.AutoCreate = AutoCreate.CreateOrUpdate;
// Optionally seed rows in the master table. This may be very helpful for
// testing or local development scenarios
// This operation is an "upsert" upon application startup
x.RegisterDatabase("tenant1", configuration.GetConnectionString("tenant1"));
x.RegisterDatabase("tenant2", configuration.GetConnectionString("tenant2"));
x.RegisterDatabase("tenant3", configuration.GetConnectionString("tenant3"));
// Tags the application name to all the used connection strings as a diagnostic
// Default is the name of the entry assembly for the application or "Marten" if
// .NET cannot determine the entry assembly for some reason
x.ApplicationName = "MyApplication";
});
// Other Marten configuration
return options;
})
// All detected changes will be applied to all
// the configured tenant databases on startup
.ApplyAllDatabaseChangesOnStartup();;
}).StartAsync();
With this tenancy model, Marten is able to discover newly added tenant databases at runtime as needed during normal transactions. And don’t fret about the extra activity, Marten is able to cache that information in memory to avoid making too many unnecessary database calls.
But what about if I need to decommission and remove a customer database? What if we need to move a tenant database at runtime? Can Marten create databases on the fly for us? How will I do that?!? And sorry, my reply is that will require at least an application restart for now, and any kind of fancier management for advanced multi-tenancy will likely go into the forthcoming “Critter Stack Pro” paid model later this year.
That’s part one. The next issue was that for users who also used Marten’s asynchronous projection feature, the “async daemon” subsystem in Marten needed to be able to discover new tenant databases in the background and ensure that all the asynchronous projections for these newly discovered databases are running in the background somewhere in the application. This led to a partial rewrite of the “async daemon” subsystem for Marten 7, but you can see the positive effect of that work in this test that “proves” that Marten is able to spin up projection building agents in the background at runtime:
[Fact]
public async Task add_tenant_database_and_verify_the_daemon_projections_are_running()
{
// In this code block, I'm adding new tenant databases to the system that I
// would expect Marten to discover and start up an asynchronous projection
// daemon for all three newly discovered databases
var tenancy = (MasterTableTenancy)theStore.Options.Tenancy;
await tenancy.AddDatabaseRecordAsync("tenant1", tenant1ConnectionString);
await tenancy.AddDatabaseRecordAsync("tenant2", tenant2ConnectionString);
await tenancy.AddDatabaseRecordAsync("tenant3", tenant3ConnectionString);
// This is a new service in Marten specifically to help you interrogate or
// manipulate the state of running asynchronous projections within the current process
var coordinator = _host.Services.GetRequiredService<IProjectionCoordinator>();
var daemon1 = await coordinator.DaemonForDatabase("tenant1");
var daemon2 = await coordinator.DaemonForDatabase("tenant2");
var daemon3 = await coordinator.DaemonForDatabase("tenant3");
// Just proving that the configured projections for the 3 new databases
// are indeed spun up and running after Marten's new daemon coordinator
// "finds" the new databases
await daemon1.WaitForShardToBeRunning("TripCustomName:All", 30.Seconds());
await daemon2.WaitForShardToBeRunning("TripCustomName:All", 30.Seconds());
await daemon3.WaitForShardToBeRunning("TripCustomName:All", 30.Seconds());
}
Now, switching over to Wolverine 2.0 and its contribution to the party, the third part to make this dynamic tenant database discovery work throughout the entire “Critter Stack” was for Wolverine to be able to also discover the new tenant databases at runtime and spin up its “durability agents” for background message scheduling and its transactional inbox/outbox support. It’s admittedly not well at all documented yet, but Wolverine has an internal system for leader election and “agent assignment” between running nodes in an application cluster to distribute work. Wolverine uses this subsystem to distribute the transactional inbox/outbox work for each tenant database across the application cluster. Look for more information on this capability as JasperFx Software will be exploiting this for a different customer engagement this year.
Marten 7.0 is released to the wild as of right now! Before getting into the highlights of what’s improved in this release, let’s go right to thanking some of the folks who made big contributions to this release either through code, testing, or feedback:
Vedran Zakanj for sponsoring Marten and also contributing ideas around schema management
And a huge thanks to Lucas Wyland for specifically sponsoring the improved LINQ provider work with an equally huge apology from me on how long that took to finish
And to many more community members who helped improve Marten throughout this very long release cycle.
Highlights
This was a huge release, if not nearly as disruptive as Marten 4 was several years ago. I do not anticipate a lot of issues for users upgrading from Marten 6 to Marten 7, but see the migration guide for more details.
The highlights of Marten 7 are:
The LINQ query support was given a large overhaul that both expanded its supported use cases and led to significantly improved performance of many common sub collection queries — which has been a large complaint and request for improvement from the Marten community for several years
A “Partial” document update capability using native PostgreSQL functionality with no JavaScript in sight! That’s been a long requested capability.
The very basic database execution pipeline underneath Marten was largely rewritten to be far more parsimonious with how it uses database connections and to take advantage of more efficient Npgsql usage. We think these changes will make Marten both more efficient overall (these changes reduced the number of object allocations by quite a bit) and help system health through using fewer database connections
We introduced Polly for resiliency to transient errors like network hiccups or a temporarily overloaded database and actually made Marten able to properly execute retries of database writes and database reads
The “async daemon” subsystem was somewhat rewritten with substantial improvements for application scalability. The asynchronous projection support also has an all new scheme for resiliency that we think will be a big improvement for our users
An option to utilize Marten’s recommended FetchForWriting() API for “write model” aggregation with asynchronous projections. This may sound like a lot of mumbo jumbo, but it’s vital because this enables the next bullet point
The ability to do zero downtime deployments of some projection changes as well as to do blue/green deployments of revisioned projections. Much more on this later this week.
A new alternative for “revisioned” documents with a numeric version as an alternative to Marten’s existing GUID based versioning scheme for optimistic concurrency
We’ll see how big of a deal this turns out to be, but Marten 7 enables the usage of Project Aspire with Marten
As time permits, I will be writing deep dive blog posts on each of the individual bullet points above over the next couple weeks — partially as a way to force the completion of some not perfectly updated documentation!
You can Place a Technical Bet on Marten
There’s frequently an understandable hesitation on the part of software shops to take a bet on an open source tool as a critical piece of their technical infrastructure — and that’s sometimes worse in the .NET ecosystem where OSS adoption isn’t as widespread. All that aside, I’m here to tell you that you can feel safe making a large technical bet on Marten because:
Marten is already a very mature project that has been in production usage since its 1.0 release in 2016
While Marten doesn’t have every single issue around production support, deployments, and schema management fixed yet, we’ve got a detailed roadmap to shore up any remaining weaknesses of the tool and we’re in this for the long haul!
PostgreSQL itself is a very successful open source project that continuously innovates and provides a very solid technical foundation for Marten itself
We’ve invested a lot of time into refining Marten’s usability over the years and we think that attention to detail shines through
JasperFx Software offers support contracts and consulting work for Marten users
In conjunction with Wolverine’s integration with Marten, the full “Critter Stack” provides a very efficient and usable stack for Event Driven Architecture using Event Sourcing and a CQRS architecture
While Marten 7.0 made some significant improvements for scalability, the forthcoming “Critter Stack Pro” commercial add on tooling will take Marten to much larger data sets and transactional throughput
Because Marten does target .NET, it’s worth pointing out that at this point, Microsoft has no technical offerings for Event Sourcing and that will absolutely contribute to Marten’s viability
What’s Next and Summary
A lot of big, important, long requested, long planned features and improvements did not make the cut for V7. I blogged last week about the current roadmap for the entire Critter Stack. Moreover, some open bugs didn’t make it into 7.0 as well. And let’s be honest, there’s going to be a slew of bug reports streaming in this week when folks try out new 7.0 features and encounter usage permutations we didn’t anticipate. I’ve finally learned my lesson and made this release after having gotten some rest to be ready for whatever the issues turn out to be in the morning.
Wolverine 2.0 will also follow shortly, but the roadmap for that is pretty well just upgrading to Marten 7, dumping .NET 6, and fixing some low hanging fruit issues and requests before a release in the next couple days.
We’ll jump on whatever those Marten 7 issues turn out to be and all the questions about “what about *my* use case I don’t see on your list!” starting tomorrow, but for right now, this was a huge release filled with all kinds of substantial improvements that for the first time included significant client sponsored requests and please don’t steal my sunshine!
Isaac Levin was kind enough to have me on his Coffee & Open Source show to talk about a variety of topics around technology and my involvement with OSS work.
I need to update my profile here and there, that picture was taken on my late grandparents farm around Christmas of 2010 outside the titular Jasper, MO
I’d say that my time in OSS has long been valuable in terms of increased technical skillset and occasionally through opportunities that arose because of my OSS tools. It’s just now though that I’m finally living out my longstanding dream to make my “Critter Stack” OSS work (Marten & Wolverine) be my actual job as part of JasperFx Software.
Just to call a few highlights and to add to our conversation after having some time to think about things:
I made a double edged bit of advice at the end to “take your shot” when you have a technical idea that could become your job, but followed by an exhortation to stop working on something that isn’t bringing you joy or opportunities.
Unfortunately, failure is an awesomely effective teacher — if you let it be. I feel like the Critter Stack tools are succeeding right now, and plenty of that is due to some harsh lessons learned from my earlier failures in OSS.
OSS projects can succeed with a mix of having a conceptual idea or approach that appeals to enough folks, a dedicated core team of contributors like Oskar and Babu, and an enthusiastic and patient community that helps with suggestions, bug reports, and contributions. I called out Wolverine especially as a tool whose usability has largely been driven by the feedback of several early adopters. Moreover, one of the hard lessons learned from my earlier failure with FubuMVC is how important it is to get enough user feedback to sand off rough edges with a tool’s usability or documentation.
I personally find it very gratifying to be working on my projects, carrying out my vision, and generally having my hand on the steering wheel of Marten and Wolverine. I’m also enjoying the hands on consulting engagements I’m doing with the current JasperFx clients and making a positive difference for them. The obvious takeaway for me — and probably for a great number of you out there as well — is that I am much happier when I feel like I have significant ownership over the work and that my contributions are respected and valued by the customer, management, product owner, or colleagues. I’ve been consistently miserable in jobs or roles where I didn’t have either of those two things.
From some notes that Oskar, Babu, and I banged out this past week, so keep your expectations for the quality of prose here! Notes in bold are my updates since this original document was banged out last weekend.
Marten 7.0
Try to release Marten 7.0 no later than early next week. This is admittedly based on JasperFx client deliverables.
Blue/green & zero downtime deployment. Ongoing work that just needs more testing at this point. This includes the projection version stuff. Actually all working locally, but my development branch is rebased on the daemon stuff, so I’d like that to go in first. Done.
Projection Snapshots – I’d really like to see this mostly land in Critter Stack Pro. Probably not happening until 2nd quarter 2024
First class subscriptions from the event store to Wolverine transports – might be in Wolverine 2.0 proper. Dunno. Not sure yet
Async projection optimizations – Probably not happening until 2nd quarter 2024
2nd level caching for aggregates
Rebuild single stream projections stream by stream
Allow for selective identity map usage of reference types.
Batched data lookups – so you can keep projections from doing chatty data access
Allow grouping logic to express optimization hints like “no data access required” or “requires aggregate state”. That could be used to optimize projection rebuilds
Wolverine 2.0
Discovery and activation of new tenant databases at runtime (client deliverable). Done.
Update to Marten 7
Project Aspire? Wolverine 2.1? This is a little more involved, so I’m not sure yet when this lands. Probably in Wolverine 2.1.
Marten 7.1
Open Telemetry Support – Sean Farrow is working on this. I don’t think it’s going to be a breaking change, so could float to 7.1. Very Basic
Sharding the event store tables – I’d love to do this sooner, and would love to stretch this in. I’m saying that we would tackle the is archived / not archived sharding in a first pass, then come back w/ fancier sharding possibilities later. This would have a potentially huge positive implication for Marten event store scalability.
The ability to “emit” new events in the async daemon during the course of processing asynchronous projections. I think this is going to take some spikes and analysis, so we gotta commit to this ASAP if it’s going into 7.0. This is falling to Marten 7.1
First class subscriptions. Hot, cold, replay, whatever. I just want a little more time and space. Does this require any breaking changes in the daemon we might want to deal with right now though? Very likely dropping to Marten 7.1
Custom event type naming strategy – it’s a breaking change to the API I think. I don’t think it’s huge though – little pluggable strategy. Can be additive.
Optimize inline projections in FetchForWriting()? Idea here is to force aggregates that are calculated Inline (or Async maybe) that are queried in FetchForWriting() be forced to use the identity map for just that document type. That does a lot to optimize the typical “aggregate handler workflow” by avoiding the current double fetching of the document when you are using lightweight sessions. Strong candidate to drop down to 7.1
Marten 7.Later
Downcasters – I vote to put this into Critter Stack Pro all the way
Marten 8.0???
More advanced Event Store partitioning
Wolverine 2.1
Likely a focus on the Wolverine.HTTP backlog
Options for strict ordering requirements of event or message processing
The Shade Tree Developer 