The feature was built for a current JasperFx Software client, and came with a wave of developments across both Marten and Wolverine to support a fairly complex, mission critical set of application integrations. The PostgreSQL transport new to Wolverine was part of this wave. Some time next week I’ll be blogging about the Marten event subscription capabilities that were built into Marten & Wolverine to support this client as well. The point being, JasperFx is wide open for business and we can help your shop succeed with challenging project work!
Wolverine now has the ability to support strict messaging order with its message listeners. Given any random listening endpoint in Wolverine, just add this directive below to make the message processing be strictly sequential (with the proviso that your error handling policies may impact the order on failures):
var host = await Host.CreateDefaultBuilder().UseWolverine(opts =>
{
opts.UseRabbitMq().EnableWolverineControlQueues();
opts.PersistMessagesWithPostgresql(Servers.PostgresConnectionString, "listeners");
opts.ListenToRabbitQueue("ordered")
// This option is available on all types of Wolverine
// endpoints that can be configured to be a listener
.ListenWithStrictOrdering();
}).StartAsync();
Some notes about the ListenWithStrictOrdering() directive you might have:
It’s supported with every external messaging broker that Wolverine supports, including Kafka, Azure Service Bus, AWS SQS, and Rabbit MQ. It is also supported with the two database backed transports (we have both kinds, Sql Server and PostgreSQL!)
When this directive is applied, Wolverine will only make the listener for each endpoint (in the case above, the Rabbit MQ named “ordered”) be active on a single node within your application. Today that distribution is just crudely spreading out the “exclusive listeners” evenly across the whole application cluster. Definitely note that the strict ordering comes at the cost of reduced throughput, so use this feature wisely! Did I mention that JasperFx Software is here and ready to work with your company on Critter Stack projects?
Every exclusive listener will quickly start up on a single node if WolverineOptions.Durability.Mode = DurabilityMode.Solo, and you may want to do that for local testing and development just to be a little quicker on cold starts
The ListenWithStrictOrdering will make the internal worker queue (Wolverine uses an internal TPL Dataflow ActionBlock in these cases) for “buffered” or “durable” endpoints be strictly sequential
You will have to have a durable message store configured for your application in order for Wolverine to perform the leadership election and “agent tracking” (what’s running where)
Summary
This is a powerful tool in the continually growing Wolverine tool belt. The strict ordering may also be used to alleviate some concurrency issues that some users have hit with event sourcing using Marten when a single stream may be receiving bursts of commands that impact the event stream. The leadership election and agent distribution in Wolverine, in conjunction with this “sticky” listener assignment, gives Wolverine a nascent ability for virtual actors that we will continue to exploit. More soon-ish!
Wolverine just got a new PostgreSQL-backed messaging transport (with the work sponsored by a JasperFx Software client!). The use case is just this, say you’re already using Wolverine to build a system with PostgreSQL as your backing database, and want to introduce some asynchronous, background processing in your system — which you could already do with just a database backed, local queue. Going farther though, let’s say that we’d like to have a competing consumers setup for our queueing for load balancing between active nodes and we’d like to do that without having to introduce some kind of new message broker infrastructure into our existing architecture.
That’s time to bring in Wolverine’s new option for asynchronous messaging just using our existing PostgreSQL database. To set that up by itself (without using Marten, but we’ll get to that in a second), it’s these couple lines of code:
var builder = WebApplication.CreateBuilder(args);
var connectionString = builder.Configuration.GetConnectionString("postgres");
builder.Host.UseWolverine(opts =>
{
// Setting up Postgresql-backed message storage
// This requires a reference to Wolverine.Postgresql
opts.PersistMessagesWithPostgresql(connectionString);
// Other Wolverine configuration
});
Of course, you’d want to setup PostgreSQL queues for Wolverine to send to and to listen to for messages to process. That’s shown below:
using var host = await Host.CreateDefaultBuilder()
.UseWolverine((context, opts) =>
{
var connectionString = context.Configuration.GetConnectionString("postgres");
opts.UsePostgresqlPersistenceAndTransport(connectionString, "myapp")
// Tell Wolverine to build out all necessary queue or scheduled message
// tables on demand as needed
.AutoProvision()
// Optional that may be helpful in testing, but probably bad
// in production!
.AutoPurgeOnStartup();
// Use this extension method to create subscriber rules
opts.PublishAllMessages().ToPostgresqlQueue("outbound");
// Use this to set up queue listeners
opts.ListenToPostgresqlQueue("inbound")
.CircuitBreaker(cb =>
{
// fine tune the circuit breaker
// policies here
})
// Optionally specify how many messages to
// fetch into the listener at any one time
.MaximumMessagesToReceive(50);
}).StartAsync();
And that’s that, we’re completely set up for messaging via the PostgreSQL database we already have with our Wolverine application!
Just a couple things to note before you run off and try to use this:
Like I alluded to earlier, the PostgreSQL queueing mechanism supports competing consumers, so different nodes at runtime can be pulling and processing messages from the PostgreSQL queues
There is a separate set of tables for each named queue (one for the actual inbound/outbound messages, and a separate table to segregate “scheduled” messages). Utilize that separation for better performance as needed by effectively sharding the message transfers
As that previous bullet point implies, the PostgreSQL transport is able to support scheduled message delivery
As in most cases, Wolverine is able to detect whether or not the necessary tables all exist in your database, and create any missing tables for you at runtime
There’s some optimizations and integration between these queues and the transactional inbox/outbox support in Wolverine for performance by reducing database chattiness whenever possible
Summary
I’m not sure I’d recommend this approach over dedicated messaging infrastructure for high volumes of messages, but it’s a way to get things done with less infrastructure in some cases and it’s a valuable tool in the Wolverine toolbox.
JasperFx Software is open for business and offering consulting services (like helping you craft modular monolith strategies!) and support contracts for both Marten and Wolverine so you know you can feel secure taking a big technical bet on these tools and reap all the advantages they give for productive and maintainable server side .NET development.
I’ve been thinking, discussing, and writing a bit lately about the whole “modular monolith” idea, starting with Thoughts on “Modular Monoliths” and continuing onto Actually Talking about Modular Monoliths. This time out I think I’d like to just put out some demos and thoughts about where Marten and Wolverine fit well into the modular monolith idea — and also some areas where I think there’s room for improvement.
First off, let’s talk about…
Modular Configuration
Both tools use the idea of a “configuration model” (what Marten Fowler coined a Semantic Model years ago) that is compiled and built from a combination of attributes in the code, explicit configuration, user supplied policies, and built in policies in baseline Marten or Wolverine as shown below with an indication of the order of precedence:
In code, when you as a user configure Marten and Wolverine inside of the Program file for your system like so:
builder.Services.AddMarten(opts =>
{
var connectionString = builder.Configuration.GetConnectionString("marten");
opts.Connection(connectionString);
// This will create a btree index within the JSONB data
opts.Schema.For<Customer>().Index(x => x.Region);
})
// Adds Wolverine transactional middleware for Marten
// and the Wolverine transactional outbox support as well
.IntegrateWithWolverine();
builder.Host.UseWolverine(opts =>
{
opts.CodeGeneration.TypeLoadMode = TypeLoadMode.Static;
// Let's build in some durability for transient errors
opts.OnException<NpgsqlException>().Or<MartenCommandException>()
.RetryWithCooldown(50.Milliseconds(), 100.Milliseconds(), 250.Milliseconds());
// Shut down the listener for whatever queue experienced this exception
// for 5 minutes, and put the message back on the queue
opts.OnException<MakeBelieveSubsystemIsDownException>()
.PauseThenRequeue(5.Minutes());
// Log the bad message sure, but otherwise throw away this message because
// it can never be processed
opts.OnException<InvalidInputThatCouldNeverBeProcessedException>()
.Discard();
// Apply the validation middleware *and* discover and register
// Fluent Validation validators
opts.UseFluentValidation();
// Automatic transactional middleware
opts.Policies.AutoApplyTransactions();
// Opt into the transactional inbox for local
// queues
opts.Policies.UseDurableLocalQueues();
// Opt into the transactional inbox/outbox on all messaging
// endpoints
opts.Policies.UseDurableOutboxOnAllSendingEndpoints();
// Connecting to a local Rabbit MQ broker
// at the default port
opts.UseRabbitMq();
// Adding a single Rabbit MQ messaging rule
opts.PublishMessage<RingAllTheAlarms>()
.ToRabbitExchange("notifications");
opts.LocalQueueFor<TryAssignPriority>()
// By default, local queues allow for parallel processing with a maximum
// parallel count equal to the number of processors on the executing
// machine, but you can override the queue to be sequential and single file
.Sequential()
// Or add more to the maximum parallel count!
.MaximumParallelMessages(10)
// Pause processing on this local queue for 1 minute if there's
// more than 20% failures for a period of 2 minutes
.CircuitBreaker(cb =>
{
cb.PauseTime = 1.Minutes();
cb.SamplingPeriod = 2.Minutes();
cb.FailurePercentageThreshold = 20;
// Definitely worry about this type of exception
cb.Include<TimeoutException>();
// Don't worry about this type of exception
cb.Exclude<InvalidInputThatCouldNeverBeProcessedException>();
});
// Or if so desired, you can route specific messages to
// specific local queues when ordering is important
opts.Policies.DisableConventionalLocalRouting();
opts.Publish(x =>
{
x.Message<TryAssignPriority>();
x.Message<CategoriseIncident>();
x.ToLocalQueue("commands").Sequential();
});
});
The nested lambdas in AddMarten() and UseWolverine() are configuring the MartenOptions and WolverineOptions models respectively (the “configuration model” in that diagram above).
I’m not aware of any one commonly used .NET idiom for building modular configuration, but I do commonly see folks using extension methods for IServiceCollection or IHostBuilder to segregate configuration that’s specific to a single module, and that’s what I think I’d propose. Assuming that we have a module in our modular monolith system for handling workflow around “incidents”, there might be an extension method something like this:
public static class IncidentsConfigurationExtensions
{
public static WebApplicationBuilder AddIncidentModule(this WebApplicationBuilder builder)
{
// Whatever other configuration, services, et al
// we need for just the Incidents module
// Extra Marten configuration
builder.Services.ConfigureMarten(opts =>
{
// I'm just adding an index for a document type within
// this module
opts.Schema.For<IncidentDetails>()
.Index(x => x.Priority);
// Purposely segregating all document types in this module's assembly
// to a separate database schema
opts.Policies.ForAllDocuments(m =>
{
if (m.DocumentType.Assembly == typeof(IncidentsConfigurationExtensions).Assembly)
{
m.DatabaseSchemaName = "incidents";
}
});
});
return builder;
}
}
Which would be called from the overall system’s Program file like so:
var builder = WebApplication.CreateBuilder(args);
builder.AddIncidentModule();
// Much more below...
Between them, the two main Critter Stack tools have a lot of support for modularity through:
Wolverine’s extension and modules allow you to easily import message handlers or HTTP endpoints from other assemblies than the main application assembly
Both Marten and Wolverine allow you to register special extensions in your IoC container that allow you to add additional configuration to both Marten (IConfigureMarten) and Wolverine (IWolverineExtension)
All of the facilities I described above can be used to separate specific configuration for different modules within the module code itself.
Modular Monoliths and Backing Persistence
In every single experience report you’ll ever find about a team trying to break up and modernize a large monolithic application the authors will invariably say that breaking apart the database was the single most challenging task. If we are really doing things better this time with the modular monolith approach, we’d probably better take steps ahead of time to make it easier to extract services by attempting to keep the persistence for each module at least somewhat decoupled from the persistence of other modules.
Going even farther in terms of separation, it’s not unlikely that some modules have quite different persistence needs and might be better served by using a completely different style of persistence than the other modules. Just as an example, one of our current JasperFx Software clients has a large monolithic application where some workflow-centric modules would be a good fit for an event sourcing approach, while other modules are more CRUD centric or reporting-centric where a straight up RDBMS approach is probably much more appropriate.
So let’s finally bring Marten and Wolverine into the mix and talk about the Good, the Bad, and the (sigh) Ugly of how the Critter Stack fits into modular monoliths:
wah, wah, wah…
Let’s start with a positive. Marten sits on top of the very robust PostgreSQL database. So in addition to Marten’s ability to use PostgreSQL as a document database and as an event store, PostgreSQL out of the box is a rock solid relational database. Heck, PostgreSQL even has some ability to be used as either a graph database! The point is that using the Marten + PostgreSQL combination gives you a lot of flexibility in terms of persistence style between different modules in a modular monolith without introducing a lot more infrastructure. Moreover, Wolverine can happily utilize its PostgreSQL-backed transactional outbox with both Entity Framework Core and Marten targeting the same PostgreSQL database in the same application.
Continuing with another positive, let’s say that we want to create some logical separation between our modules in the database, and one way to do so would be to simply keep Marten documents in separate database schemas for each module. Repeating a code sample from above, you can see that configuration below:
public static WebApplicationBuilder AddIncidentModule(this WebApplicationBuilder builder)
{
// Whatever other configuration, services, et al
// we need for just the Incidents module
// Extra Marten configuration
builder.Services.ConfigureMarten(opts =>
{
// Purposely segregating all document types in this module's assembly
// to a separate database schema
opts.Policies.ForAllDocuments(m =>
{
if (m.DocumentType.Assembly == typeof(IncidentsConfigurationExtensions).Assembly)
{
m.DatabaseSchemaName = "incidents";
}
});
});
return builder;
}
So, great, the storage for Marten documents could easily be segregated by schema. Especially considering there’s little or no referential integrity relationships between Marten document tables, it should be relatively easy to move these document tables to completely different databases later!
And with that, let’s move more into “Bad” or hopefully not too “Ugly” territory.
The event store data in Marten is all in one single set of tables (mt_streams and mt_events). So every module utilizing Marten’s event sourcing could be intermingling their events in just these tables through the one single, AddMarten() store for the application’s IHost. You could depend on marking event streams by their aggregate type like so:
public static async Task start_stream(IDocumentSession session)
{
// the Incident type argument is strictly a marker for
// Marten
session.Events.StartStream<Incident>(new IncidentLogged());
await session.SaveChangesAsync();
}
I think we could ameliorate this situation with a couple future changes:
A new flag in Marten that would make it mandatory to mark every new event stream with an aggregate type specifically to make it easier to separate the events later to extract a service and its event storage
Some kind of helper to move event streams from one database to another. It’s just not something we have in our tool belt at the moment
Of course, it would also help immeasurably if we had a way to split the event store storage for different types of event streams, but somehow that idea has never gotten any traction within Marten and never rises to the level of a high priority. Most of our discussions about sharding or partitioning the event store data has been geared around scalability — which is certainly an issue here too of course.
Marten also has its concept of “separate stores” that was meant to allow an application to interact with multiple Marten-ized databases from a single .NET process. This could be used with modular monoliths to segregate the event store data, even if targeting the same physical database in the end. The very large downside to this approach is that Wolverine’s Marten integration does not today do anything with the separate store model. So no Wolverine transactional middleware, event forwarding, transactional inbox/outbox integration, and no aggregate handler workflow. So basically everything about the full “Critter Stack” integration that makes that tooling the single most productive event sourcing development experience in all of .NET (in my obviously biased opinion). Ugly.
Randomly, I heard an NPR interview with Eli Wallach very late in his life who was the actor who played the “Ugly” character in the famous western, and I could only describe him as effusively jolly. So basically a 180 degree difference from his character!
Module to Module Communication
I’ve now spent much more time on this post than I had allotted, so it’s time to go fast…
In my last post I used this diagram to illustrate the risk of coupling modules through direct usage of internals (the red arrows):
Instead of the red arrows everywhere above, I think I’m in favor of trying to limit the module to module communication to using some mix of a “mediator” tool or an in memory message bus between modules. That’s obviously going to come with some overhead, but I think (hope) that overhead is a net positive.
For a current client, I’m recommending they further utilize MediatR as they move a little more in the direction of modularity in their current monolith. For greenfield codebases, I’d recommend Wolverine instead because I think it does much, much more.
First, Wolverine has a full set of functionality to be “just a Mediator” to decouple modules from too much of the internals of another module. Secondly, Wolverine has a lot of support for background processing through local, in memory queues that could be very advantageous in modular monoliths where Wolverine can de facto be an in memory message bus. Moreover, Wolverine’s main entry point usage is identical for messages processed locally versus messages published through external messaging brokers to external processes:
public static async Task using_message_bus(IMessageBus bus)
{
// Use Wolverine as a "mediator"
// This is normally executed inline, in process, but *could*
// also be invoking this command in an external process
// and waiting for the success or failure ack
await bus.InvokeAsync(new CategoriseIncident());
// Use Wolverine for asynchronous messaging. This could
// start by publishing to a local, in process queue, or
// it could be routed to an external message broker -- but
// the calling code doesn't have to know that
await bus.PublishAsync(new CategoriseIncident());
}
The point here is that Wolverine can potentially set your modular monolith architecture up so that it’s possible to extract or move functionality out into separate services later.
All that being said about messaging or mediator tools, some of the ugliest systems I’ve ever seen utilized messaging or proto-Mediatr command handlers between logical modules. Those systems had code that was almost indecipherable by introducing too many layers and far too much internal messaging. I think I’d say that some of the root cause of the poor system code was from getting the bounded context boundaries wrong so that the messaging was too chatty. Using high ceremony anti-corruption layers also adds a lot of mental overhead to follow information flow through several mapping transformations. One of these systems was using the iDesign architectural approach that I think naturally leads to very poorly factored software architectures and too much harmful code ceremony. I do not recommend.
I guess my only point here is that no matter what well intentioned advice people like me try to give, or any theory of how to make code more maintainable any of us might have, if you find yourself saying to yourself about code that “this is way harder than it should be” you should challenge the approach and look for something different — even if that just leads you right back to where you are now if the alternatives don’t look any better.
One important point here about both modular monoliths or a micro service strategy or a mix of the two: if two or more services/modules are chatty between themselves and very frequently have to be modified at the same time, they’re best described as a single bounded context and should probably be combined into a single service or module.
Summary
Anyway, that’s enough from me on this subject for now, and this took way longer than I meant to spend on it. Time to get my nose back to the grindstone. I am certainly very open to any feedback about the Critter Stack tools limitations for modular monolith construction and any suggestions or requests to improve those tools.
In that previous post though, the messages held in those in memory, local queues could conceivably be lost if the application is shut down unexpectedly (Wolverine will attempt to “drain” the local queues of outstanding work on graceful process shutdowns). That’s perfectly acceptable sometimes, but in other times you really need those queued up messages to be durable so that the in flight messages can be processed even if the service process is unexpectedly killed while work is in flight — so let’s opt into Wolverine’s ability to do exactly that!
To that end, let’s just assume that we’re a very typical .NET shop and we’re already using Sql Server as our backing database for the system. Knowing that, let’s add a new Nuget reference to our project:
dotnet add package WolverineFx.SqlServer
And let’s break into our Program file for the service where all the system configuration is, and expand the Wolverine configuration within the UseWolverine() call to this:
// This is good enough for what we're trying to do
// at the moment
builder.Host.UseWolverine(opts =>
{
// Just normal .NET stuff to get the connection string to our Sql Server database
// for this service
var connectionString = builder.Configuration.GetConnectionString("SqlServer");
// Telling Wolverine to build out message storage with Sql Server at
// this database and using the "wolverine" schema to somewhat segregate the
// wolverine tables away from the rest of the real application
opts.PersistMessagesWithSqlServer(connectionString, "wolverine");
// In one fell swoop, let's tell Wolverine to make *all* local
// queues be durable and backed up by Sql Server
opts.Policies.UseDurableLocalQueues();
});
Nothing else in our previous code needs to change. As a matter of fact, once you restart your application — assuming that your box can reach the Sql Server database in the appsettings.json file — Wolverine is going to happily see that those necessary tables are missing, and build them out for you in your database so that Wolverine “can just work” on its first usage. That automatic schema creation can of course be disabled and/or done with pure SQL through other Wolverine facilities, but for right now, we’re taking the easy road.
Before I get into the runtime mechanics, here’s a refresher about our first message handler:
public static class SendWelcomeEmailHandler
{
public static void Handle(SignUpRequest @event, ILogger logger)
{
// Just logging, a real handler would obviously do something real
// to send an email
logger.LogInformation("Send a Send a welcome email to {Name} at {Email}", @event.Name, @event.Email);
}
}
And the code that publishes a SignUpRequest message to a local Wolverine queue in a Minimal API endpoint:
After our new configuration up above to add message durability to our local queues, when a service client posts a SignUpRequest message is published to Wolverine as a result of a client posting valid data to the /signup Url, Wolverine will:
Persist all the necessary information about the new SignUpRequest message that Wolverine uses to process that message in the Sql Server database (this is using the “Envelope Wrapper” pattern from the old EIP book, which is quite originally called Envelope in the Wolverine internals).
If the message is successfully processed, Wolverine will delete that stored record for the message in Sql Server
If the message processing fails, and there’s some kind of retry policy in effect, Wolverine will increment the number of failed attempts in the Sql Server database (with an UPDATE statement because it’s trying to be as efficient as possible)
If the process somehow fails while the message is floating around in the in memory queues, Wolverine will be able to recover that local message from the database storage later when the system is restarted. Or if the system is running in a load balanced cluster, a different Wolverine node will be able to see that the messages are orphaned in the database and will steal that work into another node so that the messages eventually get processed
Summary and What’s Next?
That’s a lot of detail about what is happening in your system, but I’d argue that was very little code necessary to make the background processing with Wolverine be durable. And all without introducing any other new infrastructure other than the Sql Server database we were probably already using. Moreover, Wolverine can do a lot to make the necessary database setup for you at runtime so there’s hopefully very little friction getting up and running after a fresh git clone.
I’ll add at least a couple more entries to this series by looking at error handling strategies, controller the parallelism or strict ordering of message processing, a simple implementation of the Producer/Consumer pattern with Wolverine, and message scheduling.
That’s Winterfell from a Game of Thrones if you were curious. I have no earthly idea whether or not this mini-series of posts will be the slightest bit useful for anyone else, but it’s probably been good for me to read up much more on what other people think and to just flat out ponder this a lot more as it’s suddenly relevant to several different current JasperFx Software clients.
In my last post Thoughts on “Modular Monoliths” I was starting write a massive blog post about my reservations about modular monoliths and where and how the “Critter Stack” tools (Marten and Wolverine) are or aren’t already well suited for modular monolith projection construction — but I really only got around to talking about the problems with both traditional monolith structures and micro-service architectures before running out of steam. This post is to finally lay out my thoughts on “modular monoliths” with yet another 3rd post coming later to get into some specifics about how the “Critter Stack” tools (Marten and Wolverine) fit into this architectural style. I do think even the specifics of the Critter Stack tooling will help illuminate some potential challenges for folks building modular monolith systems with completely different tooling.
So let’s get started by me saying that my conceptual understanding of a “modular monolith” is that it’s a single codebase like we’ve traditionally done and consistently failed at, but this time we’ll pay more attention to isolating logical modules of functionality within that single codebase:
As I said in the earlier post, I’m very dubious about how effective the modular monolith strategy will really be for our industry because I think it’s still going to run into some very real friction. Do keep in mind that I’m coming from a primarily .NET background, and that means there’s no easy way to run multiple versions of the same library in the same process. That being said, here’s what else I’m still worried about:
You may still suffer from the same kind of friction with slow builds and sluggish IDEs you almost inevitably get from large codebases if you have to work with the whole codebase at one time — but maybe that’s not actually a necessity all the time, so let’s keep talking!
I really think it’s important for the long term health of a big system to be able to do technical upgrades or switch out technologies incrementally instead of having to upgrade the whole codebase at one time because you can never, ever (or at least very rarely) convince a non-technical product owner to let you stop and take six months to upgrade technologies without adding any new features — nor honestly should you even try to do that without some very good reasons
It’s still just as hard to find the right boundaries between modules as it was to make the proper boundaries for truly independent micro-services
A massive pet peeve of mine is hearing people exclaim something to the effect of “just use Domain Driven Design and your service and/or module boundaries will always be right!” I think the word “just” is doing a lot of work there and anybody who says that should read the classic essay Lullaby Language.
After writing my previous blog post, working on the proposals for a client that spawned this conversation in the first place, and reading up on a lot of other people’s thoughts about this subject, I’ve got a few more positive things to say.
I’m much more bullish on modular monoliths after thinking more about the Citadel and Outpost approach where you start with the assumption that some modules of the monolith will be spun out into separate processes later when the team feels like the service boundaries are truly settled enough to make that viable. To that end, I liked the way that Glenn Henriksen put this over the weekend:
I wouldn't start building a modular monolith without tinking about an exit strategy and not be afraid to use it. For instance when Conway starts laying down the law of the land.
Continuing the “Citadel” theme where you assume that you will later spawn separate “Outpost” processes later, I’m now highly concerned with building the initial system in such a way that it’s relatively easy to excise out modules to separate processes later. In the early days of Extreme Programming, we talked a little bit about the concept of “Reversibility“, which just means how easy or hard it will be to change your mind and technical direction about any given technology or approach. Likewise with the “modular monolith” approach, I actually want to think a little bit upfront about having a path to easily break out modules into separate processes or services later.
I’m probably a little more confident about introducing some level of asynchronous messaging and distributed development than some folks, so I’m going to come right out and say that I would be willing to start with some modules split into separate processes right off the bat, but ameliorate that by assuming that all these processes will have to be deployed together and will live together in one single mono-repository. To circle back to the earlier “Reversibility” theme, I think this compromise will make it much easier for teams to adjust service boundaries later as everything will be living in the same repository.
Lastly on this topic, it’s .NET-centric, but I’m hopeful that Project Aspire makes it much easier to work with this kind of distributed monolith. Likewise, I’m keeping an eye on tooling like Incrementalist as a way of better working with mono-repository codebases.
What about the Database?
There are potential dangers that might make our modular monolith decision less reversible than we’d like, but for right now let’s focus on just the database (or databases). Specifically, I’m concerned about avoiding what I’ve called the Pond Scum Anti-Pattern where a lot of different applications (or modules in our new world) float on top of a murky shared database like pond scum on brackish water in sweltering summer heat.
I grew up on farms fishing in that exact kind of farm pond, hence the metaphor:)
Taking the ideal from micro-services, I’d love it if each module had logically separate database storage such that even if they are all targeting the same physical database server, there’s some relatively easy way later to pull out the storage for an individual module and move it out later.
If scalability was an issue, I would happily go for breaking the storage for some modules out into separate databases, even though that’s a little more complexity. Some of the other folks I read in researching this topic suggested using foreign data wrappers to offload database work while still making it look to your modular monolith like it’s one big happy family database — but I personally think that’s crazy town. There’s also a very real benefit to allowing different modules to use different styles of databases or persistence based on their needs.
This probably won’t happen, but I did at least raise the possibility to a client of using event sourcing in some of their workflow-centric modules while allowing simpler modules to be remain CRUD-centric.
How Do the Modules Stay Decoupled?
Assuming we all buy off into the idea of our modules remaining loosely coupled over time such that we have a pathway to pull them out into separate “Outpost” processes later, we absolutely don’t want many of the red arrows popping up as shown below:
Mechanically, my first inclination to enforce the modularity is to say that we’ll use some kind of mediator tooling like either MediatR or my own Wolverine to handle cross module interactions. That comes with its own set of complications:
Potentially more code as is almost inevitable when purposely putting in any kind of anti-corruption layer
What to do when one module needs the data from a second module to do its work? One answer is to use Domain Queries between modules — again, probably with some kind of mediator tool. I’ve always been dubious about that strategy because of the extra code ceremony and the simple fact that any possible technique that adds abstractions between your top level code and the raw data access code has a high tendency to cause performance problems later. If you go down this path, I’d be cognizant of the potential performance penalties and look maybe for some way to batch up queries later
You potentially just say that if “Module 1 consistently needs access to data managed by Module 2” then you should probably merge the two modules. One fast way to get into trouble in any kind of complicated system is to organize first by different logical persisted entities rather than by operations. I think you’re far more likely to arrive at cleanly separate module boundaries by focusing on the command and query use cases of the system rather than dividing code up by entities like “Invoice” or “Order” or “Shipment.”
Just for now, I think there’s one last conversation to have about how a team will go about enforcing the usage of proper patterns and encapsulation of the various modules without devolving into a morass of the red arrows from the picture above. You could:
Favor internalized discipline and socialized design goals by doing whatever it takes to be able to trust the developers to naturally do the right thing. Kumbaya, up with people, stop laughing at me! I think that internalized discipline will deliver better results than high ceremony approaches that try to straight jacket developers into doing the right things, but I’m prepared to be wrong on this one
Utilize architectural tests or maybe some kind of fancy static code analysis that can spot violations of the architectural “who can talk to who” rules
Try to separate out projects or packages for modules or parts of modules to enforce rules about “who can talk to who.” I hate this approach as part of something like the Onion Architecture, and I’m probably naturally suspicious of it inside of modular monoliths too — but at least this time you’re hopefully dividing along the lines of closely related functionality rather than organizing by broad layers first.
Summary and What’s Next
My only summary is that I’m still dubious that the modular monolith idea is going to be a panacea, but this has been helpful to me personally just to think on it much harder and see what other folks are doing and saying about this architectural style.
My next and hopefully last post in this series will be taking a look at how Wolverine and Marten do or do not lend themselves to the modular monolith approach, and what might need to be improved later in these tools.
TL;DR -> I’m dubious about whether or not the currently popular “modular monolith” idea will actually pay off, but first let’s talk about why we got to this point
The pendulum in software development can frequently swing back and forth between alternatives or extremes as the community struggles to achieve better results or becomes disillusioned with one popular approach or another. It’s easy — especially for older developers like me — to give into cynicism about the hot new idea being exactly the same as something from 5-10 years earlier. That cynicism isn’t necessarily helpful at all because there are often very real and important differences in the new approach that can easily be overlooked if you are jumping to making quick comparisons between the new thing and something familiar.
I’m working with several JasperFx Software clients right now that are either wrestling with brownfield systems they wish were more modular or getting started on large efforts where they can already see the need for modularity. Quite naturally, that brings the concept of “modular monoliths” to the forefront of my mind as a possible approach to creating more maintainable software for my clients.
What Do We Want?
Before any real discussion about the pendulum swinging from monoliths to micro-services and back to modular monoliths, let’s talk a bit about what I think any given software development team really wants from their large codebase:
The codebase is very easy to “clone n’ go”, meaning that there’s very little friction in being able to configure your development environment to run the code and test suites
Fast builds, fast test runs, snappy IDE performance. Call it whatever you want, but I want developers to always be working with quick feedback cycles
The code is easy to reason about. This is absolutely vital in larger, long lived codebases as a way to avoid causing regression bugs that can easily inflict larger systems. It’s also important just to keep teams productive over time
The ability to upgrade or even replace technologies over time without requiring risky major projects just to focus on technology upgrades that your business folks are very loathe to ever approve
This one might be just me, but I want a minimum of repetitive code ceremony within the codebase. I’ve routinely seen both monolithic codebases and micro-service strategies fail in some part because of too much repetitive code ceremony cluttering up the code and slowing down development work. The “Critter Stack” tools (Marten & Wolverine) are absolutely built with this low ceremony mantra in mind.
Any given part of the codebase should be small enough that a single “two pizza” team should be able to completely own that part of the code
Some Non-Technical Stuff That Matters
I work under the assumption that most of us are really professionals and do care about trying to do good work. That being said, even for skilled developers who absolutely care about the quality of their work, there are some common organizational problems that lead to runaway technical debt:
Micromanagement – Micromanagement crushes anybody’s sense of ownership or incentive to innovate, and developers are no different. As a diehard fan of early 2000’s idealistic Extreme Programming, I of course blame modern Scrum. Bad technical leads or architects can certainly hurt as well though. Even as the most senior technical person on a team, you have to be listening to the concerns of every other team member and allowing new ideas to come from anybody
Overloaded teams – I think that keeping teams running at near their throughput capacity on new features over timeinevitably leads to overwhelming technical debt that can easily dehabilitate future work in that system. The harsh reality that many business folks don’t understand is that it’s important in the long run for development teams to have some slack time for experimentation or incremental technical debt reduction tasks
Lack of Trust – This is probably closely related or the root cause of micromanagement, but teams are far more effective when there is a strong trust relationship between the technical folks and the business. Technical teams need to be able to communicate the need to occasionally slow down on feature work to address technical concerns, and have their concerns taken seriously by the business. Of course, as technical folks, we need to work on being able to communicate our concerns in terms of business impact and always seek to maintain the trust level from the business.
Old Fashioned Monoliths
Let’s consider the past couple swings of the pendulum. First there was the simple concept of building large systems as one big codebase. The “monolith” we all fear, even though none of us likely set out to purposely create one. Doing some research today, I saw people describe old fashioned monoliths as problematic because they were in Brian Foote and Joseph Yoder’s immortal words:
A BIG BALL OF MUD is haphazardly structured, sprawling, sloppy, duct-tape and bailing wire, spaghetti code jungle.
My recent, very negative experiences with monolithic applications has been quite different though. What I’ve seen is that these monoliths had a consistent structure and clear architectural philosophy, but that the very ideas the teams had originally adopted to help keep the system maintainable were probably part of the causes for why their monolithic application was riddled with technical debt. In my opinion, I think these monolithic codebases have been difficult to work with because of:
The usage of prescriptive architectures like Clean Architecture or Onion Architecture solution templates that alternatively overcomplicated or failed to manage complexity over time because the teams did not deviate from the prescriptions
Overly relying on Ports and Adapters type thinking that led teams to introduce many more abstractions as the system grew, and that often leads to code being hard to follow or reason about. Also leads to potentially bad performance out of the sheer number of objects being created. Absolutely leads to poor performance when teams are not able to easily reason about their code’s interactions with databases because of the proliferation of abstractions
The predominance of layered architecture thinking in big systems means that closely related code is widely spread out over a big codebase
The difficulty in upgrading technologies over a monolithic codebase sheerly out of the size of the effort — and I partially blame the proliferation of common base types and marker interfaces promoted in the prescriptive Clean/Onion/Hexagonal/Ports & Adapters style architectural guidance
I hit these themes at NDC Oslo 2023 in this talk if you’re interested:
Alright, I think we can all agree on the pain of monolithic applications without enough structure, and we can agree to disagree about my negative opinions about Clean et al Architectures, but let’s move on to micro-services.
Obvious Problems with Micro-Services
I’m still bullish on the long term usefulness of micro-services — or really just “not massive services that contain just one cohesive bounded context and can actually run mostly independent.” But, let’s just acknowledge the very real downsides of micro-services that folks are running into:
Distributed development is hard, full stop. And micro-services inevitably mean more distributed development, debugging, and monitoring
It’s hard to get the service boundaries right in a complex business system, and it’s disastrous when you don’t get the boundaries right. If your team continuously finds itself having to frequently change multiple services at the same time, your boundaries are clearly wrong and you’re doing shotgun surgery — but it’s worse because you may be having to make changes in completely separate codebases and code repositories.
Again, if the boundaries aren’t quite right, you can easily get into a situation where the services have to be chatty with each other and send a lot more messages. That’s a recipe for poor performance and brittle systems.
Testing can be a lot harder if you absolutely need to do more testing between services rather than being able to depend on testing one service at a time
I’ve also seen high ceremony approaches completely defeat micro-service strategies by simply adding too much overhead to splitting up a codebase to gain any net value from splitting up the previous monolith. Again, I’m team low ceremony in almost any circumstance.
So what about Modular Monoliths?
Monoliths have been problematic, then micro-services turned out to be differently problematic. So let’s swing the pendulum back partway but focus more on making our monoliths modular for easier, more maintainable long term development. Great theory, and it’s spawning a lot of software conference talks and sage chin wagging.
This gets us to “Modular Monolith” idea that’s popular now, but I’m unfortunately dubious about the mechanics and whether or not this is just some “modular” lipstick on the old “monolith” pig.
In my next post, I’m going to try to describe my concerns and thoughts about how a “modular monolith” architecture might actually work out. I’m also concerned about how well both Marten and Wolverine are going to play within a modular monolith, and I’d like to get into some nuts and bolts about how those tools work now and how they maybe need to change to better accommodate the “modular monolith” idea.
Hey, did you know that JasperFx Software offers formal support plans for Marten and Wolverine? Not only are we making the “Critter Stack” tools be viable long term options for your shop, we’re also interested in hearing your opinions about the tools and how they should change.We’re also certainly open to help you succeed with your software development projects on a consulting basis whether you’re using any part of the Critter Stack or some completely different .NET server side tooling.
First off, let’s say that you have a simplistic document that can “self-aggregate” itself as a “Snapshot” in Marten like this:
public record InvoiceCreated(string Description, decimal Amount);
public record InvoiceApproved;
public record InvoiceCancelled;
public record InvoicePaid;
public record InvoiceRejected;
public class Invoice
{
public Invoice()
{
}
public static Invoice Create(IEvent<InvoiceCreated> created)
{
return new Invoice
{
Amount = created.Data.Amount,
Description = created.Data.Description,
// Capture the timestamp from the event
// metadata captured by Marten
Created = created.Timestamp,
Status = InvoiceStatus.Created
};
}
public int Version { get; set; }
public decimal Amount { get; set; }
public string Description { get; set; }
public Guid Id { get; set; }
public DateTimeOffset Created { get; set; }
public InvoiceStatus Status { get; set; }
public void Apply(InvoiceCancelled _) => Status = InvoiceStatus.Cancelled;
public void Apply(InvoiceRejected _) => Status = InvoiceStatus.Rejected;
public void Apply(InvoicePaid _) => Status = InvoiceStatus.Paid;
public void Apply(InvoiceApproved _) => Status = InvoiceStatus.Approved;
}
For asynchronous projections of any kind, we have a little bit of complication for testing. In a classic “Arrange, Act, Assert” test workflow, we’d like to exercise our projection — and mind you, I strongly recommend that testing happen within its integration with Marten rather than some kind of solitary unit tests with fakes — with a workflow like this:
Pump in some new events to Marten
Somehow magically wait for Marten’s asynchronous daemon running in a background thread progress to the point where it’s handled all of our newly appended events for all known, running projections
Load the expected documents that should have been persisted or updated from our new events by the projections running in the daemon, and run some assertions on the expected system state
For right now, I want to worry about the second bullet point and introduce a new (old, but it actually works correctly now) WaitForNonStaleProjectionDataAsync API introduced in Marten 7.5. You can see the new API used in this test from the new documentation on Testing Projections:
[Fact]
public async Task test_async_aggregation_with_wait_for()
{
// In your tests, you would most likely use the IHost for your
// application as it is normally built
using var host = await Host.CreateDefaultBuilder()
.ConfigureServices(services =>
{
services.AddMarten(opts =>
{
opts.Connection(
"Host=localhost;Port=5432;Database=marten_testing;Username=postgres;password=postgres;Command Timeout=5");
opts.DatabaseSchemaName = "incidents";
// Notice that the "snapshot" is running inline
opts.Projections.Snapshot<Invoice>(SnapshotLifecycle.Async);
})
// Using Solo in tests will help it start up a little quicker
.AddAsyncDaemon(DaemonMode.Solo);
}).StartAsync();
var store = host.Services.GetRequiredService<IDocumentStore>();
var invoiceId = Guid.NewGuid();
// Pump in events
using (var session = store.LightweightSession())
{
session.Events.StartStream<Invoice>(invoiceId, new InvoiceCreated("Blue Shoes", 112.24m));
await session.SaveChangesAsync();
session.Events.Append(invoiceId,new InvoiceApproved());
session.Events.Append(invoiceId,new InvoicePaid());
await session.SaveChangesAsync();
}
// Now, this is going to pause here in this thread until the async daemon
// running in our IHost is completely caught up to at least the point of the
// last event captured at the point this method was called
await store.WaitForNonStaleProjectionDataAsync(5.Seconds());
// NOW, we should expect reliable results by just loading the already
// persisted documents built by rebuilding the projection
await using var query = store.QuerySession();
// Load the document that was "projected" from the events above
// and immediately persisted to the document store
var invoice = await query.LoadAsync<Invoice>(invoiceId);
// Run assertions
invoice.Description.ShouldBe("Blue Shoes");
invoice.Status.ShouldBe(InvoiceStatus.Paid);
}
In the example projection, I’ve been capturing the timestamp in the Invoice document from the Marten event metadata:
public static Invoice Create(IEvent<InvoiceCreated> created)
{
return new Invoice
{
Amount = created.Data.Amount,
Description = created.Data.Description,
// Capture the timestamp from the event
// metadata captured by Marten
Created = created.Timestamp,
Status = InvoiceStatus.Created
};
}
But of course, if that timestamp has some meaning later on and you have any kind of business rules that may need to key off that time, it’s very helpful to be able to control the timestamps that Marten is assigning to create predictable automated tests. As of Marten 7.5, Marten uses the newer .NET TimeProvider behind the scenes, and you can replace it in testing like so:
[Fact]
public async Task test_async_aggregation_with_wait_for_and_fake_time_provider()
{
// Hang on to this for later!!!
var eventsTimeProvider = new FakeTimeProvider();
// In your tests, you would most likely use the IHost for your
// application as it is normally built
using var host = await Host.CreateDefaultBuilder()
.ConfigureServices(services =>
{
services.AddMarten(opts =>
{
opts.Connection(
"Host=localhost;Port=5432;Database=marten_testing;Username=postgres;password=postgres;Command Timeout=5");
opts.DatabaseSchemaName = "incidents";
// Notice that the "snapshot" is running inline
opts.Projections.Snapshot<Invoice>(SnapshotLifecycle.Async);
opts.Events.TimeProvider = eventsTimeProvider;
})
// Using Solo in tests will help it start up a little quicker
.AddAsyncDaemon(DaemonMode.Solo);
}).StartAsync();
var store = host.Services.GetRequiredService<IDocumentStore>();
var invoiceId = Guid.NewGuid();
// Pump in events
using (var session = store.LightweightSession())
{
session.Events.StartStream<Invoice>(invoiceId, new InvoiceCreated("Blue Shoes", 112.24m));
await session.SaveChangesAsync();
session.Events.Append(invoiceId,new InvoiceApproved());
session.Events.Append(invoiceId,new InvoicePaid());
await session.SaveChangesAsync();
}
// Now, this is going to pause here in this thread until the async daemon
// running in our IHost is completely caught up to at least the point of the
// last event captured at the point this method was called
await store.WaitForNonStaleProjectionDataAsync(5.Seconds());
// NOW, we should expect reliable results by just loading the already
// persisted documents built by rebuilding the projection
await using var query = store.QuerySession();
// Load the document that was "projected" from the events above
// and immediately persisted to the document store
var invoice = await query.LoadAsync<Invoice>(invoiceId);
// Run assertions, and we'll use the faked timestamp
// from our time provider
invoice.Created.ShouldBe(eventsTimeProvider.Start);
}
In the sample above, I used the FakeTimeProvider from the Microsoft.Extensions.TimeProvider.Testing Nuget package.
Summary
We take testability and automated testing very seriously throughout the entire “Critter Stack.” The testing of asynchronous projections has long been a soft spot that we hope is improved by the new capabilities in this post. As always, feel free to pop into the Critter Stack Discord for any questions.
Hey, did you know that JasperFx Software offers formal support plans for Marten and Wolverine? Not only are we making the “Critter Stack” tools be viable long term options for your shop, we’re also interested in hearing your opinions about the tools and how they should change.We’re also certainly open to help you succeed with your software development projects on a consulting basis whether you’re using any part of the Critter Stack or some completely different .NET server side tooling.
Hey, when you’re building grown up software systems in a responsible way, who likes effective automated testing? Me, too! Moreover, I like automated tests that are reliable — and anyone who has ever been remotely near a large automated test suite testing through the web application layer with any kind of asynchronous behavior knows exactly how painful “flake-y” tests are that suffer from timing issues.
Wolverine of course is an application framework for performing background processing and asynchronous messaging — meaning that there’s no end of the exact kind of asynchronous behavior that is notoriously hard to deal with in automated tests. At a minimum, what you need is a way to exercise the message handling within Wolverine (the “act” in the “act, arrange, assert” test pattern), but wait until all cascading activity is really complete before allowing the automated test to continue making assertions on expected outcomes. Fortunately, Wolverine has that very functionality baked into its core library. Here’s a fake saga that we recently used to fix a bug in Wolverine:
public class LongProcessSaga : Saga
{
public Guid Id { get; init; }
[Middleware(typeof(BeginProcessMiddleware))]
public static (LongProcessSaga, OutgoingMessages) Start(BeginProcess message, RecordData? sourceData = null)
{
var outgoingMessages = new OutgoingMessages();
var saga = new LongProcessSaga
{
Id = message.DataId,
};
if (sourceData is not null)
{
outgoingMessages.Add(new ContinueProcess(saga.Id, message.DataId, sourceData.Data));
}
return (
saga,
outgoingMessages
);
}
public void Handle(ContinueProcess process)
{
Continued = true;
}
public bool Continued { get; set; }
}
When the BeginProcess message is handled by Wolverine, it might also spawn a ContinueProcess message. So let’s write a test that exercises the first message, but waits until the second message that we expect to be spawned while handling the first message before allowing the test to proceed:
[Fact]
public async Task can_compile_without_issue()
{
// Arrange -- and sorry, it's a bit of "Arrange" to get an IHost
var builder = WebApplication.CreateBuilder(Array.Empty<string>());
builder.Services
.AddMarten(options =>
{
options.Connection(Servers.PostgresConnectionString);
})
.UseLightweightSessions()
.IntegrateWithWolverine();
builder.Host.UseWolverine(options =>
{
options.Discovery.IncludeAssembly(GetType().Assembly);
options.Policies.AutoApplyTransactions();
options.Policies.UseDurableLocalQueues();
options.Policies.UseDurableOutboxOnAllSendingEndpoints();
});
builder.Services.AddScoped<IDataService, DataService>();
// This is using Alba, which uses WebApplicationFactory under the covers
await using var host = await AlbaHost.For(builder, app =>
{
app.MapWolverineEndpoints();
});
// Finally, the "Act"!
var originalMessage = new BeginProcess(Guid.NewGuid());
// This is a built in extension method to Wolverine to "wait" until
// all activity triggered by this operation is completed
var tracked = await host.InvokeMessageAndWaitAsync(originalMessage);
// And now it's okay to do assertions....
// This would have failed if there was 0 or many ContinueProcess messages
var continueMessage = tracked.Executed.SingleMessage<ContinueProcess>();
continueMessage.DataId.ShouldBe(originalMessage.DataId);
}
The IHost.InvokeMessageAndWaitAsync() is part of Wolverine’s “tracked session” feature that’s descended from an earlier system some former colleagues and I developed and used at my then employer about a decade ago. The original mechanism was quite successful for our integration testing efforts of the time, and was built into Wolverine quite early. This “tracked session” feature is very heavily used within the Wolverine test suites to test Wolverine itself.
But wait, there’s more! Here’s a bigger sample from the documentation just showing you some more things that are possible:
public async Task using_tracked_sessions_advanced(IHost otherWolverineSystem)
{
// The point here is just that you somehow have
// an IHost for your application
using var host = await Host.CreateDefaultBuilder()
.UseWolverine().StartAsync();
var debitAccount = new DebitAccount(111, 300);
var session = await host
// Start defining a tracked session
.TrackActivity()
// Override the timeout period for longer tests
.Timeout(1.Minutes())
// Be careful with this one! This makes Wolverine wait on some indication
// that messages sent externally are completed
.IncludeExternalTransports()
// Make the tracked session span across an IHost for another process
// May not be super useful to the average user, but it's been crucial
// to test Wolverine itself
.AlsoTrack(otherWolverineSystem)
// This is actually helpful if you are testing for error handling
// functionality in your system
.DoNotAssertOnExceptionsDetected()
// Again, this is testing against processes, with another IHost
.WaitForMessageToBeReceivedAt<LowBalanceDetected>(otherWolverineSystem)
// There are many other options as well
.InvokeMessageAndWaitAsync(debitAccount);
var overdrawn = session.Sent.SingleMessage<AccountOverdrawn>();
overdrawn.AccountId.ShouldBe(debitAccount.AccountId);
}
As hopefully implied by the earlier example, the “tracked session” functionality also gives you:
Recursive tracking of all message activity to wait for everything to finish
Enforces timeouts in case of hanging tests that probably won’t finish successfully
The ability to probe the exact messaging activity that happened as a result of your original message
Visibility into any exceptions recorded by Wolverine during message processing that might otherwise be hidden from you. This functionality will re-throw these exceptions to fail a test unless explicitly told to ignore processing exceptions — which you may very well want to do to test error handling logic
If a test fails because of a timeout, or doesn’t reach the expected conditions, the test failure exception will show you a (hopefully) neatly formatted textual table explaining what it did observe in terms of what messages were sent, received, started, and finished executing. Again, this is to give you more visibility into test failures, because those inevitably do happen!
Last Thoughts
Supporting a complicated OSS tool like Marten or Wolverine is a little bit like being trapped in somewhere in Jurassic Park while the raptors (users, and especially creative users) are prowling around the perimeter of your tool just looking for weak spots in your tools — a genuine bug, a use case you didn’t anticipate, an awkward API, some missing documentation, or even just some wording in your documentation that isn’t clear enough. The point is, it’s exhausting and sometimes demoralizing when raptors are getting past your defenses a little too often just because you rolled out a near complete rewrite of your LINQ provider subsystem:)
Yesterday I was fielding questions from a fellow whose team was looking to move to Wolverine from one of the older .NET messaging frameworks, and he was very complimentary of the integration testing support that’s the subject of this post. My only point here is to remember to celebrate your successes to balance out the constant worry about what’s not yet great about your tool or project or codebase.
And by success, I mean a very important feature that will absolutely help teams build reliable software more productively with Wolverine that does not exist in other .NET messaging frameworks. And certainly doesn’t exist in the yet-to-be-built Microsoft eventing framework where they haven’t even considered the idea of testability.
Hey, did you know that JasperFx Software offers formal support plans for Marten and Wolverine? Not only are we making the “Critter Stack” tools be viable long term options for your shop, we’re also interested in hearing your opinions about the tools and how they should change.We’re also certainly open to help you succeed with your software development projects on a consulting basis whether you’re using any part of the Critter Stack or some completely different .NET server side tooling.
Drama came to the .NET community yet again in the past week or two when more folks noticed that Microsoft is planning to write some kind of “eventing framework” that may or may not end up competing with tools like Wolverine. By and large, that discussion seemed to be purposely designed to make my blood boil, but one comment caught my attention:
I have a fairly simple requirement where I want to fire an event and process that event in the background. A while back, I looked at many of the frameworks mentioned here, and they all seemed so convoluted. For the life of me, I could not figure out how to set up something as simple as this.
So challenge accepted! Starting with this post, I’m going to write another series of small posts to explore some of Wolverine’s rich capabilities for background processing much like the earlier Building a Critter Stack Application series.
Let’s start dirt simple and just build a little ASP.Net Core service that can use Wolverine to do some work in the background spawned by a call into an HTTP endpoint.
Let’s start by creating a new projection like so:
dotnet new webapi
Then add Wolverine — and just the core Wolverine for right now — with:
dotnet add package WolverineFx
For our simplistic service, let’s say that we have an HTTP endpoint that accepts a posted body like this type:
public record SignUpRequest(string Name, string Email);
and wants to send an email to that new user welcoming to our site. And we’d like to have that email sent in a background worker (in this case it’ll be published to a local, in memory queue and eventually run in a different thread from the original HTTP request).
Skipping ahead a little bit to the Wolverine usage, here’s a crude handler for our SignUpRequest using Wolverine:
public static class SendWelcomeEmailHandler
{
public static void Handle(SignUpRequest @event, ILogger logger)
{
// Just logging, a real handler would obviously do something real
// to send an email
logger.LogInformation("Send a Send a welcome email to {Name} at {Email}", @event.Name, @event.Email);
}
}
Now, let’s hook up Wolverine into our application with just one line of code in our Program file like so:
var builder = WebApplication.CreateBuilder(args);
// This is good enough for what we're trying to do
// at the moment
builder.Host.UseWolverine();
And finally, let’s build a little Minimal API endpoint in our system that will publish SignUpRequest as an “event” to be handled in the background by Wolverine like so:
app.MapPost("/signup", (SignUpRequest request, IMessageBus bus)
// IMessageBus is the primary entry point to Wolverine
=> bus.PublishAsync(request));
app.Run();
Alright, let’s fire up our app with F5, and run this endpoint from the Swashbuckle page:
And watching the console logging for our application, I see this log entry just proving that we did some background processing:
info: SignUpRequest[0]
Send a Send a welcome email to Jeremy at jeremy@jasperfx.net
Woohoo, we’ve successfully utilized some background processing in our new ASP.Net Core system using Wolverine!
Alright, let’s talk about some details and facts I elided in the quick start:
Our message handler is found by Wolverine by looking for public classes suffixed with “Handler”, then public methods named “Handle”. The first argument is assumed to be the message type for that handler
When I called PublishAsync() up above, Wolverine is seeking to route that SignUpRequest as a message. In this simplistic case, Wolverine is defaulting to publishing the message to a local, in memory queue specifically for the SignUpRequest type
When that message is ultimately “received” by Wolverine, Wolverine is responsible for calling into our SignUpRequestHandler.Handle() method
There’s no farther wiring to do, Wolverine puts everything else together for us based on simple defaults
There’s of course, much, much more going on in terms of error handling, logging, possible parallelizing behind the scenes of even this simple example. There’s also certainly much more powerful abilities within Wolverine for much more advanced background processing that we’ll continue to explore later in this series.
I got a little feedback over the weekend that some folks newly encountering Wolverine for the first time think that it’s harder to use than some other tools because “it doesn’t do the message routing for you.” This fortunately isn’t true, but there’s obviously some work to do on improving documentation and samples to dispel that impression.
For the sake of this post, let’s assume that you want to use Wolverine for asynchronous messaging between processes using an external messaging broker transport (or using asynchronous messaging in a single application but queueing up work in an external message broker). And while Wolverine does indeed have support for interoperability with non-Wolverine applications, let’s assume that it’s going to be Wolverine on both sides of all the message pipes.
First though, just know that for all for external transports with Wolverine, the conventional routing is opt in, meaning that you have to explicitly turn it on when you configure Wolverine within the UseWolverine() bootstrapping. Likewise, know that you can also control exactly how Wolverine configures the listening or message sending behavior in the conventionally determined endpoints.
Now then, to just go fast and make Wolverine do all the message routing for you with predictable conventions “just” see these recipes:
using var host = await Host.CreateDefaultBuilder()
.UseWolverine(opts =>
{
opts.UseRabbitMq()
// Opt into conventional Rabbit MQ routing
.UseConventionalRouting();
}).StartAsync();
In this convention, message routing is to:
Publish all messages to a Rabbit MQ exchange named after Wolverine’s message type name that’s effectively the alias for that message type. Most of the time that’s just the full name of the concrete .NET type.
Create a Rabbit MQ queue named with Wolverine’s message type name for the message type of every known message handler within your application. Wolverine also configures a binding from a Rabbit MQ exchange with that same name to the Rabbit MQ queue for that message type.
This routing behavior was absolutely influenced by similar functionality in the older MassTransit framework. Imitation is the sincerest form of flattery, plus I just agree with what they did anyway.Wolverine adds additional value through its richer resource setup model (AutoProvision / AddResourceSetupOnStartup()) and stronger model of automatic handler discovery.
That’s the quickest possible way to get started, but you have plenty of other levers, knobs, and dials to farther control the conventions as shown below:
using var host = await Host.CreateDefaultBuilder()
.UseWolverine(opts =>
{
opts.UseRabbitMq()
// Opt into conventional Rabbit MQ routing
.UseConventionalRouting(c =>
{
// Override naming conventions
c.QueueNameForListener(type => type.Name);
c.ExchangeNameForSending(type => type.Name);
// Override the configuration for each listener
c.ConfigureListeners((l, _) =>
{
l.ProcessInline().ListenerCount(5);
});
// Override the sending configuration for each subscriber
c.ConfigureSending((s, _) =>
{
s.UseDurableOutbox();
});
})
// Let Wolverine both discover all these necessary exchanges, queues,
// and binding, then also build them as necessary on the broker if
// they are missing
.AutoProvision();
}).StartAsync();
The “extra” optional configuration I used above hopefully show you how to take more exacting control over the conventional routing, but the pure defaults in the first sample will help you get up and going fast.
Let’s move on.
Azure Service Bus
Wolverine’s Azure Service Bus integration comes with a pair of conventional routing options. The simpler is to let Wolverine create and utilize an Azure Service Bus queue named after each message type name for both sending the receiving with this syntax:
using var host = await Host.CreateDefaultBuilder()
.UseWolverine(opts =>
{
opts.UseAzureServiceBus("some connection string")
.UseConventionalRouting();
}).StartAsync();
Slightly more complicated is another option to use Azure Service Bus topics and subscriptions like so:
// opts is a WolverineOptions object
opts.UseAzureServiceBusTesting()
.UseTopicAndSubscriptionConventionalRouting(convention =>
{
// Optionally control every aspect of the convention and
// its applicability to types
// as well as overriding any listener, sender, topic, or subscription
// options
});
In this usage:
Outgoing messages are routed to an Azure Service Bus topic named after Wolverine’s message type name for that type
At application startup, Wolverine will listen for an Azure Service Bus subscription for each message type from the application’s known message handlers. Likewise, that subscription will automatically be bound to a topic named after the message type name
As was the case with the Rabbit MQ conventions shown first, you have complete control over the naming conventions, the listener configuration, and the subscriber configuration.
For Wolverine’s AWS SQS integration, you can conventionally route to SQS queues named after Wolverine’s message type name (by default) like this:
As was the case with the Rabbit MQ conventions shown first, you have complete control over the naming conventions, the listener configuration, and the subscriber configuration.
Wolverine’s Kafka integration is a little bit different animal. In this case you can set up rules to publish to different Kafka topics by message type like this:
using var host = await Host.CreateDefaultBuilder()
.UseWolverine(opts =>
{
opts.UseKafka("localhost:29092").AutoProvision();
opts.PublishAllMessages().ToKafkaTopics();
opts.Services.AddResourceSetupOnStartup();
}).StartAsync();
In this case, Wolverine will send each message type to a topic name derived from the message type that’s either Wolverine’s message type name or an explicitly configured topic name configured by attribute like:
[Topic("color.purple")]
public class PurpleMessage
{
}
What if I want a completely different routing convention?!?
There’s an extension point for programmatic message routing rules in Wolverine that’s utilized by all the capabilities shown above called IMessageRouteSource, but if you think you really want to use that, maybe just head to the Wolverine Discord room and we’ll try to help you out!
Summary
Wolverine has strong support for conventional message routing using external brokers, but you need to make at least a one line of code entry in your configuration to explicitly add this behavior to your system. In all cases, Wolverine is able to help build the necessary queues, topics, subscriptions, binding, and exchanges derived by these conventions in your message broker for an efficient developer experience. Moreover, you have complete power to fine tune the usage of this conventional routing for your application.
The Shade Tree Developer 