Building a Critter Stack Application: Integrating Marten into Our Application

Hey, did you know that JasperFx Software is ready for formal support plans for Marten and Wolverine? Not only are we trying to make the “Critter Stack” tools be viable long term options for your shop, we’re also interested in hearing your opinions about the tools and how they should change.

Let’s build a small web service application using the whole “Critter Stack” and their friends, one small step at a time. For right now, the “finished” code is at CritterStackHelpDesk on GitHub.

The posts in this series are:

In the previous couple posts I’ve introduced Marten as a standalone library and some of its capabilities for persisting events and creating projected views off those events within an event sourcing persistence strategy. Today I want to end the week by simply talking about how to integrate Marten into an ASP.Net Core application.

Oskar’s Introduction to Event Sourcing – Self Paced Kit has a world of information for folks getting started with event sourcing.

Let’s start a shell of a new web service project and add a Nuget reference to Marten through:

dotnet new webapi
dotnet add package Marten

If you’ll open up the Program.cs file in your new application, find this line of code at the top where it’s just starting to configure your application:

using Marten;
// Many other using statements

var builder = WebApplication.CreateBuilder(args);

Right underneath that (it doesn’t actually matter most times what order this all happens inside the Program code, but I’m giving Marten the seat at the head of the table so to speak), add this code:

// "AddTool()" is now the common .NET idiom
// for integrating tools into .NET applications
builder.Services.AddMarten(opts =>
{
    // You always have to tell Marten what the connection string to the underlying
    // PostgreSQL database is, but this is the only mandatory piece of 
    // configuration
    var connectionString = builder.Configuration.GetConnectionString("marten");
    opts.Connection(connectionString);
    
    // We have to tell Marten about the projection we built in the previous post
    // so that Marten will "know" how to project events to the IncidentDetails
    // projected view
    opts.Projections.Add<IncidentDetailsProjection>(ProjectionLifecycle.Inline);
})
    // This is a mild optimization
    .UseLightweightSessions();;

That little bit of code is adding the necessary Marten services to your application’s underlying IoC container with the correct scoping. The main services you’ll care about are:

Service	Description	Lifetime
`IDocumentStore`	Root configuration of the Marten database	`Singleton`
`IQuerySession`	Read-only subset of the `IDocumentSession`	`Scoped`
`IDocumentSession`	Marten’s unit of work service that also exposes capabilities for querying and the event store	`Scoped`

Marten services in the IoC container

You can read more about the bootstrapping options in Marten in the documentation. If you’re wondering what “Lightweight Session” means to Marten, you can learn more about the different flavors of sessions in the documentation, but treat that as an advanced subject that’s not terribly relevant to this post.

And also, the usage of AddMarten() should feel familiar to .NET developers now as it follows the common idioms for integrating external tools into .NET applications through the generic host infrastructure that came with .NET Core. As a long term .NET developer, I cannot exaggerate how valuable I think this standardization of application bootstrapping has been for the OSS community in .NET.

Using Marten in an MVC Controller

For right now, I want to assume that many of you are already familiar with ASP.Net MVC Core, so let’s start by showing the usage of Marten within a simple controller to build the first couple endpoints to log a new incident and fetch the current state of an incident in our new incident tracking, help desk service:

public class IncidentController : ControllerBase
{
    private readonly IDocumentSession _session;

    public IncidentController(IDocumentSession session)
    {
        _session = session;
    }

    [HttpPost("/api/incidents")]
    public async Task<IResult> Log(
        [FromBody] LogIncident command
        )
    {
        var userId = currentUserId();
        var logged = new IncidentLogged(command.CustomerId, command.Contact, command.Description, userId);

        var incidentId = _session.Events.StartStream(logged).Id;
        await _session.SaveChangesAsync(HttpContext.RequestAborted);

        return Results.Created("/incidents/" + incidentId, incidentId);
    }

    [HttpGet("/api/incidents/{incidentId}")]
    public async Task<IResult> Get(Guid incidentId)
    {
        // In this case, the IncidentDetails are projected
        // "inline", meaning we can load the pre-built projected
        // view
        var details = await _session.LoadAsync<IncidentDetails>(incidentId);

        return details != null
            ? Results.Json(details)
            : Results.NotFound();
    }

    private Guid currentUserId()
    {
        // let's say that we do something here that "finds" the
        // user id as a Guid from the ClaimsPrincipal
        var userIdClaim = User.FindFirst("user-id");
        if (userIdClaim != null && Guid.TryParse(userIdClaim.Value, out var id))
        {
            return id;
        }

        throw new UnauthorizedAccessException("No user");
    }
}

It’s important to note at this point (that might change in Marten 7) that the IDocumentSession should be disposed when you’re doing using it to tell Marten to close down any open database connections. In the usage above, the scoped IoC container mechanics in ASP.Net Core are handling all the necessary object disposal for you.

Summary and What’s Next

Today we strictly just looked at how to integrate Marten services into a .NET application using the AddMarten() mechanism. Using a simple MVC Core Controller, we saw how Marten services are available and managed in the application’s IoC container and how to perform basic event sourcing actions in the context of little web service endpoints.

In later posts in this series we’ll actually replace this IncidentController with Wolverine endpoints and some “special sauce” with Marten to be much more efficient.

In the next post, I think I want to talk through the CQRS architectural style with Marten. Bear with me, but I’ll still be using explicit code with MVC Core controllers that folks are probably already familiar with to talk over the requirements and Marten capabilities in isolation. Don’t worry though, I will eventually introduce Wolverine into the mix to show how the Wolverine + Marten integration can make your code very clean and testable.

Building a Critter Stack Application: Marten Projections

The posts in this series are:

In the previous post I showed how to use the Marten library as the storage mechanism for events and event streams within an event sourcing persistence strategy. If you’re following along, you’ve basically learned how to stuff little bits of JSON into a database as the authoritative source of truth for your system. You might be asking yourself “what the @#$%@# am I supposed to do this this stuff now?” In today’s post I’m going to show you how Marten can help you derive the current state of the system from the raw event data through its usage of projections.

For more information about the conceptual role of projections in an event sourcing system, see my colleague Oskar Dudycz‘s post Guide to Projections and Read Models in Event-Driven Architecture.

Back to our help desk service, last time we created event streams representing each incident with events like:

public record IncidentLogged(
    Guid CustomerId,
    Contact Contact,
    string Description,
    Guid LoggedBy
);
 
public class IncidentCategorised
{
    public IncidentCategory Category { get; set; }
    public Guid UserId { get; set; }
}
 
public record IncidentPrioritised(IncidentPriority Priority, Guid UserId);
 
public record AgentAssignedToIncident(Guid AgentId);
 
public record AgentRespondedToIncident(        
    Guid AgentId,
    string Content,
    bool VisibleToCustomer);
 
public record CustomerRespondedToIncident(
    Guid UserId,
    string Content
);
 
public record IncidentResolved(
    ResolutionType Resolution,
    Guid ResolvedBy,
    DateTimeOffset ResolvedAt
);

Those events are directly stored in our database as our single source of truth, but we will absolute need to derive the current state of an incident to support:

User interface screens
Reports
Decision making within the help desk workflow (what event sourcing folks call the “write model”

For now, let’s say that we’d really like to have this view of a single incident:

public class IncidentDetails
{
    public Guid Id { get; set; }
    public Guid CustomerId{ get; set; }
    public IncidentStatus Status{ get; set; }
    public IncidentNote[] Notes { get; set; } = Array.Empty<IncidentNote>();
    public IncidentCategory? Category { get; set; }
    public IncidentPriority? Priority { get; set; }
    public Guid? AgentId { get; set; }
    public int Version { get; set; }
}

Let’s teach Marten how to combine the raw events describing an incident into our new IncidentDetails view. The easiest possible way to do that is to drop some new methods onto our IncidentDetails class to “teach” Marten how to modify the projected view:

public class IncidentDetails
{
    public IncidentDetails()
    {
    }

    public IncidentDetails(IEvent<IncidentLogged> logged)
    {
        Id = logged.StreamId;
        CustomerId = logged.Data.CustomerId;
        Status = IncidentStatus.Pending;
    }

    public Guid Id { get; set; }
    public Guid CustomerId{ get; set; }
    public IncidentStatus Status{ get; set; }
    public IncidentNote[] Notes { get; set; } = Array.Empty<IncidentNote>();
    public IncidentCategory? Category { get; set; }
    public IncidentPriority? Priority { get; set; }
    public Guid? AgentId { get; set; }

    // Marten itself will set this to its tracked
    // revision number for the incident
    public int Version { get; set; }

    public void Apply(IncidentCategorised categorised) => Category = categorised.Category;
    public void Apply(IncidentPrioritised prioritised) => Priority = prioritised.Priority;
    public void Apply(AgentAssignedToIncident prioritised) => AgentId = prioritised.AgentId;
    public void Apply(IncidentResolved resolved) => Status = IncidentStatus.Resolved;
    public void Apply(ResolutionAcknowledgedByCustomer acknowledged) => Status = IncidentStatus.ResolutionAcknowledgedByCustomer;
    public void Apply(IncidentClosed closed) => Status = IncidentStatus.Closed;
}

In action, the simplest way to execute the projection is to do a “live aggregation” as shown below:

static async Task PrintIncident(IDocumentStore store, Guid incidentId)
{
    await using var session = store.LightweightSession();
    
    // Tell Marten to load all events -- in order -- for the designated
    // incident event stream, then project that data into an IncidentDetails
    // view
    var incident = await session.Events.AggregateStreamAsync<IncidentDetails>(incidentId);
}

You can see a more complicated version of this projection in action by running the EventSourcingDemo project from the command line. Just see the repository README for instructions on setting up the database.

Marten is using a set of naming conventions to “know” how to pass event data to the IncidentDetails objects. As you can probably guess, Marten is calling the Apply() overloads to mutate the InvoiceDetails object for each event based on the event type. Those conventions are documented here — and yes, there are plenty of other options for using more explicit code instead of the conventional approach if you don’t care for that.

This time, with immutability!

In the example above, I purposely chose the simplest possible approach, and that led me to using a mutable structure for InvoiceDetails that kept all the details of how to project the events in the InvoiceDetails class itself. As an alternative, let’s make the InvoiceDetails be immutable as a C# record instead like so:

public record IncidentDetails(
    Guid Id,
    Guid CustomerId,
    IncidentStatus Status,
    IncidentNote[] Notes,
    IncidentCategory? Category = null,
    IncidentPriority? Priority = null,
    Guid? AgentId = null,
    int Version = 1
);

And as another alternative, let’s say you’d rather have the Marten projection logic external to the nice, clean IncidentDetails code above. That’s still possible by creating a separate class. The most common projection type is to project the events of a single stream, and for that you can subclass the Marten SingleStreamProjection base class to create your projection logic as shown below:

public class IncidentDetailsProjection: SingleStreamProjection<IncidentDetails>
{
    public static IncidentDetails Create(IEvent<IncidentLogged> logged) =>
        new(logged.StreamId, logged.Data.CustomerId, IncidentStatus.Pending, Array.Empty<IncidentNote>());

    public IncidentDetails Apply(IncidentCategorised categorised, IncidentDetails current) =>
        current with { Category = categorised.Category };

    public IncidentDetails Apply(IncidentPrioritised prioritised, IncidentDetails current) =>
        current with { Priority = prioritised.Priority };

    public IncidentDetails Apply(AgentAssignedToIncident prioritised, IncidentDetails current) =>
        current with { AgentId = prioritised.AgentId };

    public IncidentDetails Apply(IncidentResolved resolved, IncidentDetails current) =>
        current with { Status = IncidentStatus.Resolved };

    public IncidentDetails Apply(ResolutionAcknowledgedByCustomer acknowledged, IncidentDetails current) =>
        current with { Status = IncidentStatus.ResolutionAcknowledgedByCustomer };

    public IncidentDetails Apply(IncidentClosed closed, IncidentDetails current) =>
        current with { Status = IncidentStatus.Closed };
}

The exact same set of naming conventions still apply here, with Apply() methods creating a new revision of the IncidentDetails for each event, and the Create() method helping Marten to start an IncidentDetails object for the first event in the stream.

This usage does require you to register the custom projection class upfront in the Marten configuration like this:

var connectionString = "Host=localhost;Port=5433;Database=postgres;Username=postgres;password=postgres";
await using var store = DocumentStore.For(opts =>
{
    opts.Connection(connectionString);
    
    // Telling Marten about the projection logic for the IncidentDetails
    // view of the events
    opts.Projections.Add<IncidentDetailsProjection>(ProjectionLifecycle.Live);
});

Don’t worry too much about that “Live” option, we’ll dive deeper into projection lifecycles as we progress in this series.

Summary and What’s Next

Projections are a Marten feature that enable you to create usable views out of the raw event data. We used the simplest projection recipes in this post to create an IncidentDetails vew out of the raw incident events that we will use later on to build our web service.

In this sample, I was showing Marten’s ability to evaluate projected views on the fly by loading the events into memory and combining them into the final projection result on demand. Marten also has the ability to persist these projected data views ahead of time for faster querying (“Inline” or “Async” projections). If you’re familiar with the concept of materialized views in databases that support that, projections running inline or in a background process are a close analogue.

In the next post, I think I just want to talk about how to integrate Marten into an ASP.Net Core application and utilize Marten in a simple MVC Core controller — but don’t worry, before we’re done, we’re going to replace the MVC Core code with much slimmer code using Wolverine, but one new concept or tool at a time!

Building a Critter Stack Application: Marten as Event Store

The posts in this series are:

Event Sourcing

Event Sourcing is a style of persistence where the single source of truth, system state is a read only, append only sequence of all the events that resulted in a change in the system state. Using our HelpDesk incident tracking application we first started describing in the previous post on Event Storming, that results in a sequence like this:

Sequence	Incident Id	Event Type
1	1	IncidentLogged
2	1	IncidentCategorized
3	2	IncidentLogged
4	3	IncidentLogged
5	1	IncidentResolved

An event log

As you could probably guess already from the table above, the events will be stored in one single log in the sequential order they were appended. You can also see that events will be categorized by their relationship to a single logical incident. This grouping is typically called a “stream” in event sourcing.

As a first quick foray into event sourcing, let’s look at using the Marten library to create an event store for our help desk application built on top of a PostgreSQL database.

In case you’re wondering, Marten is merely a fancy library that helps you access and treat the rock solid PostgreSQL database engine as both a document database and as an event store. Marten was purposely built on PostgreSQL specifically because of the unique JSON capabilities of PostgreSQL. It’s possible that the event store portion of Marten eventually gets ported to other databases in the future (Sql Server), but it’s highly unlikely that the document database feature set would ever follow.

Using Marten as an Event Store

This code is all taken from the CritterStackHelpDesk repository, and specifically the EventSourcingDemo console project. The repository’s README file has instructions on running that project.

First off, let’s build us some events that we can later store in our new event store:

public record IncidentLogged(
    Guid CustomerId,
    Contact Contact,
    string Description,
    Guid LoggedBy
);

public class IncidentCategorised
{
    public IncidentCategory Category { get; set; }
    public Guid UserId { get; set; }
}

public record IncidentPrioritised(IncidentPriority Priority, Guid UserId);

public record AgentAssignedToIncident(Guid AgentId);

public record AgentRespondedToIncident(        
    Guid AgentId,
    string Content,
    bool VisibleToCustomer);

public record CustomerRespondedToIncident(
    Guid UserId,
    string Content
);

public record IncidentResolved(
    ResolutionType Resolution,
    Guid ResolvedBy,
    DateTimeOffset ResolvedAt
);

You’ll notice there’s a (hopefully) consistent naming convention. The event types are named in the past tense and should refer clearly to a logical event in the system’s workflow. You might also notice that these events are all built with C# records. This isn’t a requirement, but it makes the code pretty terse and there’s no reason for these events to ever be mutable anyway.

Next, I’ve created a small console application and added a reference to the Marten library like so from the command line:

dotnet new console
dotnet add package Marten

Before we even think about using Marten itself, let’s get ourselves a new, blank PostgreSQL database spun up for our little application. Assuming that you have Docker Desktop or some functional alternative on your development machine, there’s a docker compose file in the root of the finished product that we can use to stand up a new database with:

docker compose up -d

Note, and this is an important point, there is absolutely nothing else you need to do to make this new database perfectly usable for the code we’re going to write next. No manual database setup, no SQL scripts for you to run, no other command line scripts. Just write code and go.

Next, we’re going to configure Marten in code, then:

Start a new “Incident” stream with a couple events
Append additional events to our new stream

The code to do nothing but what I described is shown below:

// This matches the docker compose file configuration
var connectionString = "Host=localhost;Port=5433;Database=postgres;Username=postgres;password=postgres";

// This is spinning up Marten with its default settings
await using var store = DocumentStore.For(connectionString);

// Create a Marten unit of work
await using var session = store.LightweightSession();

var contact = new Contact(ContactChannel.Email, "Han", "Solo");
var userId = Guid.NewGuid();

// I'm telling the Marten session about the new stream, and then recording
// the newly assigned Guid for this stream
var customerId = Guid.NewGuid();
var incidentId = session.Events.StartStream(
    new IncidentLogged(customerId, contact, "Software is crashing",userId),
    new IncidentCategorised
    {
        Category = IncidentCategory.Database,
        UserId = userId
    }
    
).Id;

await session.SaveChangesAsync();

// And now let's append an additional event to the 
// new stream
session.Events.Append(incidentId, new IncidentPrioritised(IncidentPriority.High, userId));
await session.SaveChangesAsync();

Let’s talk about what I just did — and did not do — in the code above. The DocumentStore class in Marten establishes the storage configuration for a single, logical Marten-ized database. This is an expensive object to create, so there should only ever be one instance in your system.

The actual work is done with Marten’s IDocumentSession service that I created with the call to store.LightweightSession(). The IDocumentSession is Marten’s unit of work implementation and plays the same role as DbContext does inside of EF Core. When you use Marten, you queue up operations (start a new event stream, append events, etc.), then commit them in one single database transaction when you call that SaveChangesAsync() method.

For anybody old enough to have used NHibernate reading this, DocumentStore plays the same role as NHibernate’s ISessionFactory.

So now, let’s read back in the events we just persisted, and print out serialized JSON of the Marten data just to see what Marten is actually capturing:

var events = await session.Events.FetchStreamAsync(incidentId);
foreach (var e in events)
{
    // I elided a little bit of code that sets up prettier JSON
    // formatting
    Console.WriteLine(JsonConvert.SerializeObject(e, settings));
}

The raw JSON output is this:

{
  "Data": {
    "CustomerId": "314d8fa1-3cca-4984-89fc-04b24122cf84",
    "Contact": {
      "ContactChannel": "Email",
      "FirstName": "Han",
      "LastName": "Solo",
      "EmailAddress": null,
      "PhoneNumber": null
    },
    "Description": "Software is crashing",
    "LoggedBy": "8a842212-3511-4858-a3f3-dd572a4f608f"
  },
  "EventType": "Helpdesk.Api.IncidentLogged, Helpdesk.Api, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null",
  "EventTypeName": "incident_logged",
  "DotNetTypeName": "Helpdesk.Api.IncidentLogged, Helpdesk.Api",
  "IsArchived": false,
  "AggregateTypeName": null,
  "StreamId": "018c1c9b-5bd0-4273-947d-83d28c8e3210",
  "StreamKey": null,
  "Id": "018c1c9b-5f03-47f5-8c31-1d1ba70fd56a",
  "Version": 1,
  "Sequence": 1,
  "Timestamp": "2023-11-29T19:43:13.864064+00:00",
  "TenantId": "*DEFAULT*",
  "CausationId": null,
  "CorrelationId": null,
  "Headers": null
}
{
  "Data": {
    "Category": "Database",
    "UserId": "8a842212-3511-4858-a3f3-dd572a4f608f"
  },
  "EventType": "Helpdesk.Api.IncidentCategorised, Helpdesk.Api, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null",
  "EventTypeName": "incident_categorised",
  "DotNetTypeName": "Helpdesk.Api.IncidentCategorised, Helpdesk.Api",
  "IsArchived": false,
  "AggregateTypeName": null,
  "StreamId": "018c1c9b-5bd0-4273-947d-83d28c8e3210",
  "StreamKey": null,
  "Id": "018c1c9b-5f03-4a19-82ef-9c12a84a4384",
  "Version": 2,
  "Sequence": 2,
  "Timestamp": "2023-11-29T19:43:13.864064+00:00",
  "TenantId": "*DEFAULT*",
  "CausationId": null,
  "CorrelationId": null,
  "Headers": null
}
{
  "Data": {
    "Priority": "High",
    "UserId": "8a842212-3511-4858-a3f3-dd572a4f608f"
  },
  "EventType": "Helpdesk.Api.IncidentPrioritised, Helpdesk.Api, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null",
  "EventTypeName": "incident_prioritised",
  "DotNetTypeName": "Helpdesk.Api.IncidentPrioritised, Helpdesk.Api",
  "IsArchived": false,
  "AggregateTypeName": null,
  "StreamId": "018c1c9b-5bd0-4273-947d-83d28c8e3210",
  "StreamKey": null,
  "Id": "018c1c9b-5fef-4644-b213-56051088dc15",
  "Version": 3,
  "Sequence": 3,
  "Timestamp": "2023-11-29T19:43:13.909+00:00",
  "TenantId": "*DEFAULT*",
  "CausationId": null,
  "CorrelationId": null,
  "Headers": null
}

And that’s a lot of noise, so let me try to summarize the blob above:

Marten is storing each event as serialized JSON in one table, and that’s what you see as the Data leaf in each JSON document above
Marten is assigning a unique sequence number for each event
StreamId is the incident stream identity that groups the events
Each event is assigned a Version that reflects its position within its stream
Marten tracks the kind of metadata that you’d probably expect, like timestamps, optional header information, and optional causation/correlation information (we’ll use this much later in the series when I get around to discussing Open Telemetry)

Summary and What’s Next

In this post I introduced the core concepts of event sourcing, events, and event streams. I also introduced the bare bones usage of the Marten library as a way to create new event streams and append events to existing events. Lastly, we took a look at the important metadata that Marten tracks for you in addition to your raw event data. Along the way, we also previewed how the Critter Stack can reduce development time friction by very happily building out the necessary database schema objects for us as needed.

What you are probably thinking at this point is something to the effect of “So what?” After all, jamming little bits of JSON data into the database doesn’t necessarily help us build a user interface page showing a help desk technician what the current state of each open incident is. Heck, we don’t yet have any way to understand the actual current state of any incident!

Fear not though, because in the next post I’ll introduce Marten “Projections” capability that will help us create the “read side” view of the current system state out of the raw event data in whatever format happens to be most convenient for that data’s client or user.

Building a Critter Stack Application: Event Storming

I did a series of presentations a couple weeks ago showing off the usage of Wolverine and Marten to build a small service using CQRS and Event Sourcing and you can see the video above from .NET Conf 2023. Great, but I thought the talk was way too dense and I’m going to rebuild the talk from scratch before CodeMash. So I’ve got a relatively complete sample application, and we get a lot of feedback that there needs to be a single sample application that’s more realistic to show off what Marten and Wolverine can actually do. Based on other feedback, I know there’s some value in having a series of short, focused posts that build up a sample application one little concept at a time.

To that end, this post will be the start of a multi-part series showing how to use Marten and Wolverine for a CQRS architecture in an ASP.Net Core web service that also uses event sourcing as a persistence strategy.

The series so far:

I blatantly stole (with permission) this sample application idea from Oskar Dudycz. His version of the app is also on GitHub.

If you’re reading this post, it’s very likely you’re a software professional and you’re already familiar with online incident tracking applications — but hey, let’s build yet another one for a help desk company just because it’s a problem domain you’re likely (all too) familiar with!

Let’s say that you’re magically able to get your help desk business experts and stakeholders in a room (or virtual meeting) with the development team all at one time. Crazy, I know, but bear with me. Since you’re altogether, this is a fantastic opportunity to get the new system started is a very collaborative approach called Event Storming that works very well for both event sourcing and CQRS approaches.

The format is pretty simple. Go to any office supply company and get the typical pack of sticky notes like these:

Start by asking the business experts to describe events within the desired workflow that would lead to a change in state or a milestone in the business process. Try to record their terminology on orange sticky notes with a short name that generally implies a past event. In the case of an incident service, those events might be:

IncidentLogged
IncidentCategorised
IncidentResolved

This isn’t waterfall, so you can happily jump back and forth between steps here, but the next general step is to try to identify the actions or “commands” in the system that would cause each of our previously identified events. Jot these commands down on blue sticky notes with a short name in an imperative form like “LogIncident” or “CategoriseIncident”. Create some record of cause and effect by putting the blue sticky command notes just to the left of the orange sticky notes for the related events.

It’s also helpful to organize the sticky notes roughly left to right to give some context to what commands or events happen in what order (which I did not do in my crude diagram in a second).

Even though my graphic below doesn’t do this, it’s perfectly possible for the relationship between commands and events to be one command to many events.

In the course of executing these newly discovered commands, we can start to call out possible “views” of the raw event data that we might need as necessary context. We’ll record these views with a short descriptive name on green sticky notes.

After some time, our wall should be covered in sticky notes in a manner something like this:

Right off the bat, we’re learning what the DDD folks call the ubiquitous language for our business domain that can be shared between us technical folks and the business domain experts. Moreover, as we’ll see in later posts, these names from is ostensibly a requirements gathering session can translate directly to actual code artifact names.

My experience with Event Storming has been very positive, but I’d guess that it depends on how cooperative and collaborative your business partners are with this format. I found it to be a great format to talk through a system’s requirements in a way that provides actual traceability to code implementation details. In other words, when you talk with the business folks and speak in terms of an IncidentLogged, there will actually be a type in your codebase like this:

public record IncidentLogged(
    Guid CustomerId,
    Contact Contact,
    string Description,
    Guid LoggedBy
);

or LogIncident:

public record LogIncident(
    Guid CustomerId,
    Contact Contact,
    string Description
);

Help Desk API

Just for some context, I’m going to step through the creation of a web service with a handful of web service endpoints to create, read, or alter a help desk incident. In much later posts, I’ll talk about publishing internal events to take action asynchronously within the web service, and also to publish other events externally to completely different systems through Rabbit MQ queues.

The “final” code is the CritterStackHelpDesk application under my GitHub profile.

I’m not going to go near a user interface for now, but someone is working up improvements to this service to put a user interface on top with this service as a Backend for Frontend (BFF).

Summary

Event Storming can be a very effective technique for collaboratively discovering system requirements and understanding the system’s workflow with your domain experts. As developers and testers, it can also help create traceability between the requirements and the actual code artifacts without manually intensive traceability matrix documentation.

Next time…

In the next post in this new series, I’ll introduce the event sourcing functionality with just Marten completely outside of any application just to get comfortable with Marten mechanics before we go on.

Tell Us What You Want in Marten and Wolverine!

I can’t prove this conclusively, but the cure for getting the “tell me what you want, what you really, really want” out of your head is probably to go fill out the linked survey on Marten and Wolverine!

As you may know, JasperFx Software is now up and able to offer formal support contracts to help users be successful with the open source Marten and Wolverine tools (the “Critter Stack”). As the next step in our nascent plan to create a sustainable business model around the Critter Stack tools, we’d really like to elicit some feedback from our users or potential users about what features your team would be most interested in next. And to be clear, we’re specifically thinking about complex features that would be part of a paid add on model to the Critter Stack for advanced usages.

We’d love to get any feedback for us you might have in this Google Form.

Some existing ideas for paid features include:

A module for GDPR compliance
A dead letter queue browser application for Wolverine that would also help you selectively replay messages
The ability to dynamically add new tenant databases for Marten + Wolverine at runtime with no downtime
Improved asynchronous projection support in Marten, including better throughput overall, the ability to load balance the projections across running nodes
Zero downtime projection rebuilds with asynchronous Marten event store projections
The capability to do blue/green deployments with Marten event store projections
A virtual actor capability for Wolverine
A management and monitoring user interface for Wolverine + Marten that would give you insights about running nodes, active event store projections, messaging endpoint health, node assignments
DevOps recipes for the Critter Stack?

Publishing Events from Marten through Wolverine

Aren’t martens really cute?

By the way, JasperFx Software is up and running for formal support plans for both Marten and Wolverine!

Wolverine 1.11.0 was released this week (here’s the release notes) with a small improvement to its ability to subscribe to Marten events captured within Wolverine message handlers or HTTP endpoints. Since Wolverine 1.0, users have been able to opt into having Marten forward events captured within Wolverine handlers to any known Wolverine subscribers for that event with the EventForwardingToWolverine() option.

The latest Wolverine release adds the ability to automatically publish an event as a different message using the event data and its metadata as shown in the sample code below:

builder.Services.AddMarten(opts =>
{
    var connectionString = builder.Configuration.GetConnectionString("marten");
    opts.Connection(connectionString);
})
    // Adds Wolverine transactional middleware for Marten
    // and the Wolverine transactional outbox support as well
    .IntegrateWithWolverine()
    
    .EventForwardingToWolverine(opts =>
    {
        // Setting up a little transformation of an event with its event metadata to an internal command message
        opts.SubscribeToEvent<IncidentCategorised>().TransformedTo(e => new TryAssignPriority
        {
            IncidentId = e.StreamId,
            UserId = e.Data.UserId
        });
    });

This isn’t a general purpose outbox, but rather immediately publishes captured events based on normal Wolverine publishing rules immediately at the time the Marten transaction is committed.

So in this sample handler:

public static class CategoriseIncidentHandler
{
    public static readonly Guid SystemId = Guid.NewGuid();
    
    // This Wolverine handler appends an IncidentCategorised event to an event stream
    // for the related IncidentDetails aggregate referred to by the CategoriseIncident.IncidentId
    // value from the command
    [AggregateHandler]
    public static IEnumerable<object> Handle(CategoriseIncident command, IncidentDetails existing)
    {
        if (existing.Category != command.Category)
        {
            // Wolverine will transform this event to a TryAssignPriority message
            // on the successful commit of the transaction wrapping this handler call
            yield return new IncidentCategorised
            {
                Category = command.Category,
                UserId = SystemId
            };
        }
    }
}

To try to close the loop, when Wolverine handles the CategoriseIncident message, it will:

Potentially append an IncidentCategorised event to the referenced event stream
Try to transform that event to a new TryAssignPriority message
Commit the changes queued up to the underlying Marten IDocumentSession unit of work
If the transaction is successful, publish the TryAssignPriority message — which in this sample case would be routed to a local queue within the Wolverine application and handled in a different thread later

That’s a lot of text and gibberish, but all I’m trying to say is that you can make Wolverine reliably react to events captured in the Marten event store.

Critter Stack at .NET Conf 2023

JasperFx Software will be shortly announcing the availability of official support plans for Marten, Wolverine, and other JasperFx open source tools. We’re working hard to build a sustainable ecosystem around these tools so that companies can feel confident in making a technical bet on these high productivity tools for .NET server side development.

I’ll be presenting a short talk at .NET Conf 2023 entitled “CQRS with Event Sourcing using the Critter Stack.” It’s going to be a quick dive into how to use Marten and Wolverine to build a very small system utilizing a CQRS Architecture with Event Sourcing as the persistence strategy.

Hopefully, I’ll be showing off:

How Wolverine’s runtime architecture is significantly different than other .NET tools and why its approach leads to much lower code ceremony and potentially higher performance
Marten and PostgreSQL providing a great local developer story both in development and in integration testing
How the Wolverine + Marten integration makes your domain logic easily unit testable without resorting to complicated Clean/Onion/Hexagonal Architectures
Wolverine’s built in integration testing support that you’ll wish you had today in other .NET messaging tools
The built in tooling for unraveling Wolverine or Marten’s “conventional magic”

Here’s the talk abstract:

CQRS with Event Sourcing using the “Critter Stack”

Do you have a system where you think would be a good fit for a CQRS architecture that also uses Event Sourcing for at least part of its persistence strategy? Are you intimidated by the potential complexity of that kind of approach? Fear not, using a combination of the PostgreSQL-backed Marten library for event sourcing and its newer friend Wolverine for command handling and asynchronous messaging, I’ll show you how you can quickly get started with both CQRS and Event Sourcing. Once we get past the quick start, I’ll show you how the Critter Stack’s unique approach to the “Decider” pattern will help you create robust command handlers with very little code ceremony while still enjoying easy testability. Moving beyond basic command handling, I’ll show you how to reliably subscribe to and publish the events or other messages created by your command handlers through Wolverine’s durable outbox and direct subscriptions to Marten’s event storage.

Low Ceremony Web Service Development with the Critter Stack

You can’t really get Midjourney to create an image of a wolverine without veering into trademark violations, so look at the weasel and marten up there working on a website application together!

Wolverine 1.10 was released earlier this week (here’s the release notes), and one of the big additions this time around was some new recipes for combining Marten and Wolverine for very low ceremony web service development.

Before I show the new functionality, let’s imagine that you have a simple web service for invoicing where you’re using Marten as a document database for persistence. You might have a very simplistic web service for exposing a single Invoice like this (and yes, I know you’d probably want to do some kind of transformation to a view model but put that aside for a moment):

    [WolverineGet("/invoices/longhand/id")]
    [ProducesResponseType(404)] 
    [ProducesResponseType(200, Type = typeof(Invoice))]
    public static async Task<IResult> GetInvoice(
        Guid id, 
        IQuerySession session, 
        CancellationToken cancellationToken)
    {
        var invoice = await session.LoadAsync<Invoice>(id, cancellationToken);
        if (invoice == null) return Results.NotFound();

        return Results.Ok(invoice);
    }

It’s not that much code, but there’s still some repetitive boilerplate code. Especially if you’re going to care or be completist about your OpenAPI metadata. The design and usability aesthetic of Wolverine is to reduce code ceremony as much as possible without sacrificing performance or observability, so let’s look at a newer alternative.

Next, I’m going to install the new WolverineFx.Http.Marten Nuget to our web service project, and write this new endpoint using the [Document] attribute:

    [WolverineGet("/invoices/{id}")]
    public static Invoice Get([Document] Invoice invoice)
    {
        return invoice;
    }

The code up above is an exact functional equivalent to the first code sample, and even produces the exact same OpenAPI metadata (or at least tries to, OpenAPI has been a huge bugaboo for Wolverine because so much of the support inside of AspNetCore is hard wired for MVC Core). Notice though, how much less you have to do. You have a synchronous method, so that’s a little less ceremony. It’s a pure function, so even if there was code to transform the invoice data to an API specific shape, you could unit test this method without any infrastructure involved or using something like Alba. Heck, that is setting you up so that Wolverine itself is handling the “return 404 if the Invoice is not found” behavior as shown in the unit test from Wolverine itself (using Alba):

    [Fact]
    public async Task returns_404_on_id_miss()
    {
        // Using Alba to run a request for a non-existent
        // Invoice document
        await Scenario(x =>
        {
            x.Get.Url("/invoices/" + Guid.NewGuid());
            x.StatusCodeShouldBe(404);
        });
    }

Simple enough, but now let’s look at a new HTTP-centric mechanism for the Wolverine + Marten “Aggregate Handler” workflow for writing CQRS “Write” handlers using Marten’s event sourcing. You might want to glance at the previous link for more context before proceeding, or refer back to it later at least.

The main change here is that folks asked to provide the aggregate identity through a route parameter, and then to enforce a 404 response code if the aggregate does not exist.

Using an “Order Management” problem domain, here’s what an endpoint method to ship an existing order could look like:

    [WolverinePost("/orders/{orderId}/ship2"), EmptyResponse]
    // The OrderShipped return value is treated as an event being posted
    // to a Marten even stream
    // instead of as the HTTP response body because of the presence of 
    // the [EmptyResponse] attribute
    public static OrderShipped Ship(ShipOrder2 command, [Aggregate] Order order)
    {
        if (order.HasShipped) 
            throw new InvalidOperationException("This has already shipped!");
        
        return new OrderShipped();
    }

Notice the new [Aggregate] attribute on the Order argument. At runtime, this code is going to:

Take the “orderId” route argument, parse that to a Guid (because that’s the identity type for an Order)
Use that identity — and any version information on the request body or a “version” route argument — to use Marten’s FetchForWriting() mechanism to both load the latest version of the Order aggregate and to opt into optimistic concurrency protections against that event stream.
Return a 404 response if the aggregate does not already exist
Pass the Order aggregate into the actual endpoint method
Take the OrderShipped event returned from the method, and apply that to the Marten event stream for the order
Commit the Marten unit of work

As always, the goal of this workflow is to turn Wolverine endpoint methods into low ceremony, synchronous pure functions that are easily testable with unit tests.

Imagining the Ideal GraphQL Integration for Marten

I’m seeing an increasing amount of interest in exposing Marten data behind GraphQL endpoints from folks in the Critter Stack Discord channels and from current JasperFx Software clients. After having mostly let other folks handle the Marten + Hot Chocolate combination, I finally spent some significant time looking into what it takes to put Marten behind Hot Chocolate’s GraphQL handling — and I unfortunately saw some real issues for unwary users that I wrote about last week in Hot Chocolate, GraphQL, and the Critter Stack.

Today, I want to jot down my thoughts about how a good GraphQL layer for Marten could be constructed, with a couple caveats that hopefully much of this will be possible once I know much more about Hot Chocolate internals and that the rest of the Marten core team and I have zero interest in building a GraphQL layer from scratch.

Command Batching

A big part of GraphQL usage is wanting a way to aggregate queries from your client to the backend without making a lot of network round trips in a way that pretty well destines you for poor performance. Great, awesome, but on the server side, Hot Chocolate runs every query in parallel, which for Marten means opening a session for each query or serializing the usage of Marten’s sessions and therefore losing the parallelization.

Instead of that parallelization, what I’d love to do is cut in higher up in the GraphQL execution pipeline and instead, batch up the queries into a single database command. What we’ve repeatedly found over 8 years of Marten development (where did the time go?) is that batching database queries into a single network round trip to a PostgreSQL database consistently leads to better performance than making serialized requests. And that’s even with the more complex query building we do within Marten, Weasel, and Wolverine.

Streaming Marten JSON

In the cases where you don’t need to do any transformation of the JSON data being fetched by Marten into the GraphQL results (and remember, it is legal to return more fields than the client actually requested), Marten has an option for very fast HTTP services where it can just happily stream the server stored JSON data right to the HTTP response byte by byte. That’s vastly more efficient than the normal “query data, transform that to objects, then use a JSON serializer to write those objects to HTTP” mechanics.

More Efficient Parsing

Go easy commenting on this one, because this is all conjecture on my part here.

The process of going from a GraphQL query to actual results (which then have to be serialized down to the HTTP response) in Hot Chocolate + Marten is what a former colleague of mine would refer to as a “crime against computer science”:

Hot Chocolate gets the raw string for the GraphQL request that’s sorta like JSON, but definitely not compliant JSON
GraphQL is (I’m guessing) translated to some kind of intermediate model
When using a Hot Chocolate query based on returning a LINQ IQueryable — and most Hot Chocolate samples do this — Hot Chocolate is building up a LINQ Expression on the fly
Marten’s LINQ provider is then taking that newly constructed LINE Expression, and parsing that to create first an intermediate model representing the basics of the operation (are we fetching a list? limiting or skipping results? transforming the raw document data? where/order clauses?)
Marten’s LINQ provider takes Marten’s intermediate model and creates a model that represents fragments of SQL and also determines a query handler strategy for the actual results (list results? FirstOrDefault()? Single() Count()? )
Marten evaluates all these SQL fragments to build up a PostgreSQL SQL statement, executes that, and uses its query handler to resolve the actual resulting documents

If you read that list above and thought to yourself, that sounds like a ton of object allocations and overhead and I wonder if that could end up being slow, yeah, me, too.

What I’d ideally like to see is a model where Marten can take whatever GraphQL’s intermediate model is and effectively skip down from #2 straight down to #5/6. I’d also love to see some kind of way to cache “query plans” in a similar way to Marten’s compiled query mechanism where repetitive patterns of GraphQL queries can be cached to skip even more of the parsing and LINQ query/SQL generation/handler strategy selection overhead.

Batching Mutations to Marten

Betting this would be the easiest thing to pull off. Instead of depending on ambient transactions in .NET (ick), I’d like to be able to look ahead at all the incoming mutations, and if they are all Marten related, use Marten’s own unit of work mechanics and native database transactions.

Wrapping Up

That’s it for now. Not every blog post has to be War and Peace:-)

I might be back next week with an example of how to do integration testing of GraphQL endpoints with Alba — right after I learn how to do that so I can show a JasperFx client.

Wolverine now does Event Streaming with Kafka or MQTT

As part of an ongoing JasperFx client engagement, Wolverine (1.9.0) just added some new options for event streaming from Wolverine applications. The immediate need was to support messaging with the MQTT protocol for usage inside of a new system in the “Internet of Things” problem space. Knowing that a different JasperFx client is going to need to support event subscriptions with Apache Kafka, it was also convenient to finally add the much requested option for Kafka support within Wolverine while the similar MQTT work was still fresh in my mind.

While the new MQTT transport option is documented, the Kafka transport documentation is still on the way, so I’m going to focus on that first.

To get started with Kafka within a Wolverine application, add the WolverineFx.Kafka Nuget to your project. Next, add the Kafka transport option, any messaging subscription rules, and the topics you want your application to listen to with code like this:

using var host = await Host.CreateDefaultBuilder()
    .UseWolverine(opts =>
    {
        opts.UseKafka("localhost:29092");

        // Just publish all messages to Kafka topics
        // based on the message type (or message attributes)
        // This will get fancier in the near future
        opts.PublishAllMessages().ToKafkaTopics();
        
        // Or explicitly make subscription rules
        opts.PublishMessage<ColorMessage>()
            .ToKafkaTopic("colors");
        
        // Listen to topics
        opts.ListenToKafkaTopic("red")
            .ProcessInline();

        opts.ListenToKafkaTopic("green")
            .BufferedInMemory();
        

        // This will direct Wolverine to try to ensure that all
        // referenced Kafka topics exist at application start up 
        // time
        opts.Services.AddResourceSetupOnStartup();
    }).StartAsync();

I’m very sure that these two transports (and shortly a third option for Apache Pulsar) will need to be enhanced when they meet real users and unexpected use cases, but I think there’s a solid foundation ready to go.

In the near future, JasperFx Software will be ready to start offering official support contracts and relationships for both Marten and Wolverine. In the slightly longer term, we’re hoping to create some paid add on products (with support!) for Wolverine for “big, serious enterprise usage.” One of the first use cases I’d like us to tackle with that initiative will be a more robust event subscription capability from Marten’s event sourcing through Wolverine’s messaging capabilities. Adding options especially for Kafka messaging and also for MQTT, Pulsar, and maybe SignalR is an obvious foundational piece to make that a reality.