Work is continuing on the “Critter Stack 2025” round of releases, but we have finally got an alpha release of Marten 8 (8.0.0-alpha-5) that’s good enough for friendly users and core team members to try out for feedback. 8.0 won’t be a huge release, but we’re making some substantial changes to the projections subsystem and this is where I’d personally love any and all feedback about the changes so far that I’m going to try to preview in this post.
Just know that first, here are the goals of the projection changes for Marten 8.0:
- Eliminate the code generation for projections altogether and instead using dynamic Lambda compilation with FastExpressionCompiler for the remaining convention-based projection approaches. That’s complete in this alpha release.
- Expand the support for strong typed identifiers (Vogen or StronglyTypedId or otherwise) across the public API of Marten. I’m personally sick to death of this issue and don’t particularly believe in the value of these infernal things, but the user community has spoken loudly. Some of the breaking API changes in this post were caused by expanding the strong typed identifier support.
- Better support explicit code options for all projection categories (single stream projections, multi-stream projections, flat table projections, or event projections)
- Extract the basic event sourcing types, abstractions, and most of the projection and event subscription support to a new shared JasperFx.Events library that is planned to be reusable between Marten and future “Critter” tools targeting Sql Server first, then maybe CosmosDb or DynamoDb. We’ll write a better migration guide later, but expect some types you may be using today to have moved namespaces. I was concerned before starting this work for the 2nd time that it would be a time consuming boondoggle that might not be worth the effort. After having largely completed this planned work I am still concerned that this was a time consuming boondoggle and opportunity cost. Alas.
- Some significant performance and scalability improvements for asynchronous projections and projection rebuilds that are still a work in progress
Alright, on to the changes.
Single Stream Projection
Probably the most common projection type is to aggregate a single event stream into a view of that stream as either a “write model” to support decision making in commands or a “read model” to support queries or user interfaces. In Marten 8, you will still use the SingleStreamProjection base class (CustomProjection is marked as obsolete in V8), but there’s one significant change that now you have to use a second generic type argument for the identity type of the projected document (blame the proliferation of strong typed identifiers for this), with this as an example:
// This example is using the old Apply/Create/ShouldDelete conventions
public class ItemProjection: SingleStreamProjection<Item, Guid>
{
public void Apply(Item item, ItemStarted started)
{
item.Started = true;
item.Description = started.Description;
}
public void Apply(Item item, IEvent<ItemWorked> worked)
{
// Nothing, I know, this is weird
}
public void Apply(Item item, ItemFinished finished)
{
item.Completed = true;
}
public override Item ApplyMetadata(Item aggregate, IEvent lastEvent)
{
// Apply the last timestamp
aggregate.LastModified = lastEvent.Timestamp;
var person = lastEvent.GetHeader("last-modified-by");
aggregate.LastModifiedBy = person?.ToString() ?? "System";
return aggregate;
}
}
The same Apply, Create, and ShouldDelete conventions from Marten 4-7 are still supported. You can also still just put those conventional methods directly on the aggregate type just like you could in Marten 4-7.
The inline lambda options are also still supported with the same method signatures:
public class TripProjection: SingleStreamProjection<Trip, Guid>
{
public TripProjection()
{
ProjectEvent<Arrival>((trip, e) => trip.State = e.State);
ProjectEvent<Travel>((trip, e) => trip.Traveled += e.TotalDistance());
ProjectEvent<TripEnded>((trip, e) =>
{
trip.Active = false;
trip.EndedOn = e.Day;
});
ProjectEventAsync<Breakdown>(async (session, trip, e) =>
{
var repairShop = await session.Query<RepairShop>()
.Where(x => x.State == trip.State)
.FirstOrDefaultAsync();
trip.RepairShopId = repairShop?.Id;
});
}
}
So far the only different from Marten 4-7 is the additional type argument for the identity. Now let’s get into the new options for explicit code when either you just prefer that way, or your logic is too complex for the limited conventional approach.
First, let’s say that you want to use explicit code to “evolve” the state of an aggregated projection, but you won’t need any additional data lookups except for the event data. In this case, you can override the Evolve method as shown below:
public class WeirdCustomAggregation: SingleStreamProjection<MyAggregate, Guid>
{
public WeirdCustomAggregation()
{
ProjectionName = "Weird";
}
public override MyAggregate Evolve(MyAggregate snapshot, Guid id, IEvent e)
{
// Given the current snapshot and an event, "evolve" the aggregate
// to the next version.
// And snapshot can be null, just meaning it hasn't been
// started yet, so start it here
snapshot ??= new MyAggregate(){ Id = id };
switch (e.Data)
{
case AEvent:
snapshot.ACount++;
break;
case BEvent:
snapshot.BCount++;
break;
case CEvent:
snapshot.CCount++;
break;
case DEvent:
snapshot.DCount++;
break;
}
return snapshot;
}
}
I should note that you may want to explicitly configure what event types the projection is interested in as a way to optimize the projection when running in the async daemon.
Now, if you want to “evolve” a snapshot with explicit code, but you might need to do query some reference data as you do that, you can instead override the asynchronous EvolveAsync method with this signature:
public virtual ValueTask<TDoc?> EvolveAsync(TDoc? snapshot, TId id, TQuerySession session, IEvent e,
CancellationToken cancellation)
But wait, there’s (unfortunately) more options! In the recipes above, you’re assuming that the single stream projection has a simplistic lifecycle of being created, updated one or more times, then maybe being deleted and/or archived. But what if you have some kind of complex workflow where the projected document for a single event stream might be repeatedly created, deleted, then restarted? We had to originally introduce the CustomProjection mechanism to Marten 6/7 as a way of accommodating complex workflows, especially when they involved soft deletes of the projected documents. In Marten 8, we’re (for now) proposing reentrant workflows with this syntax by overriding the DetermineAction() method like so:
public class StartAndStopProjection: SingleStreamProjection<StartAndStopAggregate, Guid>
{
public StartAndStopProjection()
{
// This is an optional, but potentially important optimization
// for the async daemon so that it sets up an allow list
// of the event types that will be run through this projection
IncludeType<Start>();
IncludeType<End>();
IncludeType<Restart>();
IncludeType<Increment>();
}
public override (StartAndStopAggregate?, ActionType) DetermineAction(StartAndStopAggregate? snapshot, Guid identity,
IReadOnlyList<IEvent> events)
{
var actionType = ActionType.Store;
if (snapshot == null && events.HasNoEventsOfType<Start>())
{
return (snapshot, ActionType.Nothing);
}
var eventData = events.ToQueueOfEventData();
while (eventData.Any())
{
var data = eventData.Dequeue();
switch (data)
{
case Start:
snapshot = new StartAndStopAggregate
{
// Have to assign the identity ourselves
Id = identity
};
break;
case Increment when snapshot is { Deleted: false }:
if (actionType == ActionType.StoreThenSoftDelete) continue;
// Use explicit code to only apply this event
// if the snapshot already exists
snapshot.Increment();
break;
case End when snapshot is { Deleted: false }:
// This will be a "soft delete" because the snapshot type
// implements the IDeleted interface
snapshot.Deleted = true;
actionType = ActionType.StoreThenSoftDelete;
break;
case Restart when snapshot == null || snapshot.Deleted:
// Got to "undo" the soft delete status
actionType = ActionType.UnDeleteAndStore;
snapshot.Deleted = false;
break;
}
}
return (snapshot, actionType);
}
}
And of course, since *some* of you will do even more complex things that will require making database calls through Marten or maybe even calling into external web services, there’s an asynchronous alternative as well with this signature:
public virtual ValueTask<(TDoc?, ActionType)> DetermineActionAsync(TQuerySession session,
TDoc? snapshot,
TId identity,
IIdentitySetter<TDoc, TId> identitySetter,
IReadOnlyList<IEvent> events,
CancellationToken cancellation)
Multi-Stream Projections
Multi-stream projections are similar in mechanism to single stream projections, but there’s an extra step of “slicing” or grouping events across event streams into related aggregate documents. Experienced Marten users will be aware that the “slicing” API in Marten has not been the most usable API in the world. I think that even though it didn’t change *that* much in Marten 8, the “slicing” will still be easier to use.
First, here’s a sample multi-stream projection that didn’t change at all from Marten 7:
public class DayProjection: MultiStreamProjection<Day, int>
{
public DayProjection()
{
// Tell the projection how to group the events
// by Day document
Identity<IDayEvent>(x => x.Day);
// This just lets the projection work independently
// on each Movement child of the Travel event
// as if it were its own event
FanOut<Travel, Movement>(x => x.Movements);
// You can also access Event data
FanOut<Travel, Stop>(x => x.Data.Stops);
ProjectionName = "Day";
// Opt into 2nd level caching of up to 100
// most recently encountered aggregates as a
// performance optimization
Options.CacheLimitPerTenant = 1000;
// With large event stores of relatively small
// event objects, moving this number up from the
// default can greatly improve throughput and especially
// improve projection rebuild times
Options.BatchSize = 5000;
}
public void Apply(Day day, TripStarted e)
{
day.Started++;
}
public void Apply(Day day, TripEnded e)
{
day.Ended++;
}
public void Apply(Day day, Movement e)
{
switch (e.Direction)
{
case Direction.East:
day.East += e.Distance;
break;
case Direction.North:
day.North += e.Distance;
break;
case Direction.South:
day.South += e.Distance;
break;
case Direction.West:
day.West += e.Distance;
break;
default:
throw new ArgumentOutOfRangeException();
}
}
public void Apply(Day day, Stop e)
{
day.Stops++;
}
}
The options to use conventional Apply/Create methods or to override Evolve, EvolveAsync, DetermineAction, or DetermineActionAsync are identical to SingleStreamProjection.
Now, on to a more complicated “slicing” sample with custom code:
public class UserGroupsAssignmentProjection: MultiStreamProjection<UserGroupsAssignment, Guid>
{
public UserGroupsAssignmentProjection()
{
CustomGrouping((_, events, group) =>
{
group.AddEvents<UserRegistered>(@event => @event.UserId, events);
group.AddEvents<MultipleUsersAssignedToGroup>(@event => @event.UserIds, events);
return Task.CompletedTask;
});
}
I know it’s not that much simpler than Marten 8, but one thing Marten 8 is doing is handling tenancy grouping behind the scenes for you so that you can just focus on defining how events apply to different groupings. The sample above shaves 3-4 lines of code and a level or two of nesting from the Marten 7 equivalent.
EventProjection and FlatTableProjection
The existing EventProjection and FlatTableProjection models are supported in their entirety, but we will have a new explicit code option with this signature:
public virtual ValueTask ApplyAsync(TOperations operations, IEvent e, CancellationToken cancellation)
And of course, you can still just write a custom IProjection class to go straight down to the metal with all your own code, but that’s been simplified a little bit from Marten 7 such that you don’t have to care about whether it’s running Inline or in Async lifetimes:
public class QuestPatchTestProjection: IProjection
{
public Guid Id { get; set; }
public string Name { get; set; }
public Task ApplyAsync(IDocumentOperations operations, IReadOnlyList<IEvent> events, CancellationToken cancellation)
{
var questEvents = events.Select(s => s.Data);
foreach (var @event in questEvents)
{
if (@event is Quest quest)
{
operations.Store(new QuestPatchTestProjection { Id = quest.Id });
}
else if (@event is QuestStarted started)
{
operations.Patch<QuestPatchTestProjection>(started.Id).Set(x => x.Name, "New Name");
}
}
return Task.CompletedTask;
}
}
What’s Still to Come?
I’m admittedly cutting this post short just because I’m a good (okay, not horrible) Dad and it’s time to do bedtime in a minute. Beyond just responding to whatever feedback comes in, there’s some more test cases for the explicit coding options, more samples to write for documentation, and a seemingly endless array of use cases for strong typed identifiers.
Beyond that, there’s still a significant effort to come with Marten 8 to try some performance and scalability optimizations for asynchronous projections, but I’ll warn you all that anything too complex is likely to land in our theoretical paid add on model.
The Shade Tree Developer 