Ten Days of Critter Stack Releases

Most of the big improvements in this blog post came from JasperFx client engagements. Reach out any time to sales@jasperfx.net and we’ll happily chat with you about how we can help your shop succeed with whatever technical challenges you might have!

Let me put a stake into the ground here and say that you simply cannot (yet) vibe code yourself an equivalent of the “Critter Stack” because so much of our deep quality is the direct result of adapting to real life usages and problems over years of constant usage and continuous improvement. In the past we’ve worked with JasperFx clients or the community on issues caused by database maintenance shutdowns, way too many Kubernetes related things as pods spin up and down, and all kinds of unexpected real like episodes that have all directly led to real improvements in the tools. We’ve faced issues from sudden system surges due to unexpectedly big system inputs like import files. We’ve had to endlessly harden Marten, Polecat, and Wolverine against infrastructure hiccups and random disconnects our users have faced in real life usage. You simply cannot get that level of built in quality by “rolling your own” over a long weekend.

The last ten days have been one of the heaviest release stretches in the history of the critter stack. Every repository in the family shipped, and the three headline stories are all about the same thing: what happens when a system is under real load, on real hardware, at real scale, and something goes wrong.

Here’s what moved:

Package	Where we were on July 17	Where we are today
Wolverine	6.20.0	6.23.1
Marten	9.16.0	9.20.0
Polecat	5.1.0	5.7.0
Weasel	9.16.4	9.19.0
JasperFx / JasperFx.Events	2.28.0	2.36.2

That’s 4 Wolverine releases, 5 Marten releases, 6 Polecat releases, 3 Weasel releases, and 9 JasperFx releases in ten days. Below are the three things I most want you to know about, followed by everything else.

---

1. Marten’s Async Daemon Tells You Why It Stopped

This work was inspired by helping a JasperFx client troubleshoot issues last week

The single most frustrating failure mode in an event-sourced system is the silent one: a projection stops advancing, and the only evidence is a chart that flatlines. The daemon knew perfectly well what happened — it just had nowhere to put it.

Classified shard failures

This will be exposed through the CritterWatch user interface and MCP tools in the 1.0 RC release

The async daemon now classifies why a shard is paused or stopped and persists it, so a monitoring tool polling the database sees exactly what an in-process observer sees:

var states = await store.Storage.Database.AllProjectionProgress();
foreach (var state in states.Where(x => x.Failure != null))
{
    // ApplyEvent, EventSerialization, UnknownEventType, ProgressionOutOfOrder, or Other
    Console.WriteLine($"{state.ShardName}: {state.Failure!.Category} on {state.Failure.Event}");
}

ShardFailure is a plain, serializable record — category, the failing event’s sequence and type, the exception message and detail — deliberately not an Exception, so it survives the trip to a monitoring UI. Extended progression tracking grew four new columns to carry it (failure_category, failure_event_sequence, failure_event_type, failure_event_tenant_id), and failure_category stores the enum name rather than its ordinal so reordering the enum in a future release can never silently re-label rows an older deployment wrote.

The distinction between categories is the whole point. EventSerialization means a stored body won’t deserialize — you need a serializer or data fix. UnknownEventType means an event alias resolves to no known .NET type in this deployment — usually a missing registration or a rollback past the point where that event type was introduced. Those are different problems with different fixes, and the daemon now says which one you have. A shard that recovers clears its failure columns on the next successful start, so a supervisor built on this doesn’t keep alerting on something you fixed an hour ago.

Graceful shutdown and the drain timeout

This improvement will also help a great deal when Marten/Wolverine decides to rebalance work across a cluster of nodes as you might be scaling up or down

When a shard is stopped, the daemon doesn’t simply cancel it — it drains: lets the in-flight page of events finish applying, then flushes the progression row so the next start picks up exactly where this one left off. If that drain gets cut short, the shard restarts against a stale progression row and throws ProgressionProgressOutOfOrderException. That’s now bounded, per shard, and configurable:

// The default is 5 seconds
opts.Projections.StopAndDrainTimeout = 30.Seconds();

The motivating case is a database-per-tenant deployment with thousands of (projection × tenant) shards all trying to drain inside a Kubernetes termination grace window. A per-shard bound only helps if the process lives long enough to spend it, so pair a raised StopAndDrainTimeout with HostOptions.ShutdownTimeout and the pod’s terminationGracePeriodSeconds. Full write-up in Graceful Shutdown and the Drain Timeout.

The high-water health check became more effective

This part really only impacts users using the new per-tenant event store partitioning — but that’s going to be one of our answers for extreme scalability needs*

The high-water health check previously probed every database in a multi-tenanted store on every probe — a connection fan-out that gets ugly at a few hundred shard databases, and outright wrong when daemon distribution is spread across nodes and a node ends up probing databases it doesn’t host:

Services.AddHealthChecks().AddMartenHighWaterHealthCheck(
    staleThreshold: TimeSpan.FromSeconds(30),

    // Only probe the databases this node actually owns
    databaseFilter: db => LocallyOwnedDatabaseIdentifiers.Contains(db.Identifier),

    // Assert even under DaemonMode.ExternallyManaged (i.e. Wolverine-managed distribution)
    includeExternallyManaged: true);

Under UseTenantPartitionedEvents the check now evaluates the per-tenant HighWaterMark:<tenant> progression rows too, using the liveness heartbeat signal (the sequence-gap fallback is store-global and can’t be applied per tenant).

More reliable integration testing against asynchronous projections

This gobbledygook should really translate to “automated testing against asynchronous projections just got faster and more reliable”

Two long-tail concurrency bugs went with it: the high-water agent’s lost-wakeup race was closed rather than narrowed (jasperfx#572), and a WaitForShardState race against an already-published state was fixed (jasperfx#568). On the PostgreSQL side, Marten 9.20 added an allocation fence so an idle advisory-lock session can no longer hold gap skips open forever (marten#4953) — a fix that had a direct Wolverine counterpart, more on that below.

---

2. Wolverine’s Agent Assignment Got Its Hard Lesson

We’ve had confirmation from a JasperFx client that these changes made a dramatic improvement in how Wolverine behaved in a hugely complicated system, but I expect this to be an improvement for plenty of other users as well

This one started as an incident report (from a pretty extremely complicated usage well beyond what most people will ever experience) and turned into a nine-part fix.

The setup: a Wolverine cluster distributing thousands of Marten subscription and projection agents across nodes, under rebuild load. The symptom: Wolverine basically panicked and continuously tried to start, stop, and re-assign agents to diffent nodes because it couldn’t tell if anything was healthy or not. Nodes were being ejected while very much alive, resurrecting under new identities, and the leader re-sent the same assignments forever while nothing actually started. The database was taking roughly 96,000 telemetry inserts an hour from the churn alone — into the very database the rebuild was already saturating.

Nine separate defects fed that livelock. All of them are fixed:

The heartbeat was starved by its own work. The node heartbeat was written as the first step of the health-check loop, which also drained agent commands serially. A leader spending sixty seconds burning reply timeouts while starting thousands of subscription agents therefore delayed its own next heartbeat past StaleNodeTimeout — looking dead to its peers precisely when it was doing the most work. The heartbeat now runs on its own independent loop, so no amount of slow command work can starve it.

Resurrection restored a skeleton, not a node. When a peer deleted a still-live node’s row, every store blindly re-inserted a skeleton: fresh node number, empty capabilities, no assignments. A capability-less node is a candidate for nothing, so a 3-node cluster silently shrank to 2 for event-subscription work. MarkHealthCheckAsync now reports existence without ever inserting, and the controller re-registers with the node’s real number, its captured capabilities, and its agent assignment rows — implemented across all nine persistence stores (PostgreSQL, SQL Server, MySQL, SQLite, Oracle, RavenDB, CosmosDB, and the two in-memory/multi-tenanted wrappers).

A 2,100-agent batch answered by a 30-second timeout. Assignments went out as one mega-batch and were started serially on the receiving node — where each Marten subscription-agent start is a daemon shard spin-up with database round trips. That is hours of serial work answered by a 30-second reply window: the reply can never arrive, so the leader records nothing and re-sends the whole ~300KB batch next cycle. Batches are now chunked, started with bounded parallelism, and the reply timeout scales with chunk size.

The leader re-emitted assignments it had already sent. A leader-side pending-assignment ledger now suppresses duplicate AssignAgent commands for work already in flight, with a TTL so a start that never took still gets re-driven. That also removed the matching telemetry-write flood.

Ejection had no hysteresis. A single stale snapshot read — replica lag, a GC pause, an aggressive StaleNodeTimeout — was enough to delete a live node’s row, its in-flight envelope ownership, and its assignments. The irreversible delete now requires N consecutive stale observations, and a follower may never delete the leader’s row; only a node actually holding the leadership lock can do that.

Shutdown couldn’t finish inside a grace window. The node-shutdown drain stopped every local agent serially, so a node with thousands of shards got SIGKILLed mid-drain, abandoning unflushed daemon progression. It now fans out with bounded parallelism, passes CancellationToken.None deliberately (this is the shutdown path — a cancelled drain leaves agents half-stopped), and contains a wedged agent so it can’t abort its peers’ drain.

Every one of these knobs is on Durability, and the defaults are the ones we’d pick for you:

opts.Durability.AgentStartBatchSize        = 50;   // chunk size for assignment batches
opts.Durability.MaxAgentStartParallelism   = 10;   // bounded fan-out starting a chunk
opts.Durability.MaxAgentStopParallelism    = 10;   // symmetric, on the shutdown drain
opts.Durability.StaleNodeEjectionThreshold = 2;    // consecutive stale reads before ejection

Surfacing a paused shard

CritterWatch will use this new capability to help your systems be more resilient

Wolverine deliberately does not restart a shard the daemon paused on a poison event — restarting would fail on the identical event, so the shard would thrash instead of advance. But “we’re not going to restart it” is only defensible if you know. So Wolverine now surfaces it four ways: the agent’s health check reports the failure category, failing event, and root exception type; a NodeRecordType.AgentPaused record lands in the node-record log; IEventSubscriptionAgent.Failure exposes the ShardFailure directly; and there’s an observer hook:

public class AlertingObserver : IWolverineObserver
{
    public Task AgentPaused(Uri agentUri, ShardFailure? failure)
    {
        // Fires once per transition into the failed state -- not on every health check tick
        _alerts.Raise($"{agentUri} paused: {failure?.Category} on event {failure?.Event}");
        return Task.CompletedTask;
    }
}

Only the Other category — a database outage, a timeout, a transient bug, anything you can’t pin on a single event — is treated as potentially self-healing and auto-restarted by the stall detector. Details and the full category table are in When a Projection Fails; it applies identically to Polecat.

Agent start retries

An agent’s very first assignment can race the subsystems it depends on coming up — a subscription shard evaluated before its store’s high-water detection is running, for instance. Previously the loser of a sub-second startup race idled for a full CheckAssignmentPeriod. Now it retries locally first:

opts.Durability.AgentStartRetryAttempts = 2;                            // default; 0 disables
opts.Durability.AgentStartRetryDelay    = TimeSpan.FromMilliseconds(250); // default, × attempt number

See Agent Start Retries.

And the 6.23.1 follow-ups

Three fixes landed on top, all from running the fixed code against real deployments:

• A pending assignment is now confirmed on delivery rather than on continued assignment — a 6.23.0 regression where a pause→restart cycle left the ledger entry unconfirmed forever.
• Advisory-lock session hygiene for Marten’s gap-liveness gate, the Wolverine-side twin of marten#4953.
• Agent restriction changes are merged and persisted before health detection is kickstarted.

---

3. Polecat Got Materially Faster

Hey, we’re serious about making Polecat a first class citizen within the greater Critter Stack

Polecat — the SQL Server document database and event store — spent this window on performance, and one of the finds was some serious egg on my face.

A one-word bug that cost 6x on string identities

String identity columns in Polecat (pc_streams.id, pc_events.stream_id, tenant_id, document ids, progression names, tag values) are varchar(250). String parameters were being bound as nvarchar. SQL Server’s data-type precedence rules then convert the column side, not the parameter — so every single id lookup became CONVERT_IMPLICIT(...) over a full index scan instead of a seek.

The numbers, at 50k streams under a SQL collation: StreamIdentity.AsString appends ran at 53/sec versus 304/sec for AsGuid. Version reads were 7,814µs across 990 reads versus 42µs across 3 reads. That’s not a tuning opportunity, that’s a missing index seek on every string-keyed operation in the store.

Every bespoke site that filters a varchar column now binds through AddVarChar/AddIdParameter helpers with a fixed size for plan-cache stability: version reads, FetchStream, FetchForWriting, document exists/metadata, batched loads, DCB tag queries, natural-key operations, the daemon loader and high-water detector, progression, HiLo, and the rebuild/delete admin paths. If you use string stream keys on SQL Server, upgrade to 5.6.0 or later — this one is free.

Server-side Select() projections

On SQL Server 2025’s native json type, a “simple” Select() projection — an anonymous type or DTO composed only of (optionally nested) scalar member accesses — is now translated to a server-side JSON_OBJECT(...) and streamed with no hydrate/reserialize step at all. Emitted keys honor your serializer’s naming policy and [JsonPropertyName]; numbers stay numbers and strings stay quoted.

Two correctness guards ship regardless of whether the optimization kicks in: a non-translatable Select() falls back to a client-side transform when materialized with ToListAsync() (never a silent drop), and attempting to stream a client-side-fallback projection now throws BadLinqExpressionException instead of silently ignoring the Select and returning raw documents.

Streaming paged JSON in one round trip

Both Marten and Polecat now have the full raw-JSON streaming result family, byte-for-byte compatible with each other so clients are interchangeable:

app.MapGet("/issues/paged/{pageNumber:int}/{pageSize:int}",
    (int pageNumber, int pageSize, IQuerySession session) =>
        new StreamPaged<Issue>(session.Query<Issue>().OrderBy(x => x.Description), pageNumber, pageSize));

{"pageNumber":3,"pageSize":25,"totalItemCount":1207,"pageCount":49,"hasNextPage":true,"hasPreviousPage":true,"items":[...]}

The total row count rides along on every row via count(*) OVER() in the same query that fetches the page — so count and documents both come from a single database round trip — and the documents inside items are the already-persisted JSON, streamed straight through with no deserialize/serialize.

For infinite scroll and export feeds, StreamPagedByCursor<T> does keyset (seek) pagination with an opaque, versioned cursor, at constant cost regardless of depth:

app.MapGet("/issues/feed", (string? cursor, IQuerySession session) =>
    new StreamPagedByCursor<Issue>(
        session.Query<Issue>().OrderBy(x => x.Description).ThenBy(x => x.Id), cursor, pageSize: 25));

The terminal ordering key must be the document identity so the ordering is a total order — that’s enforced, not assumed. See Keyset (Cursor) Pagination and Polecat’s paging docs.

Marten 9.20 rounded the family out with StreamEventState and StreamEvents for streaming a single stream’s metadata and raw events (docs).

Batched event fetches

FetchStreamStatePlan and FetchStreamPlan landed in both Marten and Polecat, and Polecat gained a batched event surface it simply didn’t have — batch.Events, with FetchStreamState and FetchStream in Guid and string overloads. Both new batch items compose their SQL from the same canonical column projections and hydrate through the same readers as the standalone path, so batched and standalone can’t drift apart across a schema migration. See Batched Queries.

Native SQL Server 2025 JSON indexes

One JSON index covers many paths at once and accelerates JSON_VALUE equality, JSON_PATH_EXISTS, and JSON_CONTAINS — with no per-path computed columns:

opts.Schema.For<User>().JsonIndex(x => new { x.UserName, x.Department });
opts.Schema.For<Document>().JsonIndex();                                    // whole-document
opts.Schema.For<Article>().JsonIndex(x => x.Tags, i => i.OptimizeForArraySearch = true);

This is the SQL Server counterpart to Marten’s GinIndexJsonData(), and it requires the native json column type (SQL Server 2025) — Polecat throws a clear error rather than emitting invalid DDL if you configure one against nvarchar(max). Covering indexes landed alongside it: Index(..)/UniqueIndex(..) now carry extra members as non-key INCLUDE columns so a query can be satisfied from the index alone. JSON Indexes docs.

Polecat also picked up per-tenant managed partitioning for documents and streams, AggregateToManyAsync(), the HasTag DCB tag operator in LINQ Where(), and tenant-scoped event/tag explorer reads.

---

4. Weasel Generates EF Core Migrations Now

I myself strongly prefer the “it just works” style of migrations that Marten and later Wolverine and Polecat do, but hey, a large plurality of the .NET community is probably very used to EF Core migrations, so we’re allowing our users to jump on board that train too!

This is a bigger deal than its version number suggests. Weasel 9.18 can emit standard, compilable EF Core migration files from its own schema model — the reverse of the mapping direction it already had. Instead of Weasel applying schema changes itself via db-patch/db-apply, your team applies them with the tooling it already standardized on: dotnet ef database update, idempotent SQL scripts, migration bundles, and versioned migration files a DBA can review.

dotnet run -- db-ef-migration add AddOrderProjection

That writes migration classes with real Up() and Down() bodies, a stub DbContext per database (with __EFMigrationsHistory relocated into the critter-stack schema so it can’t collide with your application’s own EF context), and a weasel-schema-snapshot.json that the next add diffs against. Everything flows through one door — IDatabase.AllObjects() — so Marten system tables, Wolverine envelope storage, Polecat event storage, and EF-projection tables all generate the same way. Verified end-to-end on both EF 9 and EF 10.

Docs: EF Core Migration Generation and Migration Coexistence.

Weasel 9.19 also gave Oracle a first-class command builder with real statement splitting — which is what unblocked Wolverine’s Oracle durability agent running through the shared batching mechanics. ODP.NET has no DbBatch support and won’t execute ;-separated statements in a single command, so this had to be solved at the Weasel layer.

One consistent finding across the decade plus of Critter Stack development is that database query batching is very frequently advantageous for performance, and we’ve taked that very seriously over the years

---

5. JasperFx: The Shared Core

Nine JasperFx releases in ten days, because it’s where the shared event-store and daemon abstractions live. The highlights, most of which you’ve already met above through Marten and Polecat:

• ShardFailure + ShardFailureCategory (jasperfx#565/#567) — the classified reason a shard paused, exposed on ISubscriptionAgent and persisted through extended progression.
• StopAndDrainTimeout (jasperfx#564) — the configurable per-shard drain bound.
• HighWaterAgent liveness heartbeat (jasperfx#539) — a staleness surface and a local restart seam, which is what the improved health check reads.
• Lost-wakeup race closed (jasperfx#572) and the WaitForShardState race against an already-published state (jasperfx#568).
• Batched extended-progression writes (jasperfx#553/#554) — per-database-flush-interval batching, so the telemetry write path stopped being a per-shard-per-tick insert.
• Natural key extraction widened to bind IEvent<T> sources, stop fabricating aggregates, and fail loudly rather than silently (jasperfx#569).
• DCB work: IDcbAggregateRegistry for runtime discovery, serializable rich EventTagQuery as a DCB source, and a step-instrumented aggregation fold with MultiAggregateProjectionResult.
• F# support got more robust tuple/record handling and a DerivedVariable reference-propagation fix.

---

6. Everything Else

A partial list, because the window was busy:

Wolverine

• Claim checks got size-threshold auto-offload, per-message/per-endpoint store selection, and honor a DI-registered IClaimCheckStore.
• GCP Pub/Sub: named-broker support for sharded/partitioned topics, plus ListenToPubsubSubscriptionOnNamedBroker.
• Conventional routing no longer ignores named brokers.
• Oracle: the durability agent runs through the shared batching mechanics; the durable inbox binds RAW(16) Guids correctly; the message store URI uses the registered wolverinedb agent scheme.
• Redis: scheduled retries no longer vanish on an unreadable timestamp, and entries that repeatedly fail to deserialize get dead-lettered instead of looping.
• NServiceBus interop: the EnclosedMessageTypes header is split before resolution, shared across Azure Service Bus, SNS, SQS, and the database transports.
• HTTP: a raft of OpenAPI and binding fixes — [FromQuery] on arrays and collections, case-insensitive enum array parsing, 415 instead of 404 when no Content-Type reaches an [AcceptsContentType] route, no duplicate description of route-bound [FromQuery]/[FromHeader] parameters, fail-fast when an endpoint advertises a body its HTTP method can’t carry, and explicitly-routed chains mapped inside the constructor so PublishMessage/SendMessage endpoints get their metadata.
• IHost.ClearAllWolverineStorageAsync(), and resources setup provisions message storage even under AutoCreate.None.
• Exclusive listener inboxes are now recovered on the listening node.

Marten

• TimescaleDB support — projection and document hypertables, folded into core Marten.
• Natural keys hardened: the previous key row is retired when the key changes, and the foreign key guard is scoped to its own table.
• Simple LINQ Select() projections translate to jsonb_build_object (the Postgres side of the same optimization Polecat got).
• ETag / If-None-Match (304) support on StreamOne and StreamAggregate.
• Tenant-scoped event and tag explorer reads.

---

Upgrading

In this case, everything in the critter stack moved in lockstep — Wolverine 6.23.1 pins Marten 9.20.0, Polecat 5.7.0, and JasperFx 2.36.1+, so upgrading Wolverine pulls the rest forward for you. If you’re on a Marten-or-Polecat-only application, take Marten 9.20.0 / Polecat 5.7.0 directly.

Nothing here is a breaking change. The agent-assignment work is entirely behavioral and needs no configuration to benefit from; the new Durability knobs exist for tuning, not for opting in. The classified shard-failure columns require extended progression tracking, which is still off by default — turn it on with Events.EnableExtendedProgressionTracking if you want database-visible per-shard health, and note that the per-tenant high-water health check needs it too.

If you’re running the critter stack at any real scale, CritterWatch consumes all of the new failure surfacing described above without any work on your part.

---

Closing Thoughts

I would dearly appreciate it if the world could slow down a bit in the next couple weeks so that release cadence can come back to Earth. I’d also appreciate it if everybody else could chill out a bit in their OSS activity so GitHub actions can be more performant and responsive for me and the Critter Stack community!