
Wolverine 6.30 is out. It is a big release with one clear headline: a new endpoint mode, EndpointMode.NativeAck, plus the supporting cast it needed to be trustworthy — lease renewal, an additional idempotency guard, partitioning, and a five-node chaos harness to address a reported issue. This release came with feature requests from JasperFx Software clients, and just to remind folks, being a JasperFx support client means that like Homer below, you have the secret handshake to get your Critter Stack needs and requests to the top of our priority list:
There is also a crop of transport fixes, two long-standing multi-tenancy gaps closed, and a couple of new HTTP and event sourcing conveniences. And inevitably, we also smuggled in some support for our soon forthcoming “Event Modeling” visualization support across the Critter Stack and more for CritterWatch visualization.
You’ve obviously detected that this post was largely drafted by AI of course, but the other side effect of AI usage is that we actually find many more little issues as we work and this 6.30 release includes quite a few bug fixes that spilled out of AI assisted efforts to improve our continuous integration safety nets and an embarrassing number of combinatorial bugs in the existing code that were discovered during the “Native Acks” effort.
Let’s take the tour.
🚀 The Fourth Endpoint Mode
This mode came about from interactions with a JasperFx client who needed to optimize the throughput of listening endpoints but still needed an “at least once” delivery guarantee. The existing Durable mode gives you the delivery guarantee and parallelization, but the extra database overhead was problematic for a flood of small messages that needed to be handled quickly. The Buffered mode gives you the maximal throughput, but a lesser delivery guarantee and is mostly suitable for “fire and forget” messages. Our Inline mode gives you the “at least once” guarantee without the database overhead of Durable, but limited parallelism and throughput. Our client used a different configuration change in Wolverine that alleviated their message pressure, but this new mode was going to be the next thing we tried for them if they needed it.
Wolverine has had three listener modes for a long time, and if you have configured a busy endpoint you have probably felt the shape of the tradeoff. You want parallelism. You want group-ordered processing. You do not want to lose messages when a pod gets rescheduled. And you would very much like to not stand up a database to get all three.
Until 6.30 you could have any two.
| Broker ack timing | Loss window | Parallelism | Group partitioning | DB cost | |
|---|---|---|---|---|---|
Inline | after handler success | none | ListenerCount only | none | none |
NativeAck | after handler success | none | MaximumParallelMessages | ✔️ | none |
BufferedInMemory | at receipt, before the handler | crash loses buffered messages | MaximumParallelMessages | ✔️ | none |
Durable | after the inbox insert | none | MaximumParallelMessages | ✔️ | inbox insert + mark-handled |
NativeAck fills the empty cell: Buffered’s throughput and partitioning with Inline’s no-loss guarantee, and no database involvement.
opts.ListenToRabbitQueue("webhooks") .ProcessInParallelWithNativeAcks() .PartitionProcessingByGroupId(PartitionSlots.Five) .MaximumParallelMessages(10);
The mechanism is exactly what the name says. The broker delivery is held unacknowledged while the message flows through an in-memory, optionally group-partitioned execution block, and is settled natively from the completion continuation — acked on handler success, nacked or dead-lettered on terminal failure. Nothing is written to a database. Nothing is acknowledged ahead of its handler.
Three consequences follow, and all three are the point rather than side effects:
- Back pressure is the broker’s prefetch window, not
BufferingLimits. The broker stops delivering once its unacked ceiling is reached, so there is noBackPressureAgentat all. - A dying node loses nothing. Anything queued but not yet completed is still unacknowledged, so closing the channel or crashing hands every one of those deliveries back to the broker.
- Shutdown redelivers, but redelivery is not duplicate execution. More on that below, because it is the part everyone gets wrong — including, until this release, our own documentation.
The guarantee, stated exactly
I want to be precise here rather than reassuring, because this is the sort of claim people build on:
Protection against intra-group concurrency is the hard guarantee. Strict sequential processing in original delivery order is not.
The sequential lane per group slot structurally guarantees that no two messages sharing a group id execute concurrently on the owning node. Original-order processing is not guaranteed under failure, requeue, or broker redelivery — a failed or redelivered message re-enters its lane later, never concurrently. That is the honest contract for native-ack retry semantics on every broker. If you need strict order under failure, keep the durable inbox.
📊 What a Redeployment Actually Costs
This was from the community.
Earlier versions of the docs said a rolling deploy produced “duplicate deliveries bounded by the prefetch depth.” That was honest reasoning. It was also never measured — and when we finally measured it, the figure turned out to be the wrong one to quote. It is the bound on redeliveries, and it overstates handler-visible duplicates by more than an order of magnitude.
So #3713 built the harness: a five node cluster, group-partitioned across five RabbitMQ slots, under a sustained flood deep enough that the broker was holding a full unacked window at every disruption, reading the broker’s own messages_unacknowledged at that instant. Ranges are across four consecutive runs.
| Scenario | Unacked at disruption | Duplicate executions | Rate |
|---|---|---|---|
| Steady state, no disruption | 0 | 0 | 0% |
| Rolling deploy, all 5 nodes drained and replaced | 180 | 0 | 0% |
| One node killed outright mid-flood | 180 | 3–4 | ~0.1% |
| Two hard kills plus two rolling replacements | 180 | 7–9 | ~0.05% |
Two findings matter more than the headline percentage:
A graceful rolling deploy costs zero duplicate executions. Draining settles the handlers that were already running, so nothing runs twice. The prefetch window is redelivered — those messages simply had not executed yet, so they run for the first time, and your handler cannot tell them from any other first delivery.
A hard kill costs about one duplicate per busy lane, not one per unacked message. Only handlers that were mid-flight when the connection died can run twice, and that population is the partition slot count, not the prefetch depth. Three or four per killed node against an unacked window of 180 — a factor of roughly 45, and it barely moved between runs.
Handlers still need to be idempotent; at-least-once is the contract and 0.1% of a flood is not a small number of messages. But size that work against in-flight lanes rather than against prefetch. The full writeup is in What a redeployment actually costs.
🔌 Transport Support: Opt-In and Default-Closed
Wolverine actually supports more messaging technology options than any other messaging tool in .NET, and man, that’s not always a blessing.
A transport must settle each delivery individually and tolerate settling out of order, because the execution block completes messages in handler-completion order rather than delivery order. Not every broker can express that, so support is opt-in and default-closed — calling ProcessInParallelWithNativeAcks() on a transport that has not claimed the mode throws at configuration time rather than degrading silently.
Seven transports qualified and shipped in this release:
| Transport | Docs | Issue |
|---|---|---|
| RabbitMQ | Native Ack Endpoints | #3708 |
| Amazon SQS | Native Ack Processing | #4050 |
| Azure Service Bus | Native ack endpoints | #4051 |
| NATS JetStream | Native Acks with Parallel Processing | #4053 |
| Redis Streams | Native Acks with Parallel Processing | #4046 |
| Pulsar | Native Ack Processing | #4047 |
| GCP Pub/Sub | Concurrency and flow control | #4052 |
Kafka cannot and is out of scope. A cumulative offset commit has no way to express a gap. This is not an oversight to be fixed later — it is what the storage model means.
Two transports also refuse the mode for particular endpoints whose own settings contradict it, again at bootstrap rather than at runtime. Pulsar’s AcknowledgeCumulative() reintroduces exactly the gap-less commit problem that disqualifies Kafka. And an SQS FIFO queue exists to guarantee ordering within a message group, which native-ack lanes deliberately do not preserve — and which partitioning by group id does not rescue, because SQS blocks a message group behind its own in-flight head. Both combinations are rejected by name.
Brokers that put a clock on an unsettled delivery
This is about protecting you from problems that arise from long executing messages when using any kind of native broker acknowledgement. This isn’t an issue at all in our Durable mode, but becomes an issue using any other endpoint mode. We had to get more serious about this with the advent or our new NativeAck mode in 6.30.
On SQS, Azure Service Bus, JetStream and Pub/Sub, an unsettled delivery is on a timer — visibility timeout, lock duration, AckWait. Wolverine renews that clock for every delivery still sitting in a lane, for as long as it sits there (#4048).
This is unconditional under this mode and not something you opt into. Lane queue time is unbounded by design, so an un-renewed native-ack endpoint would be a duplicate-delivery generator by construction rather than merely at risk under a slow handler. A transport that declares such a clock but does not implement renewal is refused at startup.
SQS is a good illustration of how far the per-transport tuning goes — MaxNumberOfMessages defaults down under this mode:
opts.ListenToSqsQueue("webhooks") .ProcessInParallelWithNativeAcks() .PartitionProcessingByGroupId(PartitionSlots.Five) .MaximumParallelMessages(10);
Instead of the usual 10, a native-ack endpoint receives twice the number of lanes that can be busy at once, clamped to the SQS maximum. Under every other mode the surplus messages in a batch are deleted before their handlers run, so a full batch is free and saves API calls. Here each one sits in a lane holding an unsettled delivery to renew and to redeliver on a crash. Setting the property explicitly always wins.
🛡️ The In-Memory Idempotency Guard
The durable inbox deduplicates on the primary key of its incoming table. NativeAck has no such table and is at-least-once by design, so 6.30 adds the non-durable analogue (#3710): an opt-in, bounded, in-memory set of the message ids this process has already handled on this endpoint.
opts.ListenToRabbitQueue("webhooks") .ProcessInParallelWithNativeAcks() .PartitionProcessingByGroupId(PartitionSlots.Five) // Opt in. Both arguments are optional; these are the defaults. .WithInMemoryIdempotency(window: 5.Minutes(), maxTracked: 100_000);
Or everywhere at once:
opts.Policies.AllListeners(x => x.WithInMemoryIdempotency());
Read the limits before you rely on it. The guard is per process and in memory, and three consequences follow — none of which is a bug:
- A restart forgets everything. The very deploy that produces the redelivery burst also empties the guard on the node that starts up. It protects a running process against a redelivery it saw itself.
- A second node never knew. With competing consumers, a redelivery can land on a different node than the original.
- Eviction is generational, not exact. An id is remembered for at least half the window and at most the whole window — less if a flood of unique ids hits
maxTrackedfirst.
The promise is at-least-once delivery with best-effort deduplication, not exactly-once. If you need hard deduplication across restarts and nodes, that is what the durable inbox is for. Memory is bounded by construction — two rotating hash sets, no per-entry timestamps, no LRU bookkeeping, roughly single-digit megabytes at the 100,000 default. Details in In-Memory Idempotency Guard.
⚠️ The Fix You Might Actually Be Affected By
Buried in the supporting work is #3712, and it is worth pulling out because it may already apply to you.
Wolverine now validates listener configuration coherence at bootstrap instead of silently ignoring settings that the chosen mode cannot honor. In writing those checks we discovered that RabbitMQ queues default to Inline — and Inline supports neither parallelism nor group partitioning.
Which means: if you built a sharded topology with PublishToShardedRabbitQueues() and did not explicitly add ConfigureListening(x => x.BufferedInMemory()), your partitioning was silently doing nothing. The configuration was accepted and ignored. Now it is rejected with an error that says so.
A companion fix (#4022) catches the related case where a local queue configured with ProcessInline() was accepted at configuration time and then threw a message-less NotSupportedException from deep inside agent startup.
The partitioning guide covers the topology options, including the NativeAck variant added in #3709 — partitioned clustering without the durable inbox, across all ten transports that support the sharded topology.
🏢 Multi-Tenancy: Conjoined EF Core Under a Marten-Owned Store
#4044 is a small feature with a genuinely sneaky root cause.
If Marten owns your message storage through IntegrateWithWolverine(), Wolverine’s message store is built from Marten’s NpgsqlDataSource and never sees a connection string. And NpgsqlDataSource.ConnectionString deliberately omits the password — so there is no string the conjoined DbContext could be configured with that the database would actually accept. The obvious fix (pull the connection string off the data source) produces an authentication failure at first use.
So there is a new DbDataSource overload that carries the credentials through intact:
var builder = Host.CreateApplicationBuilder(); var configuration = builder.Configuration; builder.UseWolverine(opts => { // Marten owns the message storage here, so Wolverine's message store is built // from Marten's NpgsqlDataSource rather than from a connection string opts.Services.AddMarten(m => { m.Connection(configuration.GetConnectionString("main")!); }).IntegrateWithWolverine(); // ...which means the conjoined DbContext has to be configured from that same // DbDataSource. NpgsqlDataSource.ConnectionString deliberately omits the password, // so the connection string overload cannot authenticate in this setup opts.Services.AddDbContextWithWolverineManagedConjoinedTenancy<ConjoinedItemsDbContext>( (builder, dataSource) => { builder.UseNpgsql((NpgsqlDataSource)dataSource); }, AutoCreate.CreateOrUpdate); });
Registering the connection string overload in this setup now fails fast at startup with an error naming this overload, rather than surfacing later as an authentication failure.
A second defect on the same path is fixed too: IntegrateWithWolverine() never registered the tenant partitioning provider, so PartitionPerTenant() failed outright. Both are covered in With Marten Owning the Message Store.
🌐 HTTP and Event Sourcing
Marten concurrency conflicts as 409
This was a JasperFx client request.
An endpoint using [WriteAggregate] can lose an optimistic concurrency race — two clients posting to the same aggregate at once. Without a handler that escapes as an unhandled 500, even though nothing went wrong: optimistic concurrency did its job. 409 Conflict is the honest status.
6.30 ships a documented, tested recipe (#3764). The OnException middleware convention is all you need — but there are two exception types, and the second is easy to miss:
public static class MartenConcurrencyExceptionMiddleware { // Marten's optimistic concurrency failures -- EventStreamUnexpectedMaxEventIdException from // the event store, and document level concurrency violations -- all derive from // JasperFx.ConcurrencyException, so one handler covers them public static ProblemDetails OnException(ConcurrencyException ex) { return new ProblemDetails { Status = 409, Title = "Conflict", Detail = ex.Message }; } // StreamLockedException does NOT derive from ConcurrencyException -- it is a MartenException -- // so the FetchForExclusiveWriting path needs its own handler. Catching only ConcurrencyException // silently misses it public static ProblemDetails OnException(StreamLockedException ex) { return new ProblemDetails { Status = 409, Title = "Conflict", Detail = ex.Message }; } }
That second handler is the whole reason this is a documented recipe rather than a one-liner in a FAQ. Marten.Exceptions.StreamLockedException — what FetchForExclusiveWriting throws on a contended stream — derives from MartenException, not ConcurrencyException. A recipe that catches only the latter silently leaves the exclusive locking path returning 500s. See Recipe: Marten Concurrency Conflicts as 409.
[StreamState] and [StreamEvents]
New parameter attributes for handlers whose read is the raw stream rather than the folded aggregate — timeline views, audit endpoints, anything [ReadModel] cannot express (#3627).
They are store-agnostic across Marten, Polecat and Fisher, and on Marten both fetches are batched into a single round trip.
Event Model slices per route
HttpChainDescriptor and GrpcRpcDescriptor now carry the slice the route is, so a consumer walking endpoint by endpoint sees it next to the route rather than only through the assembled model (#4000).
🐛 Transport Fixes
The AI tools are sometimes good about finding combinatorial or lurking bugs while doing other work. It’s annoying, but I always ask Claude to file issues for any unrelated problems it finds while doing any work — then immediately turn around and try to address them.
A good crop this time, and several of them share a theme worth naming: silent failure. Each of these was doing the wrong thing without reporting anything.
Pulsar
- Requeue, scheduled retry and dead-letter routing were simply unimplemented (#3797).
- A global native-resiliency failure rule was swallowing every user-configured error policy in the entire application (#4079, also reported as #4075). A plain local queue in a
UsePulsar()host got exactly one attempt. If you use Pulsar and have ever wondered why anOnExceptionpolicy seemed inert, this is why. - Hot-tail listeners silently dropped deferred messages in every mode (#4060).
GCP Pub/Sub
- Listener shutdown could hang waiting on in-flight callbacks (#4065).
- Exhausting
MaxTotalAckExtensionsilently delivered a concurrent duplicate rather than reporting anything (#4066). - Effective listener concurrency was not what the configuration implied — the flow-control bound is global per
SubscriberClientrather than per inner client (#4067). Now documented in Concurrency and flow control, along with the sharp edge that ordering keys cap concurrency at the number of distinct group ids. PubsubTopicOptions.OrderBygained a configuration surface (#4087, docs).
Redis
DeleteStreamEntryOnAcksilently never acked on Redis < 8.2, whereXACKDELis unsupported (#4058).
Ack reliability
- A shared ack-attempt budget across stacked retry blocks, plus terminal-failure classification for Azure Service Bus and SQS so a permanent settle failure stops rather than burning the whole budget (#4012 — partially delivered; the remaining items are tracked there for 6.31).
⬆️ Upgrading
This release is additive. EndpointMode.NativeAck is opt-in per endpoint and default-closed per transport, and MaximumBrokerRedeliveries defaults to off.
Requires JasperFx 2.55.0.
The one thing to look at before upgrading is #3712, described above — the new listener coherence validation will reject at startup a configuration it previously accepted and ignored. That is a change in behavior, but the configuration it rejects was never doing what it looked like it was doing.
What’s Next
NativeAck is probably overdue and another option for Wolverine usage. We’ll be releasing a new version of our curated AI Skills this week that builds in decision making about endpoint usage in Wolverine applications.
Find us in the Critter Stack Discord or open an issue on GitHub.








