Wolverine 6.13 dropped yesterday with quite a few refinements to our existing integration with Kafka in Wolverine. We had, of course, probably fallen into the trap of just trying to make Kafka behave like a Rabbit MQ analogue. It was already on my radar to get enable more idiomatic Kafka capabilities, and a community request this week was a little impetus to finally go do this. I will admit that we could possibly stand to reorganize the Kafka related documentation inside of https://wolverinefx.io, but for right now, I’m hoping we can declare victory on our Kafka integration story for a bit.
Smarter Batch Sending
I’m beyond embarrassed over this one. For whatever reason, at some point we had disabled real batch sending with Kafka during some kind of troubleshooting and a user spotted that in the past couple weeks.That major oversight is now addressed. That should increase outbound throughput to Kafka from a Wolverine application by a potentially considerable amount (~20X improvement according to our benchmarking). Ouch, but hey, it’s better now!
#3150 — Kafka: commit-strategy overhaul with CommitMode replaces that with an explicit, opt-in strategy that defaults to the idiomatic non-blocking path:
opts.UseKafka(connectionString) .ConfigureListeners(l => l.CommitOffsets(CommitMode.StoreThenAutoFlush));
The four modes:
| Mode | What it does | When to reach for it |
|---|---|---|
StoreThenAutoFlush (default) | EnableAutoOffsetStore=false + StoreOffset per completed message; Kafka’s background committer flushes on AutoCommitIntervalMs | The new default — idiomatic Kafka throughput |
PerMessage | Synchronous commit of the message’s own offset | Strict at-least-once on low-volume topics |
BatchCount(n) | Commit watermark every N messages | High-volume topics where you want a tunable lever |
BatchInterval(t) | Commit watermark every T elapsed | Bursty traffic |
A subtle but important correctness fix rides along: CompleteAsync and the DLQ paths now commit the message’s specific TopicPartitionOffset (offset + 1), not the consumer’s global position. That was a prerequisite for every concurrency feature below.
If you’d already set EnableAutoCommit=true on the Kafka client, Wolverine now respects that and issues no manual commits at all — the previous transport blanket-overrode it.
And: in-flight-safe watermarks for every mode
In Wolverine’s default buffered listener (handlers running at MaxDegreeOfParallelism), messages can complete out of order. The original Batch strategy tracked an in-flight watermark; the new StoreThenAutoFlush and PerMessage strategies initially did not, which meant a fast-completing offset 11 could advance the committed position past a still-in-flight offset 10 — and on a crash, that 10 would be silently dropped.
#3161 — in-flight-safe offset watermark for all commit strategies routes all three manual strategies through a per-partition OffsetWatermark. The committable position is now the lowest still-in-flight offset, or high-water + 1 when nothing is in flight. It never advances past in-flight work, it’s monotonic across re-seeks, and it tolerates the offset gaps that compacted or read_committed transactional topics produce.
Scale-out, the way Kafka actually wants you to do it
The next two PRs make Kafka’s own group coordinator the recommended path to scale Wolverine handlers across nodes.
#3139 — Cooperative-sticky rebalancing + static membership
Two opt-in knobs that any production Kafka deployment will recognize:
opts.UseKafka(connectionString) .UseCooperativeStickyAssignment() // incremental rebalances .UseStaticMembership(); // POD_NAME → HOSTNAME → machine name
UseCooperativeStickyAssignment()setspartition.assignment.strategy = CooperativeSticky, so a rebalance only moves the partitions that need to move — the rest of the group keeps working uninterrupted.UseStaticMembership()setsgroup.instance.idso a rolling restart of the same pod doesn’t churn the partition map. Instance id is resolved fromPOD_NAME→HOSTNAME→ machine name (the k8s StatefulSet idiom), and Wolverine logs the resolved id at startup so you can verify per-node uniqueness.
Both are opt-in so you don’t break a live rolling upgrade by silently switching assignment strategies. The Kafka docs section now spells out the two-step rolling onto cooperative-sticky.
#3140 — Opt-in intra-partition concurrency by key
The second concurrency lever. Within a single partition assigned to your node, process messages with different keys concurrently while preserving strict ordering per key:
opts.ListenToKafkaTopic("orders") .ProcessConcurrentlyByKey(PartitionSlots: 8);
The trick is that this reuses Wolverine’s existing durable sharded execution — it forces the durable inbox, persists each envelope in consumption order, commits the Kafka offset on persist (the specific-offset fix from #3150), and shards inbox processing by the message key. The inbox is the reliability boundary, so a crash or rebalance can’t lose in-flight work.
#3146 — First-class AutoOffsetReset + ephemeral hot-tail
Cold start and live-tail consumption are now first-class:
opts.ListenToKafkaTopic("metrics").BeginAtEarliest(); // or .BeginAtLatest() opts.ListenToKafkaTopic("events").TailFromLatest(); // broadcast/fan-out
TailFromLatest() is the interesting one — the listener joins a unique per-process consumer group ({ServiceName}-hot-tail-{guid}) at the tail with EnableAutoCommit=true. Every node receives every message, no commits, no replay. This is the Kafka-local equivalent of a broadcast subscription, and it’s perfect for cache invalidation, ephemeral notifications, or dashboards. The trade-off (throwaway consumer groups left behind on the broker) is called out in the docs.
Replay, finally as a discrete operation
#3147 — Bounded one-shot replay via Assign lets you replay a window of a topic’s history back through the normal Wolverine handler pipeline without disturbing the live consumer group:
// Programmatic await host.ReplayKafkaTopicAsync(new KafkaReplayRequest { Topic = "orders", FromTimestamp = DateTimeOffset.UtcNow.AddHours(-2), });
# CLI dotnet run -- kafka-replay orders --from-timestamp 2026-06-18T10:00:00Z
Under the covers, KafkaReplay spins up a throwaway Assign()-based consumer with a unique group id and EnableAutoCommit=false, resolves per-partition start/end from explicit offsets or OffsetsForTimes, seeks to the start, and feeds every record through runtime.Pipeline.InvokeAsync — the same envelope mapping and handlers as live consumption. Each partition pauses at its end boundary. The live group’s committed offsets are untouched.
Live seek of a running group-subscribed listener and a CritterWatch control pane are explicit follow-ups.
Non-blocking tiered retries
This is the one a lot of users have been asking for. #3148 — Non-blocking tiered retry topics via OnException DSL:
opts.OnException<TransientException>() .MoveToKafkaRetryTopic(1.Seconds(), 30.Seconds(), 5.Minutes());
On a matching failure the message is produced to a tiered fixed-delay retry topic ({source}.retry.{delay}), the source offset is committed so the partition keeps flowing — no head-of-line blocking, and a delayed consumer reprocesses it through the normal handler pipeline once the tier delay elapses. After the last tier it lands in the existing Kafka DLQ. Tier, attempt, and exception metadata travel in headers.
Two design notes worth calling out:
- The continuation self-guards: if a non-Kafka listener somehow hits this rule it falls back to a normal inline retry, so the policy can never cross transports. The Kafka transport scans
opts.Policies.FailuresatConnectAsyncand warns at startup if non-Kafka listeners are present. - The core got one small generic hook —
IFailureActions.ContinueWith(IContinuationSource)— so transport-specific continuations can plug into the standard error DSL discoverably. This was the gap Pulsar’s resiliency support had to work around; that pattern is now first-class.
Exactly-once building blocks (the cheap ones)
#3149 — Idempotent producer + read_committed + EOS docs ships the cheap, opt-in pieces and — just as importantly — documents Wolverine’s actual exactly-once story so you reach for the right tool:
opts.UseKafka(connectionString) .UseIdempotentProducer() // producer→broker dedupe .UseReadCommitted(); // skip records from aborted Kafka txns
The new docs section leads with the durable inbox/outbox as the recommended path for DB-backed apps — that’s effectively-once across DB + Kafka, which Kafka transactions can’t span — then covers the idempotent producer, read_committed, the handler-idempotency reality, and a clear non-goal callout pointing DB-free Kafka→Kafka EOS users at Kafka Streams.
A transactional read-process-write EOS engine remains an explicit non-goal for Wolverine.
One small but annoying bug
#3151 — Fix ExtendConsumerConfiguration inheritance, contributed by @Ferchke7: a regression from a recent PR where ExtendConsumerConfiguration() created an empty topic-level ConsumerConfig, which Kafka then preferred over the parent, silently dropping any global consumer settings configured via UseKafka(...).ConfigureClient(...). Now the topic config is layered properly: parent → existing topic → extension callback.
Where we are vs. where this leaves the .NET Kafka story
After yesterday, Wolverine has:
- ✅ Idiomatic non-blocking commits, four selectable strategies, in-flight-safe watermarks
- ✅ Native scale-out via cooperative-sticky + static membership
- ✅ Second-tier concurrency by message key within a partition
- ✅ First-class cold-start / hot-tail consumption
- ✅ Bounded replay through the normal handler pipeline, without touching the live group
- ✅ Non-blocking tiered retry topics wired into the standard error DSL
- ✅ Idempotent producer +
read_committed+ an honest EOS story built on the durable inbox/outbox
The remaining gap the umbrella tracks is the transactional read-process-write EOS engine — explicitly a non-goal — and we’d rather just focus on Wolverine having a great transactional inbox/outbox integration with Kafka and all of our supported messaging options.
Upgrade to Wolverine 6.13.0 (6.13.1 was completely unrelated to the Kafka support), tweak nothing, and you’ll already see the throughput bump from the new default commit strategy. Then pick the levers that match your topic shape.
— Jeremy
The Shade Tree Developer 