Retrofitting existing xUnit.Net tests to Bobcat

Marten’s async daemon is the subsystem that runs projections and subscriptions in the background. It’s also been historically been the single hardest thing in Marten to test. Every test has to publish events, stand up a daemon (sometimes two, in hot/cold mode), wait for shards to catch up, and then check what the projections wrote. The DaemonTests project has also been for years one of our worst sources of flaky tests and I never trusted it to run in CI until earlier this year.

So when Bobcat needed a real test suite to prove itself against before 1.0, DaemonTests was one of the obvious guinea pigs (another was Wolverine’s leadership compliance tests that I’ll write about next). If projected specifications and the Supervisor could make that suite readable and reliable, they could hopefully do it anywhere.

This post is about how that went: the mechanics of how the tests were decorated, what the specifications look like in preview and in results, where we got real bugs out of it, and how the suite runs in CI today under the Supervisor’s retry budget.

I’ve already introduced Bobcat and the Bobcat Supervisor in earlier posts, so now let’s move on to a “what it looks like on a real codebase” follow up. All of the code is public in the Marten repository, and I’ll link to the actual files as I go.

Fair warning: a lot of the Bobcat work on DaemonTests was done by AI agents working from Bobcat issues, with me reviewing the pull requests. I’m going to quote the agents’ own write ups in a couple of places because, honestly, they documented the trade offs better than I would have.

No need to rewrite tests

Bobcat was initially going to be an alternative Gherkin tool with possibly a more code centric model where specifications could be “projected” from the raw test code — with a 3rd mode using a fluent interface kind of approach where you embedded Lambda functions inside of Given/When/Then() methods. It quickly became clear that the “projected” model was going to be the most usable, so we simplified Bobcat by dropping the early Gherkin model. And the less said about the Fluent Interface + nested Lambda approach, that better. I despise tools like that. Even the ones that I might have written myself years ago.

The constraint that shaped all of Bobcat’s “projected specification” model is that a big, old test suite has to be able to adopt it one class at a time, with no mandatory base class, no signature changes, and no rewrite. DaemonTests is exactly that kind of suite. It’s got hundreds of tests, a lot of them written years apart by different people, and nobody was going to sign up to rewrite them in a new DSL.

The first pull request back in September rendered two of the test classes as specifications. Eighteen tests started reporting themselves as Given/When/Then steps, and not one test body was edited. All of the steps came from decorating five helper methods on the shared DaemonContext base class and marking the two classes with one attribute.

The second pull request in October moved that work to Bobcat 1.0, routed the projection waits through Bobcat’s catch-up diagnostics, added a handful of brand new specifications written with the Critter Stack vocabulary, and rendered three more classes. Today the suite references Bobcat 1.0.0 and WolverineFx.Bobcat 6.55.0, and bobcat preview lists 30 specifications across six features:

Feature: Building an aggregate projection
Feature: Catching up projections and subscriptions
Feature: Rebuilding an inline projection
Feature: Resuming a failed rebuild
Feature: Skipping events that fail to apply
Feature: The async daemon keeps read models current

That’s a small fraction of the 393 tests in the project, but you have to start somewhere!

Mechanics, part one: decorate the helpers you already call

Almost every daemon test calls the same handful of helper methods on DaemonContext: start the daemon, publish the events, wait for a shard to reach a sequence number, rebuild a projection, compare every expected aggregate against what the projection actually wrote. Those helpers are the vocabulary of the suite. So that’s where the Bobcat attributes went:

[When("the projection daemon is running")]
internal async Task<IProjectionDaemon> StartDaemon()
{
    var daemon = theStore.Tenancy.Default.Database.As<MartenDatabase>()
        .StartProjectionDaemon(theStore);
    await daemon.StartAllAsync();
    _daemon = daemon;
    return daemon;
}
[Given("the events are published")]
internal async Task PublishSingleThreaded()
{
    await PublishSingleThreaded<Trip>();
}
[Given("the events are published on {threads} threads")]
internal Task PublishMultiThreaded(int threads)
{
    // ...
}
[Then("the {shardName} shard has reached sequence {sequence}")]
internal async Task<ShardState> WaitForShard(IProjectionDaemon daemon, string shardName, long sequence,
    Type readModel, TimeSpan timeout = default)
{
    // ...
}
[When("the {projectionName} projection is rebuilt")]
internal Task RebuildProjection(IProjectionDaemon daemon, string projectionName)
{
    return daemon.RebuildProjectionAsync(projectionName, CancellationToken.None);
}
[Then("every expected aggregate matches the actual")]
internal async Task CheckAllExpectedAggregatesAgainstActuals()
{
    // ...
}

[Given], [When], [Then], [Check] and the keywordless [Step] are all the same StepAttribute with a different keyword. The {placeholder} names in the sentence bind to the method’s parameters by name, and they’re filled in from the argument at each call site. A Type argument renders as the type’s short name, so WaitForCatchUp(typeof(Trip)) reads as “the Trip read model has caught up”. The docs on step helpers cover the binding rules, and the placeholder rules are worth reading because a Guid or a record renders differently than you might expect.

Then the test class gets one attribute:

[BobcatFeature("Building an aggregate projection")]
public class build_aggregate_projection: DaemonContext
{
    // 17 existing [Fact] tests, untouched
}

And a test like this one, which has not changed, now renders as a five step specification:

[Fact]
public async Task rebuild_the_projection()
{
    NumberOfStreams = 10;
    Logger.LogDebug("The expected number of events is {NumberOfEvents}", NumberOfEvents);
    StoreOptions(x => x.Projections.Add(new TripProjectionWithCustomName(), ProjectionLifecycle.Async));
    var agent = await StartDaemon();
    await PublishSingleThreaded();
    await WaitForShard(agent, "TripCustomName:All", NumberOfEvents, typeof(Trip), 30.Seconds());
    Logger.LogDebug("About to rebuild TripCustomName:All");
    await RebuildProjection(agent, "TripCustomName");
    Logger.LogDebug("Done rebuilding TripCustomName:All");
    await CheckAllExpectedAggregatesAgainstActuals();
}

And the results look like this in the console (we use Spectre.Console under the covers, so there’s color coded results and just take my word for it that it’s pretty!):

  rebuild the projection OK
  ────────────────────────────────
    ✓ When  the projection daemon is running (120ms)
    ✓ Given the events are published (67ms)
    ✓ Then  the TripCustomName:All shard has reached sequence 231 (248ms)
    ✓ When  the TripCustomName projection is rebuilt (25ms)
    ✓ Then  every expected aggregate matches the actual (21ms)

Yes, the When comes before the Given in that one, because that’s the order the test actually does things in. Bobcat renders what ran. It doesn’t editorialize. To be honest, I’ve never liked the Given/When/Then nomenclature anyway. To each their own.

The trick that makes this work with zero changes to the test bodies is that Bobcat.Generators is a source generator that emits C# interceptors for every call to a decorated helper. There’s no reflection or stack walking at runtime. Each call site is known at build time, which is also what lets bobcat preview show the specification before anything has run.

An interceptor has to be an extension method, and an extension method can’t see a protected member. So every decorated helper on DaemonContext had to be widened from protected to internal. That’s a C# language rule and not a Bobcat design choice, and Bobcat reports a protected helper as a BOBCAT039 diagnostic so you find out at build time rather than wondering why a step is missing. In Marten’s case that was eight helpers, and InternalsVisibleTo already existed for a sibling project, so nothing new was exposed. The Bobcat docs call this out because it’s the one thing you’ll actually have to touch.

Mechanics, part two: marker comments

Decorated helpers give you the work a test does. They don’t give you the narrative, and a lot of daemon tests have a middle section that’s just plain Marten code with no helper to hang a step on. For those, Bobcat reads marker comments: any // comment that opens with Given, When, Then, And or But as a whole word is a declared step. Every other comment is left alone, which is the property that makes this usable on a real suite that’s already full of explanatory comments like // Gotta do this to have it mixed up.

The clearest example in DaemonTests is when_skipping_events_in_daemon, which specifies what happens when a projection throws on some events and the daemon is configured to skip them:

[BobcatFeature("Skipping events that fail to apply")]
public class when_skipping_events_in_daemon : DaemonContext
{
    // ...
    [Given("the names are published to a running daemon, the bad ones failing to apply")]
    internal async Task<IProjectionDaemon> PublishTheEvents()
    {
        var daemon = await StartDaemon();
        var events = theNames.Select(name => new NameEvent {Name = name})
            .OfType<object>()
            .ToArray();
        theSession.Events.StartStream(Guid.NewGuid(), events);
        await theSession.SaveChangesAsync();
        await daemon.Tracker.WaitForHighWaterMark(theNames.Length);
        await WaitForShard(daemon, "CollateNames:All", theNames.Length, typeof(NamesByLetter));
        await WaitForShard(daemon, "NamedDocuments:All", theNames.Length, typeof(NamedDocument), 5.Minutes());
        return daemon;
    }
    [Fact]
    public async Task see_the_dead_letter_events()
    {
        var daemon = await PublishTheEvents();
        // When the daemon stops, draining the dead letter events it queued up
        await daemon.StopAllAsync();
        // Then each skipped event is a dead letter for the projection that skipped it
        var skipped = await theSession.Query<DeadLetterEvent>().ToListAsync(TestContext.Current.CancellationToken);
        var namedDocumentsSkipped = skipped.Where(x => x.ProjectionName == "NamedDocuments" && x.ShardName == "All")
            .Select(x => x.EventSequence).OrderBy(x => x).ToList();
        namedDocumentsSkipped.ShouldBe([4L, 5, 6, 7, 11, 14]);
        var collateNamesSkipped = skipped.Where(x => x.ProjectionName == "CollateNames" && x.ShardName == "All")
            .Select(x => x.EventSequence).OrderBy(x => x).ToList();
        collateNamesSkipped.ShouldBe([4L, 5, 6, 7]);
    }
}

There are three different Bobcat mechanisms in that one test, and they compose:

1. PublishTheEvents() is a [Given] helper that lives on the test class itself rather than on DaemonContext. It calls StartDaemon() and WaitForShard(), which are decorated too, so those render nested underneath the Given with their own timings.
2. The two // comments are marker comments. They declare what the test is doing in the stretch of code that calls no helper.
3. The two ShouldBe() calls are projected assertions, which is the next section.

Here’s what that renders as from bobcat run:

  see the dead letter events OK
  ────────────────────────────────────
    ✓ Given the names are published to a running daemon, the bad ones failing to apply (306ms)
      ✓ When  the projection daemon is running (15ms)
      ✓ Then  the CollateNames:All shard has reached sequence 15 (44ms)
      ✓ And   the NamedDocuments:All shard has reached sequence 15
    · When  the daemon stops, draining the dead letter events it queued up — declared
    ✓ Then  each skipped event is a dead letter for the projection that skipped it (4ms)
      ✓ And   namedDocumentsSkipped should be [4L, 5, 6, 7, 11, 14] (4ms)
      ✓ And   collateNamesSkipped should be [4L, 5, 6, 7]

Notice the When the daemon stops line is marked declared with no tick and no timing. Nothing decorated ran inside that comment’s region, so Bobcat has nothing to report about it other than that the test claims to do it. The documentation is pretty strict about keeping “declared” and “recorded” separate, and I think that’s right. A step that borrowed a verdict from its neighbors would be exactly the kind of inference that makes test reports lie to you.

Marker comments are also how we narrated the longest, most involved daemon test in the suite, rebuild_mode_start_resumes_after_a_failure, which proves that a failed rebuild can be restarted from its committed progress without folding any event twice. That one is pure Marten and Postgres code with no shared helpers, and the comments carry the whole specification:

// Given a hundred streams of one event each
// When a Rebuild-mode run fails on event 60
// Then the committed progress and the failure are recorded by extended progression tracking
// And every event at or below the committed mark is in a document already
// When the shard is stopped, the fault cleared, and the Rebuild-mode run started again
// Then every event is folded into its document exactly once
// And the failure is cleared
// And no event below the first run's committed mark was folded again by the second run
// And the restarted run keeps following the store past where the rebuild started

Some of those comments, like the “Given a hundred streams” line, aren’t “meaningful” because they map to code. They’re meaningful because they’re the sentence a maintainer needs in order to understand what the next ten lines are for. Bobcat just happens to also render them.

One gotcha I tripped over while grabbing screenshots for this post: a marker comment wraps onto the // lines directly below it, up to a blank line or a line of code (dogfooding for the win I guess). That’s handy for long sentences, but it also means an explanatory comment sitting right under a marker becomes part of the step’s sentence. A couple of the steps in that rebuild test render with a trailing explanation glued on for that reason. Leave a blank line between the marker and the explanation and it stays a comment.

Mechanics, part three: your Shouldly assertions are steps too

We’ll add support for other assertion libraries later if there’s any demand, but I prefer Shouldly and that’s what the Critter Stack code already uses, so here we are.

DaemonTests has one line in its csproj that does a surprising amount of work:

<!-- A statement-level Shouldly assertion inside a [BobcatFeature] class renders as its own
     Then step, and a run of consecutive assertions is all evaluated before the test fails,
     so a red daemon spec shows every disagreement rather than the first. Classes without
     [BobcatFeature] are untouched. -->
<BobcatProjectAssertions>true</BobcatProjectAssertions>

With that on, any statement-level Shouldly call inside a [BobcatFeature] class becomes a Then step with a sentence built from the C# expression as written, a verdict, a duration, and an expected/actual cell where the comparison allows it. jNames.Names.OrderBy(x => x).ShouldBe([...]) renders as “jNames.Names.OrderBy(x => x) should be [“Jack”, “Jane”, “Jeremy”, “Jill”]” with no attribute and no helper. Nothing moves.

The part that matters more than the rendering is that a run of consecutive assertions is all evaluated before the test fails. A plain Shouldly test throws on the first ShouldBe and never reaches the rest. With projected assertions, every assertion in the run executes, and the test fails once at the end with all of the wrongs in the failure. For daemon tests that check three or four things about shard state after a long setup, that’s the difference between one slow red/green cycle and four of them. The Shouldly page in the Bobcat docs is the whole story, including the closed set of comparisons that get a cell and the ones (ShouldBeTrue, ShouldBeOfType) that render as a step with no cell because any cell would be making something up.

Two things to know from using this on a real suite:

• An assertion inside a loop is a step per iteration, because each one really ran. The “the shards should still be running” spec renders shard.Status should be AgentStatus.Running once per shard. If you don’t want that, assert once over the whole collection outside the loop, and the docs say the same. The rebuild-resume test does exactly that, and says why in a comment:

// One assertion over the whole set rather than one per event: with BobcatProjectAssertions on, a
// per-event ShouldBe would render as sixty steps
var foldedAgain = committedSequences
    .Select(e => (Index: ((RestartableIncrement)e.Data).Index, e.Sequence))
    .Where(x => RestartableTallyProjection.Applications[x.Index] != 1)
    .Select(x => $"event {x.Index} at sequence {x.Sequence}")
    .ToList();
foldedAgain.ShouldBeEmpty();

• Only a statement is projected. x.ShouldNotBeNull().Name.ShouldBe("a") consumes the first assertion’s result, so Bobcat leaves a fluent chain alone rather than risk reporting a NullReferenceException in place of the real failure.

There’s also a comment in when_skipping_events_in_daemon explaining why each subject is pulled into a local first: on Bobcat 1.0.0-alpha.1 a projected assertion whose subject spanned lines broke the build because the generator didn’t escape the newlines in the subject string. That’s the kind of thing that dogfooding on a real suite is for, and it’s why the comments in these files tend to say why and not just what.

All the Bobcat-specific bits in one place

For the whole DaemonTests project, this is the entire Bobcat surface area:

That’s it. No base class was imposed, and catch_up_timeout_diagnostics deliberately isn’t a [BobcatFeature] because those tests expect the error a recorded specification would report, which is a nice illustration that opting in is per class.

The one dependency cost worth being honest about: Bobcat.Xunit 1.0 is built on xunit.v3 4.x over Microsoft.Testing.Platform 2, and the two xunit lines can’t share a test project. Marten’s other 23 test projects are still on xunit.v3 3.x, so DaemonTests takes 4.x through a VersionOverride and the rest of the repo moves on its own schedule. The compatibility page spells out what pairs with what.

Waiting on the daemon

The most useful change in the second PR wasn’t a rendering change at all. Daemon tests spend most of their lives waiting for a projection shard to catch up, and when a wait times out, Marten’s own exception used to say roughly “the shards haven’t started within the timeout.” That tells you nothing about which shard, where it stood, or whether the daemon even ran.

DaemonContext now has two waits that route through Bobcat’s catch-up diagnosis:

/// <summary>
/// Wait for every async projection in <c>theStore</c> to catch up before reading
/// <paramref name="readModel"/>. On a timeout this throws Bobcat's
/// <see cref="ProjectionCatchUpTimeoutException"/>, which names each shard that was behind and
/// where it stood (<c>'Trip:All' at 412 of 1000</c>), rather than the bare TimeoutException
/// Marten's own wait throws. A stalled daemon test then says which shard stalled.
/// </summary>
[Given("the {readModel} read model has caught up")]
internal async Task WaitForCatchUp(Type readModel, TimeSpan timeout = default)
{
    if (timeout == default) timeout = 30.Seconds();
    try
    {
        await EventStores.WaitForNonStaleProjectionsAsync(theStore, timeout,
            token: TestContext.Current.CancellationToken);
    }
    catch (TimeoutException e) when (e is not ProjectionCatchUpTimeoutException)
    {
        throw await catchUpTimedOut($"the {readModel.Name} read model has caught up", readModel, timeout, e);
    }
}

We have pre-canned helpers for this in our Wolverine.Bobcat package that’s meant to be our specifications library for event sourced applications with the Critter Stack.

And when one of those waits times out, this is the message you get, from a test in catch_up_timeout_diagnostics that forces the timeout on purpose by never starting the daemon:

Waiting for 'the TripCustomName:All shard has reached sequence 43' timed out after 2s, before reading the Trip.
Behind: 'TripCustomName:All' at 0 of 43. This is the environment (a daemon behind, stopped or not running), not
evidence about the Trip itself. The store said: TimeoutException: Shard TripCustomName:All did not reach
sequence number 43 in the time allowed

“This is the environment, not evidence about the Trip itself” is the sentence I care about. When an AI agent, or a tired human, reads that failure, the next move is to look at the daemon and not at the projection code.

And it paid for itself before the PR was even reviewed. From the agent’s write up:

The first two full gate runs failed the new specs with Behind: 'ShipmentSummary:All' at 0 of 4, which meant the hosted daemon had never run its shard. The cause was that the fixture cleaned event data after the Solo daemon started. That restarted the sequence underneath a shard that had already loaded its old progress, so the shard sat “ahead” of every new event. Isolated runs won that race and full runs lost it.

That’s a real test fixture bug of exactly the kind that produces a “flaky” test that nobody can reproduce on their own machine. The fix was a one liner (clean the store before StartAsync()), and the comment in ShipmentsHost now explains the race for the next person. The diagnosis is what made it findable.

New specifications with the Critter Stack vocabulary

Everything above is about retrofitting existing tests. For new daemon specifications, we used CritterStackSpec, the store-only base class from WolverineFx.Bobcat, against an application that hosts its own daemon with AddAsyncDaemon(DaemonMode.Solo). This is from async_daemon_specifications:

[BobcatFeature("The async daemon keeps read models current")]
public class async_daemon_specifications: CritterStackSpec, IClassFixture<ShipmentsHost>
{
    public async_daemon_specifications(ShipmentsHost app): base(app.Host)
    {
        ProjectionTimeout = 30.Seconds();
    }
    [Fact]
    public async Task an_async_projection_catches_up_with_an_appended_event()
    {
        var id = Guid.NewGuid();
        await GivenEvents<ShipmentSummary>(id, new ShipmentPacked(id, 3));
        await WhenActing("the shipment is shipped", services => append(services, id, new ShipmentShipped(id)));
        ThenEvents(new ShipmentShipped(id));
        await ThenReadModel<ShipmentSummary>(id, Specify<ShipmentSummary>()
            .With(x => x.Items, 3)
            .With(x => x.Shipped, true));
    }
    [Fact]
    public async Task an_async_projection_deletes_the_read_model_its_stream_cancels()
    {
        var id = Guid.NewGuid();
        await GivenEvents<ShipmentSummary>(id, new ShipmentPacked(id, 2));
        await WhenActing("the shipment is cancelled", services => append(services, id, new ShipmentCancelled(id)));
        ThenEvents(new ShipmentCancelled(id));
        await ThenNoReadModel<ShipmentSummary>(id);
    }
}

Nothing in there waits by hand. ThenReadModel opens with Bobcat’s own catch-up step, which is the same diagnosis as above if it times out. And this is where the “data centric” side of Bobcat shows up, because the events and the read model render as grids:

The third spec in that file, a rebuild through the running daemon, had to go on a host of its own, and the class comment says why: JasperFx’s rebuild stops the shard’s continuous agent, rebuilds with dedicated agents, and doesn’t start the continuous agent again, so any spec that ran after the rebuild on a shared host timed out at “‘ShipmentSummary:All’ at 2 of 4”. Whether a hosted daemon should resume the shard after a rebuild is a question for Marten rather than for the specification, and I’d rather have that written down in the test than silently worked around.

What it looks like: preview, run, and the UI

Because every step is known at build time, you can see what the suite specifies without running it. bobcat preview prints the whole thing to the console with the templates unfilled and the helper each step binds to:

  rebuild the projection
  ────────────────────────────────
      ○ When  the projection daemon is running
        ↳ DaemonContext.StartDaemon
      ○ Given the events are published
        ↳ DaemonContext.PublishSingleThreaded
      ○ Then  the {shardName} shard has reached sequence {sequence}
        ↳ DaemonContext.WaitForShard
      ○ When  the {projectionName} projection is rebuilt
        ↳ DaemonContext.RebuildProjection
      ○ Then  every expected aggregate matches the actual
        ↳ DaemonContext.CheckAllExpectedAggregatesAgainstActuals

The same preview is available in the Bobcat UI once you’ve got a resident runner attached for the suite. This is bobcat ui plus bobcat resident src/DaemonTests/bin/Debug/net10.0/DaemonTests, and every feature and specification gets its own run button:

I like this view a lot more than I expected to. Seventeen tests about building an aggregate projection, readable in one screen by somebody who has never opened the C#.

Running a couple of features with bobcat run src/DaemonTests --feature "The async daemon keeps read models current" "Skipping events that fail to apply" gives you the run page, with the passing specs folded to one line each:

And when something’s wrong, the failed scenario is opened on its specification with the disagreement right there. I broke one expectation on purpose for this screenshot, so the aggregate is expected to contain a “Joe” that was never published:

The whole run also lands in a JSON results file with a published schema, and bobcat run --html writes a self contained results page you can attach to a CI artifact. Those are what an agent reads when it’s working on a red build, and the bobcat tool page covers the rest of the commands.

While grabbing these screenshots I noticed the UI’s scenario page renders the marker-comment steps ahead of the recorded helper steps and flattens the nesting, where the console gets the order right. That’s bobcat#810, and it’s a reminder that 1.0 means the model is stable, not that I’m done finding things.

The Supervisor, retry budgets, and the flaky test that earned its retry

Now the other half of the story. Readable specifications don’t help if the suite doesn’t run reliably, and DaemonTests was historically the least reliable project in Marten. It had its own workflow (daemon.yml) that shelled straight out to dotnet test, which meant the most timing-sensitive suite in the repo was the one with no retry story at all.

Today every Marten test target runs through the Bobcat Supervisor from the Nuke build, and the daemon suite is one job in the matrix of tests.yml. The whole configuration lives in build/SupervisedTests.cs, and the interesting part is small:

csharp

const int MaxRetriesPerRun = 25;
const int MaxAttemptsPerTest = 3;
static readonly TimeSpan StallThreshold = TimeSpan.FromMinutes(5);
void configureSupervisor(Supervisor supervisor, TestLedger ledger)
{
    supervisor.RetryBudget = DisableTestRetry
        ? RetryBudget.None
        : new RetryBudget { MaxAttemptsPerTest = MaxAttemptsPerTest, MaxRetriesPerRun = MaxRetriesPerRun };
    // The policy only ever retries in a fresh process, so idle lane workers are released
    // before the retry passes. Without it, workers+1 test hosts sit resident at once — which is
    // what OOM-killed 16GB GitHub runners on Wolverine, both times during a retry.
    supervisor.ReleaseIdleLanes = true;
    // Report-only observability. A test in flight past the threshold is named at once (a wedged
    // daemon test used to be visible only as a job hitting its 20 minute cap), and a one-line
    // heartbeat every 30s says how far a run got and what has been running longest.
    supervisor.StallThreshold = StallThreshold;
    supervisor.StallAction = StallAction.Report;
    supervisor.HeartbeatInterval = TimeSpan.FromSeconds(30);
    // Longest-processing-time-first lane balancing from the previous run's ledger, keyed by
    // display name (bobcat#730). No ledger, or no match, degrades to balancing by test count,
    // and the run report's first lines say how many durations matched.
    supervisor.KnownTestDurations = ledger.KnownDurations();
}

A few things about that policy that I think are important, and that took us a while to get right:

Retries are opt-in, per test, and always in a fresh process. Bobcat’s default failure policy only retries a test that carries [Trait("Retry", "N")]. Everything else that fails, fails. Up until we made that change, every failure in Marten’s CI was retried, which made “flaky” and “broken” impossible to tell apart without going and opening the log. A retry runs in a fresh process because a failed daemon test usually leaves a half-built store, schema objects, or a running daemon behind it. The fresh process is the reset.

The retry budget is the operator’s ceiling, not the test’s. MaxAttemptsPerTest = 3 is the most any test gets, including the first attempt. A test’s own [Trait("Retry", "N")] can lower that but never raise it. MaxRetriesPerRun = 25 caps the whole run, and the build’s own comment says how to read it: a run sitting at the cap isn’t “25 flaky tests,” it’s 25 flaky tests plus an unknown tail.

Stall detection is report-only. A test in flight past five minutes is named in the log, the job summary, and the run report, but nothing gets killed. Before this, a wedged daemon test was only visible as a job hitting its 20 minute cap with no indication of which test wedged. Nothing in Marten’s suites legitimately runs that long as one test.

A pass on a retry is never laundered into a green tick. This is the RetryLedger, and its header comment is the best summary of the problem I’ve read:

Nothing in the GitHub UI distinguishes a job that passed cleanly from a job that passed after retrying nine tests. Both render as a green tick, and a Serilog warning buried in the log of a job that PASSED is a warning nobody has a reason to go read. Without this, “we added retries” quietly becomes “we stopped being able to tell whether the suite works”.

So every supervised run writes its retry count to the GitHub job summary, emits a ::warning annotation when the count is nonzero so it shows up on a green run, and uploads a JSON ledger that a flakiness roll-up job aggregates across every job in the workflow. Deliberately not in there is failing the build past a retry threshold. The value is in the visibility, not in a cliff that turns master red on a bad day.

What the ledger says about the daemon suite right now

Here’s the daemon job’s ledger from the most recent green master run as I write this (November 2nd):

The one test that passed on a retry was Bug_catching_up_apply_errors.should_catch_apply_event_exceptions, and it’s the best example I have of how the opt-in policy is supposed to work. That test carries [Trait("Retry", "3")], and the comment above it explains how it earned that:

// Retry is opt-in under the supervisor (JasperFx/bobcat#565), and this test earned it with
// evidence: in 33 green master runs (2026-09 to 2026-10-15) its first attempt failed in 13 —
// "Task should throw System.AggregateException but did not" — and passed on a fresh-process
// retry every time. It is still reported as FLAKY in the retry ledger and the job summary;
// the trait keeps a known intermittent from turning the job red, it does not hide it. The
// projection throws deterministically, so a CatchUpAsync that returns cleanly is worth an
// investigation of its own (state left by an earlier test in the same process is the lead).
[Fact]
[Trait("Retry", "3")]

That’s the whole philosophy in one comment. The ledger gave us the evidence (13 first-attempt failures in 33 runs, same message every time, always passes on a fresh process). The trait keeps a known intermittent from turning the job red. The ledger still reports it as flaky on every run so it can’t quietly disappear. And the comment leaves the next investigator a lead. A second test in the same file picked up the same trait a couple of weeks later after one miss turned a whole job red for a pull request that couldn’t possibly have touched it.

One worker, and a local gate

The daemon job runs supervised at one worker. The issue that drove this work measured two lanes green locally, 374 of 374 twice, at 5m10s against 7m09s with retries off, but raising the worker count in CI is a separate decision we want to make on CI evidence rather than on a laptop. Marten’s heavier suites do split across lanes with a database per lane, and the parallel-ready suites page is the checklist we run a suite through before we let it.

The last piece is that the same supervisor configuration runs locally. ./build.sh LocalGate runs the logical suites of tests.yml on your own machine through Bobcat’s local gate, with the same opt-in retry policy, the same stall report, and the same lane balancing from the ledger, before you push. The implementation is in build/LocalGate.cs and the header comment there is, again, the documentation.

What I’d tell you to do with your own suite

If you’ve got a big integration suite with a lot of history, this is the order I’d go in, because it’s the order that worked on DaemonTests:

1. Put the suite under the Supervisor first, with retries off (--disable-test-retry in Marten’s build), and read the ledger for a couple of weeks. You’ll find out which tests are actually flaky and which are just broken.
2. Give the proven-flaky tests a [Trait("Retry", "N")] with a comment that says what the evidence was. Keep the ledger reporting them.
3. Pick the test classes that explain how the system is supposed to behave, decorate the three or four shared helpers they all call, and mark the classes with [BobcatFeature]. Don’t touch the test bodies.
4. Add marker comments where a test’s narrative isn’t carried by a helper, and turn on BobcatProjectAssertions.
5. Replace your bare TimeoutException waits with something that says what was behind. If you’re on Marten, Polecat or Fisher, WolverineFx.Bobcat has that built in.
6. Write the new specifications with CritterStackSpec or WolverineSpec, and let the existing tests stay existing tests.

The code for all of this is public:

• The suite: src/DaemonTests in Marten, with DaemonContext.cs holding the decorated helpers and Specifications/ holding the CritterStackSpec specs and the catch-up diagnostics
• The two pull requests: #5363 (the first rendering, no test bodies edited) and #5690 (Bobcat 1.0, catch-up diagnostics, CritterStackSpec), plus the Bobcat issue that drove the second one
• The build: build/SupervisedTests.cs, build/RetryLedger.cs, build/LocalGate.cs and .github/workflows/tests.yml
• The Bobcat docs: Writing Specifications, Shouldly assertions as steps, the bobcat tool, the Bobcat UI, Reliable Integration Testing and Continuous Integration for the Supervisor, and Local Gates

Bobcat is MIT licensed, 1.0 is out, and DaemonTests is the suite I’d point you at if you want to see what it looks like on code that wasn’t written for it.