Flow-based specifications with Bobcat

Babu added a new feature to our JasperFx CRM website to allow folks to download trial licenses to CritterWatch to try it out before purchasing. It’s a small feature with a long tail: a prospect fills in a form, we email them a verification link, they click it, and in one transaction we provision an account and a contact, grant a trial license, sign them in, open a follow-up task for sales, and publish a couple of notifications. There’s a one-trial-per-person rule with the usual email games (plus-addressing, Gmail dots, free-mail domains), a one-trial-per-company rule, and a kill switch. I realized pretty quickly that it’s a good little problem domain to demonstrate our new Bobcat library for writing human readable specifications.

The integration tests for it were already fine. 545 lines of xUnit and Alba, every rule covered, green. But now, let’s add Bobcat to the existing tests and make the content and meaning of those tests more easily reviewable by humans. While Bobcat is most certainly jam packed with features to make “Table Driven Testing” easier, today I want to talk about using Bobcat for “Flow Based Testing”

What a flow-based spec is

I might be using the nomenclature wrong here, but once upon a time even before Cucumber dominated this space, we talked about expressing these kinds of automated tests as either “flow based” meaning it read like a paragraph of prose or “table based” where you mostly specified tests in tables. In the end, I think this is an “open ended wrench vs. close ended wrench” kind of choice where you need both approaches in different scenarios. I’m going to issue a promise for a better example of table driven tests with Bobcat later and just dive into this sample.

Most of the specs in the Critter Stack samples are one act: arrange some events, send one command, assert what happened. The trial is a little more complicated and the thing under test is the sequence:

  1. the prospect submits the trial form
  2. nothing is created yet, but a verification email goes out
  3. the prospect follows the link
  4. now the account, contact, license, sign-in cookie, sales task and notifications all appear, together

A Bobcat spec can act more than once. Every earlier act arranges state; the last act is the one every Then judges. So a flow-based spec reads top to bottom as the story the prospect lives through, and the assertions at the bottom are about the moment that matters.

The setup

Bobcat supports TUnit, NUnit, and MSTest too, but we’ve put more effort so far into xUnit.Net and TUnit. So far, I’m very happy with our not entirely originally planned decision to dump Gherkin and “just project” specifications from plain old xunit tests by sprinkling some Bobcat attributes and meaningful comments into the existing code. I think this is going to make Bobcat much easier for folks to adopt than it would have been with our original idea of being yet another Gherkin tool.

Anyway, our new spec project sits beside the existing test project (we haven’t completely replaced the old tests yet):

<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<IsTestProject>true</IsTestProject>
<TestingPlatformDotnetTestSupport>true</TestingPlatformDotnetTestSupport>
<UseMicrosoftTestingPlatformRunner>true</UseMicrosoftTestingPlatformRunner>
<!-- Render statement-level Shouldly assertions as specification steps. -->
<BobcatProjectAssertions>true</BobcatProjectAssertions>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.NET.Test.Sdk" />
<PackageReference Include="xunit.v3" />
<PackageReference Include="xunit.runner.visualstudio" />
<PackageReference Include="Bobcat" />
<PackageReference Include="Bobcat.Xunit" />
<PackageReference Include="Bobcat.Generators" />
<PackageReference Include="WolverineFx.Bobcat" />
<PackageReference Include="WolverineFx.Bobcat.Http" />
<PackageReference Include="Alba" />
<PackageReference Include="Shouldly" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\..\src\JasperFX.Portal\JasperFX.Portal.csproj" />
<ProjectReference Include="..\JasperFX.Portal.TestSupport\JasperFX.Portal.TestSupport.csproj" />
</ItemGroup>
</Project>

Two things worth calling out. Microsoft.Testing.Platform 2 on the .NET 10 SDK insists on dotnet test running in its own mode, so the repository’s global.json grows a "test": { "runner": "Microsoft.Testing.Platform" } section, which in turn means the old xUnit 2 project had to move to xUnit v3 as well. That move touched exactly one file (the fixture’s InitializeAsync and DisposeAsync now return ValueTask). And the TestSupport project is new: the portal’s test host, its fakes for Stripe, the package registry, Cloudflare Turnstile and email, and the seed helpers moved out of the xUnit project into a plain class library so both runners share one host. Nothing about the fakes changed.

The host fixture for the specs is the same shape as the one in the old project:

public sealed class PortalFixture : IAsyncLifetime
{
private PortalTestHost? _inner;
public PortalTestHost Portal => _inner ?? throw new InvalidOperationException("The test host has not started.");
public IAlbaHost Host => Portal.Host;
public async ValueTask InitializeAsync() => _inner = await PortalTestHost.StartAsync();
public async ValueTask DisposeAsync() { if (_inner is not null) await _inner.DisposeAsync(); }
}
[Collection(PortalCollection.Name)]
public abstract class PortalSpec : WolverineHttpSpec
{
protected PortalSpec(PortalFixture app) : base(app.Host)
{
Portal = app.Portal;
Timeout = TimeSpan.FromSeconds(20); // first request to a Razor page compiles the view
}
protected PortalTestHost Portal { get; }
protected IDocumentStore Store => Portal.Store;
protected FakeEmailSender Email => Portal.Email;
}

WolverineHttpSpec is the vocabulary from WolverineFx.Bobcat.Http: WhenPosted, WhenReading, ThenHandled, ThenMessageScheduled and friends, over an Alba host. One Postgres Testcontainer per run, no reset between specs, unique emails per spec. Exactly what the old suite did.

The steps

This is the part that makes a flow readable. The mechanics of driving two Razor pages, reading a token out of a fake outbox, and querying Marten all live in step helpers on a TrialSpec base class. The helper’s attribute text is what the rendered specification says; the method body is how it’s done, and the assertions inside it are ordinary Shouldly.

public abstract class TrialSpec(PortalFixture app) : PortalSpec(app)
{
private string _lastPage = string.Empty;
private IScenarioResult? _lastResponse;
private long _sequenceBeforeAct;
[Given("the Turnstile check will {outcome}")]
public void TurnstileWill(TurnstileOutcome outcome) => Portal.FakeTurnstile.Next = outcome;
[When("{email} requests a CritterWatch trial as an individual named {name}")]
public Task RequestsATrialAsAnIndividual(string email, string name) =>
RequestTrial(new Dictionary<string, string>
{
["sku"] = TrialSku, ["AccountType"] = "individual",
["BuyerName"] = name, ["Email"] = email, ["BillingCountry"] = "US",
});
private async Task RequestTrial(Dictionary<string, string> form, bool clearLimits = true)
{
if (clearLimits) await ContactSeed.ClearRateLimitWindowsAsync(Store);
var result = await Portal.Host.Scenario(s =>
{
s.Post.FormData(form).ToUrl("/trial");
s.StatusCodeShouldBe(200);
});
_lastPage = await result.ReadAsTextAsync();
SpecLog.Write($"POST /trial for {form["Email"]} -> 200");
}
[When("{email} follows the verification link")]
public async Task FollowsTheVerificationLink(string email)
{
var token = VerificationTokenFor(email).ShouldNotBeNull($"a verification email reached {email}");
// The verification POST is the act: it provisions the customer, grants
// the trial, and publishes the notifications the tracked session waits for.
await MarkActStart();
await WhenTracked("the emailed token is posted to /trial/verify", async () =>
{
_lastResponse = await Portal.Host.Scenario(s =>
{
s.Post.FormData(new Dictionary<string, string> { ["Token"] = token }).ToUrl("/trial/verify");
s.IgnoreStatusCode();
});
_lastPage = await _lastResponse.ReadAsTextAsync();
SpecLog.Write($"POST /trial/verify -> {_lastResponse.Context.Response.StatusCode}");
});
}
[Then("the page says {text}")]
public void ThePageSays(string text) => _lastPage.ShouldContain(text, Case.Insensitive);
[Then("the account holds a 30-day CritterWatch trial license and nothing was sold")]
public async Task<Entitlement> TheAccountHoldsATrialLicense(Account account)
{
await using var read = Store.QuerySession();
var entitlements = await read.Query<Entitlement>().Where(e => e.AccountId == account.Id).ToListAsync();
var trial = entitlements.ShouldHaveSingleItem();
trial.Sku.ShouldBe(TrialSku);
trial.Source.ShouldBe(EntitlementSources.SelfServeTrial);
trial.Kind.ShouldBe(FulfillmentKind.CritterWatchLicense);
trial.PurchaseId.ShouldBeNull();
(trial.PeriodEnd - DateTimeOffset.UtcNow).TotalDays.ShouldBe(30, 1);
(await read.LoadAsync<RevealableToken>(trial.Id)).ShouldNotBeNull("the license key is waiting to be revealed");
(await read.Query<Purchase>().Where(p => p.AccountId == account.Id).ToListAsync()).ShouldBeEmpty();
return trial;
}
}

Three habits in there that I’d recommend for any flow:

  • WhenTracked for the act that cascades. The verification page publishes two messages and schedules a third. Wrapping the HTTP call in WhenTracked runs it inside a Wolverine tracked session, so the spec waits for every handler the act caused before the Then steps run, ThenHandled<T>() and ThenMessageScheduled<T>() have something to judge, and the run records the message activity (more on that below).
  • Shouldly in the helpers, as you’d write it anywhere. Bobcat captures a failed assertion as the step’s failure cell, with Shouldly’s own message, and marks the steps after it as not run. You don’t need a second assertion vocabulary to get a rendered specification.
  • SpecLog.Write for what the system did. Logs render under the step for a failed scenario (or with BOBCAT_VERBOSE_REPORTS=1), so a red spec says what the page answered, not just what was expected.

The spec

With the steps in place the happy path is the flow, in order, with nothing in it that isn’t the business rule:

[BobcatFeature("Self-serve CritterWatch trial")]
public sealed class SelfServeTrial(PortalFixture app) : TrialSpec(app)
{
[BobcatSpec(typeof(StartTrialRequest), Domain = "Trials")]
public async Task an_individual_requests_verifies_and_starts_a_trial()
{
var email = CompanyEmail("sam");
TurnstileWill(TurnstileOutcome.Passed);
await RequestsATrialAsAnIndividual(email, "Sam Solo");
ThePageSays("Check your email");
AVerificationEmailWasSentTo(email);
await NothingExistsForYet(email);
await FollowsTheVerificationLink(email);
IsRedirectedHomeSignedIn(email);
await TheActAppended(typeof(AccountCreated), typeof(ContactCreated), typeof(EntitlementGranted),
typeof(TaskCreated), typeof(ContactSignedIn));
var account = await IsAContactOnAnAccount(email, ContactRole.Member, "Sam Solo", AccountKind.Individual, "Sam Solo");
await TheAccountHoldsATrialLicense(account);
await TheOneTrialLocksAreTaken(email);
await AFollowUpTaskIsAssigned(3, account);
ThenMessageScheduled<TaskDueReminder>();
ThenHandled<SendTrialStartedNotification>();
ThenHandled<NotifyAdminsTrialStarted>();
ReceivedTheTrialReadyEmail(email);
OnlyThePrimaryAdminWasAlerted("Sam Solo");
}
}

The refusals read the same way. The one-trial-per-company rule, for instance, is a flow where the second act is supposed to do nothing:

[BobcatSpec(typeof(StartTrialRequest), Domain = "Trials")]
public async Task a_colleague_on_the_same_company_domain_is_refused()
{
var domain = $"{Guid.NewGuid():N}.test";
TurnstileWill(TurnstileOutcome.Passed);
await AlreadyTookATrial($"first@{domain}");
await RequestsATrialForAnOrganization($"second@{domain}", "Same Corp", "Second Person");
await FollowsTheVerificationLink($"second@{domain}");
ThePageSays("already had a CritterWatch trial");
await TheActAppendedNoEvents();
await NothingExistsForYet($"second@{domain}");
}

AlreadyTookATrial is a [Given] helper that runs the whole happy-path flow for the first colleague. In the rendered spec it shows as one Given with the request and the verification nested under it, which is exactly the right level of detail for a precondition.

What it renders

bobcat run tests/JasperFX.Portal.Specs --verbose prints every specification as it ran. This is the happy path, verbatim apart from the message-activity table that follows it, which gets its own section next:

Feature: Self-serve CritterWatch trial
═══════════════════════════════════════
an individual requests verifies and starts a trial OK
────────────────────────────────────────────────────────────
✓ Given the Turnstile check will Passed (1ms)
✓ When sam@1f27b29e….test requests a CritterWatch trial as an individual named Sam Solo (84ms)
Logs:
POST /trial for sam@1f27b29e….test -> 200
✓ Then the page says Check your email (5ms)
✓ And a verification email was sent to sam@1f27b29e….test
✓ And nothing exists for sam@1f27b29e….test yet (7ms)
✓ When sam@1f27b29e….test follows the verification link (175ms)
✓ And the emailed token is posted to /trial/verify (172ms)
Logs:
POST /trial/verify -> 302
✓ Then sam@1f27b29e….test is redirected home, signed in (4ms)
✓ And the act appended AccountCreated, ContactCreated, EntitlementGranted, TaskCreated, ContactSignedIn (8ms)
✓ And sam@1f27b29e….test is a Member contact named Sam Solo on a Individual account named Sam Solo (4ms)
✓ And the account holds a 30-day CritterWatch trial license and nothing was sold (15ms)
✓ And the one-trial locks are taken for sam@1f27b29e….test and its company domain (1ms)
✓ And a follow-up task is assigned to the primary admin, due in 3 days (2ms)
✓ And TaskDueReminder is scheduled (1ms)
✓ scheduled: yes
✓ And SendTrialStartedNotification is handled (2ms)
✓ handled: yes
✓ And NotifyAdminsTrialStarted is handled
✓ handled: yes
✓ And sam@1f27b29e….test received the trial-ready email
✓ And only the primary admin was alerted that Sam Solo started a trial
Succeeded with Rights: 18, Wrongs: 0, Errors: 0

The whole feature is eleven specifications, about four seconds after the container is up.

And this is what a failure looks like when a Shouldly assertion inside a helper disagrees (I changed the expected license kind on purpose for this run):

  an individual requests verifies and starts a trial FAILED
  ────────────────────────────────────────────────────────────
    ✓ Given the Turnstile check will Passed (1ms)
    ✓ When  sam@7d0a1913….test requests a CritterWatch trial as an individual named Sam Solo (112ms)
    ...
    ✓ And   sam@7d0a1913….test is a Member contact named Sam Solo on a Individual account named Sam Solo (7ms)
    ✗ And   the account holds a 30-day CritterWatch trial license and nothing was sold (16ms)
      trial.Kind
          should be
      FulfillmentKind.RegistryToken
          but was
      FulfillmentKind.CritterWatchLicense
        ✗ trial.Kind: expected 'FulfillmentKind.RegistryToken', got 'FulfillmentKind.CritterWatchLicense'
    ○ Then  the one-trial locks are taken for {email} and its company domain — not run
    ○ Then  a follow-up task is assigned to the primary admin, due in {days} days — not run
    ○ Then  {email} received the trial-ready email — not run
    ○ Then  only the primary admin was alerted that {name} started a trial — not run
  Failed with Rights: 10, Wrongs: 1, Errors: 0

The step that failed carries the Shouldly message as its cell, and every step after it is listed as not run, so the reader sees both what went wrong and how much of the flow never got exercised. That rendering shows up in the test runner’s output too: dotnet test prints the same tree under the failed test, so a CI log has it without any extra tooling.

The message activity report is for your AI agent

We think and hope that functionality like this in Bobcat will be a killer advantage for doing AI assisted development. In a later blog post I’ll discuss a bit more about how you can emit your own custom telemetry into test results as well.

Look again at the bottom of that failure. Below the tree, the run prints a Message activity table. It’s there on the console for every scenario bobcat run --verbose shows and for every failed scenario dotnet test prints, and it’s in the JSON results and the UI. Here it is for the happy path, trimmed to the columns that fit a blog (i.e. it looks better in real life, but you get the point):

  Message activity
  ┌─────────┬───────────────────┬──────────────────────────────┬─────────┬──────────────────┐
  │ at (ms) │ event             │ message                      │ attempt │ correlation id   │
  ├─────────┼───────────────────┼──────────────────────────────┼─────────┼──────────────────┤
  │ 142     │ Sent              │ SendTrialStartedNotification │ 0       │ 357c8f36bc708c0a │
  │ 143     │ Sent              │ NotifyAdminsTrialStarted     │ 0       │ 357c8f36bc708c0a │
  │ 143     │ Sent (scheduled)  │ TaskDueReminder              │ 0       │ 357c8f36bc708c0a │
  │ 144     │ Received          │ SendTrialStartedNotification │ 0       │ 357c8f36bc708c0a │
  │ 144     │ Received          │ NotifyAdminsTrialStarted     │ 0       │ 357c8f36bc708c0a │
  │ 145     │ ExecutionStarted  │ NotifyAdminsTrialStarted     │ 0       │ 357c8f36bc708c0a │
  │ 145     │ ExecutionStarted  │ SendTrialStartedNotification │ 0       │ 357c8f36bc708c0a │
  │ 153     │ ExecutionFinished │ NotifyAdminsTrialStarted     │ 1       │ 357c8f36bc708c0a │
  │ 154     │ MessageSucceeded  │ NotifyAdminsTrialStarted     │ 1       │ 357c8f36bc708c0a │
  │ 165     │ ExecutionFinished │ SendTrialStartedNotification │ 1       │ 357c8f36bc708c0a │
  │ 165     │ MessageSucceeded  │ SendTrialStartedNotification │ 1       │ 357c8f36bc708c0a │
  └─────────┴───────────────────┴──────────────────────────────┴─────────┴──────────────────┘

The full table also carries the destination, the service, the message id, the causation id (the id of the message that caused this one, blank for a message the act sent itself), the saga id, and the OpenTelemetry trace and activity ids. One row per envelope record, in order. It’s a causal chain: find the act’s own rows, follow what they caused, see where each attempt ended.

This is deliberate, and it’s a part of Bobcat that came from its Storyteller ancestor to add more context to potentially failing tests for easier troubleshooting just in case it failed. A human reading a red spec usually wants the assertion message and maybe the stack trace. An AI agent fixing a red spec needs more than that, because it can’t set a breakpoint and poke around. It needs to know what the act actually did: which messages went out, which handlers ran, how many attempts, whether something was only scheduled, whether a cascade never happened. The message activity report, together with the Events appended and HTTP exchanges reports that WolverineFx.Bobcat records alongside it, is that answer, written out in the one place the agent is already reading, and keyed by the same correlation, causation and trace ids the running application’s telemetry uses.

The Critter Stack AI Skills already know how to read it. The critterstack-sdd-bobcat-troubleshooting skill teaches an agent to diagnose a red Bobcat specification from what the run wrote out, without rerunning under a debugger: the gathered disagreements on the failing step, then the message activity grid as a causal chain, the events appended, the HTTP exchanges, the step logs, and how to ask a running bobcat ui for the same thing as JSON (bobcat failing-spec --uid "Feature/Scenario"). And the ids in that grid are the bridge to the wolverine-observability-command-line-diagnostics skill, which covers the same application’s command line: dotnet run describe for handler discovery, routing and transport configuration, and the event-query and stream-query commands for reading the event store from a shell or a test loop. An agent that sees TaskDueReminder only ever Sent (scheduled) in a spec, and then can’t find a handler for it in describe, has found the bug in two commands, with no human in the loop.

That’s the iteration loop we’re building for: a spec goes red, the agent reads the tree, the cells, the logs and the activity, forms a hypothesis, checks it against the application’s own diagnostics, changes the code, and reruns only what was red with bobcat run --failed. Every piece of that is text on a console or JSON on a port, and none of it needed a sink of our own.

With bobcat ui running on :5525, the run also lands on the board, where each spec is linked to the Event Model slice its command binds to (StartTrialRequest here, from the [BobcatSpec(typeof(...))] attribute):

The Bobcat UI run page: eleven passing specifications for the self-serve CritterWatch trial, each bound to its slice

And a single scenario’s page is the same tree with the step logs and, below it, the message activity the tracked session saw, which is the view I’d paste into a pull request:

One specification in the Bobcat UI: the steps, the step logs showing the POSTs, and the message activity table for the notifications the act caused

Rough edges, honestly

This is a retrofit onto Razor Pages, not a greenfield Wolverine.HTTP app, and a few things weren’t perfectly smooth.

Shouldly inside a helper stops at the first disagreement. Bobcat’s own SpecAssert.Check records a cell and carries on, so one red run reports every wrong in a step, and on a green run each check shows as a ✓ cell under its step. A Shouldly call throws, so the step ends at the first failing assertion and the passing checks render no cells. We went with Shouldly anyway because (that’s what Jeremy strongly prefers!) it’s what the rest of the suite uses and the failure message is better; if you want every disagreement from one run, SpecAssert.Gather(() => x.ShouldBe(y)) gives you both.

WhenTracked doesn’t see events a Razor page commits. The vocabulary reference says that after WhenTracked every event committed during the tracked session counts as the act’s, so ThenEmitted(...) should work. It didn’t here: the page handler appends through the IDocumentSession it gets from DI, and those events came back MISSING even though ThenHandled<T>() for the messages the same handler published passed. The workaround is a [Then] helper that records the event store’s high-water sequence before the act and reads what was appended past it, which is what TheActAppended and TheActAppendedNoEvents above do. It’s filed; the pattern is fine to keep.

The MTP runner is all or nothing. You can’t have one project on Microsoft.Testing.Platform and another on VSTest under the same dotnet test. The xUnit v3 move was trivial for us; budget for it anyway.

Where this goes next

We’re mildly rebuilding and expanding our homegrown CRM system partially to improve JasperFx’s own automation (because the cobbler’s children currently have no shoes) and partly to flesh out an all new guided set of tutorials on the Critter Stack’s new “Event Modeling” capabilities, our ability to scaffold applications, Bobcat specifications, and how every part of that is glued together by our AI Skills for our new AI-centric development world.

Where to go from here

  • Bobcat is MIT and documented at jasperfx.github.io/bobcat; the Wolverine vocabulary used here is on the Specifications with Bobcat page.
  • The Critter Stack AI Skills are the agent-side half of this story. The introduction post explains what they are and how to install them into your agent, and they’re sold through the portal: Solo for individual developers, Team for teams under ten, and Team (Large) for bigger teams. Every support plan includes them.
  • Support. If you’d like help putting Bobcat, Wolverine or Marten to work on your own system, or you’d rather have us on call when a spec goes red for a reason that isn’t yours, that’s what the JasperFx support plans are for. Consulting engagements and workshops are at jasperfx.net/services.

Retrofitting existing xUnit.Net tests to Bobcat

Marten’s async daemon is the subsystem that runs projections and subscriptions in the background. It’s also been historically been the single hardest thing in Marten to test. Every test has to publish events, stand up a daemon (sometimes two, in hot/cold mode), wait for shards to catch up, and then check what the projections wrote. The DaemonTests project has also been for years one of our worst sources of flaky tests and I never trusted it to run in CI until earlier this year.

So when Bobcat needed a real test suite to prove itself against before 1.0, DaemonTests was one of the obvious guinea pigs (another was Wolverine’s leadership compliance tests that I’ll write about next). If projected specifications and the Supervisor could make that suite readable and reliable, they could hopefully do it anywhere.

This post is about how that went: the mechanics of how the tests were decorated, what the specifications look like in preview and in results, where we got real bugs out of it, and how the suite runs in CI today under the Supervisor’s retry budget.

I’ve already introduced Bobcat and the Bobcat Supervisor in earlier posts, so now let’s move on to a “what it looks like on a real codebase” follow up. All of the code is public in the Marten repository, and I’ll link to the actual files as I go.

Fair warning: a lot of the Bobcat work on DaemonTests was done by AI agents working from Bobcat issues, with me reviewing the pull requests. I’m going to quote the agents’ own write ups in a couple of places because, honestly, they documented the trade offs better than I would have.

No need to rewrite tests

Bobcat was initially going to be an alternative Gherkin tool with possibly a more code centric model where specifications could be “projected” from the raw test code — with a 3rd mode using a fluent interface kind of approach where you embedded Lambda functions inside of Given/When/Then() methods. It quickly became clear that the “projected” model was going to be the most usable, so we simplified Bobcat by dropping the early Gherkin model. And the less said about the Fluent Interface + nested Lambda approach, that better. I despise tools like that. Even the ones that I might have written myself years ago.

The constraint that shaped all of Bobcat’s “projected specification” model is that a big, old test suite has to be able to adopt it one class at a time, with no mandatory base class, no signature changes, and no rewrite. DaemonTests is exactly that kind of suite. It’s got hundreds of tests, a lot of them written years apart by different people, and nobody was going to sign up to rewrite them in a new DSL.

The first pull request back in September rendered two of the test classes as specifications. Eighteen tests started reporting themselves as Given/When/Then steps, and not one test body was edited. All of the steps came from decorating five helper methods on the shared DaemonContext base class and marking the two classes with one attribute.

The second pull request in October moved that work to Bobcat 1.0, routed the projection waits through Bobcat’s catch-up diagnostics, added a handful of brand new specifications written with the Critter Stack vocabulary, and rendered three more classes. Today the suite references Bobcat 1.0.0 and WolverineFx.Bobcat 6.55.0, and bobcat preview lists 30 specifications across six features:

Feature: Building an aggregate projection
Feature: Catching up projections and subscriptions
Feature: Rebuilding an inline projection
Feature: Resuming a failed rebuild
Feature: Skipping events that fail to apply
Feature: The async daemon keeps read models current

That’s a small fraction of the 393 tests in the project, but you have to start somewhere!

Mechanics, part one: decorate the helpers you already call

Almost every daemon test calls the same handful of helper methods on DaemonContext: start the daemon, publish the events, wait for a shard to reach a sequence number, rebuild a projection, compare every expected aggregate against what the projection actually wrote. Those helpers are the vocabulary of the suite. So that’s where the Bobcat attributes went:

[When("the projection daemon is running")]
internal async Task<IProjectionDaemon> StartDaemon()
{
var daemon = theStore.Tenancy.Default.Database.As<MartenDatabase>()
.StartProjectionDaemon(theStore);
await daemon.StartAllAsync();
_daemon = daemon;
return daemon;
}
[Given("the events are published")]
internal async Task PublishSingleThreaded()
{
await PublishSingleThreaded<Trip>();
}
[Given("the events are published on {threads} threads")]
internal Task PublishMultiThreaded(int threads)
{
// ...
}
[Then("the {shardName} shard has reached sequence {sequence}")]
internal async Task<ShardState> WaitForShard(IProjectionDaemon daemon, string shardName, long sequence,
Type readModel, TimeSpan timeout = default)
{
// ...
}
[When("the {projectionName} projection is rebuilt")]
internal Task RebuildProjection(IProjectionDaemon daemon, string projectionName)
{
return daemon.RebuildProjectionAsync(projectionName, CancellationToken.None);
}
[Then("every expected aggregate matches the actual")]
internal async Task CheckAllExpectedAggregatesAgainstActuals()
{
// ...
}

[Given], [When], [Then], [Check] and the keywordless [Step] are all the same StepAttribute with a different keyword. The {placeholder} names in the sentence bind to the method’s parameters by name, and they’re filled in from the argument at each call site. A Type argument renders as the type’s short name, so WaitForCatchUp(typeof(Trip)) reads as “the Trip read model has caught up”. The docs on step helpers cover the binding rules, and the placeholder rules are worth reading because a Guid or a record renders differently than you might expect.

Then the test class gets one attribute:

[BobcatFeature("Building an aggregate projection")]
public class build_aggregate_projection: DaemonContext
{
// 17 existing [Fact] tests, untouched
}

And a test like this one, which has not changed, now renders as a five step specification:

[Fact]
public async Task rebuild_the_projection()
{
NumberOfStreams = 10;
Logger.LogDebug("The expected number of events is {NumberOfEvents}", NumberOfEvents);
StoreOptions(x => x.Projections.Add(new TripProjectionWithCustomName(), ProjectionLifecycle.Async));
var agent = await StartDaemon();
await PublishSingleThreaded();
await WaitForShard(agent, "TripCustomName:All", NumberOfEvents, typeof(Trip), 30.Seconds());
Logger.LogDebug("About to rebuild TripCustomName:All");
await RebuildProjection(agent, "TripCustomName");
Logger.LogDebug("Done rebuilding TripCustomName:All");
await CheckAllExpectedAggregatesAgainstActuals();
}

And the results look like this in the console (we use Spectre.Console under the covers, so there’s color coded results and just take my word for it that it’s pretty!):

  rebuild the projection OK
  ────────────────────────────────
    ✓ When  the projection daemon is running (120ms)
    ✓ Given the events are published (67ms)
    ✓ Then  the TripCustomName:All shard has reached sequence 231 (248ms)
    ✓ When  the TripCustomName projection is rebuilt (25ms)
    ✓ Then  every expected aggregate matches the actual (21ms)

Yes, the When comes before the Given in that one, because that’s the order the test actually does things in. Bobcat renders what ran. It doesn’t editorialize. To be honest, I’ve never liked the Given/When/Then nomenclature anyway. To each their own.

The trick that makes this work with zero changes to the test bodies is that Bobcat.Generators is a source generator that emits C# interceptors for every call to a decorated helper. There’s no reflection or stack walking at runtime. Each call site is known at build time, which is also what lets bobcat preview show the specification before anything has run.

An interceptor has to be an extension method, and an extension method can’t see a protected member. So every decorated helper on DaemonContext had to be widened from protected to internal. That’s a C# language rule and not a Bobcat design choice, and Bobcat reports a protected helper as a BOBCAT039 diagnostic so you find out at build time rather than wondering why a step is missing. In Marten’s case that was eight helpers, and InternalsVisibleTo already existed for a sibling project, so nothing new was exposed. The Bobcat docs call this out because it’s the one thing you’ll actually have to touch.

Mechanics, part two: marker comments

Decorated helpers give you the work a test does. They don’t give you the narrative, and a lot of daemon tests have a middle section that’s just plain Marten code with no helper to hang a step on. For those, Bobcat reads marker comments: any // comment that opens with Given, When, Then, And or But as a whole word is a declared step. Every other comment is left alone, which is the property that makes this usable on a real suite that’s already full of explanatory comments like // Gotta do this to have it mixed up.

The clearest example in DaemonTests is when_skipping_events_in_daemon, which specifies what happens when a projection throws on some events and the daemon is configured to skip them:

[BobcatFeature("Skipping events that fail to apply")]
public class when_skipping_events_in_daemon : DaemonContext
{
// ...
[Given("the names are published to a running daemon, the bad ones failing to apply")]
internal async Task<IProjectionDaemon> PublishTheEvents()
{
var daemon = await StartDaemon();
var events = theNames.Select(name => new NameEvent {Name = name})
.OfType<object>()
.ToArray();
theSession.Events.StartStream(Guid.NewGuid(), events);
await theSession.SaveChangesAsync();
await daemon.Tracker.WaitForHighWaterMark(theNames.Length);
await WaitForShard(daemon, "CollateNames:All", theNames.Length, typeof(NamesByLetter));
await WaitForShard(daemon, "NamedDocuments:All", theNames.Length, typeof(NamedDocument), 5.Minutes());
return daemon;
}
[Fact]
public async Task see_the_dead_letter_events()
{
var daemon = await PublishTheEvents();
// When the daemon stops, draining the dead letter events it queued up
await daemon.StopAllAsync();
// Then each skipped event is a dead letter for the projection that skipped it
var skipped = await theSession.Query<DeadLetterEvent>().ToListAsync(TestContext.Current.CancellationToken);
var namedDocumentsSkipped = skipped.Where(x => x.ProjectionName == "NamedDocuments" && x.ShardName == "All")
.Select(x => x.EventSequence).OrderBy(x => x).ToList();
namedDocumentsSkipped.ShouldBe([4L, 5, 6, 7, 11, 14]);
var collateNamesSkipped = skipped.Where(x => x.ProjectionName == "CollateNames" && x.ShardName == "All")
.Select(x => x.EventSequence).OrderBy(x => x).ToList();
collateNamesSkipped.ShouldBe([4L, 5, 6, 7]);
}
}

There are three different Bobcat mechanisms in that one test, and they compose:

  1. PublishTheEvents() is a [Given] helper that lives on the test class itself rather than on DaemonContext. It calls StartDaemon() and WaitForShard(), which are decorated too, so those render nested underneath the Given with their own timings.
  2. The two // comments are marker comments. They declare what the test is doing in the stretch of code that calls no helper.
  3. The two ShouldBe() calls are projected assertions, which is the next section.

Here’s what that renders as from bobcat run:

  see the dead letter events OK
  ────────────────────────────────────
    ✓ Given the names are published to a running daemon, the bad ones failing to apply (306ms)
      ✓ When  the projection daemon is running (15ms)
      ✓ Then  the CollateNames:All shard has reached sequence 15 (44ms)
      ✓ And   the NamedDocuments:All shard has reached sequence 15
    · When  the daemon stops, draining the dead letter events it queued up — declared
    ✓ Then  each skipped event is a dead letter for the projection that skipped it (4ms)
      ✓ And   namedDocumentsSkipped should be [4L, 5, 6, 7, 11, 14] (4ms)
      ✓ And   collateNamesSkipped should be [4L, 5, 6, 7]

Notice the When the daemon stops line is marked declared with no tick and no timing. Nothing decorated ran inside that comment’s region, so Bobcat has nothing to report about it other than that the test claims to do it. The documentation is pretty strict about keeping “declared” and “recorded” separate, and I think that’s right. A step that borrowed a verdict from its neighbors would be exactly the kind of inference that makes test reports lie to you.

Marker comments are also how we narrated the longest, most involved daemon test in the suite, rebuild_mode_start_resumes_after_a_failure, which proves that a failed rebuild can be restarted from its committed progress without folding any event twice. That one is pure Marten and Postgres code with no shared helpers, and the comments carry the whole specification:

// Given a hundred streams of one event each
// When a Rebuild-mode run fails on event 60
// Then the committed progress and the failure are recorded by extended progression tracking
// And every event at or below the committed mark is in a document already
// When the shard is stopped, the fault cleared, and the Rebuild-mode run started again
// Then every event is folded into its document exactly once
// And the failure is cleared
// And no event below the first run's committed mark was folded again by the second run
// And the restarted run keeps following the store past where the rebuild started

Some of those comments, like the “Given a hundred streams” line, aren’t “meaningful” because they map to code. They’re meaningful because they’re the sentence a maintainer needs in order to understand what the next ten lines are for. Bobcat just happens to also render them.

One gotcha I tripped over while grabbing screenshots for this post: a marker comment wraps onto the // lines directly below it, up to a blank line or a line of code (dogfooding for the win I guess). That’s handy for long sentences, but it also means an explanatory comment sitting right under a marker becomes part of the step’s sentence. A couple of the steps in that rebuild test render with a trailing explanation glued on for that reason. Leave a blank line between the marker and the explanation and it stays a comment.

Mechanics, part three: your Shouldly assertions are steps too

We’ll add support for other assertion libraries later if there’s any demand, but I prefer Shouldly and that’s what the Critter Stack code already uses, so here we are.

DaemonTests has one line in its csproj that does a surprising amount of work:

<!-- A statement-level Shouldly assertion inside a [BobcatFeature] class renders as its own
Then step, and a run of consecutive assertions is all evaluated before the test fails,
so a red daemon spec shows every disagreement rather than the first. Classes without
[BobcatFeature] are untouched. -->
<BobcatProjectAssertions>true</BobcatProjectAssertions>

With that on, any statement-level Shouldly call inside a [BobcatFeature] class becomes a Then step with a sentence built from the C# expression as written, a verdict, a duration, and an expected/actual cell where the comparison allows it. jNames.Names.OrderBy(x => x).ShouldBe([...]) renders as “jNames.Names.OrderBy(x => x) should be [“Jack”, “Jane”, “Jeremy”, “Jill”]” with no attribute and no helper. Nothing moves.

The part that matters more than the rendering is that a run of consecutive assertions is all evaluated before the test fails. A plain Shouldly test throws on the first ShouldBe and never reaches the rest. With projected assertions, every assertion in the run executes, and the test fails once at the end with all of the wrongs in the failure. For daemon tests that check three or four things about shard state after a long setup, that’s the difference between one slow red/green cycle and four of them. The Shouldly page in the Bobcat docs is the whole story, including the closed set of comparisons that get a cell and the ones (ShouldBeTrue, ShouldBeOfType) that render as a step with no cell because any cell would be making something up.

Two things to know from using this on a real suite:

  • An assertion inside a loop is a step per iteration, because each one really ran. The “the shards should still be running” spec renders shard.Status should be AgentStatus.Running once per shard. If you don’t want that, assert once over the whole collection outside the loop, and the docs say the same. The rebuild-resume test does exactly that, and says why in a comment:
// One assertion over the whole set rather than one per event: with BobcatProjectAssertions on, a
// per-event ShouldBe would render as sixty steps
var foldedAgain = committedSequences
.Select(e => (Index: ((RestartableIncrement)e.Data).Index, e.Sequence))
.Where(x => RestartableTallyProjection.Applications[x.Index] != 1)
.Select(x => $"event {x.Index} at sequence {x.Sequence}")
.ToList();
foldedAgain.ShouldBeEmpty();
  • Only a statement is projected. x.ShouldNotBeNull().Name.ShouldBe("a") consumes the first assertion’s result, so Bobcat leaves a fluent chain alone rather than risk reporting a NullReferenceException in place of the real failure.

There’s also a comment in when_skipping_events_in_daemon explaining why each subject is pulled into a local first: on Bobcat 1.0.0-alpha.1 a projected assertion whose subject spanned lines broke the build because the generator didn’t escape the newlines in the subject string. That’s the kind of thing that dogfooding on a real suite is for, and it’s why the comments in these files tend to say why and not just what.

All the Bobcat-specific bits in one place

For the whole DaemonTests project, this is the entire Bobcat surface area:

Bobcat, Bobcat.Generators, Bobcat.XunitThe core package, the interceptor generator, and the xUnit v3 runner adapter that records every [BobcatFeature] class
WolverineFx.BobcatCritterStackSpec, CritterStackScenario and the EventStores helpers, which moved out of Bobcat core in 1.0.0-rc.2. Despite the name, it works against a Marten-only application
[BobcatFeature("...")] on a classNames the feature and opts the class in. The {Feature}/{Scenario} identity is the feature title plus the method name with underscores read as spaces (spec identities)
[Given], [When], [Then] on helper methodsThe step helpers, with {placeholder} bound to parameters by name. [BobcatStep(..., Keyword = "Given")] is the older spelling and still works
// Given, // When, // Then, // And commentsDeclared steps, read at compile time
<BobcatProjectAssertions>true</BobcatProjectAssertions>Statement-level Shouldly calls become Then steps

That’s it. No base class was imposed, and catch_up_timeout_diagnostics deliberately isn’t a [BobcatFeature] because those tests expect the error a recorded specification would report, which is a nice illustration that opting in is per class.

The one dependency cost worth being honest about: Bobcat.Xunit 1.0 is built on xunit.v3 4.x over Microsoft.Testing.Platform 2, and the two xunit lines can’t share a test project. Marten’s other 23 test projects are still on xunit.v3 3.x, so DaemonTests takes 4.x through a VersionOverride and the rest of the repo moves on its own schedule. The compatibility page spells out what pairs with what.

Waiting on the daemon

The most useful change in the second PR wasn’t a rendering change at all. Daemon tests spend most of their lives waiting for a projection shard to catch up, and when a wait times out, Marten’s own exception used to say roughly “the shards haven’t started within the timeout.” That tells you nothing about which shard, where it stood, or whether the daemon even ran.

DaemonContext now has two waits that route through Bobcat’s catch-up diagnosis:

/// <summary>
/// Wait for every async projection in <c>theStore</c> to catch up before reading
/// <paramref name="readModel"/>. On a timeout this throws Bobcat's
/// <see cref="ProjectionCatchUpTimeoutException"/>, which names each shard that was behind and
/// where it stood (<c>'Trip:All' at 412 of 1000</c>), rather than the bare TimeoutException
/// Marten's own wait throws. A stalled daemon test then says which shard stalled.
/// </summary>
[Given("the {readModel} read model has caught up")]
internal async Task WaitForCatchUp(Type readModel, TimeSpan timeout = default)
{
if (timeout == default) timeout = 30.Seconds();
try
{
await EventStores.WaitForNonStaleProjectionsAsync(theStore, timeout,
token: TestContext.Current.CancellationToken);
}
catch (TimeoutException e) when (e is not ProjectionCatchUpTimeoutException)
{
throw await catchUpTimedOut($"the {readModel.Name} read model has caught up", readModel, timeout, e);
}
}

We have pre-canned helpers for this in our Wolverine.Bobcat package that’s meant to be our specifications library for event sourced applications with the Critter Stack.

And when one of those waits times out, this is the message you get, from a test in catch_up_timeout_diagnostics that forces the timeout on purpose by never starting the daemon:

Waiting for 'the TripCustomName:All shard has reached sequence 43' timed out after 2s, before reading the Trip.
Behind: 'TripCustomName:All' at 0 of 43. This is the environment (a daemon behind, stopped or not running), not
evidence about the Trip itself. The store said: TimeoutException: Shard TripCustomName:All did not reach
sequence number 43 in the time allowed

“This is the environment, not evidence about the Trip itself” is the sentence I care about. When an AI agent, or a tired human, reads that failure, the next move is to look at the daemon and not at the projection code.

And it paid for itself before the PR was even reviewed. From the agent’s write up:

The first two full gate runs failed the new specs with Behind: 'ShipmentSummary:All' at 0 of 4, which meant the hosted daemon had never run its shard. The cause was that the fixture cleaned event data after the Solo daemon started. That restarted the sequence underneath a shard that had already loaded its old progress, so the shard sat “ahead” of every new event. Isolated runs won that race and full runs lost it.

That’s a real test fixture bug of exactly the kind that produces a “flaky” test that nobody can reproduce on their own machine. The fix was a one liner (clean the store before StartAsync()), and the comment in ShipmentsHost now explains the race for the next person. The diagnosis is what made it findable.

New specifications with the Critter Stack vocabulary

Everything above is about retrofitting existing tests. For new daemon specifications, we used CritterStackSpec, the store-only base class from WolverineFx.Bobcat, against an application that hosts its own daemon with AddAsyncDaemon(DaemonMode.Solo). This is from async_daemon_specifications:

[BobcatFeature("The async daemon keeps read models current")]
public class async_daemon_specifications: CritterStackSpec, IClassFixture<ShipmentsHost>
{
public async_daemon_specifications(ShipmentsHost app): base(app.Host)
{
ProjectionTimeout = 30.Seconds();
}
[Fact]
public async Task an_async_projection_catches_up_with_an_appended_event()
{
var id = Guid.NewGuid();
await GivenEvents<ShipmentSummary>(id, new ShipmentPacked(id, 3));
await WhenActing("the shipment is shipped", services => append(services, id, new ShipmentShipped(id)));
ThenEvents(new ShipmentShipped(id));
await ThenReadModel<ShipmentSummary>(id, Specify<ShipmentSummary>()
.With(x => x.Items, 3)
.With(x => x.Shipped, true));
}
[Fact]
public async Task an_async_projection_deletes_the_read_model_its_stream_cancels()
{
var id = Guid.NewGuid();
await GivenEvents<ShipmentSummary>(id, new ShipmentPacked(id, 2));
await WhenActing("the shipment is cancelled", services => append(services, id, new ShipmentCancelled(id)));
ThenEvents(new ShipmentCancelled(id));
await ThenNoReadModel<ShipmentSummary>(id);
}
}

Nothing in there waits by hand. ThenReadModel opens with Bobcat’s own catch-up step, which is the same diagnosis as above if it times out. And this is where the “data centric” side of Bobcat shows up, because the events and the read model render as grids:

The third spec in that file, a rebuild through the running daemon, had to go on a host of its own, and the class comment says why: JasperFx’s rebuild stops the shard’s continuous agent, rebuilds with dedicated agents, and doesn’t start the continuous agent again, so any spec that ran after the rebuild on a shared host timed out at “‘ShipmentSummary:All’ at 2 of 4”. Whether a hosted daemon should resume the shard after a rebuild is a question for Marten rather than for the specification, and I’d rather have that written down in the test than silently worked around.

What it looks like: preview, run, and the UI

Because every step is known at build time, you can see what the suite specifies without running it. bobcat preview prints the whole thing to the console with the templates unfilled and the helper each step binds to:

  rebuild the projection
  ────────────────────────────────
      ○ When  the projection daemon is running
        ↳ DaemonContext.StartDaemon
      ○ Given the events are published
        ↳ DaemonContext.PublishSingleThreaded
      ○ Then  the {shardName} shard has reached sequence {sequence}
        ↳ DaemonContext.WaitForShard
      ○ When  the {projectionName} projection is rebuilt
        ↳ DaemonContext.RebuildProjection
      ○ Then  every expected aggregate matches the actual
        ↳ DaemonContext.CheckAllExpectedAggregatesAgainstActuals

The same preview is available in the Bobcat UI once you’ve got a resident runner attached for the suite. This is bobcat ui plus bobcat resident src/DaemonTests/bin/Debug/net10.0/DaemonTests, and every feature and specification gets its own run button:

I like this view a lot more than I expected to. Seventeen tests about building an aggregate projection, readable in one screen by somebody who has never opened the C#.

Running a couple of features with bobcat run src/DaemonTests --feature "The async daemon keeps read models current" "Skipping events that fail to apply" gives you the run page, with the passing specs folded to one line each:

And when something’s wrong, the failed scenario is opened on its specification with the disagreement right there. I broke one expectation on purpose for this screenshot, so the aggregate is expected to contain a “Joe” that was never published:

The whole run also lands in a JSON results file with a published schema, and bobcat run --html writes a self contained results page you can attach to a CI artifact. Those are what an agent reads when it’s working on a red build, and the bobcat tool page covers the rest of the commands.

While grabbing these screenshots I noticed the UI’s scenario page renders the marker-comment steps ahead of the recorded helper steps and flattens the nesting, where the console gets the order right. That’s bobcat#810, and it’s a reminder that 1.0 means the model is stable, not that I’m done finding things.

The Supervisor, retry budgets, and the flaky test that earned its retry

Now the other half of the story. Readable specifications don’t help if the suite doesn’t run reliably, and DaemonTests was historically the least reliable project in Marten. It had its own workflow (daemon.yml) that shelled straight out to dotnet test, which meant the most timing-sensitive suite in the repo was the one with no retry story at all.

Today every Marten test target runs through the Bobcat Supervisor from the Nuke build, and the daemon suite is one job in the matrix of tests.yml. The whole configuration lives in build/SupervisedTests.cs, and the interesting part is small:

csharp

const int MaxRetriesPerRun = 25;
const int MaxAttemptsPerTest = 3;
static readonly TimeSpan StallThreshold = TimeSpan.FromMinutes(5);
void configureSupervisor(Supervisor supervisor, TestLedger ledger)
{
supervisor.RetryBudget = DisableTestRetry
? RetryBudget.None
: new RetryBudget { MaxAttemptsPerTest = MaxAttemptsPerTest, MaxRetriesPerRun = MaxRetriesPerRun };
// The policy only ever retries in a fresh process, so idle lane workers are released
// before the retry passes. Without it, workers+1 test hosts sit resident at once — which is
// what OOM-killed 16GB GitHub runners on Wolverine, both times during a retry.
supervisor.ReleaseIdleLanes = true;
// Report-only observability. A test in flight past the threshold is named at once (a wedged
// daemon test used to be visible only as a job hitting its 20 minute cap), and a one-line
// heartbeat every 30s says how far a run got and what has been running longest.
supervisor.StallThreshold = StallThreshold;
supervisor.StallAction = StallAction.Report;
supervisor.HeartbeatInterval = TimeSpan.FromSeconds(30);
// Longest-processing-time-first lane balancing from the previous run's ledger, keyed by
// display name (bobcat#730). No ledger, or no match, degrades to balancing by test count,
// and the run report's first lines say how many durations matched.
supervisor.KnownTestDurations = ledger.KnownDurations();
}

A few things about that policy that I think are important, and that took us a while to get right:

Retries are opt-in, per test, and always in a fresh process. Bobcat’s default failure policy only retries a test that carries [Trait("Retry", "N")]. Everything else that fails, fails. Up until we made that change, every failure in Marten’s CI was retried, which made “flaky” and “broken” impossible to tell apart without going and opening the log. A retry runs in a fresh process because a failed daemon test usually leaves a half-built store, schema objects, or a running daemon behind it. The fresh process is the reset.

The retry budget is the operator’s ceiling, not the test’s. MaxAttemptsPerTest = 3 is the most any test gets, including the first attempt. A test’s own [Trait("Retry", "N")] can lower that but never raise it. MaxRetriesPerRun = 25 caps the whole run, and the build’s own comment says how to read it: a run sitting at the cap isn’t “25 flaky tests,” it’s 25 flaky tests plus an unknown tail.

Stall detection is report-only. A test in flight past five minutes is named in the log, the job summary, and the run report, but nothing gets killed. Before this, a wedged daemon test was only visible as a job hitting its 20 minute cap with no indication of which test wedged. Nothing in Marten’s suites legitimately runs that long as one test.

A pass on a retry is never laundered into a green tick. This is the RetryLedger, and its header comment is the best summary of the problem I’ve read:

Nothing in the GitHub UI distinguishes a job that passed cleanly from a job that passed after retrying nine tests. Both render as a green tick, and a Serilog warning buried in the log of a job that PASSED is a warning nobody has a reason to go read. Without this, “we added retries” quietly becomes “we stopped being able to tell whether the suite works”.

So every supervised run writes its retry count to the GitHub job summary, emits a ::warning annotation when the count is nonzero so it shows up on a green run, and uploads a JSON ledger that a flakiness roll-up job aggregates across every job in the workflow. Deliberately not in there is failing the build past a retry threshold. The value is in the visibility, not in a cliff that turns master red on a bad day.

What the ledger says about the daemon suite right now

Here’s the daemon job’s ledger from the most recent green master run as I write this (November 2nd):

Tests393
Clean passes392
Passed on retry1
Retries performed1
Failed0
Wall clock5m 48s, one lane

The one test that passed on a retry was Bug_catching_up_apply_errors.should_catch_apply_event_exceptions, and it’s the best example I have of how the opt-in policy is supposed to work. That test carries [Trait("Retry", "3")], and the comment above it explains how it earned that:

// Retry is opt-in under the supervisor (JasperFx/bobcat#565), and this test earned it with
// evidence: in 33 green master runs (2026-09 to 2026-10-15) its first attempt failed in 13 —
// "Task should throw System.AggregateException but did not" — and passed on a fresh-process
// retry every time. It is still reported as FLAKY in the retry ledger and the job summary;
// the trait keeps a known intermittent from turning the job red, it does not hide it. The
// projection throws deterministically, so a CatchUpAsync that returns cleanly is worth an
// investigation of its own (state left by an earlier test in the same process is the lead).
[Fact]
[Trait("Retry", "3")]

That’s the whole philosophy in one comment. The ledger gave us the evidence (13 first-attempt failures in 33 runs, same message every time, always passes on a fresh process). The trait keeps a known intermittent from turning the job red. The ledger still reports it as flaky on every run so it can’t quietly disappear. And the comment leaves the next investigator a lead. A second test in the same file picked up the same trait a couple of weeks later after one miss turned a whole job red for a pull request that couldn’t possibly have touched it.

One worker, and a local gate

The daemon job runs supervised at one worker. The issue that drove this work measured two lanes green locally, 374 of 374 twice, at 5m10s against 7m09s with retries off, but raising the worker count in CI is a separate decision we want to make on CI evidence rather than on a laptop. Marten’s heavier suites do split across lanes with a database per lane, and the parallel-ready suites page is the checklist we run a suite through before we let it.

The last piece is that the same supervisor configuration runs locally. ./build.sh LocalGate runs the logical suites of tests.yml on your own machine through Bobcat’s local gate, with the same opt-in retry policy, the same stall report, and the same lane balancing from the ledger, before you push. The implementation is in build/LocalGate.cs and the header comment there is, again, the documentation.

What I’d tell you to do with your own suite

If you’ve got a big integration suite with a lot of history, this is the order I’d go in, because it’s the order that worked on DaemonTests:

  1. Put the suite under the Supervisor first, with retries off (--disable-test-retry in Marten’s build), and read the ledger for a couple of weeks. You’ll find out which tests are actually flaky and which are just broken.
  2. Give the proven-flaky tests a [Trait("Retry", "N")] with a comment that says what the evidence was. Keep the ledger reporting them.
  3. Pick the test classes that explain how the system is supposed to behave, decorate the three or four shared helpers they all call, and mark the classes with [BobcatFeature]. Don’t touch the test bodies.
  4. Add marker comments where a test’s narrative isn’t carried by a helper, and turn on BobcatProjectAssertions.
  5. Replace your bare TimeoutException waits with something that says what was behind. If you’re on Marten, Polecat or Fisher, WolverineFx.Bobcat has that built in.
  6. Write the new specifications with CritterStackSpec or WolverineSpec, and let the existing tests stay existing tests.

The code for all of this is public:

Bobcat is MIT licensed, 1.0 is out, and DaemonTests is the suite I’d point you at if you want to see what it looks like on code that wasn’t written for it.

Building Bobcat Specifications

Bobcat 1.0, the Critter Stack’s new tool for executable specifications (automated tests that are human readable for easier review) was released just yesterday. As I wrote about in the release post yesterday, Bobcat is the “spiritual” successor to a mostly forgotten OSS project of mine called Storyteller that was itself trying to be a better version of FitNesse which was trying to solve the same problems of making human readable but executable specifications that has mostly been popularized by Gherkin tools like Cucumber.

I’ve admittedly just lifted this from the documentation website, but here’s an introduction to authoring a specification with Bobcat:

What you will build: one business rule, specified three ways in plain C# against a plain .NET class, with no Critter Stack and nothing hosted. You start from an xUnit test you could have written yesterday, make it a specification without touching its body, give it a vocabulary so the next test reads the same way, and then write the data-heavy cases as tables. By the end you have seen what the run prints for a pass, a wrong value, a wrong row and a thrown exception.

Bobcat specifications are code. There is no second file and no binding layer between the specification and the test that executes it: the test is the specification. It keeps running on its own runner, in your IDE’s test explorer, and Bobcat renders it as readable steps with a verdict per step.

That is a deliberate compromise. A Gherkin or FitNesse tool theoretically lets a non-technical domain expert author a specification directly, in a file of their own; a projected test is C#, so authoring stays with the developers and the domain expert reviews the rendered specification instead.

There is a little bit of an exemption though for being able to express table centric tests through Markdown tables directly embedded inside of C# (we’ll try to add F# later, but for right now, we’re using C# specific features). The folks who did use Storyteller effectively usually mention the effectiveness of Storyteller’s support for table driven testing for data intensive tests, the “set verification” feature, and the ability to express decision tree tests in Storyteller and now Bobcat with declarative markdown tables.

I should point out that the source generator with Bobcat can (hopefully) validate the Markdown table text at compile time to help eliminate those pesky binding problems that were so nasty with Gherkin or FitNesse.

The project setup is Getting Started: the three packages, the runner adapter, and global.json. This page assumes that and starts at the first test. Every sample on it is compiled and run in Bobcat’s own repository, in src/Bobcat.Xunit.Samples, and every output block is what that run printed.

The rule

A bank account that may not be overdrawn:

public record Transaction(string Kind, decimal Amount);
public class Account
{
public decimal Balance { get; private set; }
public List<Transaction> Transactions { get; } = new();
public void Deposit(decimal amount)
{
Balance += amount;
Transactions.Add(new Transaction("Deposit", amount));
}
public bool TryWithdraw(decimal amount)
{
if (amount > Balance) return false;
Balance -= amount;
Transactions.Add(new Transaction("Withdrawal", amount));
return true;
}
}

Step 1: a test you already have, as a specification

Here is the test as most teams would write it, with Shouldly:

[Fact]
public void a_withdrawal_larger_than_the_balance_is_refused_as_most_teams_write_it()
{
var account = new Account();
account.Deposit(100);
var accepted = account.TryWithdraw(200);
accepted.ShouldBeFalse();
account.Balance.ShouldBe(100);
}

Make it a specification without changing a line of its logic. Mark the class [BobcatFeature("Withdrawals")], and say what each part of the body does in a comment that opens with Given, When or Then:

[Fact]
public void a_withdrawal_larger_than_the_balance_is_refused_narrated()
{
// Given an account with a balance of 100
var account = new Account();
account.Deposit(100);
// When 200 is withdrawn
var accepted = account.TryWithdraw(200);
// Then the withdrawal is refused and the balance is still 100
accepted.ShouldBeFalse();
account.Balance.ShouldBe(100);
}

On xUnit, [BobcatFeature] is all it takes: the Bobcat.Xunit package opens a scenario around every test in the class and closes it with the verdict xUnit reported. On TUnit, NUnit and MSTest, put [BobcatScenario] beside it. (The sample project references Bobcat by project rather than by package, so it states [BobcatScenario] itself.) Run dotnet test in a terminal and the run prints the specification it produced:

Feature: Withdrawals
═════════════════════
a withdrawal larger than the balance is refused narrated OK
────────────────────────────────────────────────────────────
· Given an account with a balance of 100 — declared
· When 200 is withdrawn — declared
✓ Then the withdrawal is refused and the balance is still 100 (3ms)
✓ Then accepted should be false (1ms)
✓ And account.Balance should be 100 (2ms)

The declared is honest. A marker comment declares a step; nothing observed the Given or the When, so they have no duration and no verdict of their own. The Then is different only because the sample project opts into projecting Shouldly assertions as steps, so the two calls under the comment rendered beneath it with their own verdicts. Without that opt-in the Then would read declared as well, and when a Shouldly assertion threw, the scenario would be red with the exception and nothing narrower: no expected and actual pair, and the second assertion never reached. That is the limit of this style, and also its whole appeal. Opting an existing suite in costs one attribute per class and some comments (Marker comments, Projecting the assertions you already wrote).

Keep the comments, add the helpers

The two styles compose. A comment gives a test its narrative, and a decorated helper called underneath it renders nested under that sentence, with its own timing and verdict (They are different steps, and they nest). Most suites end up with both.

Step 2: a vocabulary

The second withdrawal spec should read like the first, and the tenth like the second. Move the mechanics into step helpers on a class of their own: methods decorated with [Given], [When] or [Then], carrying a sentence whose {placeholders} are filled from the arguments of those names:

private readonly Account _account = new();
private bool _accepted;
[Given("an account with a balance of {balance}")]
internal void AnAccountWithABalanceOf(decimal balance) => _account.Deposit(balance);
[When("{amount} is withdrawn")]
internal void IsWithdrawn(decimal amount) => _accepted = _account.TryWithdraw(amount);
[Then("the withdrawal is {outcome}")]
internal void TheWithdrawalIs(string outcome)
=> SpecAssert.Check("outcome", _accepted ? "accepted" : "refused", outcome);
[Then("the balance is {balance}")]
internal void TheBalanceIs(decimal balance) => SpecAssert.Check("balance", _account.Balance, balance);

The specification is now four calls:

[Fact]
public void a_withdrawal_larger_than_the_balance_is_refused()
{
_steps.AnAccountWithABalanceOf(100);
_steps.IsWithdrawn(200);
_steps.TheWithdrawalIs("refused");
_steps.TheBalanceIs(100);
}

Every call to a decorated helper is a recorded step: Bobcat’s source generator intercepts the call at build time, so there is no reflection and no registration. Two details carry weight:

  • The helpers are internal, not private. The generated interceptor is an extension method, which cannot reach a private member. That one keyword is the whole adoption cost on an existing helper class (BOBCAT039).
  • SpecAssert.Check records instead of throwing. It writes an expected/actual cell onto the step in progress and returns, so the specification keeps going and reports every disagreement it has. The adapter fails the test once, at the end, with all of them (Checks that gather instead of throwing).

Now write a specification whose expectations are wrong. A withdrawal of 30 from 100 is accepted and leaves 70, and this spec says otherwise, twice:

[Fact]
public void a_withdrawal_within_the_balance_is_refused()
{
_steps.AnAccountWithABalanceOf(100);
_steps.IsWithdrawn(30);
_steps.TheWithdrawalIs("refused");
_steps.TheBalanceIs(100);
}
  a withdrawal within the balance is refused FAILED
  ────────────────────────────────────────────────────
    ✓ Given an account with a balance of 100
    ✓ When  30 is withdrawn
    ✗ Then  the withdrawal is expected 'refused', got 'accepted' (1ms)
    ✗ And   the balance is expected '100', got '70' (1ms)
  Failed with Rights: 2, Wrongs: 2, Errors: 0

Both wrongs are there. A cell named for the placeholder it belongs to renders in the sentence, where the value sits, and the test runner’s own failure message says the same thing:

failed Bobcat.Xunit.Samples.Specs.WithdrawalSpecs.a_withdrawal_within_the_balance_is_refused (38ms)
Xunit.MicrosoftTestingPlatform.XunitException: Bobcat.Engine.SpecAssertionException : 2 specification steps failed:
Then the withdrawal is refused => outcome: expected 'refused', got 'accepted'
And the balance is 100 => balance: expected '100', got '70'

A helper that throws is different. An exception is an error, not a wrong, and it stops the scenario: the step is marked !, the exception is printed once at the bottom, and every step after it renders as ○ … — not run with its placeholders unfilled, because its arguments were never evaluated (Report everything, then stop).

Step 3: the boundary as a decision table

The interesting cases of the overdraft rule are the ones at the edge: exactly the balance, a penny over. Four specs of four lines each would say less than one table. A helper that takes a StepTable receives the whole table from the call site, and TableRunner.Run calls a row method once per row:

[Then("withdrawals are decided as")]
internal Task WithdrawalsAreDecidedAs(StepTable table) => TableRunner.Run(this, nameof(decide), table);
private string decide(decimal balance, decimal amount)
{
var account = new Account();
account.Deposit(balance);
return account.TryWithdraw(amount) ? "accepted" : "refused";
}

The balance and amount columns bind to the parameters of those names. The returned value is compared against the one column no parameter names, outcome, so the table is a decision table with a verdict per row:

[Fact]
public Task the_overdraft_boundary() => _steps.WithdrawalsAreDecidedAs("""
| balance | amount | outcome |
| 100 | 99.99 | accepted |
| 100 | 100 | accepted |
| 100 | 100.01 | refused |
""");

The table is pipe-delimited text, read by an implicit conversion to StepTable. A markdown table pastes in unchanged, alignment row and all, so the table in the pull request and the table in the test are the same text. Give one row the wrong answer and the grid says which:

[Fact]
public Task the_overdraft_boundary_with_a_wrong_row() => _steps.WithdrawalsAreDecidedAs("""
| balance | amount | outcome |
| 100 | 99.99 | accepted |
| 100 | 100 | refused |
| 100 | 100.01 | refused |
""");
  the overdraft boundary with a wrong row FAILED
  ─────────────────────────────────────────────────
    ✗ Then  withdrawals are decided as (4ms)
╭───┬─────────┬────────┬────────────────────────────────────┬─────────╮
│ # │ balance │ amount │ outcome                            │ Verdict │
├───┼─────────┼────────┼────────────────────────────────────┼─────────┤
│ 1 │ 100     │ 99.99  │ accepted                           │   OK    │
│ 2 │ 100     │ 100    │ expected 'refused', got 'accepted' │  FAIL   │
│ 3 │ 100     │ 100.01 │ refused                            │   OK    │
╰───┴─────────┴────────┴────────────────────────────────────┴─────────╯
  Failed with Rights: 2, Wrongs: 1, Errors: 0

A row that throws is that row’s failure and the remaining rows still run. out parameters are output columns too, so one row method can judge several results (Decision tables).

Step 4: the history as a set verification

The account’s transaction list is a collection, and a collection claim is a set verification: the expected rows are a table, the system’s rows are matched against them, and every row gets its own verdict.

[Then("the transactions are")]
internal void TheTransactionsAre(StepTable expected)
=> SetVerificationComparer.Verify(_account.Transactions, expected, ordered: true);
[Fact]
public void the_history_records_what_happened()
{
_steps.AnAccountWithABalanceOf(100);
_steps.IsWithdrawn(30);
_steps.IsWithdrawn(200);
_steps.TheTransactionsAre("""
| Kind | Amount |
| Deposit | 100 |
| Withdrawal | 30 |
""");
}

The refused withdrawal must leave no trace, and the table says so by not listing it. Now a specification that forgets a withdrawal that did happen:

[Fact]
public void the_history_the_specification_did_not_expect()
{
_steps.AnAccountWithABalanceOf(100);
_steps.IsWithdrawn(30);
_steps.TheTransactionsAre("""
| Kind | Amount |
| Deposit | 100 |
""");
}
  the history the specification did not expect FAILED
  ──────────────────────────────────────────────────────
    ✓ Given an account with a balance of 100
    ✓ When  30 is withdrawn
    ✗ Then  the transactions are (4ms)
╭───┬────────────┬────────┬─────────╮
│ # │ Kind       │ Amount │ Verdict │
├───┼────────────┼────────┼─────────┤
│ 1 │ Deposit    │ 100    │   OK    │
│ 2 │ Withdrawal │ 30     │  EXTRA  │
╰───┴────────────┴────────┴─────────╯
  Failed with Rights: 4, Wrongs: 1, Errors: 0

EXTRA is a row the system produced that the specification does not describe. MISSING is a row the specification describes that the system never produced, shown with the values it expected. FAIL is a matched row with a wrong cell, and ORDER is a row in the wrong place when order was part of the claim. Name keyColumns: to say which columns identify a row, and leave ordered: out when order is not the claim (Set verification).

Step 5: one specification per example

A decision table judges many rows inside one specification, with one verdict. When each example is something a person may want to run, see fail or track on its own, make each row its own specification instead. [BobcatTheory] is xUnit’s [Theory] plus the scenario bracket, and [BobcatExamples] writes its rows as the same table:

[BobcatTheory]
[BobcatExamples("""
| balance | amount | outcome |
| 100 | 100 | accepted |
| 100 | 100.01 | refused |
""")]
public void a_withdrawal_is_decided(decimal balance, decimal amount, string outcome)
{
_steps.AnAccountWithABalanceOf(balance);
_steps.IsWithdrawn(amount);
_steps.TheWithdrawalIs(outcome);
}

Each row runs as its own test with its own identity and its own rendered steps:

  a withdrawal is decided (balance: 100, amount: 100, outcome: accepted) OK
  ────────────────────────────────────────────────────────────
    ✓ Given an account with a balance of 100
    ✓ When  100 is withdrawn
    ✓ Then  the withdrawal is accepted (1ms)

Columns bind to parameters by name, and because the table is a constant, the build checks it: a column naming no parameter or a cell the parameter’s type cannot read is a compile error rather than a surprise at run time (A theory’s examples as a table).

Reading and running

  • The run prints its specifications when a terminal is attached and nothing is listening on the monitor wire. BOBCAT_SPEC_CONSOLE=1 prints anyway, into CI’s captured output; BOBCAT_SPEC_CONSOLE_FILE=spec.txt writes the report to a file instead (Reading the specification the run produced).
  • bobcat run renders the specification tree in the terminal, and bobcat preview shows every specification without executing anything, because the steps are known at build time (The bobcat tool).
  • Your IDE’s test explorer shows each specification as the ordinary test it is, so you run and debug it there (Integrating Bobcat with Your IDE).

Three things to know before you start

  • xUnit v3, TUnit, NUnit or MSTest (3.10+). xUnit v2 has no adapter and is not waiting for one. Its BeforeAfterTestAttribute cannot hand over a test result, which is the seam the adapter needs (xUnit v2 is not supported).
  • Use the shipped runner adapter, Bobcat.Xunit, Bobcat.TUnit, Bobcat.NUnit or Bobcat.MSTest. Marten’s hand-rolled one never set the verdict, so every scenario it published was a clean pass, red tests included (What you have to add).
  • Declared is not executed. A marker comment describes what the test does; it does not make the test do it. The two can drift, and a comment can never carry a cell. When a step is worth a verdict of its own, it is worth a helper (Declared is not executed).