Flow-based specifications with Bobcat

Babu added a new feature to our JasperFx CRM website to allow folks to download trial licenses to CritterWatch to try it out before purchasing. It’s a small feature with a long tail: a prospect fills in a form, we email them a verification link, they click it, and in one transaction we provision an account and a contact, grant a trial license, sign them in, open a follow-up task for sales, and publish a couple of notifications. There’s a one-trial-per-person rule with the usual email games (plus-addressing, Gmail dots, free-mail domains), a one-trial-per-company rule, and a kill switch. I realized pretty quickly that it’s a good little problem domain to demonstrate our new Bobcat library for writing human readable specifications.

The integration tests for it were already fine. 545 lines of xUnit and Alba, every rule covered, green. But now, let’s add Bobcat to the existing tests and make the content and meaning of those tests more easily reviewable by humans. While Bobcat is most certainly jam packed with features to make “Table Driven Testing” easier, today I want to talk about using Bobcat for “Flow Based Testing”

What a flow-based spec is

I might be using the nomenclature wrong here, but once upon a time even before Cucumber dominated this space, we talked about expressing these kinds of automated tests as either “flow based” meaning it read like a paragraph of prose or “table based” where you mostly specified tests in tables. In the end, I think this is an “open ended wrench vs. close ended wrench” kind of choice where you need both approaches in different scenarios. I’m going to issue a promise for a better example of table driven tests with Bobcat later and just dive into this sample.

Most of the specs in the Critter Stack samples are one act: arrange some events, send one command, assert what happened. The trial is a little more complicated and the thing under test is the sequence:

1. the prospect submits the trial form
2. nothing is created yet, but a verification email goes out
3. the prospect follows the link
4. now the account, contact, license, sign-in cookie, sales task and notifications all appear, together

A Bobcat spec can act more than once. Every earlier act arranges state; the last act is the one every Then judges. So a flow-based spec reads top to bottom as the story the prospect lives through, and the assertions at the bottom are about the moment that matters.

The setup

Bobcat supports TUnit, NUnit, and MSTest too, but we’ve put more effort so far into xUnit.Net and TUnit. So far, I’m very happy with our not entirely originally planned decision to dump Gherkin and “just project” specifications from plain old xunit tests by sprinkling some Bobcat attributes and meaningful comments into the existing code. I think this is going to make Bobcat much easier for folks to adopt than it would have been with our original idea of being yet another Gherkin tool.

Anyway, our new spec project sits beside the existing test project (we haven’t completely replaced the old tests yet):

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <IsTestProject>true</IsTestProject>
    <TestingPlatformDotnetTestSupport>true</TestingPlatformDotnetTestSupport>
    <UseMicrosoftTestingPlatformRunner>true</UseMicrosoftTestingPlatformRunner>
    <!-- Render statement-level Shouldly assertions as specification steps. -->
    <BobcatProjectAssertions>true</BobcatProjectAssertions>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" />
    <PackageReference Include="xunit.v3" />
    <PackageReference Include="xunit.runner.visualstudio" />
    <PackageReference Include="Bobcat" />
    <PackageReference Include="Bobcat.Xunit" />
    <PackageReference Include="Bobcat.Generators" />
    <PackageReference Include="WolverineFx.Bobcat" />
    <PackageReference Include="WolverineFx.Bobcat.Http" />
    <PackageReference Include="Alba" />
    <PackageReference Include="Shouldly" />
  </ItemGroup>
  <ItemGroup>
    <ProjectReference Include="..\..\src\JasperFX.Portal\JasperFX.Portal.csproj" />
    <ProjectReference Include="..\JasperFX.Portal.TestSupport\JasperFX.Portal.TestSupport.csproj" />
  </ItemGroup>
</Project>

Two things worth calling out. Microsoft.Testing.Platform 2 on the .NET 10 SDK insists on dotnet test running in its own mode, so the repository’s global.json grows a "test": { "runner": "Microsoft.Testing.Platform" } section, which in turn means the old xUnit 2 project had to move to xUnit v3 as well. That move touched exactly one file (the fixture’s InitializeAsync and DisposeAsync now return ValueTask). And the TestSupport project is new: the portal’s test host, its fakes for Stripe, the package registry, Cloudflare Turnstile and email, and the seed helpers moved out of the xUnit project into a plain class library so both runners share one host. Nothing about the fakes changed.

The host fixture for the specs is the same shape as the one in the old project:

public sealed class PortalFixture : IAsyncLifetime
{
    private PortalTestHost? _inner;
    public PortalTestHost Portal => _inner ?? throw new InvalidOperationException("The test host has not started.");
    public IAlbaHost Host => Portal.Host;
    public async ValueTask InitializeAsync() => _inner = await PortalTestHost.StartAsync();
    public async ValueTask DisposeAsync() { if (_inner is not null) await _inner.DisposeAsync(); }
}
[Collection(PortalCollection.Name)]
public abstract class PortalSpec : WolverineHttpSpec
{
    protected PortalSpec(PortalFixture app) : base(app.Host)
    {
        Portal = app.Portal;
        Timeout = TimeSpan.FromSeconds(20);   // first request to a Razor page compiles the view
    }
    protected PortalTestHost Portal { get; }
    protected IDocumentStore Store => Portal.Store;
    protected FakeEmailSender Email => Portal.Email;
}

WolverineHttpSpec is the vocabulary from WolverineFx.Bobcat.Http: WhenPosted, WhenReading, ThenHandled, ThenMessageScheduled and friends, over an Alba host. One Postgres Testcontainer per run, no reset between specs, unique emails per spec. Exactly what the old suite did.

The steps

This is the part that makes a flow readable. The mechanics of driving two Razor pages, reading a token out of a fake outbox, and querying Marten all live in step helpers on a TrialSpec base class. The helper’s attribute text is what the rendered specification says; the method body is how it’s done, and the assertions inside it are ordinary Shouldly.

public abstract class TrialSpec(PortalFixture app) : PortalSpec(app)
{
    private string _lastPage = string.Empty;
    private IScenarioResult? _lastResponse;
    private long _sequenceBeforeAct;
    [Given("the Turnstile check will {outcome}")]
    public void TurnstileWill(TurnstileOutcome outcome) => Portal.FakeTurnstile.Next = outcome;
    [When("{email} requests a CritterWatch trial as an individual named {name}")]
    public Task RequestsATrialAsAnIndividual(string email, string name) =>
        RequestTrial(new Dictionary<string, string>
        {
            ["sku"] = TrialSku, ["AccountType"] = "individual",
            ["BuyerName"] = name, ["Email"] = email, ["BillingCountry"] = "US",
        });
    private async Task RequestTrial(Dictionary<string, string> form, bool clearLimits = true)
    {
        if (clearLimits) await ContactSeed.ClearRateLimitWindowsAsync(Store);
        var result = await Portal.Host.Scenario(s =>
        {
            s.Post.FormData(form).ToUrl("/trial");
            s.StatusCodeShouldBe(200);
        });
        _lastPage = await result.ReadAsTextAsync();
        SpecLog.Write($"POST /trial for {form["Email"]} -> 200");
    }
    [When("{email} follows the verification link")]
    public async Task FollowsTheVerificationLink(string email)
    {
        var token = VerificationTokenFor(email).ShouldNotBeNull($"a verification email reached {email}");
        // The verification POST is the act: it provisions the customer, grants
        // the trial, and publishes the notifications the tracked session waits for.
        await MarkActStart();
        await WhenTracked("the emailed token is posted to /trial/verify", async () =>
        {
            _lastResponse = await Portal.Host.Scenario(s =>
            {
                s.Post.FormData(new Dictionary<string, string> { ["Token"] = token }).ToUrl("/trial/verify");
                s.IgnoreStatusCode();
            });
            _lastPage = await _lastResponse.ReadAsTextAsync();
            SpecLog.Write($"POST /trial/verify -> {_lastResponse.Context.Response.StatusCode}");
        });
    }
    [Then("the page says {text}")]
    public void ThePageSays(string text) => _lastPage.ShouldContain(text, Case.Insensitive);
    [Then("the account holds a 30-day CritterWatch trial license and nothing was sold")]
    public async Task<Entitlement> TheAccountHoldsATrialLicense(Account account)
    {
        await using var read = Store.QuerySession();
        var entitlements = await read.Query<Entitlement>().Where(e => e.AccountId == account.Id).ToListAsync();
        var trial = entitlements.ShouldHaveSingleItem();
        trial.Sku.ShouldBe(TrialSku);
        trial.Source.ShouldBe(EntitlementSources.SelfServeTrial);
        trial.Kind.ShouldBe(FulfillmentKind.CritterWatchLicense);
        trial.PurchaseId.ShouldBeNull();
        (trial.PeriodEnd - DateTimeOffset.UtcNow).TotalDays.ShouldBe(30, 1);
        (await read.LoadAsync<RevealableToken>(trial.Id)).ShouldNotBeNull("the license key is waiting to be revealed");
        (await read.Query<Purchase>().Where(p => p.AccountId == account.Id).ToListAsync()).ShouldBeEmpty();
        return trial;
    }
}

Three habits in there that I’d recommend for any flow:

• WhenTracked for the act that cascades. The verification page publishes two messages and schedules a third. Wrapping the HTTP call in WhenTracked runs it inside a Wolverine tracked session, so the spec waits for every handler the act caused before the Then steps run, ThenHandled<T>() and ThenMessageScheduled<T>() have something to judge, and the run records the message activity (more on that below).
• Shouldly in the helpers, as you’d write it anywhere. Bobcat captures a failed assertion as the step’s failure cell, with Shouldly’s own message, and marks the steps after it as not run. You don’t need a second assertion vocabulary to get a rendered specification.
• SpecLog.Write for what the system did. Logs render under the step for a failed scenario (or with BOBCAT_VERBOSE_REPORTS=1), so a red spec says what the page answered, not just what was expected.

The spec

With the steps in place the happy path is the flow, in order, with nothing in it that isn’t the business rule:

[BobcatFeature("Self-serve CritterWatch trial")]
public sealed class SelfServeTrial(PortalFixture app) : TrialSpec(app)
{
    [BobcatSpec(typeof(StartTrialRequest), Domain = "Trials")]
    public async Task an_individual_requests_verifies_and_starts_a_trial()
    {
        var email = CompanyEmail("sam");
        TurnstileWill(TurnstileOutcome.Passed);
        await RequestsATrialAsAnIndividual(email, "Sam Solo");
        ThePageSays("Check your email");
        AVerificationEmailWasSentTo(email);
        await NothingExistsForYet(email);
        await FollowsTheVerificationLink(email);
        IsRedirectedHomeSignedIn(email);
        await TheActAppended(typeof(AccountCreated), typeof(ContactCreated), typeof(EntitlementGranted),
            typeof(TaskCreated), typeof(ContactSignedIn));
        var account = await IsAContactOnAnAccount(email, ContactRole.Member, "Sam Solo", AccountKind.Individual, "Sam Solo");
        await TheAccountHoldsATrialLicense(account);
        await TheOneTrialLocksAreTaken(email);
        await AFollowUpTaskIsAssigned(3, account);
        ThenMessageScheduled<TaskDueReminder>();
        ThenHandled<SendTrialStartedNotification>();
        ThenHandled<NotifyAdminsTrialStarted>();
        ReceivedTheTrialReadyEmail(email);
        OnlyThePrimaryAdminWasAlerted("Sam Solo");
    }
}

The refusals read the same way. The one-trial-per-company rule, for instance, is a flow where the second act is supposed to do nothing:

[BobcatSpec(typeof(StartTrialRequest), Domain = "Trials")]
public async Task a_colleague_on_the_same_company_domain_is_refused()
{
    var domain = $"{Guid.NewGuid():N}.test";
    TurnstileWill(TurnstileOutcome.Passed);
    await AlreadyTookATrial($"first@{domain}");
    await RequestsATrialForAnOrganization($"second@{domain}", "Same Corp", "Second Person");
    await FollowsTheVerificationLink($"second@{domain}");
    ThePageSays("already had a CritterWatch trial");
    await TheActAppendedNoEvents();
    await NothingExistsForYet($"second@{domain}");
}

AlreadyTookATrial is a [Given] helper that runs the whole happy-path flow for the first colleague. In the rendered spec it shows as one Given with the request and the verification nested under it, which is exactly the right level of detail for a precondition.

What it renders

bobcat run tests/JasperFX.Portal.Specs --verbose prints every specification as it ran. This is the happy path, verbatim apart from the message-activity table that follows it, which gets its own section next:

Feature: Self-serve CritterWatch trial
═══════════════════════════════════════
  an individual requests verifies and starts a trial OK
  ────────────────────────────────────────────────────────────
    ✓ Given the Turnstile check will Passed (1ms)
    ✓ When  sam@1f27b29e….test requests a CritterWatch trial as an individual named Sam Solo (84ms)
      Logs:
        POST /trial for sam@1f27b29e….test -> 200
    ✓ Then  the page says Check your email (5ms)
    ✓ And   a verification email was sent to sam@1f27b29e….test
    ✓ And   nothing exists for sam@1f27b29e….test yet (7ms)
    ✓ When  sam@1f27b29e….test follows the verification link (175ms)
      ✓ And   the emailed token is posted to /trial/verify (172ms)
        Logs:
          POST /trial/verify -> 302
    ✓ Then  sam@1f27b29e….test is redirected home, signed in (4ms)
    ✓ And   the act appended AccountCreated, ContactCreated, EntitlementGranted, TaskCreated, ContactSignedIn (8ms)
    ✓ And   sam@1f27b29e….test is a Member contact named Sam Solo on a Individual account named Sam Solo (4ms)
    ✓ And   the account holds a 30-day CritterWatch trial license and nothing was sold (15ms)
    ✓ And   the one-trial locks are taken for sam@1f27b29e….test and its company domain (1ms)
    ✓ And   a follow-up task is assigned to the primary admin, due in 3 days (2ms)
    ✓ And   TaskDueReminder is scheduled (1ms)
        ✓ scheduled: yes
    ✓ And   SendTrialStartedNotification is handled (2ms)
        ✓ handled: yes
    ✓ And   NotifyAdminsTrialStarted is handled
        ✓ handled: yes
    ✓ And   sam@1f27b29e….test received the trial-ready email
    ✓ And   only the primary admin was alerted that Sam Solo started a trial
  Succeeded with Rights: 18, Wrongs: 0, Errors: 0

The whole feature is eleven specifications, about four seconds after the container is up.

And this is what a failure looks like when a Shouldly assertion inside a helper disagrees (I changed the expected license kind on purpose for this run):

  an individual requests verifies and starts a trial FAILED
  ────────────────────────────────────────────────────────────
    ✓ Given the Turnstile check will Passed (1ms)
    ✓ When  sam@7d0a1913….test requests a CritterWatch trial as an individual named Sam Solo (112ms)
    ...
    ✓ And   sam@7d0a1913….test is a Member contact named Sam Solo on a Individual account named Sam Solo (7ms)
    ✗ And   the account holds a 30-day CritterWatch trial license and nothing was sold (16ms)
      trial.Kind
          should be
      FulfillmentKind.RegistryToken
          but was
      FulfillmentKind.CritterWatchLicense
        ✗ trial.Kind: expected 'FulfillmentKind.RegistryToken', got 'FulfillmentKind.CritterWatchLicense'
    ○ Then  the one-trial locks are taken for {email} and its company domain — not run
    ○ Then  a follow-up task is assigned to the primary admin, due in {days} days — not run
    ○ Then  {email} received the trial-ready email — not run
    ○ Then  only the primary admin was alerted that {name} started a trial — not run
  Failed with Rights: 10, Wrongs: 1, Errors: 0

The step that failed carries the Shouldly message as its cell, and every step after it is listed as not run, so the reader sees both what went wrong and how much of the flow never got exercised. That rendering shows up in the test runner’s output too: dotnet test prints the same tree under the failed test, so a CI log has it without any extra tooling.

The message activity report is for your AI agent

We think and hope that functionality like this in Bobcat will be a killer advantage for doing AI assisted development. In a later blog post I’ll discuss a bit more about how you can emit your own custom telemetry into test results as well.

Look again at the bottom of that failure. Below the tree, the run prints a Message activity table. It’s there on the console for every scenario bobcat run --verbose shows and for every failed scenario dotnet test prints, and it’s in the JSON results and the UI. Here it is for the happy path, trimmed to the columns that fit a blog (i.e. it looks better in real life, but you get the point):

  Message activity
  ┌─────────┬───────────────────┬──────────────────────────────┬─────────┬──────────────────┐
  │ at (ms) │ event             │ message                      │ attempt │ correlation id   │
  ├─────────┼───────────────────┼──────────────────────────────┼─────────┼──────────────────┤
  │ 142     │ Sent              │ SendTrialStartedNotification │ 0       │ 357c8f36bc708c0a │
  │ 143     │ Sent              │ NotifyAdminsTrialStarted     │ 0       │ 357c8f36bc708c0a │
  │ 143     │ Sent (scheduled)  │ TaskDueReminder              │ 0       │ 357c8f36bc708c0a │
  │ 144     │ Received          │ SendTrialStartedNotification │ 0       │ 357c8f36bc708c0a │
  │ 144     │ Received          │ NotifyAdminsTrialStarted     │ 0       │ 357c8f36bc708c0a │
  │ 145     │ ExecutionStarted  │ NotifyAdminsTrialStarted     │ 0       │ 357c8f36bc708c0a │
  │ 145     │ ExecutionStarted  │ SendTrialStartedNotification │ 0       │ 357c8f36bc708c0a │
  │ 153     │ ExecutionFinished │ NotifyAdminsTrialStarted     │ 1       │ 357c8f36bc708c0a │
  │ 154     │ MessageSucceeded  │ NotifyAdminsTrialStarted     │ 1       │ 357c8f36bc708c0a │
  │ 165     │ ExecutionFinished │ SendTrialStartedNotification │ 1       │ 357c8f36bc708c0a │
  │ 165     │ MessageSucceeded  │ SendTrialStartedNotification │ 1       │ 357c8f36bc708c0a │
  └─────────┴───────────────────┴──────────────────────────────┴─────────┴──────────────────┘

The full table also carries the destination, the service, the message id, the causation id (the id of the message that caused this one, blank for a message the act sent itself), the saga id, and the OpenTelemetry trace and activity ids. One row per envelope record, in order. It’s a causal chain: find the act’s own rows, follow what they caused, see where each attempt ended.

This is deliberate, and it’s a part of Bobcat that came from its Storyteller ancestor to add more context to potentially failing tests for easier troubleshooting just in case it failed. A human reading a red spec usually wants the assertion message and maybe the stack trace. An AI agent fixing a red spec needs more than that, because it can’t set a breakpoint and poke around. It needs to know what the act actually did: which messages went out, which handlers ran, how many attempts, whether something was only scheduled, whether a cascade never happened. The message activity report, together with the Events appended and HTTP exchanges reports that WolverineFx.Bobcat records alongside it, is that answer, written out in the one place the agent is already reading, and keyed by the same correlation, causation and trace ids the running application’s telemetry uses.

The Critter Stack AI Skills already know how to read it. The critterstack-sdd-bobcat-troubleshooting skill teaches an agent to diagnose a red Bobcat specification from what the run wrote out, without rerunning under a debugger: the gathered disagreements on the failing step, then the message activity grid as a causal chain, the events appended, the HTTP exchanges, the step logs, and how to ask a running bobcat ui for the same thing as JSON (bobcat failing-spec --uid "Feature/Scenario"). And the ids in that grid are the bridge to the wolverine-observability-command-line-diagnostics skill, which covers the same application’s command line: dotnet run describe for handler discovery, routing and transport configuration, and the event-query and stream-query commands for reading the event store from a shell or a test loop. An agent that sees TaskDueReminder only ever Sent (scheduled) in a spec, and then can’t find a handler for it in describe, has found the bug in two commands, with no human in the loop.

That’s the iteration loop we’re building for: a spec goes red, the agent reads the tree, the cells, the logs and the activity, forms a hypothesis, checks it against the application’s own diagnostics, changes the code, and reruns only what was red with bobcat run --failed. Every piece of that is text on a console or JSON on a port, and none of it needed a sink of our own.

With bobcat ui running on :5525, the run also lands on the board, where each spec is linked to the Event Model slice its command binds to (StartTrialRequest here, from the [BobcatSpec(typeof(...))] attribute):

And a single scenario’s page is the same tree with the step logs and, below it, the message activity the tracked session saw, which is the view I’d paste into a pull request:

Rough edges, honestly

This is a retrofit onto Razor Pages, not a greenfield Wolverine.HTTP app, and a few things weren’t perfectly smooth.

Shouldly inside a helper stops at the first disagreement. Bobcat’s own SpecAssert.Check records a cell and carries on, so one red run reports every wrong in a step, and on a green run each check shows as a ✓ cell under its step. A Shouldly call throws, so the step ends at the first failing assertion and the passing checks render no cells. We went with Shouldly anyway because (that’s what Jeremy strongly prefers!) it’s what the rest of the suite uses and the failure message is better; if you want every disagreement from one run, SpecAssert.Gather(() => x.ShouldBe(y)) gives you both.

WhenTracked doesn’t see events a Razor page commits. The vocabulary reference says that after WhenTracked every event committed during the tracked session counts as the act’s, so ThenEmitted(...) should work. It didn’t here: the page handler appends through the IDocumentSession it gets from DI, and those events came back MISSING even though ThenHandled<T>() for the messages the same handler published passed. The workaround is a [Then] helper that records the event store’s high-water sequence before the act and reads what was appended past it, which is what TheActAppended and TheActAppendedNoEvents above do. It’s filed; the pattern is fine to keep.

The MTP runner is all or nothing. You can’t have one project on Microsoft.Testing.Platform and another on VSTest under the same dotnet test. The xUnit v3 move was trivial for us; budget for it anyway.

Where this goes next

We’re mildly rebuilding and expanding our homegrown CRM system partially to improve JasperFx’s own automation (because the cobbler’s children currently have no shoes) and partly to flesh out an all new guided set of tutorials on the Critter Stack’s new “Event Modeling” capabilities, our ability to scaffold applications, Bobcat specifications, and how every part of that is glued together by our AI Skills for our new AI-centric development world.

Where to go from here

• Bobcat is MIT and documented at jasperfx.github.io/bobcat; the Wolverine vocabulary used here is on the Specifications with Bobcat page.
• The Critter Stack AI Skills are the agent-side half of this story. The introduction post explains what they are and how to install them into your agent, and they’re sold through the portal: Solo for individual developers, Team for teams under ten, and Team (Large) for bigger teams. Every support plan includes them.
• Support. If you’d like help putting Bobcat, Wolverine or Marten to work on your own system, or you’d rather have us on call when a spec goes red for a reason that isn’t yours, that’s what the JasperFx support plans are for. Consulting engagements and workshops are at jasperfx.net/services.