Somewhat coincidentally, there’s a new Storyteller 4.1.1 release up today that improves the Storyteller spec editor UI quite a bit. To use the techniques shown in this post, you’ll want to at least be on 4.1 (for some bug fixes to problems found in writing this blog post).

One of our goals with the Storyteller 4.0 release was to shorten the time and effort it takes to go from authoring or capturing specification text to a fully automated execution with backing code. As part of that, Joe McBride and I built in a new feature that lets you create or modify the specification language for Storyteller with markdown completely outside of the backing C# code.

Great, but how about a demonstration to make that a bit more concrete? I’m working on a Jasper feature today to effectively provide a form of content negotiation within the service bus to try to select the most efficient serialization format for a given message. At the moment we need to specify and worry about:

• What serialization formats are available?
• What is the preferred formats for the application in order?
• Are there any format preferences for the outgoing channel where the message is going to be sent?
• Did the user explicitly choose which serialization format to use for the message?
• If this message is a response to an original message sent from somewhere else, did the original sender specify its preferred list of serialization formats?

Okay, so back to Storyteller. Step #1 is to design the specification language I’ll need to describe the desired serialization selection logic to Storyteller Fixture’s and Grammar’s. That led to a markdown file like this that I added with the “New Fixture” link from the Storyteller UI:

# Serializer Selection

## AvailableSerializers
### The available serializers are {mimetypes}

## Preference
### The preferred serializer order is {mimetypes}

## SerializationChoice
### Outgoing Serialization Choice
|table  |content     |channel               |envelope               |selection|
|default|NULL        |NULL                  |EMPTY                  |EMPTY    |
|header |Content Type|Channel Accepted Types|Envelope Accepted Types|Selection|

This is definitely the kind of scenario that lends itself to being expressed as a decision table, so I’ve described a Table grammar for the main inputs and the expected serialization format selection.

Now, without writing any additional C# code, I can switch to writing up acceptance tests for the new serialization selection logic. I think in this case it’s a little bit easier to go straight to the specification markdown file, so here’s the first specification as that:

# Serialization Selection Rules

[SerializerSelection]
|> AvailableSerializers text/xml; text/json; text/yaml
|> Preference text/json; text/yaml
|> SerializationChoice
    [rows]
    |content   |channel               |envelope               |selection|
    |NULL      |EMPTY                 |EMPTY                  |text/json|
    |NULL      |text/xml, text/yaml   |EMPTY                  |text/xml |
    |NULL      |EMPTY                 |text/xml, text/yaml    |text/xml |
    |text/xml  |EMPTY                 |EMPTY                  |text/xml |
    |text/xml  |text/json, text/other |text/yaml              |text/xml |
    |text/other|EMPTY                 |EMPTY                  |NULL     |
    |NULL      |text/other, text/else |EMPTY                  |NULL     |
    |NULL      |text/other, text/json |EMPTY                  |text/json|
    |NULL      |EMPTY                 |text/other             |NULL     |
    |NULL      |EMPTY                 |text/other, text/json  |text/json|
    |NULL      |text/yaml             |text/xml               |text/xml |

In the Storyteller UI, this specification is rendered as this:

At this point, it’s time for me to write the backing Fixture code. Using the new Fixture & Grammar Explorer page in Storyteller 4, I can export a stubbed version of the Fixture code I’ll need to implement:

    public class SerializerSelectionFixture : StoryTeller.Fixture
    {
        public void AvailableSerializers(string mimetypes)
        {
            throw new System.NotImplementedException();
        }

        public void Preference(string mimetypes)
        {
            throw new System.NotImplementedException();
        }

        [StoryTeller.Grammars.Tables.ExposeAsTable("Outgoing Serialization Choice")]
        public void SerializationChoice(string content, string channel, string envelope, string selection)
        {
            throw new System.NotImplementedException();
        }              
    }

That’s only Storyteller’s guess at what the matching code should be, but in this case it’s good enough with just one tweak to the “SerializationChoice” method you can see in the working code for the class above.

Now I’ve got a specification for the desired functionality and even a stub of the test harness. Time for coffee, standup, and then actually writing the real code and fleshing out the SerializerSelectionFixture class shown above. Back in a couple hours….

…which turned into a week or two of Storyteller bugfixes, but here’s the results of the specification as rendered in the results:

EDIT: Nice coincidence, there’s a new podcast today with Matthew Groves and I talking about Storyteller we recorded at CodeMash 2017.

Before I introduce the Storyteller 4.1 release, I’ve got to talk about the art of making OSS releases. I admittedly got impatient to get the big Storyteller 4.0 release out the door last month to time it with a trip to my company’s main office. Not quite a month later, I’m having to push Storyteller 4.1 this morning with some key usability changes and some significant bug fixes that make the tool much more usable. Depending on how you want to look at it, I think you can say two different things about my Storyteller 4.0 release:

1. I probably should have dogfooded it longer on my own projects before releasing it and I might have earned Storyteller a bad first impression from some folks.
2. By releasing when I did, I got valuable feedback from early users and a couple significant pull requests fixing issues that I might not have found on my own.

So, was I too hasty or not on releasing 4.0 last month? I’m going to give myself a pass just this one time because the feedback from early adopters was so helpful, but next time I roll out something as big as Storyteller 4 that had to swap out so much of its architecture, I think I’ll do more dogfooding and just kick out early alphas. I’m also in a position where I can drop alpha tools onto some of our internal teams and let them find problems, but I honestly try not to let that happen too much.

Storyteller 4.1

I just pushed a round of Nuget updates for Storyteller 4.1 that added some convenience functionality and quite a few bug fixes, a couple of which were somewhat severe. The new Nugets today include:

1. Storyteller 4.1
2. StorytellerRunnerCsproj 4.1.0.506 (it’s still using my old pre-dotnet cli mechanisms for building Nuget’s within TeamCity builds, if you’re wondering why the version is so different)
3. StorytellerRunner 1.1
4. dotnet-storyteller 1.1
5. dotnet-stdocs 1.0.0

The entire release notes and issues can be found here. The highlights are:

• Storyteller completely disables the file watching on binary files when you’re using Storyteller in the dotnet CLI mode, and it’s been somewhat relaxed in the older AppDomain mode to prevent unnecessary CPU usage. If you’re using the dotnet CLI mode, just know that you have to manually rebuild the underlying system. Fortunately, that can be done at any time in the Storyteller UI with the “ctrl+shift+b” shortcut (suspiciously similar to VS.Net). You can also force a system recycle before running a specification from any specification page with the “ctrl+2” shortcut.
• While we’re still committed to doing a dotnet test adapter for Storyteller when we feel that VS2017 is stable, for the meantime, Storyteller 4.1 introduces a new class called “StorytellerRunner” that you can use to run specifications directly from within your IDE.
• Storyteller can more readily deal with file paths with spaces in the path. Old timers like me still think that’s unnatural, but Storyteller is going to adapt to the world that is here;)
• A new “SimpleSystem” super class for users to more quickly customize system bootstrapping, teardown, and more readily apply actions immediately before or after specification runs.

New Constellation of Storyteller Extensions

All of these are in flight, but a couple are going into early usage this week, so here’s what’s in store in the near future:

1. Storyteller.AspNetCore — new library that allows you to control an ASP.Net Core application from within Storyteller. So far, all it does is handle the application bootstrapping and teardown, but we’re hoping to quickly add some integrated diagnostics to the Storyteller HTML results for HTTP requests. This does use on the “also in flight” Alba project.
2. Storyteller.RDBMS — I talked about it a little here. Right now I’ve tested it against Postgresql and one of my teammates at work is starting to use it against Sql Server this week.
3. Storyteller.Selenium — this is a little farther back on the back burner, but we’re building up a Selenium helper for Storyteller. Lots of folks ask questions about integrating Storyteller and Selenium, so this might move up the priority list.

As I wrote about a couple weeks back, we’re looking to be a bit more Agile with our relational database development. Storyteller is generally our tool of choice for automated testing when the problem domain involves a lot of data setup and where the declarative data checking becomes valuable. To take the next step toward more test automation against both our centralized database and the related applications, I’ve been working on a new package for Storyteller to enable easy integration of relational database manipulation and insertions. While I don’t have anything released to Nuget yet, I was hoping to get a little bit of feedback from others who might be interested in this new package — and have something to show other developers at work;)

As a super simplistic example, I’ve been retrofitting some Storyteller coverage against the Hilo sequence generation in Marten. That feature really only has two database objects:

1. mt_hilo: a table just to track which “page” of sequential numbers has been reserved
2. mt_get_next_hi: a stored procedure (I know, but let it go for now) that’s used to reserve and fetch the next page for a named entity

Those objects are shown below:

DROP TABLE IF EXISTS public.mt_hilo CASCADE;
CREATE TABLE public.mt_hilo (
	entity_name			varchar CONSTRAINT pk_mt_hilo PRIMARY KEY,
	hi_value			bigint default 0
);

CREATE OR REPLACE FUNCTION public.mt_get_next_hi(entity varchar) RETURNS int AS $$
DECLARE
	current_value bigint;
	next_value bigint;
BEGIN
	select hi_value into current_value from public.mt_hilo where entity_name = entity;
	IF current_value is null THEN
		insert into public.mt_hilo (entity_name, hi_value) values (entity, 0);
		next_value := 0;
	ELSE
		next_value := current_value + 1;
		update public.mt_hilo set hi_value = next_value where entity_name = entity;
	END IF;

	return next_value;
END
$$ LANGUAGE plpgsql;

As a tiny proof of concept, I wanted to have a Storyteller specification just to test the happy path of the objects above. In the Fixture class for the Hilo sequence objects, I need grammars to:

1. Verify that there is no existing data in mt_hilo at the beginning of the spec
2. Call the mt_get_next_hi function with a given entity name and verify the page number returned from the function
3. Do a set verification of the exact rows in the mt_hilo table at the end of the spec

To implement the desired specification language for the steps above, I wrote this class using the new Storyteller.RDBMS bits:

    public class HiloFixture : PostgresqlFixture
    {
        public HiloFixture()
        {
            Title = "The HiLo Objects";
        }

        public override void SetUp()
        {
            WriteTrace("Deleting from mt_hilo");
            Runner.Execute("delete from mt_hilo");
        }

        public IGrammar NoRows()
        {
            return NoRowsIn("There should be no rows in the mt_hilo table", "public.mt_hilo");
        }

        public RowVerification CheckTheRows()
        {
            return VerifyRows("select entity_name, hi_value from mt_hilo")
                .Titled("The rows in mt_hilo should be")
                .AddField("entity_name")
                .AddField("hi_value");
        }

        public IGrammarSource GetNextHi(string entity)
        {
            return Sproc("mt_get_next_hi")
                .Format("Get the next Hi value for entity {entity} should be {result}")
                .CheckResult<int>();
        }
    }

A couple other notes on the class above:

• You might notice that I’m cleaning out the mt_hilo table in the Fixture.Setup() method. I do this to quietly establish a known starting state at the beginning of the specification execution
• It’s not shown here, but part of your setup for this tooling is to tell Storyteller what the database connection string is. I haven’t exactly settled on the final mechanism for this yet.
• The HiloFixture class subclasses the PostgresqlFixture class that provides some helpers for defining grammars against a Postgresql database. I’m developing against Postgresql at the moment (just so I can code on OSX), but this new package will target Sql Server as well out of the box because that’s what we need it for at work;)

Now that we’ve got the Fixture, I wrote this specification shown in Storyteller’s markdown flavored persistence:

# Read and Write

[Hilo]

In the initial state, there should be no data

|> NoRows
|> GetNextHi entity=foo, result=0
|> GetNextHi entity=bar, result=0
|> GetNextHi entity=foo, result=1
|> CheckTheRows
    [rows]
    |entity_name|hi_value|
    |foo        |1       |
    |bar        |0       |

Finally, here’s what the result of running the specification above looks like:

Where do I foresee this being used?

I think the main usage for us is with some of our services that are tightly coupled to a Sql Server database. I see us using this tool to set up test data and be able to verify expected database state changes when our C# services execute.

I also see this for testing stored procedure logic when we deem that valuable, especially when the data setup and verification requires a lot of steps. I say that because Storyteller turns the expression of the specification into a declarative form. That’s also valuable because it helps you to decouple the expression of the specification from changes to the database structure. I.e., using Storyteller means that you can more easily handle scenarios like a database table getting a new non-null column with no default that would break any hard coded Sql statements.

I’d of course prefer not to have a lot of business logic in sproc’s, but if we are going to have mission critical sproc’s in production, I’d really prefer to have some test coverage over them.

Storyteller is a long running project for authoring human readable, executable specifications for .Net projects. The new 4.0 release is meant to make Storyteller easier to use and consume for non technical folks and to improve developer’s ability to troubleshoot specification failures.

After about 5 months of effort, I was finally able to cut the 4.0 Nugets for Storyteller this morning and the very latest documentation updates. If you’re completely new to Storyteller, check out our getting started page or this webcast. If you’re coming from Storyteller 3.0, just know that you will need to first convert your specifications to the new 4.0 format. The Storyteller Fixture API had no breaking changes, but the bootstrapping steps are a little bit different to accommodate the dotnet CLI.

You can see the entire list of changes here, or the big highlights of this release are:

• CoreCLR Support! Storyteller 4.0 can be used on either .Net 4.6 projects or projects that target the CoreCLR. As of now, Storyteller is now a cross platform tool. You can read more about my experiences migrating Storyteller to the CoreCLR here.
• Embraces the dotnet CLI. I love the new dotnet cli and wish we’d had it years ago. There is a new “dotnet storyteller” CLI extensibility package that takes the place of the old ST.exe console tool in 3.0 that should be easier to set up for new users.
• Markdown Everywhere! Storyteller 4.0 changed the specification format to a Markdown plus format, added a new capability to design and generate Fixture’s with markdown, and you can happily use markdown text as prose within specifications to improve your ability to communicate intentions in Storyteller specifications.
• Stepthrough Mode. Integration tests can be very tricky to debug when they fail. To ease the load, Storyteller 4.0 adds the new Stepthrough mode that allows you manually walk through all the steps of a Storyteller specification so you can examine the current state of the system under test as an aid in troubleshooting.
• Asynchronous Grammars. It’s increasingly an async-first kind of world, so Storyteller follows suit to make it easier for you to test asynchronous code.
• Performance Assertions. Storyteller already tracks some performance data about your system as specifications run, so why not extend that to applying assertions about expected performance that can fail specifications on your continuous integration builds?

Other Things Coming Soon(ish)

• A helper library for using Storyteller with ASP.Net Core applications with some help from Alba. I’m hoping to recreate some of the type of diagnostics integration we have today with Storyteller and our FubuMVC applications at work for our newer ASP.net Core projects.
• A separate package of Selenium helpers for Storyteller
• An extension specifically for testing relational database code
• A 4.1 release with the features I didn’t get around to in 4.0;)

How is Storyteller Different than Gherkin Tools?

First off, can we just pretend for a minute that Gherkin/Cucumber tools like SpecFlow may not be the absolute last word for automating human readable, executable specifications?

By this point, I think most folks associate any kind of acceptance test driven development or truly business facing Behavioral Driven Development with the Gherkin approach — and it’s been undeniably successful. Storyteller on the other hand, was much more influenced by Fitnesse and could accurately be described as a much improved evolution of the old FIT model.

SpecFlow is the obvious comparison for Storyteller and by far the most commonly used tool in the .Net space. The bottom line for me with Storyteller vs. SpecFlow is that I think that Storyteller is far more robust technically in how you can approach the automated testing aspect of the workflow. SpecFlow might do the business/testing to development workflow a little better (but I’d dispute that one too with the release of Storyteller 4.0), but Storyteller has much, much more functionality for instrumenting, troubleshooting, and enforcing performance requirements of your specifications. I strongly believe that Storyteller allows you to tackle much more complex automated testing scenarios than other options.

Here is a more detailed list about how Storyteller differs from SpecFlow:

• Storyteller is FOSS. So on one hand, you don’t have to purchase any kind of license to use it, but you’ll be dependent upon the Storyteller community for support.
• Instead of parsing human written text and trying to correlate that to the right calls in the code, Storyteller specifications are mostly captured as the input and expected output. Storyteller specifications are then “projected” into human readable HTML displays.
• Storyteller is much more table centric than Gherkin with quite a bit of functionality for set-based assertions and test data input.
• Storyteller has a much more formal mechanism for governing the lifecycle of your system under test with the specification harness rather than depending on an application being available through other means. I believe that this makes Storyteller much more effective at development time as you cycle through code changes when you work through specifications.
• Storyteller does not enforce the “Given/When/Then” verbiage in your specifications and you have much more freedom to construct the specification language to your preferences.
• Storyteller has a user interface for editing specifications and executing specifications interactively (all React.js based now). The 4.0 version makes it much easier to edit the specification files directly, but the tool is still helpful for execution and troubleshooting.
• We do not yet have direct Visual Studio.Net integration like SpecFlow (and I’m somewhat happy to let them have that one;)), but we will develop a dotnet test adapter for Storyteller when the dust settles on the VS2017/csproj churn.
• Storyteller has a lot of functionality for instrumenting your specifications that’s been indispensable for troubleshooting specification failures and even performance problems. The built in performance tracking has consistently been one of our most popular features since it was introduced in 3.0.