Storyteller 3: Executable Specifications and Living Documentation for .Net

tl;dr: The open source Storyteller 3 is an all new version of an old tool that my shop (and others) use for customer facing acceptance tests, large scale test automation, and “living documentation” generation for code-centric systems.

A week from today I’m giving a talk at .Net Unboxed on Storyteller 3, an open source tool largely built by myself and my colleagues for creating, running, and managing Executable Specifications against .Net projects based on what we feel are the best practices for automated testing based on over a decade of working with automated integration testing. As I’ll try to argue in my talk and subsequent blog posts, the most complete approach in the .Net ecosystem for reliably and economically writing large scale automated integration tests.

My company and a couple other early adopters have been using Storyteller for daily work since June and the feedback has been pleasantly positive so far. Now is as good of time as any to make a public beta release for the express purpose of getting more feedback on the tool so we can continue to improve the tool prior to an official 3.0 release in January.

If you’re interested in kicking the tires on Storyteller, the latest beta as of now is 3.0.0.279-alpha available on Nuget.org. For help getting started, see our tutorial and getting started pages.

Some highlights:

It’s improved a lot since then, but I gave a talk at work in March previewing Storyteller 3 that at least discusses the goals and philosophy behind the tool and Storyteller’s approach to acceptance tests and integration tests.

A Brief History

I had a great time at Codemash this year catching up with old friends. I was pleasantly surprised when I was there to be asked several times about the state of Storyteller, an OSS project others had originally built in 2008 as a replacement for FitNesse as our primary means of expressing and executing automated customer facing acceptance tests. Frankly, I always thought that Storyteller 1 and the incrementally better Storyteller 2 were failures in terms of usability and I was so burnt out on working with it that I had largely given up on it and ignored it for years.

Unfortunately, my shop has a large investment in Storyteller tests and our largest and most active project was suffering with heinously slow and unreliable Storyteller regression test suites that probably caused more harm than good with their support costs. After a big town hall meeting to decide whether to scrap and replace Storyteller with something else, we instead decided to try to improve Storyteller to avoid having to rewrite all of our tests. The result has been an effective rewrite of Storyteller with an all new client. While trying very hard to mostly preserve backward compatibility with the previous version in its public API’s, the .Net engine is also a near rewrite in order to squeeze out as much performance and responsiveness as we could.

Roadmap

The official 3.0 release is going to happen in early January to give us a chance to possibly get more early user feedback and maybe to get some more improvements in place. You can see the currently open issue list on GitHub. The biggest things outstanding on our roadmap are:

  • Modernize the client technology to React.js v14 and introduce Redux and possibly RxJS as a precursor to doing any big improvements to the user interface and trying to improve the performance of the user interface with big specification suites
  • A “step through” mode in the interactive specification running so users can step through a specification like you would in a debugger
  • The big one, allow users to author the actual specification language in the user interface editor with some mechanics to attach that language to actual test support code later
Advertisements

9 thoughts on “Storyteller 3: Executable Specifications and Living Documentation for .Net

  1. Pingback: Dew Drop – October 20, 2015 (#2115) | Morning Dew

  2. Pingback: How I’m Documenting OSS Projects | The Shade Tree Developer

  3. Pingback: Adaptech Solutions | Behaviour Driven Design From the Ground Up (Part 1)

  4. John Woodard

    So my main question about Storyteller is how we can use this with a CI tool like Team City. Our FIT tests (not FitNesse) are currently running in a custom console application that spits out Team City specific indications as to tests that pass and fail, with the resulting HTML outputs as artifacts so I can see what exactly happened to the failing tests. Can I do the same with Storyteller?

    Reply
    1. jeremydmiller Post author

      @John,

      Storyteller wouldn’t be very useful if it didn’t have CI support;) See the “st run” command described in http://storyteller.github.io/documentation/ci/. It’s just a command line call that will drop a single HTML file with all the results (filterable summary table + drill down into individual specs). We run that from our TeamCity builds and just keep the HTML file as an artifact. You can add a tab to the CI results for that file, or happily open it from the artifacts links.

      – Jeremy

      Reply
      1. John Woodard

        So do you get any indication that the tests are Pass/Fail without viewing the HTML document? For the FIT tests, I basically wrote my own FIT test runner using the FIT library as a console application that output the Test tags that TeamCity integrates into. That way I can see (and be informed by TeamCity) the failing tests as Build Failures (Exactly like what happens when NUnit tests fail) so I can see what tests are failing even while things are still running. I also can, at a glance, look at my entire bank of builds and see what is working and what is failing.

      2. jeremydmiller Post author

        The build would pass or fail based on the exit code of “st run”, and yes, there is a flag to spit out the TeamCity console output that it uses to report test progress and failures.

  5. Pingback: Moving Storyteller to the CoreCLR and going Cross Platform | The Shade Tree Developer

  6. Pingback: Storyteller 3.0 Official Release — and on to 4.0 | The Shade Tree Developer

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s