Tag Archives: Marten

Marten V4 Preview: Command Line Administration

TL;DR — It’s going to be much simpler in V4 to incorporate Marten’s command line administration tools into your .Net Core application.

In my last post I started to lay out some of the improvements in the forthcoming Marten V4 release with our first alpha Nuget release. In this post, I’m going to show the improvements to Marten’s command line package that can be used for some important database administration and schema migrations.

Unlike ORM tools like Entity Framework (it’s a huge pet peeve of mine when people describe Marten as an ORM), Marten by and large tries to allow you to be as productive as possible by keeping your focus on your application code instead of having to spend much energy and time on the details of your database schema. At development time you can just have Marten use its AutoCreate.All mode and it’ll quietly do anything it needs to do with your Postgresql database to make the document storage work at runtime.

For real production though, it’s likely that you’ll want to explicitly control when database schema changes happen. It’s also likely that you won’t want your application to have permissions to change the underlying database schema on the fly. To that end, Marten has quite a bit of functionality to export database schema updates for formal database migrations.

We’ve long supported an add on package called Marten.CommandLine that let’s you build your own command line tool to help manage these schema updates, but to date it’s required you to build a separate console application parallel to your application and has probably not been that useful to most folks.

In V4 though, we’re exploiting the Oakton.AspNetCore library that allows you to embed command line utilities directly into your .Net Core application. Let’s make that concrete with a small sample application in Marten’s GitHub repository.

Before I dive into that code, Marten v3.12 added a built in integration for Marten into the .Net Core generic HostBuilder that we’re going to depend on here. Using the HostBuilder for configuring and bootstrapping Marten into your application allows you to use the exact same Marten configuration and application configuration in the Marten command utilities without any additional work.

This sample application was built with the standard dotnet new webapi template. On top of that, I added a reference to the Marten.CommandLine library.

.Net Core applications tend to be configured and bootstrapped by a combination of a Program.Main() method and a StartUp class. First, here’s the Program.Main() method from the sample application:

public class Program
{
// It's actually important to return Task<int>
// so that the application commands can communicate
// success or failure
public static Task<int> Main(string[] args)
{
return CreateHostBuilder(args)

// This line replaces Build().Start()
// in most dotnet new templates
.RunOaktonCommands(args);
}

public static IHostBuilder CreateHostBuilder(string[] args) =>
Host.CreateDefaultBuilder(args)
.ConfigureWebHostDefaults(webBuilder =>
{
webBuilder.UseStartup<Startup>();
});
}

Note the signature of the Main() method and how it uses the RunOaktonCommands() method to intercept the command line arguments and execute named commands (with the default being to just run the application like normal).

Now, the Startup.ConfigureServices() method with Marten added in is this:

public void ConfigureServices(IServiceCollection services)
{
    // This is the absolute, simplest way to integrate Marten into your
    // .Net Core application with Marten's default configuration
    services.AddMarten(Configuration.GetConnectionString("Marten"));
}

Now, to the actual command line. As long as the Marten.CommandLine assembly is referenced by your application, you should see the additional Marten commands. From your project’s root directory, run dotnet run -- help and we see there’s some additional Marten-related options:

Oakton command line options with Marten.CommandLine in play

And that’s it. Now you can use dotnet run -- dump to export out all the SQL to recreate the Marten database schema, or maybe dotnet run -- patch upgrade_staging.sql --e Staging to create a SQL patch file that would make any necessary changes to upgrade your staging database to reflect the current Marten configuration (assuming that you’ve got an appsettings.Staging.json file with the right connection string pointing to your staging Postgresql server).

Check out the Marten.CommandLine documentation for more information on what it can do, but expect some V4 improvements to that as well.

Marten V4 Preview: Linq and Performance

Marten is an open source library for .Net that allows developers to treat the robust Postgresql database as a full featured and transactional document database (NoSQL) as well as supporting the event sourcing pattern of application persistence.

After a false start last summer, development on the long awaited and delayed Marten V4.0 release is heavily in flight and we’re making a lot of progress. The major focus of the remaining work is improving the event store functionality (that I’ll try to blog about later in the week if I can). We posted the first Marten V4 alpha on Friday for early adopters — or folks that need Linq provider fixes ASAP! — to pull down and start trying out. So far the limited feedback has been a nearly seamless upgrade.

You can track the work and direction of things through the GitHub issues that are already done and the ones that are still planned.

For today though, I’d like to focus on what’s been done so far in V4 in terms of making Marten simply better and faster at its existing feature set.

Being Faster by Doing Less

One of the challenging things about Marten’s feature set is the unique permutations of what exactly happens when you store, delete, or load document to and from the database. For example, some documents may or may not be:

On top of that, Marten supports a couple different flavors of document sessions:

  • Query-only sessions that are strictly read only querying
  • The normal session that supports an internal identity map functionality that caches previously loaded documents
  • Automatic dirty checking sessions that are the heaviest Marten sessions
  • “Lightweight” sessions that don’t use any kind of identity map caching or automatic dirty checking for faster performance and better memory usage — at the cost of a little more developer written code.

The point here is that there’s a lot of variability in what exactly happens in Marten when you save, load, or delete a document with Marten. In the current version, Marten uses a combination of runtime if/then logic, some “Nullo” classes, and a little bit of “Expression to Lambda” runtime compilation.

For V4, I completely re-wired the internals to use C# code generated and compiled at runtime using Roslyn’s runtime compilation capabilities. Marten is using the LamarCompiler and LamarCodeGeneration libraries as helpers. You can see these two libraries and this technique in action in a talk I gave at NDC London in 2019.

The end result of all this work is that we can generated the tightest possible C# handling code and the tightest possible SQL for the exact permutation of document storage characteristics and session type. Along the way, we’ve striven to reduce the number of dictionary lookups, runtime branching logic, empty Nullo objects, and generally the number of computer instructions that would have to be executed by the underlying processor just to save, load, or delete a document.

So far, so good. It’s hard to say exactly how much this is going to impact any given Marten-using application, but the existing test suite clearly runs faster now and I’m not seeing any noticeable issue with the “cold start” of the initial, one time code generation and compilation (that was a big issue in early Roslyn to the point where we ripped that out of pre 1.0 Marten, but seems to be solved now).

If anyone is curious, I’d be happy to write a blog post diving into the guts of how that works. And why the new .Net source generator feature wouldn’t work in this case if anyone wants to know about that too.

Linq Provider Almost-Rewrite

To be honest, I think Marten’s existing Linq provider (pre-V4) is pretty well stuck at the original proof of concept stage thrown together 4-5 years ago. The number of open issues where folks had hit limitations in the Linq provider support built up — especially with anything involving child collections on document types.

For V4, we’ve heavily restructured the Linq parsing and SQL generation code to address the previous shortcomings. There’s a little bit of improvement in the performance of Linq parsing and also a little bit of optimization of the SQL generated by avoiding unnecessary CASTs. Most of the improvement has been toward addressing previously unsupported scenarios. A potential improvement that we haven’t yet exploited much is to make the SQL generation and Linq parsing more able to support custom value types and F#-isms like discriminated unions through a new extensibility mechanism that teaches Marten about how these types are represented in the serialized JSON storage.

Querying Descendent Collections

Marten pre-V4 didn’t handle querying through child collections very well and that’s been a common source of user issues. With V4, we’re heavily using the Common Table Expression query support in Postgresql behind the scenes to make Linq queries like this one shown below possible:

var results = theSession.Query<Top>()
.Where(x => x.Middles.Any(b => b.Bottoms.Any()))
.ToList();

I think that at this point Marten can handle any combination of querying through child collections through any number of levels with all possible query operators (Any() / Count()) and any supported Where() fragment within the child collection.

Multi-Document Includes

Marten has long had some functionality for fetching related documents together in one database round trip for more efficient document reading. A long time limitation in Marten is that this Include() capability was only usable for logical “one to one” or “many to one” document relationships. In V4, you can now use Include() querying for “one to many” relationships as shown below:

[Fact]
public void include_many_to_list()
{
var user1 = new User { };
var user2 = new User { };
var user3 = new User { };
var user4 = new User { };
var user5 = new User { };
var user6 = new User { };
var user7 = new User { };

theStore.BulkInsert(new User[]{user1, user2, user3, user4, user5, user6, user7});

var group1 = new Group
{
Name = "Odds",
Users = new []{user1.Id, user3.Id, user5.Id, user7.Id}
};

var group2 = new Group {Name = "Evens", Users = new[] {user2.Id, user4.Id, user6.Id}};

using (var session = theStore.OpenSession())
{
session.Store(group1, group2);
session.SaveChanges();
}

using (var query = theStore.QuerySession())
{
var list = new List<User>();

query.Query<Group>()
.Include(x => x.Users, list)
.Where(x => x.Name == "Odds")
.ToList()
.Single()
.Name.ShouldBe("Odds");

list.Count.ShouldBe(4);
list.Any(x => x.Id == user1.Id).ShouldBeTrue();
list.Any(x => x.Id == user3.Id).ShouldBeTrue();
list.Any(x => x.Id == user5.Id).ShouldBeTrue();
list.Any(x => x.Id == user7.Id).ShouldBeTrue();
}
}

This was a longstanding request from users, and to be honest, we had to completely rewrite the Include() internals to add this support. Again, we used Common Table Expression SQL statements in combination with per session temporary tables to pull this off.

Compiled Queries Actually Work

I think the Compiled Query feature is unique in Marten. It’s probably easiest and best to think of it as a “stored procedure” for Linq queries in Marten. The value of a compiled query in Marten is:

  1. It potentially cleans up the application code that has to interact with Marten queries, especially for more complex queries
  2. It’s potentially some reuse for commonly executed queries
  3. Mostly though, it’s a significant performance improvement because it allows Marten to “remember” the Linq query plan.

While compiled queries have been supported since Marten 1.0, there’s been a lot of gap between what works in Marten’s Linq support and what functions correctly inside of compiled queries. With the advent of V4, the compiled query planning was rewritten with a new strategy that so far seems to support all of the Linq capabilities of Marten. We think this will make the compiled query feature much more useful going forward.

Here’s an example compiled query that was not possible before V4:

public class FunnyTargetQuery : ICompiledListQuery<Target>
{
public Expression<Func<IMartenQueryable<Target>, IEnumerable<Target>>> QueryIs()
{
return q => q
.Where(x => x.Flag && x.NumberArray.Contains(Number));
}

public int Number { get; set; }
}

And in usage:

var actuals = session.Query(new FunnyTargetQuery{Number = 5}).ToArray();

Multi-Level SelectMany because why not?

Marten has long supported the SelectMany() keyword in the Linq provider support, but in V4 it’s much more robust with the ability to chain SelectMany() clauses n-deep and do that in combination with any kind of Count() / Distinct() / Where() / OrderBy() Linq clauses. Here’s an example:

[Fact]
public void select_many_2_deep()
{
var group1 = new TargetGroup
{
Targets = Target.GenerateRandomData(25).ToArray()
};

var group2 = new TargetGroup
{
Targets = Target.GenerateRandomData(25).ToArray()
};

var group3 = new TargetGroup
{
Targets = Target.GenerateRandomData(25).ToArray()
};

var groups = new[] {group1, group2, group3};

using (var session = theStore.LightweightSession())
{
session.Store(groups);
session.SaveChanges();
}

using var query = theStore.QuerySession();

var loaded = query.Query<TargetGroup>()
.SelectMany(x => x.Targets)
.Where(x => x.Color == Colors.Blue)
.SelectMany(x => x.Children)
.OrderBy(x => x.Number)
.ToArray()
.Select(x => x.Id).ToArray();

var expected = groups
.SelectMany(x => x.Targets)
.Where(x => x.Color == Colors.Blue)
.SelectMany(x => x.Children)
.OrderBy(x => x.Number)
.ToArray()
.Select(x => x.Id).ToArray();

loaded.ShouldBe(expected);
}

Again, we pulled that off with Common Table Expression statements.

Calling Generic Methods from Non-Generic Code in .Net

Somewhat often (or at least it feels that way this week) I’ll run into the need to call a method with a generic type argument from code that isn’t generic. To make that concrete, here’s an example from Marten. The main IDocumentSession service has a method called Store() that directs Marten to persist one or more documents of the same type. That method has this signature:

void Store<T>(params T[] entities);

That method would typically be used like this:

using (var session = store.OpenSession())
{
    // The generic constraint for "Team" is inferred from the usage
    session.Store(new Team { Name = "Warriors" });
    session.Store(new Team { Name = "Spurs" });
    session.Store(new Team { Name = "Thunder" });

    session.SaveChanges();
}

Great, and easy enough (I hope), but Marten also has this method where folks can add a heterogeneous mix of any kind of document types all at once:

void StoreObjects(IEnumerable<object> documents);

Internally, that method groups the documents by type, then delegates to the property Store<T>() method for each document type — and that’s where this post comes into play.

(Re-)Introducing Baseline

Baseline is a library available on Nuget that provides oodles of little helper extension methods on common .Net types and very basic utilities that I use in almost all my projects, both OSS and at work. Baseline is an improved subset of what was long ago FubuCore (FubuCore was huge, and it also spawned Oakton), but somewhat adapted to .Net Core.

I wanted to call this library “spackle” because it fills in usability gaps in the .Net base class library, but Jason Bock beat me to it with his Spackle library of extension methods. Since I expected this library to be used as a foundational piece from within basically all the projects in the JasperFx suite, I chose the name “Baseline” which I thought conveniently enough described its purpose and also because there’s an important throughway near the titular Jasper called “Baseline”. I don’t know for sure that it’s the basis for the name, but the Battle of Carthage in the very early days of the US Civil War started where this road is today.

Crossing the Non-Generic to Generic Divide with Baseline

Back to the Marten StoreObjects(object[]) calling Store<T>(T[]) problem. Baseline has a helper extension method called CloseAndBuildAs<T>() method I frequently use to solve this problem. It’s unfortunately a little tedious, but first design a non-generic interface that will wrap the calls to Store<T>() like this:

internal interface IHandler
{
void Store(IDocumentSession session, IEnumerable<object> objects);
}

And a concrete, open generic type that implements IHandler:

internal class Handler<T>: IHandler
{
public void Store(IDocumentSession session, IEnumerable<object> objects)
{
// Delegate to the Store<T>() method
session.Store(objects.OfType<T>().ToArray());
}
}

Now, the StoreObjects() method looks like this:

public void StoreObjects(IEnumerable<object> documents)
{
assertNotDisposed();

var documentsGroupedByType = documents
.Where(x => x != null)
.GroupBy(x => x.GetType());

foreach (var group in documentsGroupedByType)
{
// Build the right handler for the group type
var handler = typeof(Handler<>).CloseAndBuildAs<IHandler>(group.Key);
handler.Store(this, group);
}
}

The CloseAndBuildAs<T>() method above does a couple things behind the scenes:

  1. It creates a closed type for the proper Handler<T> based on the type arguments passed into the method
  2. Uses Activator.CreateInstance() to build the concrete type
  3. Casts that object to the interface supplied as a generic argument to the CloseAndBuildAs<T>() method

The method shown above is here in GitHub. It’s not shown, but there are some extra overloads to also pass in constructor arguments to the concrete types being built.

Marten Quickstart with .Net Core HostBuilder

The Marten Community just released Marten 3.12 with a mix of new functionality and plenty of bug fixes this past week. In particular, we added some new extension methods directly into Marten for integration into .Net Core applications that are bootstrapped by the new generic host builder in .Net Core 3.*.

There’s a new runnable, sample project in GitHub called AspNetCoreWebAPIWithMarten that contains all the code from this blog post.

For a small sample ASP.Net Core web service project using Marten’s new integration, let’s start a new web service project with the dotnet new webapi template. Doing this gives you some familiar files that we’re going to edit in a bit:

  1. appsettings.json — standard configuration file for .Net Core
  2. Program.cs — has the main command line entry point for .Net Core applications. We aren’t actually going to touch this right now, but there will be some command line improvements to Marten v4.0 soon that will add some important development lifecycle utilities that will require a 2-3 line change to this file. Soon.
  3. Startup.cs — the convention based Startup class that holds most of the configuration and bootstrapping for a .Net Core application.

Marten does sit on top of Postgresql, so let’s add a docker-compose.yml file to the codebase for our local development database server like this one:

version: '3'
services:
 postgresql:
 image: "clkao/postgres-plv8:latest"
ports:
 - "5433:5432"

At the command line, run docker-compose up -d to start up your new Postgresql database in Docker.

Next, we’ll add a reference to the latest Marten Nuget to the main project. In the appsettings.json file, I’ll add the connection string to the Postgresql container we defined above:

"ConnectionStrings": {
  "Marten": "Host=localhost;Port=5433;Database=postgres;Username=postgres;password=postgres"
}

Finally, let’s go the Startup.ConfigureServices() method and add this code to register Marten services:

public class Startup
{
    // We need the application configuration to get
    // our connection string to Marten
    public Startup(IConfiguration configuration)
    {
        Configuration = configuration;
    }

    public IConfiguration Configuration { get; }

    public void ConfigureServices(IServiceCollection services)
    {
         // This line registers Marten services with all default
         // Marten settings
         var connectionString = Configuration.GetConnectionString("Marten");
         services.AddMarten(connectionString);

         // And some other stuff we don't care about in this post...

And that’s it, we’ve got Marten configured in its most basic “getting started” usage with these services in our application’s IoC container:

  1. IDocumentStore with a Singleton lifetime. The document store can be used to create sessions, query the configuration of Marten, generate schema migrations, and do bulk inserts.
  2. IDocumentSession with a Scoped lifetime for all read and write operations. By default, this is done with the IDocumentStore.OpenSession() method and the session created will have the identity map behavior
  3. IQuerySession with a Scoped lifetime for all read operations against the document store.

Now, let’s build out a very rudimentary issue tracking web service to capture and persist new issues as well as allow us to query for existing, open issues:

public class IssueController : ControllerBase
{
// This endpoint captures, creates, and persists
// new Issue entities with Marten
[HttpPost("/issue")]
public async Task<IssueCreated> PostIssue(
[FromBody] CreateIssue command,
[FromServices] IDocumentSession session)
{
var issue = new Issue
{
Title = command.Title,
Description = command.Description
};

// Register the new Issue entity
// with Marten
session.Store(issue);

// Commit all pending changes
// to the underlying database
await session.SaveChangesAsync();

return new IssueCreated
{
IssueId = issue.Id
};
}

[HttpGet("/issues/{status}/")]
public Task<IReadOnlyList<IssueTitle>> Issues(
[FromServices] IQuerySession session,
IssueStatus status)
{
// Query Marten's underlying database with Linq
// for all the issues with the given status
// and return an array of issue titles and ids
return session.Query<Issue>()
.Where(x => x.Status == status)
.Select(x => new IssueTitle {Title = x.Title, Id = x.Id})
.ToListAsync();
}

// Return only new issues
[HttpGet("/issues/new")]
public Task<IReadOnlyList<IssueTitle>> NewIssues(
[FromServices] IQuerySession session)
{
return Issues(session, IssueStatus.New);
}
}

And that is actually a completely working little application. In its default settings, Marten will create any necessary database tables on the fly, so we didn’t have to worry about much database setup other than having the new Postgresql database started in a Docker container. Likewise, we didn’t have to subclass any kind of Marten service like you would with Entity Framework Core and we most certainly didn’t have to create any kind of Object/Relational Mapping just to get going. Additionally, we didn’t have to care too awfully much about how to get the right Marten services integrated into our application’s IoC container with the right scoping because that’s all handled for us with the UseMarten() extension method that comes out of the box with Marten 3.12 and greater.

In a follow up post later this week, I’ll show you how to customize the Marten service registrations for per-HTTP request multi-tenancy and traceable, contextual logging of Marten-related SQL execution.

Marten v4.0 Planning Document (Part 1)

As I wrote about a couple weeks ago in a post called Kicking off Marten V4 Development, the Marten team is actively working on the long delayed v4.0 release with planned improvements for performance, the Linq support, and a massive planned jump in capability for the event store functionality. This post is a the result of a long comment thread and many discussions between the Marten community.

We don’t yet have a great consensus about the exact direction that the event store improvements are going to go, so I’m strictly focusing on improvements to the Marten internals and the Document Database support. I’ll follow up with a part 2 on the event store as that starts to gel.

If you’re interested in Marten, here’s some links:

 

Pulling it Off

We’ve got the typical problems of needing to address incoming pull requests and bug issues in master while probably needing to have a long lived branch for v4.

As an initial plan, let’s:

  1. Start with the unit testing improvements as a way to speed up the build before we dive into much more of this? This is in progress with about a 25% reduction in test throughput time so far in this pull request
  2. Do a bug sweep v3.12 release to address as many of the tactical problems as we can before branching to v4
  3. Possibly do a series of v4, then v5 releases to do this in smaller chunks? We’ve mostly said do the event store as v4, then Linq improvements as v5 — Nope, full speed ahead with a large v4 release in order to do as many breaking changes as possible in one release
  4. Extract the generic database manipulation code to its own library to clean up Marten, and speed up our builds to make the development work be more efficient.
  5. Do the Event Store v4 work in a separate project built as an add on from the very beginning, but leave the existing event store in place. That would enable us to do a lot of work and mostly be able to keep that work in master so we don’t have long-lived branch problems. Break open the event store improvement work because that’s where most of the interest is for this release.

Miscellaneous Ideas

  • Look at some kind of object pooling for the DocumentSession / QuerySession objects?
  • Ditch the document by document type schema configuration where Document A can be in one schema, and Document “B” is in another schema. Do that, and I think we open the door for multi-tenancy by schema.
  • Eliminate ManagedConnection altogether. I think it results in unnecessary object allocations and it’s causing more harm that help as it’s been extended over time. After studying that more today, it’s just too damn embedded. At least try to kill off the Execute methods that take in a Lambda. See this GitHub issue.
  • Can we consider ditching < .Net Core or .Net v5 for v4? The probable answer is “no,” so let’s just take this off the table.
  • Do a hunt for classes in Marten marked public that should be internal. Here’s the GitHub issue.
  • Make the exceptions a bit more consistent

Dynamic Code Generation

If you look at the pull request for Document Metadata and the code in Marten.Schema.Arguments you can see that our dynamic Expression to Lambda compilation code is getting extremely messy, hard to reason with, and difficult to extend.

Idea: Introduce a dependency on LamarCodeGeneration and LamarCompiler. LamarCodeGeneration has a strong model for dynamically generating C# code at runtime. LamarCompiler adds runtime Roslyn support to compile assemblies on the fly and utilities to attach/create these classes. We could stick with Expression to Lambda compilation, but that can’t really handle any kind of asynchronous code without some severe pain and it’s far more difficult to reason about (Jeremy’s note: I’m uniquely qualified to make this statement unfortunately).

What gets dynamically generated today:

  • Bulk importer handling for a single entity
  • Loading entities and tracking entities in the identity map or version tracking

What could be generated in the future:

  • Document metadata properties — but sad trombone, that might have to stay with Expressions if the setters are internal/private :/
  • Much more of the ISelector implementations, especially since there’s going to be more variability when we do the document metadata
  • Finer-grained manipulation of the IIdentityMap

Jeremy’s note: After doing some detailed analysis through the codebase and the spots that would be impacted by the change to dynamic code generation, I’m convinced that this will lead to significant performance improvements by eliminating many existing runtime conditional checks and casts

Track this work here.

Unit Testing Approach

This is in progress, and going well.

If we introduce the runtime code generation back into Marten, that’s unfortunately a non-trivial “cold start” testing issue. To soften that, I suggest we get a lot more aggressive with reusable xUnit.Net class fixtures between tests to reuse generated code between tests, cut way down on the sheer number of database calls by not having to frequently check the schema configuration, and other DocumentStore overhead.

A couple other points about this:

  • We need to create more unique document types so we’re not having to use different configurations for the same document type. This would enable more reuse inside the testing runtime
  • Be aggressive with separate schemas for different configurations
  • We could possibly turn on xUnit.net parallel test running to speed up the test cycles

Document Metadata

  • From the feedback on GitHub, it sounds like the desire to extend the existing metadata to tracking data like correlation identifiers, transaction ids, user ids, etc. To make this data easy to query on, I would prefer that this data be separate columns in the underlying storage
  • Use the configuration and tests from pull request for Document Metadata, but use the Lamar-backed dynamic code generation from the previous section to pull this off.
  • I strongly suggest using a new dynamic codegen model for the ISelector objects that would be responsible for putting Marten’s own document metadata like IsDeleted or TenantId or Version onto the resolved objects (but that falls apart if we have to use private setters)
  • I think we could expand the document metadata to allow for user defined properties like “user id” or “transaction id” much the same way we’ll do for the EventStore metadata. We’d need to think about how we extend the document tables and how metadata is attached to a document session

My thought is to designate one (or maybe a few?) .Net type as the “metadata type” for your application like maybe this one:

    public class MyMetadata
    {
        public Guid CorrelationId { get; set; }
        public string UserId { get; set; }
    }

Maybe that gets added to the StoreOptions something like:

var store = DocumentStore.For(x => {
    // other stuff

    // This would direct Marten to add extra columns to
    // the documents and events for the metadata properties
    // on the MyMetadata type.

    // This would probably be a fluent interface to optionally fine tune
    // the storage and applicability -- i.e., to all documents, to events, etc.
    x.UseMetadataType<MyMetadata>();
});

Then at runtime, you’d do something like:

session.UseMetadata<MyMetadta>(metadata);

Either through docs or through the new, official .Net Core integration, we have patterns to have that automatically set upon new DocumentSession objects being created from the IoC to make the tracking be seemless.

Extract Generic Database Helpers to its own Library

  • Pull everything to do with Schema object generation, difference detection, and DDL generation to a separate library (IFeatureSchemaISchemaObject, etc.). Mostly to clean out the main library, but also because this code could easily be reused outside of Marten. Separating it out might make it easier to test and extend that functionality, which is something that occasionally gets requested. There’s also the possibility of further breaking that into abstractions and implementations for the long run of getting us ready for Sql Server or other database engine support. The tests for this functionality are slow, and rarely change. It would be advantageous to get this out of the main Marten library and testing project.
  • Pull the ADO.Net helper code like CommandBuilder and the extension methods into a small helper library somewhere else (I’m nominating the Baseline repository). This code is copied around to other projects as it is, and it’s another way of getting stuff out of the main library and the test suite.

Track this work in this GitHub issue.

F# Improvements

We’ll have a virtual F# subcommittee to be watching this work for F#-friendliness:

HostBuilder Integration

We’ll bring Joona-Pekka Kokko’s ASP.Net Core integration library into the main repository and make that the officially blessed and documented recipe for integrating Marten into .Net Core applications based on the HostBuilder in .Net Core. I suppose we could also multi-target IWebHostBuilder for ASP.Net Core 2.*.

That HostBuilder integration could be extended to:

  • Optionally set up the Async Daemon in an IHostedService — more on this in the Event Store section
  • Optionally register some kind of IDocumentSessionBuilder that could be used to customize session construction?
  • Have some way to have container resolved IDocumentSessionListener objects attached to IDocumentSession. This is to have an easy recipe for folks who want events broadcast through messaging infrastructure in CQRS architectures

See the GitHub issue for this.

Command Line Support

The Marten.CommandLine package already uses Oakton for command line parsing. For easier integration in .Net Core applications, we could shift that to using the Oakton.AspNetCore package so the command line support can be added to any ASP.net Core 2.* or .Net Core 3.* project by installing the Nuget. This might simplify the usage because you’d no longer need a separate project for the command line support.

There are some long standing stories about extending the command line support for the event store projection rebuilding. I think that would be more effective if it switches over to Oakton.AspNetCore.

See the GitHub issue

Linq

This is also covered by the Linq Overhaul issue.

  • Bring back the work in the linq branch for the revamped IField model within the Linq provider. This would be advantageous for performance, cleans up some conditional code in the Linq internals, could make the Linq support be aware of Json serialization customizations like [JsonProperty], and probably helps us deal more with F# types like discriminated unions.
  • Completely rewrite the Include() functionality. Use Postgresql Common Table Expression and UNION queries to fetch both the parent and any related documents in one query without needing to do any kind of JOIN s that complicate the selectors. There’d be a column for document type the code could use to switch. The dynamic code generation would help here. This could finally knock out the long wished for Include() on child collections feature. This work would nuke the InnerJoin stuff in the ISelector implementations, and that would hugely simplify a lot of code.
  • Finer grained code generation would let us optimize the interactions with IdentityMap. For purely query sessions, you could completely skip any kind of interaction with IdentityMap instead of wasting cycles on nullo objects. You could pull out a specific IdentityMap<TEntity, TKey> out of the larger identity map just before calling selectors to avoid some repetitive “find the right inner dictionary” on each document resolved.
  • Maybe introduce a new concept of ILinqDialect where the Expression parsing would just detect what logical thing it finds (like !BoolProperty), and turns around and calls this ILinqDialect to get at a WhereFragment or whatever. This way we could ready ourselves to support an alternative json/sql dialect around JSONPath for Postgresql v12+ and later for Sql Server vNext. I think this would fit into the theme of making the Linq support more modular. It should make the Linq support easier to unit test as we go. Before we do anything with this, let’s take a deep look into the EF Core internals and see how they handle this issue
  • Consider replacing the SelectMany() implementation with Common Table Expression sql statements. That might do a lot to simplify the internal mechanics. Could definitely get us to an n-deep model.
  • Do the Json streaming story because it should be compelling, especially as part of the readside of a CQRS architecture using Marten’s event store functionality.
  • Possibly use a PLV8-based JsonPath polyfill so we could use sql/json immediately in the Linq support. More research necessary.

Partial Updates

Use native postgresql partial JSON updates wherever possible. Let’s do a perf test on that first though.

Using Postgresql Advisory Locks for Leader Election

If you’re running an application with a substantial workload, or just want some kind of high availability, you’re probably running that application across multiple servers (heretofore called “nodes” because who knows where they’re physically running these days). That’s great and all, but it’s not too uncommon that you’ll need to make some kind of process run on only one of those nodes at any one time.

As an example, the Marten event store functionality as a feature to support asynchronous projection builders called the “async daemon” (because I thought that sounded cool at the time). The async daemon is very stateful, and can only function while running on one node at a time — but it doesn’t have any existing infrastructure to help you manage that. What we know we need to do for the upcoming Marten v4.0 release is to provide “leader election” to make sure the async daemon is actively building projections on only one node and can be activated or fail over to another node as needed to guarantee that exactly one node is active at all times.

From Wikipedia, Leader Election “is the process of designating a single process as the organizer of some task distributed among several computers.” There’s plenty of existing art to do this, but it’s not for the feint of heart. In the past, I tried to do this with FubuMVC using a custom implementation of the Bully Algorithm. Microsoft’s microservices pattern guidance has some .Net centric approaches to leader election. Microsoft’s new Dapr tool is supposed to support leader election some day.

From my previous experience, building out and especially testing custom election infrastructure was very difficult. As a far easier approach, I’ve used Advisory Locks in Postgresql in Jasper (I’m also using the Sql Server equivalents as well) as what I think of as a “poor man’s leader election.”

An advisory lock in Postgresql is an arbitrary, application-managed lock on a named resource. Postgresql simply tracks these locks as a distributed lock such that only one active client can hold the lock at any one time. These locks can be held either at:

  1. The connection level, such that the lock, once obtained, is held as long as the database connection is open.
  2. The transaction level, such that a lock obtained within the course of one Postgresql transaction is held until the transaction is committed, rolled back, or the connection is lost.

As an example, Jasper‘s “Durability Agent” is a constantly running process in Jasper applications that tries to read and process any persisted messages persisted in a Postgresql or Sql Server database. Since you certainly don’t want a unique message to be processed by more than one node, the durability uses advisory locks to try to temporarily take sole ownership of replaying persisted messages with a workflow similar to this sequence diagram:

Transaction Scoped Advisory Lock Usage

That’s working well so far for Jasper, but in Marten v4.0, we want to use the connection scoped advisory lock for leader election of a long running process for the async daemon.

Sample Usage for Leader Election

Before you look at any of these code samples, just know that this is over-simplified to show the concept, isn’t in production, and would require a copious amount of error handling and logging to be production worthy.

For Marten v4.0, we’ll use the per-connection usage to ensure that the new version of the async daemon will only be running on one node (or at least the actual “leader” process that distributes and assigns work across other nodes if we do it well). The async daemon process itself is probably going to be a .Net Core IHostedService that runs in the background.

As just a demonstrator, I’ve pushed up a little project called AdvisoryLockSpike to GitHub just to show the conceptual usage. First let’s say that the actual worker bee process of the async daemon implements this interface:

public enum ProcessState
{
    Active,
    Inactive,
    Broken
}

public interface IActiveProcess : IDisposable
{
    Task<ProcessState> State();
    
    
    // The way I've done this before, the
    // running code does all its work using
    // the currently open connection or at
    // least checks the connection to "know"
    // that it still has the leadership role
    Task Start(NpgsqlConnection conn);
}

Next, we need something around that to actually deal with the mechanics of trying to obtain the global lock and starting or stopping the active process. Since that’s a background process within an application, I’m going to use the built in BackgroundService in .Net Core with this little class:

public class LeaderHostedService<T> : BackgroundService
    where T : IActiveProcess
{
    private readonly LeaderSettings<T> _settings;
    private readonly T _process;
    private NpgsqlConnection _connection;

    public LeaderHostedService(LeaderSettings<T> settings, T process)
    {
        _settings = settings;
        _process = process;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        // Don't try to start right off the bat
        await Task.Delay(_settings.FirstPollingTime, stoppingToken);
            
        _connection = new NpgsqlConnection(_settings.ConnectionString);
        await _connection.OpenAsync(stoppingToken);
        
        while (!stoppingToken.IsCancellationRequested)
        {
            var state = await _process.State();
            if (state != ProcessState.Active)
            {
                // If you can take the global lock, start
                // the process
                if (await _connection.TryGetGlobalLock(_settings.LockId, cancellation: stoppingToken))
                {
                    await _process.Start(_connection);
                }
            }

            // Start polling again
            await Task.Delay(_settings.OwnershipPollingTime, stoppingToken);
        }

        if (_connection.State != ConnectionState.Closed)
        {
            await _connection.DisposeAsync();
        }

    }
}

To fill in the blanks, the TryGetGlobalLock() method is an extension method helper to call the underlying pg_try_advisory_lock function in Postgresql to try to obtain a global advisory lock for the configured lock id. That extension method is shown below:

// Try to get a global lock with connection scoping
public static async Task<bool> TryGetGlobalLock(this DbConnection conn, int lockId, CancellationToken cancellation = default(CancellationToken))
{
    var c = await conn.CreateCommand("SELECT pg_try_advisory_lock(:id);")
        .With("id", lockId)
        .ExecuteScalarAsync(cancellation);

    return (bool) c;
}

Raw ADO.Net is so verbose and unusable out of the box that I’ve built up a set of extension methods to streamline its usage that you might observe above if you notice that that isn’t quite out of the box ADO.Net.

I’m generally a fan of strong typed configuration, and .Net Core makes that easy now, so I’ll use this class to represent the configuration:

public class LeaderSettings<T> where T : IActiveProcess
{
    public TimeSpan OwnershipPollingTime { get; set; } = 5.Seconds();
    
    // It's a random number here so that if you spin
    // up multiple nodes at the same time, they won't
    // all collide trying to grab ownership at the exact
    // same time
    public TimeSpan FirstPollingTime { get; set; } 
        = new Random().Next(100, 3000).Milliseconds();
    
    // This would be something meaningful
    public int LockId { get; set; }
    
    public string ConnectionString { get; set; }
}

In this approach, the background services will be constantly polling to try to take over as the async daemon if the async daemon is not active somewhere else. If the current async daemon node fails, the connection will drop and the global advisory lock is released and ready for another node to take over. We’ll see how this goes, but the early feedback from my own usage on Jasper and other Marten contributors other projects is positive. With this approach, we hope to enable teams to use the async daemon on multi-node deployments of their application with just Marten out of the box and without having to have any kind of sophisticated infrastructure for leader election.

Kicking off Marten v4 Development

banner

If you’re not familiar with Marten, it’s a library that allows .Net developers to treat Postgresql as a Document Database and also as an Event Store. For a quick background on Marten, here’s my presentation on Marten from Dotnetconf 2018.

The Marten community is gearing up to work on the long awaited v4.0 release, which I think is going to be the most ambitious release of Marten since the original v1.0 release in September of 2016. Marten is blessed with a strong user community, and we’ve got a big backlog of issues, feature requests, and suggestions for improvements that have been building up for quite some time.

First, some links for anyone who wants to be a part of that conversation:

Now would be a perfect time to add any kind of feedback or requests for Marten. We do have an unofficial “F# advisory board” this time around, but the more the merrier on that side. There’ll be plenty of opportunities for folks to contribute, whether that’s in coding, experimenting with early alphas, or just being heard in the discussions online.

Now, on to the release plans:

Event Sourcing Improvements

The biggest area of focus in Marten v4.0 is likely going to be the event store functionality. At the risk of being (rightly) mocked, I’d sum up our goals here as “Make the Marten Event Store be Web Scale!”

There’s a lot of content on the aforementioned Marten v4.0 discussion, and also on an older GitHub discussion on Event Store improvements for v4.0. The highlights (I think) are:

  • Event metadata (a long running request)
  • Partitioning Support
  • Improved projection support, including the option to project event data to flat database tables
  • A lot of improvements (near rewrite) to the existing support for asynchronous projections including multi-tenancy support, running across multiple application nodes, performance optimizations, and projection rebuild improvements
  • New support for asynchronous projection builders using messaging as an alternative to the polling async daemon. The first cut of this is very likely to be based around Jasper (I’m using this as a way to push Jasper forward too of course) and either Rabbit MQ or Azure Service Bus. If things go well, I’d hope that that expands to at least Jasper + Kafka and maybe parallel support for NServiceBus and MassTransit.
  • Snapshotting
  • The ability to rebuild projections with zero downtime

Linq Support Improvements

I can’t overstate how complex building and maintaining a comprehensive Linq provider can be just out of the sheer number of usage permutations.

Marten v4.0 is going to include a pretty large overhaul of the Linq support. Along the way, we’re hopefully going to:

  • Rewrite the functionality to include related documents altogether and finally support Include() on collections that’s been a big user request for 3+ years.
  • Improve the efficiency of the generated SQL
  • More variable data type behavior within the Linq support. You can read more about the mechanics on this issue. This also includes being quite a bit smarter about the Json serialization and how that varies for different .Net data types. As an example, I think these changes will allow Marten’s Linq support to finally deal with things like F# Discriminator Union types.
  • Improve the internals by making it more modular and a little more performant in how it handles marshaling data from the database to C# objects.
  • Improve Marten’s ability to query within child collections
  • More flexibility in event stream identifier strateggies

 

Other Improvements

  • Document Metadata improvements, both exposing Marten’s metadata and user customizable metadata for things like user id or correlation identifiers
  • An option to stream JSON directly to .Net Stream objects like HTTP responses without going through serialization first or even a JSON string.
  • Optimizing the Partial update support to use Postgresql native operators when possible
  • Formal support for .Net Core integration through the generic HostBuilder mechanism

 

Planning

Right now it’s all about the discussions and approach, but that’s going pretty well right now. The first thing I’m personally going to jump into is a round of automated test performance just to make everything go faster later.

The hope is that a subsequent V5 release might finally take Marten into supporting other database engines besides Postgresql, with Sql Server being the very obvious next step. We might take some preliminary steps with V4 to make the multi-database engine support be more feasible later with fewer breaking API changes, but don’t hold me to that.

I don’t have any idea about timing yet, sorry.

It’s an OSS Nuget Release Party! (Jasper v1.0, Lamar, Alba, Oakton)

My bandwidth for OSS work has been about zero for the past couple months. With the COVID-19 measures drastically reducing my commute and driving kids to school time, getting to actually use some of my projects at work, and a threat from an early adopter to give up on Jasper if I didn’t get something out soon, I suddenly went active again and got quite a bit of backlog things, pull requests, and bug fixes out.

My main focus in terms of OSS development for the past 3-4 years has been a big project called “Jasper” that was originally going to be a modernized .Net Core successor to FubuMVC. Just by way of explanation, Jasper, MO is my ancestral hometown and all the other projects I’m mentioning here are named after either other small towns or landmarks around the titular “Jasper.”

Alba

Alba is a library for HTTP contract testing with ASP.Net Core endpoints. It does quite a bit to reduce the repetitive code from using TestServer by itself in tests and makes your tests be much more declarative and intention revealing. Alba v3.1.1 was released a couple weeks ago to address a problem exposing the application’s IServiceProvider from the Alba SystemUnderTest. Fortunately for once, I caught this one myself while dogfooding Alba on a project at work.

Alba originated with the code we used to test FubuMVC HTTP behavior back in the day, but was modernized to work with ASP.Net Core instead of OWIN, and later retrofitted to just use TestServer under the covers.

Baseline

Baseline is a grab bag of utility code and extension methods on common .Net types that you can’t believe is missing from the BCL, most of which is originally descended from FubuCore.

As an ancillary project, I ripped out the type scanning and assembly finding code from Lamar into a separate BaselineTypeDiscovery Nuget that’s used by most of the other libraries in this post. There was a pretty significant pull request in the latest BaselineTypeDiscovery v1.1.0 release that should improve the application start up time for folks that use Lamar to discover assemblies in their application.

Oakton

Oakton is a command parsing library for .Net that was lifted originally from FubuCore that is used by Jasper. Oakton v2.0.4 and Oakton.AspNetCore v2.1.3 just upgrade the assembly discovery features to use the newer BaselineTypeDiscovery release above.

Lamar

Lamar is a modern, fast, ASP.Net Core compliant IoC container and the successor to StructureMap. I let a pretty good backlog of issues and pull requests amass, so I took some time yesterday to burn that down and the result is Lamar v4.2. This release upgrades the type scanning, fixes some bugs, and added quite a few fixes to the documentation website.

Jasper

Jasper at this point is a command executor ala MediatR (but much more ambitious) and a lightweight messaging framework — but the external messaging will mature much more in subsequent releases.

This feels remarkably anti-climactic seeing as how it’s been my main focus for years, but I pushed Jasper v1.0 today, specifically for some early adopters. The documentation is updated here. There’s also an up to date repository of samples that should grow. I’ll make a much bigger deal out of Jasper when I make the v1.1 or v2.0 release sometime after the Coronavirus has receded and my bandwidth slash ambition level is higher. For right now I’m just wanting to get some feedback from early users and let them beat it up.

Marten

There’s nothing new to say from me about Marten here except that my focus on Jasper has kept me from contributing too much to Marten. With Jasper v1.0 out, I’ll shortly (and finally) turn my attention to helping with the long planned Marten v4 release. For a little bit of synergy, part of my plans there is to use Jasper for some of the advanced Marten event store functionality we’re planning.

 

 

 

 

 

 

My (Big) OSS Plans for 2020

It’s now a yearly thing for me to blog about my aspirations and plans for various OSS projects at the beginning of the year. I was mostly on the nose in 2018, and way, way off in 2019. I’m hoping and planning for a much bigger year in 2020 as I’ve gotten much more enthusiastic and energetic about some ongoing efforts recently.

Most of my time and ambition next year is going toward Jasper, Marten, and Storyteller:

Jasper

Jasper is a toolkit for common messaging scenarios between .Net applications with a robust in process command runner that can be used either with or without the messaging. Jasper wholeheartedly embraces the .Net Core 3.0 ecosystem rather than trying to be its own standalone framework.

Jasper has been gestating for years, I almost gave up on it completely early last year, and purposely set it aside until after .Net Core 3.0 was released. However, I came back to it a few months ago with fresh eyes and full of new ideas from doing a lot of messaging scenario work for Calavista clients. I’m very optimistic right now about Jasper from a purely technical perspective. I’m furiously updating documentation, building sample projects, and dealing with last minute API improvements in an effort to kick out the big v1.0 release sometime in January of 2020.

Marten

Marten is a library that allows .Net developers to treat the outstanding Postgresql database as both a document database and an event store. Marten is mostly supported by a core team, but I’m hoping to be much more involved again this year. The big thing is a proposed v4 release that looks like it’s mostly going to be focused on the event sourcing functionality. There’s an initial GitHub issue for the overall work here, and I want to write a bigger post soon on some ideas about the approach. There’s going to be some new functionality, but the general theme is to make the event sourcing be much more scalable. Marten is a team effort now, and there’s plenty of room for more contributors or collaborators.

For some synergy, I’m also planning on building out some reference applications that use Jasper to integrate Marten with cloud based queueing and load distribution for the asynchronous projection support. I’m excited about this work just to level up on my cloud computing skills.

Storyteller

Storyteller is a tool I’ve worked on and used for what I prefer to call executable specification in .Net (but other folks would call Behavior Driven Development, which I don’t like only because BDD is overloaded). I did quite a bit of preliminary work last year on a proposed Storyteller v6 that would bring it more squarely into the .Net Core world, I wrote a post last year called Spitballing the Future of Storyteller that laid out all my thoughts. I liked where it was heading, but I got distracted by other things.

For more synergy, Storyteller v6 will use Jasper a little bit for its communication between the user interface and the specification process. It also dovetails nicely with my need to update my Javascript UI skillset.

 

Smaller Projects

Lamar — the modern successor to StructureMap and ASP.Net Core compliant IoC container. I will be getting a medium sized maintenance release out very soon as I’ve let the issue list back up. I’m only focused on dealing with problems and questions as they come in.

EDIT 1/6/2020 –> There’s a Lamar 4.1 release out!

Alba — a library that wraps TestServer to make integration testing against ASP.Net Core HTTP endpoints easier. The big work late last year was making it support ASP.Net Core v3. I don’t have any plans to do much with it this year, but that could change quickly if I get to use it on a client project this year.

Oakton — yet another command line parser. It’s used by Jasper, Storyteller, and the Marten command line package. I feel like it’s basically done and said so last year, but I added some specific features for ASP.Net Core applications and might add more along those lines this year.

StructureMap — Same as last year. I answer questions here and there, but otherwise it’s finished/abandoned

FubuMVC — A fading memory, and I’m pleasantly surprised when someone mentions it to me about once or twice a year

 

My integration testing challenges this week

EDIT 3/12: All these tests are fixed, and it wasn’t as bad as I’d thought it was going to be, but the extra logging and faster change code/run test under debugger cycle definitely helped.

This was meant to be a short post, or at least an easy to write post on my part, but it spilled out from integration testing and into some Storyteller mechanics, semi-advanced xUnit.Net usage, ASP.Net Core logging integration, and even a tepid defense of compositional architectures wrapped around an IoC container.

I’ve been working feverishly the past couple of months to push Jasper to a 1.0 release. As (knock on wood) the last big epic, I’ve been working on a large overhaul and redesign of the message persistence behind Jasper’s durable messaging. The existing code was fairly well covered by integration tests, so I felt confident that I could make the large scale changes and use the big bang integration tests to ensure the intended functionality still worked as designed.

I assume that y’all can guess how this has turned out. After a week of code changes and fixing any and all unit test and intermediate integration test failures, I got to the point where I was ready to run the big bang integration tests and, get this, they didn’t pass on the first attempt! I know, shocking right? Moreover, the tests involve all kinds of background processing and even multiple logical applications (think ASP.Net Core IWebHost objects) being started up and shut down during the tests, so it wasn’t exactly easy to spot the source of my test failures.

I thought it’d be worth talking about how I first stopped and invested in improving my test harness to make it faster to launch the test harness and to capture a lot more information about what’s going on in all the multithreading/asynchronous madness going on but…

First, an aside on Debugger Hell

You probably want to have some kind of debugger in your IDE of choice. You also want to be reasonably good using that tool, know how to use its features, and conversant in its supported keyboard shortcuts. You also want to try really hard to avoid needing to use your debugger too often because that’s just not an efficient way to get things done. Many folks in the early days of Agile development, including me, described debugger usage as a code or testing “smell.” And while “smell” is mostly used today as a pejorative to put down something that you just don’t like, it was originally just meant as a warning you should pay more attention to in your code or approach.

In the case of debugger usage, it might be telling you that your testing approach needs to be more fine grained or that you are missing crucial test coverage at lower levels. In my case, I’ll be looking for places where I’m missing smaller tests on elements of the bigger system and fill in those gaps before getting too worked up trying to solve the big integration test failures.

Storyteller Test Harness

For these big integration tests, I’m using Storyteller as my testing tool (think Cucumber,  but much more optimized for integration testing as opposed to being cute). With Storyteller, I’ve created a specification language that lets me script out message failover scenarios like the one shown below (which is currently failing as I write this):

JasperFailoverSpec

In the specification above, I’m starting and stopping Jasper applications to prove out Jasper’s ability to fail over and recover pending work from one running node to another using its Marten message persistence option. At runtime, Jasper has a background “durability agent” constantly running that is polling the database to determine if there is any unclaimed work to do or known application nodes are down (using advisory locks through Postgresql if anybody would ever be interested in a blog post about just that). Hopefully that’s enough description just to know that this isn’t particularly an easy scenario to test or code.

In my initial attempts to diagnose the failing Storyteller tests I bumped into a couple problems and sources of friction:

  1. It was slow and awkward mechanically to get the tests running under the debugger (that’s been a long standing problem with Storyteller and I’m finally happy with the approach shown later in this post)
  2. I could tell quickly that exceptions were being thrown and logged in the background processing, but I wasn’t capturing that log output in any kind of usable way

Since I’m pretty sure that the tests weren’t going to get resolved quickly and that I’d probably want to write even more of these damn tests, I believed that I first needed to invest in better visibility into what was happening inside the code and a much quicker way to cycle into the debugger. To that end, I took a little detour and worked on some Storyteller improvements that I’ve been meaning to do for quite a while.

Incorporating Logging into Storyteller Results

Jasper uses the ASP.Net Core logging abstractions for its own internal logging, but I didn’t have anything configured except for Debug and Console tracing to capture the logs being generated at runtime. Even with the console output, what I really wanted was all the log information correlated with both the individual test execution and which receiver or sender application the logging was from.

Fortunately, Storyteller has an extensibility model to capture custom logging and instrumentation directly into its test results. It turned out to be very simple to whip together an adapter for ASP.Net Core logging that captured the logging information in a way that can be exposed by Storyteller.

You can see the results in the image below. The table below is just showing all the logging messages received by ILogger within the “Receiver1” application during one test execution. The yellow row is an exception that was logged during the execution that I might not have been able to sense otherwise.

AspNetLoggingInStoryteller

For the implementation, ASP.Net Core exposes the ILoggerProvider service such that you can happily plug in as many logging strategies as you want to an application in a combinatorial way. On the Storyteller side of things, you have the Report interface that let’s you plug in custom logging that can expose HTML output into Storyteller’s results.

Implementing that crudely came out as a single class that implements both adapter interface (here’s a gist of the whole thing):

public class StorytellerAspNetCoreLogger : Report, ILoggerProvider

The actual logging just tracks all the calls to ILogger.Log() as little model objects in memory:

public void Log<TState>(LogLevel logLevel, EventId eventId, TState state, Exception exception, Func<TState, Exception, string> formatter)
{
    var logRecord = new LogRecord
    {
        Category = _categoryName,
        Level = logLevel.ToString(),
        Message = formatter(state, exception),
        ExceptionText = exception?.ToString()
    };

    // Just keep all the log records in an in memory list
    _parent.Records.Add(logRecord);
}

Fortunately enough, in the Storyteller Fixture code for the test harness I bootstrap the receiver and sender applications per test execution, so it’s really easy to just add the new StorytellerAspNetCoreLogger to both the Jasper applications and the Storyteller test engine:

var registry = new ReceiverApp();
registry.Services.AddSingleton<IMessageLogger>(_messageLogger);

var logger = new StorytellerAspNetCoreLogger(key);

// Tell Storyteller about the new logger so that it'll be
// rendered as part of Storyteller's results
Context.Reporting.Log(logger);

// This is bootstrapping a Jasper application through the 
// normal ASP.Net Core IWebHostBuilder
return JasperHost
    .CreateDefaultBuilder()
    .ConfigureLogging(x =>
    {
        x.SetMinimumLevel(LogLevel.Debug);
        x.AddDebug();
        x.AddConsole();
        
        // Add the logger to the new Jasper app
        // being built up
        x.AddProvider(logger);
    })

    .UseJasper(registry)
    .StartJasper();

And voila, the logging information is now part of the test results in a useful way so I can see a lot more information about what’s happening during the test execution.

It sucks that my code is throwing exceptions instead of just working, but at least I can see what the hell is going wrong now.

Get the debugger going quickly

To be honest, the friction of getting Storyteller tests running under a debugger has always been a drawback to Storyteller — especially compared to how fast that workflow is with tools like xUnit.Net that integrate seamlessly into your IDE. You’ve always been able to just attach your debugger to the running Storyteller process, but I’ve always found that to be clumsy and slow — especially when you’re trying to quickly cycle between attempted fixes and re-running the tests.

I made some attempts in Storyteller 5 to improve the situation (after we gave up on building a dotnet test adapter because that model is bonkers), but that still takes some set up time to make it work and even I have to always run to the documentation to remember how to do it. Sometime this weekend the solution for a quick xUnit.Net execution wrapper around Storyteller popped into my head and it honestly took about 15 minutes flat to get things working so that I could kick off individual Storyteller specifications from xUnit.Net as shown below:

StorytellerWithinXUnit

Maybe that’s not super exciting, but the end result is that I can rerun a specification after making changes with or without debugging with nothing but a simple keyboard shortcut in the IDE. That’s a dramatically faster feedback cycle than what I had to begin with.

Implementation wise, I just took advantage of xUnit.Net’s [MemberData] feature for parameterized tests and Storyteller’s StorytellerRunner class that was built to allow users to run specifications from their own code. After adding a new xUnit.Net test project and referencing the original Storyteller specification project named “StorytellerSpecs, ” I added the code file shown below in its entirety::

// This only exists as a hook to dispose the static
// StorytellerRunner that is hosting the underlying
// system under test at the end of all the spec
// executions
public class StorytellerFixture : IDisposable
{
    public void Dispose()
    {
        Runner.SpecRunner.Dispose();
    }
}

public class Runner : IClassFixture<StorytellerFixture>
{
    internal static readonly StoryTeller.StorytellerRunner SpecRunner;

    static Runner()
    {
        // I'll admit this is ugly, but this establishes where the specification
        // files live in the real StorytellerSpecs project
        var directory = AppContext.BaseDirectory
            .ParentDirectory()
            .ParentDirectory()
            .ParentDirectory()
            .ParentDirectory()
            .AppendPath("StorytellerSpecs")
            .AppendPath("Specs");

        SpecRunner = new StoryTeller.StorytellerRunner(new SpecSystem(), directory);
    }

    // Discover all the known Storyteller specifications
    public static IEnumerable<object[]> GetFiles()
    {
        var specifications = SpecRunner.Hierarchy.Specifications.GetAll();
        return specifications.Select(x => new object[] {x.path}).ToArray();
    }

    // Use a touch of xUnit.Net magic to be able to kick off and
    // run any Storyteller specification through xUnit
    [Theory]
    [MemberData(nameof(GetFiles))]
    public void run_specification(string path)
    {
        var results = SpecRunner.Run(path);
        if (!results.Counts.WasSuccessful())
        {
            SpecRunner.OpenResultsInBrowser();
            throw new Exception(results.Counts.ToString());
        }
    }
}

And that’s that. Something I’ve wanted to have for ages and failed to build, done in 15 minutes because I happened to remember something similar we’d done at work and realized how absurdly easy xUnit.Net made this effort.

Summary

  • Sometimes it’s worthwhile to take a step back from trying to solve a problem through debugging and invest in better instrumentation or write some automation scripts to make the debugging cycles faster rather than just trying to force your way through the solution
  • Inspiration happens at random times
  • Listen to that niggling voice in your head sometimes that’s telling you that you should be doing things differently in your code or tests
  • Using a modular architecture that’s composed by an IoC container the way that ASP.net Core does can sometimes be advantageous in integration testing scenarios. Case in point is how easy it was for me to toss in an all new logging provider that captured the log information directly into the test results for easier test failure resolution

And when I get around to it, these little Storyteller improvements will end up in Storyteller itself.