Skip to content

Building an EventStore with User Defined Projections on top of Postgresql and Node.js

I did an internal talk on the tooling and concepts in this post at our Salt Lake City office a couple months ago. The recording, for what it’s worth, is here. I’m assuming that you’re at least somewhat familiar with the concepts of Event Sourcing and CQRS, but if you’re not, there are links to descriptive explanations of these concepts in the body of the post.

For most of the 2000’s my goto strategy for application persistence was to use some sort of object relational mapping to persist and read the object structures that I wanted to work with in my code. Sometimes I used hand rolled code to do the mapping, and other times my teams used NHibernate. In the past couple years I’ve been on projects that used the RavenDb document database with mixed success. I’ve also worked on a couple codebases that used an event sourcing strategy to persist meaningful business events, sometimes with RavenDb as the underlying storage engine and another project that uses an older version of NEventStore with Sql Server as the storage mechanism.

For various reasons, we’ve chosen to use a Node.js based stack to rewrite an old WPF application that is a suitable candidate for event sourcing on the backend (Corey Kaylor explained his take on this decision in a blog post). Since we already wanted to replace Sql Server (and probably RavenDb) with Postgresql in the long run, at Corey’s suggestion I have been working on and off to try leveraging  to create a new event store suitable for Node.js development that supports user-defined projections. Lacking all originality, I’m calling this new library “pg-events.”  You can find pg-events hosted under my GitHub account (my very first foray back into OSS post-FubuMVC).

 

Feature Set

  • Support the basic event sourcing pattern by appending the raw business events as JSON to the event store
  • Track events by a “stream” of related events that probably relates directly to some kind of business concept or workflow
  • Support user-defined projections of the raw event data to create “read side” views for clients
  • Support aggregated views of a stream (really just another projection). Use a basic snapshotting strategy of the aggregate state for efficiency
  • Build time tooling to initialize a postgresql database with the custom schema objects and import javascript libraries to postgresql
  • A crude, partial implementation of CommonJS that runs within postgresql

 

Conceptual Architecture

The first thing to know is that we’re making a very large bet on the portability of Javascript code and the ability to run at least a subset of this new event store code hosted in Postgresql, Node.js, embedded in other programming, or even potentially in a browser. The user-defined projections could potentially be executed in any of the pieces below, and we think that flexibility will pay off down the road for both performance and scalability tuning.

So far, I think the end state is going to consist of these four pieces:

 

pg-events

  1. Postgresql Database
    1. Custom schema objects largely based on Greg Young’s Building an Event Store paper to store events and stream metadata.
    2. Tables to persist projected views. Most projection views will be persisted to separate tables instead of one giant “pge_views” table for better query performance
    3. Stored procedures to update and query data in the event storage tables, mostly using postgresql’s Javascript support
    4. Executes and updates aggregate snapshots and synchronous projections. More on that in the following section.
  2. A Node.js Client
    1. Exposes methods to append events to the store
    2. Exposes methods to query for projected views, aggregate snapshots, and raw event stream data
    3. See https://github.com/jeremydmiller/pg-events/wiki/Client for more information
  3. Admin CLI Tool
    1. Build the necessary schema objects into a Postgresql
    2. Loads the user defined projections and other pg-events libraries into the database
    3. Can reset the event storage and projected view tables to an empty state for testing
    4. Eventually, this tool will also support “recapitulation” to rebuild projection data from the raw events when the definition of a projection changes
  4. Background Projection Runner
    1. Executes and updates projected views in a background process. This is my very next coding adventure. I’m going to build it out first with Node.js, then try my hand at implementing it again with a standalone Golang executable that uses an embedded V8 engine to execute the projections. Expect my twitter feed to be entertaining when I’m able to start that work. I’ll blog about this later when I know what it’s going to actually look like;)

 

 

User Defined Projections 

We looked at EventStore at first and I definitely liked their first class support for user defined projections. Our implementation of projections is very obviously influenced by EventStore’s.

I think that the event sourcing efforts I’ve been a part of have been successful overall, but “projecting” the raw event stream into a persisted read side or view model has been challenging. For pg-events, we’re expressing the projections with simple transformation functions that will take in the initial state and the raw event data and simply return the new state (it’s a logical fold left operation for the projections that work across multiple events).

For a sample event sourcing domain, I’ve been using the idea of a quest from the way too many fantasy books I’ve read over my lifetime. During a quest, our heroes might record events like “QuestStarted”, “MembersJoined”, “MembersDeparted”, or “TownReached.” To know or understand the exact composition of a quest party at any time, we need to replay some of the change events (Gandalf stayed behind to fight the Balrog, Boromir was killed, Frodo and Sam ran off, Gollum joined up, etc.) for the quest.

Say we write a projection for a new view across the events in a single quest called “Party” just to understand the membership. From the unit tests, that projection looks like:

require("../../lib/projections")
	.projectStream({
		name: 'Party',
		stream: 'Quest', 
		mode: 'sync',

		$init: function(){
			return {
				active: true,
				traveled: 0,
				location: null,
				members: []
			}
		},

		QuestStarted: function(state, evt){
			state.active = true;
			state.location = evt.location;
			state.members = evt.members.slice(0);

			state.members.sort();
		},

		TownReached: function(state, evt){
			state.location = evt.location;
			state.traveled += evt.traveled;
		},

		EndOfDay: function(state, evt){
			state.traveled += evt.traveled;
		},

		QuestEnded: function(state, evt){
			state.active = false;
			state.location = evt.location;
		},

		MembersJoined: function(state, evt){
			state.members = state.members.concat(evt.members);
			state.members.sort();
		},

		MembersDeparted: function(state, evt){
			state.location = evt.location;

			for (var i = 0; i < evt.members.length; i++){
				var index = state.members.indexOf(evt.members[i]);
				state.members.splice(index, 1);
			}

			state.members.sort();
		}


	});

You’ll notice that there’s a field called “mode” with a value of “sync.” Using the portability of Javascript, we’re planning for these modes:

  1. sync – A ‘sync’ projection will be executed synchronously inside postgresql within the same transaction as the event capture
  2. async – In progress. An ‘async’ projection will be calculated in a background process instead of at event capture time (Eventual Consistency).
  3. live – Forthcoming. These projections will only be calculated upon demand. I’m not yet sure if we’ll do the actual projection transformations within the database or the Node.js client. I guess we could allow two different “live” modes if there’s value in doing that.

So, eventual consistency killing you in your current event sourcing efforts because you hit errors by querying off of stale data? Opt for synchronous projections. Have lots of writes, but relatively few reads? Use asynchronous or even live projections that are only calculated on demand. Have lots of reads but very few writes? I think I would again opt for synchronous projections.

I worked on a system a couple years ago in a failed startup that ran projections in a browser to do historical point in time simulations. I don’t see any reason why we couldn’t do something similar in pg-events if that is ever valuable.

Believe it or not, I have a decent start on documenting the projection support at https://github.com/jeremydmiller/pg-events/wiki/Projections.

 

Why Postgresql? 

Postgresql is the people pleaser of database engines. Want all the normal RDBMS capabilities? Would you be more productive using Postgresql as a document database? Want to write stored procedures with a language that closely resembles Oracle’s PL/SQL (no, a thousand times no, never again)?  Would you even want to use Javascript inside the database itself? Regardless of how you answered any of those questions, Postgresql is trying really hard to be what you want. In our case, I like that we can use postgresql as an event store, a document database for things that don’t fit into the event sourcing model, and a classic RDBMS if that’s what we want in some circumstances.

Mostly though, we like that Postgresql has a proven track record and we suspect that the DevOps support tools will be more effective than we’ve experienced with other OSS database tools.

Of course, the only reason why pg-events is viable in the first place is that postgresql has outstanding JSON support and the ability to author stored procedures with Javascript using Google’s V8 engine. With our project timeline, it’s also safe to assume that Postgresql 9.4 with its significant improvements to the JSON storage will be available before we go live to production.

 

Why CQRS isn’t crazy

I’ll feely admit that the first time I saw Greg Young talking about the Command Query Responsibility Segregation (CQRS) style of architecture in 2008 I thought it was nuts. Specifically, I was afraid that doing the transformations between the “write side” model and the “read side” model consumed by the clients would lead to far too much repetitive “left hand, right hand” code. The reality is, of course, is that I was already doing a lot of work to map database tables to object graphs, transforming domain model objects to DTO’s to send over the wire, and crafting database views to transform our raw data into something more conducive to reporting requirements. In a way, CQRS just explicitly calls out a large part of software development efforts that is often overlooked. If we simply accept the idea that different consumers and producers of the persisted state in our system naturally have different needs as far as how the same information is written, structured, and consumed, CQRS isn’t really “crazy talk” or extra work. One of the biggest differences is that with event sourcing + CQRS you probably try to pre-build and persist the read side views instead of trying to create views or DTO’s on the fly from the “one true database model.”

 

Some thoughts on Relational Databases

I’m very much in the camp that says that the database is strictly for persistence and your business logic and/or user interface should never be tightly coupled to whatever the database is, so the idea of just consuming the raw database tabular data in business logic code is a non-starter for me — not to mention that a flat database table structure is very rarely the exact structure that you’d want in your business logic code outside of CRUD-centric applications. I’ve been a part of technical arguments with database-centric folks for so long that I’m simply happy to say “agree to disagree” on these issues and let us all go on our way.

There’s a tremendous amount of inertia and investment in tooling in our industry in regards to the usage of relational databases as the de facto standard for just about all persistence needs. Additionally, most developers, testers, and even the business people seem to naturally understand the relational database model. Even so, as alternative models like document or graph databases build up more tooling, acceptance, and developer familiarity, I think that relational databases will eventually be consigned to reporting applications or pure CRUD applications (but even then I prefer document databases).

That being said, I think that the future really is “polyglot persistence” and that our children are going to laugh at us in decades to come when we explain how we built systems against relational databases.

StructureMap 3.1

I pushed a new minor release version 3.1 of the StructureMap, StructureMap.Web, StructureMap.AutoMocking, StructureMap.AutoMocking.Moq, and StructureMap.AutoFactory packages to Nuget.org this morning. You can see a list of the closed issues in this release here.

Thank you to Matt Honeycutt, Marco Cordeiro, and Jimmy Bogard for their help in making this incremental release.

 

Future Plans

  • The documentation is still in flight and will probably be so for quite some time. What little is there so far is up at http://structuremap.github.io.
  • Xamarin support. StructureMap 3.0 is already built with PCL compliance and runs on WP8, so getting it to run on Xamarin runtimes should be a piece of cake, right? Right?
  • Continue to make bug fix releases as needed. I hate how many bugs have popped up since the 3.0 release, but at least it’s much easier to make incremental releases in the Nuget era than it was with Sourceforge back in the day.

I’m still holding the line on not strong naming StructureMap unless someone does the pull request to support multiple signed and unsigned versions of the Nuget. I’m starting to get asked about a signed version every couple weeks and I still don’t want to do that. You’re always welcome to just clone the repository and sign the code yourself.

 

Building with IContext.Root

A couple StructureMap users have asked over the years to support contextual resolution of injected loggers with tools like log4net or NLog. While StructureMap 2.5+ supported this pattern, some of the support got lost in the big restructuring work for 3.0 and this release brings it back.

Say you’re using a logging tool that allows you to specify different logging rules and mechanisms by namespaces, types, or assemblies. Your logging tool probably has some construct like the following code to build the right logger for a given type:

    public static class LogManager
    {
        // Give me a Logger with the correctly
        // configured logging rules and sources
        // matching the type I'm passing in
        public static Logger ForType(Type type)
        {
            return new Logger(type);
        } 
    }

Now, let’s say that you want StructureMap to inject a Logger into constructor arguments of the objects it’s going to build. If you want to create a Logger that’s suitable for the topmost concrete type being built by a service location request to StructureMap. You could use code similar to this:

            // IContext.RootType is new for 3.1
            var container = new Container(_ => {
                _.For<Logger>()
                    .Use(c => LogManager.ForType(c.RootType))
                    .AlwaysUnique(); 
            });

If you wanted to build the Logger individually to match each type in the object graph being created, use this code instead:

            // Resolve the logger for the type one level up
            container = new Container(_ => {
                _.For<Logger>().Use(c => LogManager.ForType(c.ParentType))
                    .AlwaysUnique();
            });

The AlwaysUnique() lifecycle is important here to force StructureMap to create a new Logger instance every time one is necessary in the object graph to prevent the very first Logger created from being shared throughout the entire object graph. This is one of the very few use cases for the “unique” lifecycle.

 

Child Containers

New for StructureMap 3.0 are “child containers,” which should not be confused for nested containers — and all of that would be much more clear if I’d ever get around to writing the big blog post on nested container behavior that I’ve promised a half dozen people this year. Child containers are meant for stateful client development where you might want to pop a child container for a region, pane, or specific view of the application to override some of the main application services while being able to gracefully fallback to the application container for everything else.

 

Better IEnumerable<T>/IList<T>/T[] Support

Since at least version 2.5, if StructureMap encounters constructor or setter arguments of IEnumerable<T>, IList<T>, or T[] in a concrete type where the dependency is not explicitly configured, those arguments will be fulfilled by creating an enumerable of all configured instances of the type T in the container in the order in which they were registered. Great, and it was valuable in several usages within FubuMVC, but other folks are wanting to resolve the enumeration types directly or as Lazy<IList<T>> or Func<T[]>. StructureMap 3.1 will now resolve enumeration types that are not explicitly configured by returning all the known configured instances of the type T. To make that concrete, see the acceptance tests for this behavior.

Composable Generators in Javascript

I’m getting to work with Node.js for the first time and generally enjoying the change of pace from .Net — especially the part where my full Mocha test suite that includes integration tests runs faster than my souped up i7/SSD box can compile even the simplest .Net library. For my part, I’ve been playing with a new library to support event sourcing user defined with projections by leveraging Postgresql’s native JSON features and embedded Javascript support (not much to see yet, but the code is here if you’re curious).

Since it is Node.js, every action you do that touches the database or file system really wants to be asynchronous. I started by using promises with the Bluebird library, and it was okay for simple cases. However, the code became very hard to follow when I started chaining more than 3-4 operations together. Just this week I finally got a chance to try out the usage of the new ES6 generators feature with my codebase in the places where I was having to combine so many asynchronous operations and I’m very happy with the results so far.

To my eyes, generators are a big improvement in readability over promises. Consider two versions of the same method from my event store library. First up, this is the original version using promises to execute a stored procedure in Postgresql (yep, I said sproc) then transform the asynchronous result to return to the calling client:

	append: function(){
		var message = this.toEventMessage(arguments);

		return pg.connectAsync(this.options.connection).spread(function(client, release){
			return client.queryAsync('select pge_append_event($1)', [message])
				.finally(release);
		})
		.then(function(result){
			return Promise.resolve(result.rows[0].pge_append_event);
		});
	},

Now, see the newer equivalent method using generators with some help from the postgres-gen library:

	append: function(){
		var message = this.toEventMessage(arguments);

		return this.db.transaction(function*(t){
			return (yield t.query('select pge_append_event(?)', message)).rows[0].pge_append_event;
		});
	},

I’m going to claim that the generator version on the bottom is cleaner and easier to write and understand than the version that only uses promises above. Some of that is due to my usage of the postgres-gen library to smooth out the interaction with Postgresql, but I think that not having to nest Javascript functions is a big win (yes I know that Coffeescript or arrow functions with Traceur would help too by making the inline function syntax cleaner, but I’d still rather avoid the nesting anyway).

The biggest difference to me came when I wanted to start dynamically composing several asynchronous operations together. As I said earlier, I’m trying to build an event store library with user defined projections. To test the projection support I needed to repeatedly store a sequence of events and then fetch and evaluate the value of the calculated projection to verify the new functionality. I very quickly realized that embedding the original promise mechanics into the code was making the tests laborious to write and hard to read. To that end, I built a tiny DSL just for testing. You can see the raw code in Github here, but consider this sample:

	scenario('can update a view across a stream to reflect latest', function(x){
		var id = uuid.v4();

                // stores several pre-canned events to the event store
		x.append(id, 'Quest', e1_1, e1_2, e1_3);

                // does an assertion that the named view for the event
                // stream above exactly matches this document
		x.viewShouldBe(id, 'Party', {
			active: true,
			location: 'Baerlon',
			traveled: 16,
			members: ['Egwene', 'Mat', 'Moiraine', 'Perrin', 'Rand', 'Thom']
		});

                // do some more events and check the new view
		x.append(id, e1_4, e1_5);
		x.viewShouldBe(id, 'Party', {
			active: true,
			location: 'Shadar Logoth',
			traveled: 31,
			members: ['Egwene', 'Mat', 'Moiraine', 'Perrin', 'Rand']
		});

	});

Every call to the append() or viewShouldBe() methods requires an asynchronous operation to either post or query data from the underlying Postgresql database. Originally, I implemented the testing DSL above by creating an empty promise, then running that promise through all the “steps” defined in the scenario to chain additional promise operations. Earlier this week I got a chance to switch the testing DSL to using generators underneath the language and that made a big difference.

The first thing to know is that you can embed a generator function inside another generator by using the yield* keyword to designate that you want an inner generator function to be executed inline. Knowing that, the way that I implemented the testing DSL shown above was to create an empty array called steps as a member on a scenario object. In the testing expression methods like the append(), I just pushed a new generator function into the steps array like this:

	this.append = function(){
		var message = client.toEventMessage(arguments);
		self.lastId = message.id;

		// Adding a new generator function
		// to the steps array
		this.steps.push(function*(){
			yield client.append(message);
		});
	}

After the scenario is completely defined, my testing DSL just executes the steps as a single generator function as shown in the code below:

	// I'm using the Bluebird Promise.coroutine()
	// method to treat this single aggregated
	// generator method as a single promise
	// that can be happily executed and tracked
	// by the Mocha test harness
	this.execute = function(client){
		return Promise.coroutine(function*(){
			this.steps.forEach(function(s){
				yield* s;
			});
		});
	}

You can see the commit diff of the code here that demonstrates the difference between using just promises and using a generator to compose the asynchronous operation. The full implementation for the “scenario” testing DSL is here.

As an aside, if you’re playing Design Pattern Bingo at home, my testing DSL uses Fowler’s Nested Closure pattern to first define the steps of a test specification in a nested function before executing the steps. I’ve used that pattern successfully several times for little one off testing tools and seen plenty of other folks do similar things.

 

Using generators as a composable Russian Doll Model with Koa.js

My organization settled on a Node.js based stack for a rewrite of an older .Net system, but when we first started discussing a possible new platform Node.js was almost summarily dismissed because of its callback hell problems. Fortunately, I got a tip on twitter to look at the Koa.js framework and its support for generators to avoid the older callback hell problems and I was sold. Since our timeline is long enough that we feel safe betting on ES6 Harmony features, we’re starting with Koa.js.

One of the things I feel was successful in FubuMVC was our use of what I’ve always called the Russian Doll model. In this model, we could compose common behaviors like validation, authorization, and transaction management into reusable “behaviors” that could be applied individually or conventionally to the handlers for each HTTP route in a similar manner to aspect oriented programming (but with object composition instead of IL weaving or dynamic proxies). One of the things that appeals to me about Koa.js is their usage of generators as a middleware strategy that’s conceptually equivalent to FubuMVC’s old behavior model (the current Connect middleware is “single pass” where the Russian Doll model requires nesting the middleware handlers to do optional before and after operations).

At some point I’d like to experiment with building the FubuMVC “BehaviorGraph” model of configuring middleware per route for better fine grained control, but that’s a ways off.

React.js plays nicely with other tools and other thoughts

tl;dr: I’m not completely sure about React.js yet just because it’s such a different approach, but I really appreciate how easy it is to use it in conjunction with other client side tools.

My shop is looking very hard at moving to React.js for new development and I’m scrambling a bit to catch up as I’ve been mostly on the server side for years. Partially as a learning experience, I’ve been using React.js to rebuild the UI for our FubuMVC.Diagnostics package in FubuMVC 2.0 as a single page application. While my usage of React.js on the diagnostics is mostly about read only data displays, I did hit a case where I wanted a “typeahead” type search bar. While a google search will turn up a couple nascent projects to build typeahead type components purely with React.js, none looked to me to be nearly as mature and usable as the existing typeahead.js query plugin*.

Fortunately, it turns out to be very easy to use jquery plugins inside of React classes so I just used typeahead.js as is. For the StructureMap diagnostics, I wanted a search bar that allowed users to search for all the assemblies, namespaces, and types configured in the main application container with typeahead recommendations. Since I was going to want to reuse that search bar on a couple different little screens, I did things the idiomatic way React wants to be used anyway and created the small component shown below:

In the not unlikely chance that the code below is mangled and unreadable, you can find the source code in GitHub here.

var SearchBox = React.createClass({
	componentDidMount: function(){
		var element = this.getDOMNode();
		$(element).typeahead({
		  minLength: 5,
		  highlight: true
		},
		{
		  name: 'structuremap',
		  displayKey: 'value',
		  
		  // Just a little object that implements the necessary 
		  // signature to plug into typeahead.js
		  source: FubuDiagnostics.StructureMapSearch.findMatches,
		  templates: {
			empty: '<div class="empty-message">No matches found</div>',
			suggestion: _.template('<div><p><%= display%> - <small><%= type%></small></p><p><small><%= value%></small></p></div>')
		  }
		});
		
                // Behind the scenes, this is just delegating to Backbone's router
                // to 'navigate' the main pane of the page to a different view
		$(element).on('typeahead:selected', function(jquery, option){
			var url = 'structuremap/search-results/' + option.type + '/' + option.value;
			FubuDiagnostics.navigateTo(url);
		});
	},
	
	componentWillUnmount: function(){
		var element = this.getDOMNode();
		$(element).typeahead('destroy');
	},

	render: function(){
		return (
			<input type="search" name="search" ref="input" 
                          className="form-control typeahead structuremap-search" 
                          placeholder="Search the application container" />
		);
	}
});

 

There really isn’t that much to the code above, but the salient points are:

  • The componentDidMount() method fires just after React.js renders the component, giving me the chance to apply the typeahead jquery plugin to the DOM element.
  • In this case the component is a single HTML element, so I could just use the getDOMNode() method to get at the actual HTML DOM element. In more complicated components you can also use the refs collection.
  • I cleaned up and deactivated the typeahead plugin in the componentWillUnmount() method whenever React.js is tearing down the component to clean up after myself

To use the component above, I just embedded it into another component as more or less a custom tag in the jsx markup:

var Summary = React.createClass({
	render: function(){
		var items = [];
		
		this.props.assemblies.forEach(function(assem, i){		
			items.push(AssemblySummaryItem(assem));
			
			assem.namespaces.forEach(function(ns){
				items.push(NamespaceSummaryItem(ns));
			});
			
		});
		
		return (
			<div>
                        
                        // This is all you have to do to include it
			<SearchBox />
			
			<hr />
			
			<ul className="list-group">
				{items}
			</ul>
			
			</div>
		);
	}
});

 

So what do I think of React.js so far?

Um, well, React is interesting. I know that my initial reaction was similar to many others: “you seriously want me to embed HTML markup directly into JavaScript?” Once you get past that, it’s been very easy for straight forward usages of rendering JSON from the server into HTML displays. I won’t make any permanent judgement of React.js until I use it on something with much more complicated user interactions.

To speak to one of React.js’s major design philosophies, I’ve never liked two way model binding in any user interface technology I’ve ever used because I think it so easily obfuscates the behavior of screens and causes unexpected side effects. I’m not completely sure how well the one way data flow thing is going to work out in bigger applications, but I’m at least initially on board with the React.js team’s basic design philosophy.

One thing to consider is that React.js is much more of a library that you’d use as opposed to an all encompassing framework like Angular.js or Ember.js that you have to wrap your application around. The downside of that is that React.js doesn’t really do anything to encourage separation of concerns or help you structure your application beyond the view layer, but the old presentation patterns that came out of older technologies still apply and are still useful if you just treat React.js as the view technology. You’re just on your own to do your own thinking. The upside of React.js being just a library is how easy it is to use it in conjunction with other tools like the entire jquery ecosystem or bits of Backbone.js or even as something within frameworks like Angular or Ember.

I’ve also done a little bit of a proof of concept of using React.js as the view layer in a possible rewrite of the Storyteller UI into a pure web application. That work will probably generate much more interesting blog posts later as it’s going to require much deeper hierarchies of components, a lot more complicated user interaction, and will push us into exploring React.js usage with separated presentation patterns and much more unit testing.

At least in the circles I run in, there’s been a bit of a meme the past couple years that Functional Programming is the one true path to software programming and a corresponding parallel backlash against almost everything to do with OOP. If React.js becomes more popular, I think that it might help start a rediscovery of the better ways to do OOP. Namely a focus on thinking about responsibilities, roles, collaboration patterns, and object composition as a way to structure complicated React.js components instead of the stupid “Class-centric,” inheritance laden, data-centric approach to OOP that’s been so easy for the FP guys to caricature.

 

* I’m not really working much on the client side, so what do I really know? All the same, I think the backlash against jquery feels like a bit of throwing the baby out with the bathwater. Especially if you’re constraining the DOM manipulation to the purely view layer of your code.

Integration Testing with FubuMVC and OWIN

tl;dr: Having an OWIN host to run FubuMVC applications in process made it much easier to write integration tests and I think OWIN will end up being a very positive thing for the .Net development community.

FubuMVC is still dead-ish as an active OSS project, but the things we did might still be useful to other folks so I’m continuing my series of FubuMVC lessons learned retrospectives — and besides, I’ve needed to walk a couple other developers through writing integration tests against FubuMVC applications in the past week so writing this post is clearly worth my time right now.

One of the primary design goals of FubuMVC was testability of application code, and while I think we succeeded in terms of simpler, cleaner unit tests from the very beginning (especially compared to other web frameworks in .Net), there are so many things that can only be usefully tested and verified by integration tests that execute the entire HTTP stack.

Quite admittedly, FubuMVC was initially very weak in this area — partially because the framework itself only ran on top of ASP.Net hosting. While we had good unit test coverage from day one, our integration testing story had to evolve as we went in roughly these steps:

  1. Haphazardly build sample usages of features in an ASP.Net application that had to run hosted in either IIS or IISExpress that had to be executed manually. Obviously not a great solution for regression testing or for setting up test scenarios either.
  2. A not completely successful attempt to run FubuMVC on the early “Delegate of Doom” version of the OWIN specification and the old Kayak HTTP server. It worked just well enough to get my hopes up but didn’t really fly (there were some scenarios where it would just hang). If you follow me on Twitter and seen me blast OWIN as a “mystery meat” API, it’s largely because of lingering negativity from our first attempt.
  3. Running FubuMVC on top of Web API’s Self Host libraries. This was a nice win as it enabled us to embed FubuMVC applications in process for the first time, and our integration test coverage improved quickly. The Self Host option was noticeably slow and never worked on Mono* for us.
  4. Just in time for the 1.0 release, we finally made a successful attempt at OWIN hosting with the less insane OWIN 1.0 specification, and we’ve ran most of our integration tests and acceptance tests on Katana ever since. Moving to Katana and OWIN was much faster than the Web API Self Host infrastructure and at least in early versions, worked on Mono (the Katana team has periodically broken Mono support in later versions).
  5. For the forthcoming 2.0 release I built a new “InMemoryHost” model that can execute a single HTTP request at a time using our OWIN support without needing any kind of HTTP server.

I cannot overstate how valuable it has been to have an embedded hosting model has been for automated testing. Debugging test failures, using the application services exactly as the real application is configured, and setting up test data in databases or injecting in fake services is much easier with the embedded hosting models.

Here’s a real problem that I hit early this year. FubuMVC’s content negotiation (conneg) logic for selecting readers and writers was originally based only on the HTTP accepts and content-type headers, which is great and all except for how many ill behaved clients there are out there that don’t play nice with the HTTP specification. I finally went into our conneg support earlier this year and added support for overriding the accepts header, with an in the box default that looked for a query string on the request like “?format=json” or “format=xml.” While there are some unit tests for the internals of this feature, this is exactly the kind of feature that really has to be tested through an entire HTTP request and response to verify correctness.

If you’re having any issues with the formatting of the code samples, you can find the real code on GitHub at the bottom of this page.

 

I started by building a simple GET action that returned a simple payload:

    public class OverriddenConnegEndpoint
    {
        public OverriddenResponse get_conneg_override_Name(OverriddenResponse response)
        {
            return response;
        }
    }

    public class OverriddenResponse
    {
        public string Name { get; set; }
    }

Out of the box, the new “conneg/override/{Name}” route up above would respond with either a json or xml serialization of the output model based on the value of the accepts header, with “application/json” being the default representation in the case of wild cards for the accepts header. In the functionality, content negotiation needs to also look out for the new query string rules.

The next step is to define our FubuMVC application with a simple IApplicationSource class in the same assembly:

    public class SampleApplication : IApplicationSource
    {
        public FubuApplication BuildApplication()
        {
            return FubuApplication.DefaultPolicies().StructureMap();
        }
    }

The role of the IApplicationSource class in a FubuMVC application is to bootstrap the IoC container for the application and define any custom policies or configuration of a FubuMVC application. By using an IApplicationSource class, you’re establishing a reusable configuration that can be quickly applied to automated tests to make your tests as close to the production deployment of your application as possible for more realistic testing. This is crucial for FubuMVC subsystems like validation or authorization that mix in behavior to routes and endpoints off of conventions determined by type scanning or configured policies.

 

Using Katana and EndpointDriver with FubuMVC 1.0+

First up, let’s write a simple NUnit test for overriding the conneg behavior with the new “?format=json” trick, but with Katana and the FubuMVC 1.0 era EndpointDriver object:

        [Test]
        public void with_Katana_and_EndpointDriver()
        {
            using (var server = EmbeddedFubuMvcServer
                .For<SampleApplication>())
            {
                server.Endpoints.Get("conneg/override/Foo?format=json", acceptType: "text/html")
                    .ContentTypeShouldBe(MimeType.Json)
                    .ReadAsJson<OverriddenResponse>()
                    .Name.ShouldEqual("Foo");
            }
        }

Behind the scenes, the EmbeddedFubuMvcServer class spins up the application and a new instance of a Katana server to host it. The “Endpoints” object exposes a fluent interface to define and execute an HTTP request that will be executed with .Net’s built in WebClient object. The EndpointDriver fluent interface was originally built specifically to test the conneg support in the FubuMVC codebase itself, but is usable in a more general way for testing your own application code written on top of FubuMVC.

 

Using the InMemoryHost and Scenarios in FubuMVC 2.0

EndpointDriver was somewhat limited in its coverage of common HTTP usage, so I hoped to completely replace it in FubuMVC 2.0 with a new “scenario” model somewhat based on the Play Framework’s Play Specification tooling in Scala. I knew that I also wanted a purely in memory hosting model for integration tests to avoid the extra time it takes to spin up an instance of Katana and sidestep potential port contention issues.

The result is the same test as above, but written in the new “Scenario” style concept:

        [Test]
        public void with_in_memory_host()
        {
            // The 'Scenario' testing API was not completed,
            // so I never got around to creating more convenience
            // methods for common things like deserializing JSON
            // into .Net objects from the response body
            using (var host = InMemoryHost.For<SampleApplication>())
            {
                host.Scenario(_ => {
                    _.Get.Url("conneg/override/Foo?format=json");
                    _.Request.Accepts("text/html");

                    _.ContentTypeShouldBe(MimeType.Json);
                    _.Response.Body.ReadAsText()
                        .ShouldEqual("{\"Name\":\"Foo\"}");
                });
            }
        }

 

The newer Scenario concept was an attempt to make HTTP centric testing be more declarative. C# is not as expressive as Scala is, but I was still trying to make the test expression as clean and readable as possible and the syntax above probably would have evolved after more usage. The newer Scenario concept also has complete access to FubuMVC 2.0’s raw HTTP abstractions so that you’re not limited at all in what kinds of things you can express in the integration tests.

If you want to see more examples of both EmbeddedFubuMvcServer/EndpointServer and InMemoryHost/Scenario in action, please see the FubuMVC.IntegrationTesting project on GitHub.

 

Last Thoughts

If you choose to use either of these tools in your own FubuMVC application testing, I’d highly recommend doing something at the testing assembly level to cache the InMemoryHost or EmbeddedFubuMvcServer so that they can be used across test fixtures to avoid the nontrivial cost of repeatedly initializing your application.

While I’m focusing on HTTP centric testing here, using either tool above also has the advantage of building your application’s IoC container out exactly the way it should be in production for more accurate integration testing of underlying application services.

 

If I had to do it all over again…

We would have had an embedded hosting model from the very beginning, even if it had been on a fake, “only one HTTP request at a time” model. Moreover, if we were to ever reboot a new FubuMVC like web framework in the future KVM, I would vote for wrapping any new framework completely around the OWIN signature as our one and only model of an HTTP request. In any theoretical future effort, I’d invest time from the very beginning in something like FubuMVC 2.0’s InMemoryHost model early to make integration and acceptance testing easier and faster.

With the recent release of StructureMap 3, I’d also opt for a new child container model such that you can fake out application services on a test by test basis and rollback to the previous container state without having to re-initialize the entire application each time for faster testing.

 

* Mono support was a massive time sink for the FubuMVC project and never really paid off. Maybe having Xamarin involved much earlier in the new KVM .Net runtime and the emphasis on PCL supportwill make that situation much better in the future.

Some Thoughts on Collective Ownership and Knowledge

This is a mild rewrite of an old blog post of mine from 2005 that I think is still relevant today. While this was originally about pair programming, I’ve tried to rewrite this to be more about working collaboratively in general. 

My shop is having some constructive internal discussions about our development process and practices. While I don’t think that Pair Programming is likely to be a big part of our daily routine, I think we could still use a bigger dose of collective ownership in our various code base’s and less silo-ing of developers. Since I’m a typical introverted developer who doesn’t handle hours of pair programming and I don’t think my attitude is uncommon at all, we need to look for ways to get the same type of shared understanding that you would get from 100% paired programming.

One of my suggestions is at a minimum to stop assigning coding stories to a single developer. Even if they aren’t pair programming all the time, they can still collaborate on the approach and split tasks so they can code in parallel. On one hand this should increase our team’s shared understanding of the codebase, and on the other hand it’ll definitely help us work discrete stories serially to completion instead of having so many half-done stories laying around at any one time.

 

“This is the Way We Do It”

My favorite metaphor for software design these days is a fishing tackle box.  I want a place for everything and everything in its place.  When I need a top water lure, I know exactly where to look.  I put data access code here, business logic there, and hook the two things up like this.  When I’m creating new code I want to organize it along a predictable structure that anyone else on the project will instantly recognize.  When other coders are making changes I want them to follow the same basic organization so I can find and understand their code later.

Each application is a little bit different so the application’s design is always going to vary.  In any agile project you should hit an inflection point in the team’s velocity that I think of as the “This is the Way We Do It” moment.  Things become smoother.  There are fewer surprises.  Story estimates become more consistent and accurate.  When any pair starts a new story, they understand the general pattern of the system structure and can make the mechanical implementation with minimal fuss.

You really want to get to this point as soon as you can.  There are two separate issues to address before you can reach this inflection point:

  1. Determining the pattern and architecture for the system under development
  2. Socializing the design throughout the development team

To the first point, pairing allows you bring to bear the knowledge and experience of everybody on the team to the work at hand.  It’s just not possible for any one developer to understand every technology and design pattern in the world. By having every developer active in the project design, you can often work out a workable approach faster than a solo architect ever could.  On the one project I’ve done with theoretical 100% pairing, we had a couple of developers with a lot of heavy client experience and me with more backend and web development experience.  By pairing together with our disparate knowledge we could rapidly create a workable general design strategy for the system as a whole by bringing a wider skill set to any coding task.

Right now I’m working some with a system that’s being built by two different teams that work in two very different ways with very different levels of understanding of some of the core architecture. That’s really not an ideal situation and it’s causing plenty of grumbling. There’s a couple specific subsystems that generate complaints about their usability. In one case it just needs to be a full redesign, but that really needs to take place with more than one or two people involved. In the other case, the problems might be from the team that didn’t build the subsystem not understanding how it works and why it was built that way — or it might be because the remaining developer who worked on it doesn’t completely understand the problems that the other team is having. Regardless of what the actual problem is, more active collaboration between the teams might help that subsystem be more usable.

If you’re a senior developer or the technical lead, one of your responsibilities is fostering an understanding of the technical direction to the other developers.  Nothing else I’ve ever done as a lead (design sessions, documentation, presentations, “do this,” etc.) beats working shoulder to shoulder with other developers as a mechanism for creating a shared understanding of the project strategy.  By making every developer be involved or at least exposed to the thinking behind the design, they’ll have much more contextual information about the design.

We have a great policy of doing internal brown bag presentations at our development office and it’s giving our guys a lot more experience in presentation skills and opportunities to learn about new tools and techniques along the way. While I’d like to see that continue and I like learning about programming languages we don’t yet use, I also think our teams should probably speak about their own work much more often to try to create better shared understanding of the architectures and code structures that they use every day.

Part of any explanation or knowledge sharing about your system’s architecture, structure, and build processes needs to include a discussion of why you’ve chosen or arrived at that architecture. One unpleasant fact I’ve discovered over and over again is that the more detailed instructions you give to another developer, the worse the code is that comes back to you.  I simply can’t do the thinking for someone else, especially if I’m trying to do all the thinking upfront independently of the feedback you’d naturally get from working through the problem at hand.  If the developer doing the work understands the “why” of your design or instructions, they’ll often do a better job and make improvements as they go — and that’s especially important if you gave out instructions that turn out to be the wrong approach and they should really do something different altogether.

For example, I’ve spent the last couple years learning where some of the more rarely used cooking utensils and gadgets in our kitchen go. Instead of trying to memorize where each thing I find unloading the dishwasher goes, my wife has explained her rationale for how the kitchen is organized and I’m somewhat able to get things put up in the right place now. Organizing code isn’t all that different.

 

Improving Our Code vs. Defensiveness about My Code

Don’t for one second discount the psychological advantages of pair programming or even just a more collaborative coding process.  Formal or even just peer code reviews can be nasty affairs.  They can often amount to a divide between the prosecution and the accused.  In my admittedly limited experience, they’ve been largely blown off or devolve into meaningless compliance checks with coding style standards.  Even worse is the fact that they are generally used as a gating process just prior to the next stage of the waterfall, eliminating the usefulness because it’s too late to make any kind of big change.

The collective ownership achieved with pair programming can turn this situation on its head.  My peers and I can now start to talk about how to improve our code instead of being defensive or sensitive to criticism about my code.  Since we’ve all got visibility now into the majority of the code, we can have informed conversations about the technical direction overall.  The in depth code review happens in real time, so problems are caught sooner.  Add in the shifting of different coders through different areas of the code and you end up with more eyes on any important piece of code. The ability to be self-critical about existing code, without feeling defensive, helps to continuously improve the system design and code.  I think this is one of the primary ways in which agile development can lead to a better, more pleasant workplace.

Adventures in Custom Testing Infrastructure

tl;dr: Sometimes the overhead of writing custom testing infrastructure can lead to easier development

 

Quick Feedback Cycles are Key

It’d be nice if someday I could write all my code perfectly in both structure and function the first time through, but for now I have to rely on feedback mechanisms to tell me when the code isn’t working correctly. That being said, I feel the most productive when I have the tightest feedback cycle between making a change in code and knowing how it’s actually working — and by “quick” I mean both the time it takes for me to setup the feedback cycle and how long the feedback cycle itself takes.

While I definitely like using quick twitch feedback tools like REPL’s or auto-reloading/refreshing web tools like our own fubu run or Mimosa.js’s “watch” command, my primary feedback mechanism for code centric tasks is usually automated tests. That being said, it helps when the tests are mechanically easy to write and run quickly enough that you can get into a nice “red/green/refactor” cycle. For whatever reasons, I’ve hit several problem domains in the last couple years where it was laborious in my time to set up the preconditions and testing inputs and also to measure and assert on the expected outcomes.

 

Maybe Invest in Some Custom Testing Infrastructure?

In some cases I knew right away that testing a feature was going to be a problem, so I started by asking myself “how do I wish I could express the test setup and assertions.” If it seems feasible, I’ll write custom ObjectMother if that’s possible or Test Data Builder‘s for the data setup in more complex cases. I’ve occasionally resorted to building little interpreters that read text and create data structures or files (I do this more often for hierarchical data than anything else I think) or perform assertions on the final state.

You can see an example of this in my old Storyteller2 codebase. Storyteller is a tool for automated acceptance tests and includes a tree view pane in the UI with the inevitable hierarchy of tests organized by suites in an n-deep hierarchy like:

Top Level Suite
  - Suite 1
    -Suite 2
    -Suite 3
      - Test 1
      - Test 2

In the course of building the Storyteller client, I needed to write a series of tests on the tree view state that had to start with a known hierarchy of suites and test files as inputs. After performing actions like filtering or receiving state updates within the UI, I needed to assert on the expected display in this test explorer pane (which tests and suites were visible and were they marked as running, failed, successful, or unknown).

First, to deal with the setup of the hierarchical data I created a little custom class that read flat text data and turned that into the desired hierarchy:

            hierarchy =
                StoryTeller.Testing.DataMother.BuildHierarchy(
                    @"
t1,Success
t2,Failure
t3,Success
s1/t4,Success
s1/t5,Success
s1/t6,Failure
s1/s2/t7,Success
s1/s2/t7,Success
");

Then in the “assertion” part of the test I created a custom specification class that could again read its expectations expressed as flat text and assert that the resulting tree view exactly matched the specified state:

        [Test]
        public void the_child_nodes_are_constructed_with_the_empty_suite()
        {
            var spec =
                new TreeNodeSpecification(
                    @"
suite:Empty
suite:s1
test:s1/t4
test:s1/t5
test:s1/t6
test:t1
test:t2
test:t3
");

            spec.AssertMatch(view.TestNode);
        }

As I recall, writing the simple text parsing classes just to make the expression of the automated tests made it pretty easy to add new behavior quickly. In this case, the time investment upfront for the custom testing infrastructure paid off.

 

FubuMVC’s View Engine Support

A couple months ago I finally got to carve off some time to finally go overhaul the view engine support code in FubuMVC. My main goals were to cut the unnecessarily complex internal code down to something more manageable as a precursor to optimizing both runtime performance and FubuMVC’s time to initialize an application. Since I was about to start monkeying around quite a bit with the internals of code that many of our users depend on, it’s a good thing that we had an existing suite of integration tests that acted as acceptance tests (think layouts, partials, HTML helpers, and our conventional attachment of views to routes) so that in theory I could safely make the restructuring changes without breaking existing behavior.

Going in though, I knew that there was some significant drawbacks to using our existing mechanism for testing the view engine support and I wasn’t looking forward to the inevitable test failures or formulating new integration tests.

 

Problems with the Existing Test Suite

In order to write end to end tests against the view engine support we had been effectively writing little mini FubuMVC applications inside our integration test libraries. Quite naturally, that often meant adding several view files and folders to simulate all the different permutations for layout rendering, using partials, sharing views from external Bottles (a superset of Area’s for you ASP.Net MVC folks), and view profiles (mobile vs. desktop for example). In the test fixtures we would spin up a FubuMVC application with Katana, run HTTP requests, and make assertions against the content that should or should not be present in the HTTP response body.

It wasn’t terrible, but it came with a serious drawbacks:

  1. It wasn’t complete and I’d need to add additional tests
  2. It was expensive in mechanical effort to create those little mini FubuMVC applications that had to be spread over so many different files and even folders
  3. Understanding the tests when something went wrong could be difficult because the expression of the test was effectively split over so many files

 

The New Approach

Before going too far into the code changes against the view engine support, I built a new test harness that would allow me to express in one testing class file:

  1. What all the views and layouts were in the entire system including the content of the views
  2. What the views were in external Bottles loaded into the application
  3. If necessary, configure a complete FubuMVC application if the defaults weren’t sufficient for the test
  4. Declare what content should and should not be rendered when certain routes were executed

The end result was a base class I called ViewIntegrationContext. Mechanically, I made TestFixture classes deriving from this abstract class. In the constructor function of the test fixture classes I would specify the location, content, and view model of any number of Spark or Razor views. When the test fixture class was first executed, it would:

  1. Create a brand new folder using a guid as the name to host the new “application” to avoid collisions with existing test runs (while the new test harness does try to clean up after itself, I’ve learned not to be very trusting of the file system during automated tests)
  2. Write out the Spark and Razor files based on the data specified in the constructor function to the new application folder
  3. Optionally load content Bottles and FubuMVC configurations inside the test harness (ignore that for now if you would, but it was a huge win for me)
  4. Load a new FubuMVC application in memory with the root directory pointing to our new folder for just this test

For each test, the ViewIntegrationContext object uses FubuMVC 2.0’s brand new in memory test harness (somewhat inspired by PlaySpecification from Scala) to execute a “Scenario” where I could declaratively specify what url to render and assert what content should or should not be present in the HTML output.

To make this concrete, the very simplest test to check that FubuMVC really can render a Spark view looks like this:

    [TestFixture]
    public class Simple_rendering : ViewIntegrationContext
    {
        public Simple_rendering()
        {
            SparkView<BreatheViewModel>("Breathe")
                .Write(@"
<p>This is real output</p>
<h2>${Model.Text}</h2>");
        }

        [Test]
        public void can_render()
        {
            Scenario.Get.Input(new AirInputModel{TakeABreath = true});
            Scenario.ContentShouldContain("<h2>Breathe in!</h2>");
        }
    }

    public class AirEndpoint
    {
        public AirViewModel TakeABreath(AirRequest request)
        {
            return new AirViewModel { Text = "Take a {0} breath?".ToFormat(request.Type) };
        }

        public BreatheViewModel get_breathe_TakeABreath(AirInputModel model)
        {
            var result = model.TakeABreath
                ? new BreatheViewModel { Text = "Breathe in!" }
                : new BreatheViewModel { Text = "Exhale!" };

            return result;
        }
    }

    public class AirRequest
    {
        public AirRequest()
        {
            Type = "deep";
        }

        public string Type { get; set; }
    }

    public class AirInputModel
    {
        public bool TakeABreath { get; set; }
    }

    public class AirViewModel
    {
        public string Text { get; set; }
    }

    public class BreatheViewModel : AirViewModel
    {

    }

 

So did this payoff? Heck yeah it did, especially for scenarios where I needed to build out multiple views and layouts. The biggest win for me was that the tests were completely self-contained instead of spread out over so many files and folders. Even better yet, the new in memory Scenario support in FubuMVC made the actual tests very declarative with decently descriptive failure messages.

 

It’s Not All Rainbows and Unicorns

I cherry picked some examples that I felt went well, but there have been some other times when I’ve gone down a rabbit hole of building custom testing infrastructure only to see it be a giant boondoggle. There’s a definite bit of overhead to writing this kind of tooling and you always have to consider whether you’ll save time in the whole compared to writing more crude or repetitive testing code. While I tend to be aggressive about building custom test harnesses, you might accurately call it a speculative exercise and hold off until you feel some pain in your testing.

Moreover, any kind of custom test harness where you decouple the expression of the test (inputs, actions, and assertions) from the actual code that’s being exercised obfuscates your traceability back to the actual code. I’ve seen plenty of cases where the “goodness” of making the expression of the test prettier and more declarative was more than offset by how hard it was to debug test failures because of the extra mental overhead of connecting the meaning of the test to the code that should be implementing it. It’s for that reason that I’ve never been a big fan of most Behavior Driven Development tools for testing that isn’t customer facing.

 

 

 

Follow

Get every new post delivered to your Inbox.

Join 29 other followers