TL;DR We’re able to utilize RavenDb‘s support for embedded databases, some IoC trickery, and our FubuMVC.RavenDb library to make automated testing far simpler by quickly spinning up a brand new database for each individual automated test to have complete control over the state of our system. Oh, and removing ASP.Net and relational databases out of the equation makes automated functional testing far easier too.
Known inputs and expected outcomes is the mantra of successful automated testing. This is generally pretty simple with unit tests and more granular integration tests, but sooner or later you’re going to want to exercise your application stack with a persistent database. You cannot sustain your sanity, much less be successful, while doing automated testing if you cannot easily put your system in a known state before you try to exercise the system. Stateful elements of your application architecture includes things like queues, the file system, and in memory caches, but for this post I’m only concerned with controlling the state of the application database.
On my last several projects we’ve used some sort of common test setup action to roll back our database to a near empty state before a test adds the exact data to the database that it needs as part of the test execution (the “arrange” part of arrange, act, and assert). You can read more about the ugly stuff I’ve tried in the past at the bottom of this post, but I think we’ve finally arrived at a solution for this problem that I think is succeeding.
Our Solution
First, we’re using RavenDb as a schema-less document database. We also use StructureMap to compose the services in our system, and RavenDb’s IDocumentStore is built and scoped as a singleton. In functional testing scenarios, we run our entire application (FubuMVC website hosted with an embedded web server, RavenDb, our backend service) in the same AppDomain as our testing harness, so it’s very simple for us to directly alter the state of the application. Before each test, we:
- Eject and dispose any preexisting instance of IDocumentStore from our main StructureMap container
- Replace the default registration of IDocumentStore with a new, completely empty instance of RavenDb’s EmbeddedDocumentStore
- Write a little bit of initial state into the new database (a couple pre-canned logins and tenants).
- Continue to the rest of the test that will generally start by adding test specific data using our normal repository classes helpfully composed by StructureMap to use the new embedded database
I’m very happy with this solution for a couple different reasons. First, it’s lightning fast compared with other mechanics I’ve used and describe at the bottom of this post. Secondly, using a schema-less database means that we don’t have much maintenance work to do to keep this database cleansing mechanism up to date with new additions to our persistent domain model and event store — and I think this is a significant source of friction when testing against relational databases.
Show me some code!
I won’t get into too much detail, but we use StoryTeller2 as our test harness for functional testing. The “arrange” part of any of our functional tests gets expressed like this taken from one of our tests for our multi-tenancy support:
---------------------------------------------- |If the system state is | |The users are | |Username |Password |Clients | |User1 |Password1|ClientA, ClientB, ClientC| |User2 |Password2|ClientA, ClientB | ----------------------------------------------
In the test expressed above, the only state in the system is exactly what I put into the “arrange” section of the test itself. The “If the system state is” DSL is implemented by a Fixture class that runs this little bit of code in its setup:
- public override void SetUp(ITestContext context)
- {
- // There’s a bit more than this going on here, but the service below
- // is part of our FubuPersistence library as a testing hook to
- // wipe the slate clean in a running application
- _reset = Retrieve<ICompleteReset>();
- _reset.ResetState();
- }
As long as my team is using our “If the system state is” fixture to setup the testing state, the application database will be set back to a known state before every single test run — making the automated tests far more reliable than other mechanisms I’ve used in the past.
The ICompleteReset interface originates from the FubuPersistence project that was designed in no small part to make it simpler to completely wipe out the state of your running application. The ResetState() method looks like this:
- public void ResetState()
- {
- // Shutdown any type of background process in the application
- // that is stateful or polling before resetting the database
- _serviceResets.Each(x => {
- trace(“Stopping services with {0}”, x.GetType().Name);
- x.Stop();
- });
- // The call to replace the database
- trace(“Clearing persisted state”);
- _persistence.ClearPersistedState();
- // Load any basic state that has to exist for all tests.
- // I’m thinking that this is nothing but a couple default
- // login credentials and maybe some static lookup list
- // data
- trace(“Loading initial data”);
- _initialState.Load();
- // Restart any and all background processes to run against the newly
- // created database
- _serviceResets.Each(x => {
- trace(“Starting services with {0}”, x.GetType().Name);
- x.Start();
- });
- }
The method _persistence.ClearPersistedState() called above to rollback all persistence is implemented by our RavenDbPersistedState class. That method does this:
- public void ClearPersistedState()
- {
- // _container is the main StructureMap IoC container for the
- // running application. The line below will
- // eject any existing IDocumentStore from the container
- // and dispose it
- _container.Model.For<IDocumentStore>().Default.EjectObject();
- // RavenDbSettings is another class from FubuPersistence
- // that just controls the very intial creation of a
- // RavenDb IDocumentStore object. In this case, we’re
- // overriding the normal project configuration from
- // the App.config with instructions to use an
- // EmbeddedDocumentStore running completely
- // in memory.
- _container.Inject(new RavenDbSettings
- {
- RunInMemory = true
- });
- }
The code above doesn’t necessarily create a new database, but we’ve set ourselves up to use a brand new embedded, in memory database whenever something does request a running database from the StructureMap container. I’m not going to show this code for the sake of brevity, but I think it’s important to note that the RavenDb database construction will use your normal mechanisms for bootstrapping and configuring an IDocumentStore including all the hundred RavenDb switches and pre-canned indices.
All the code shown here is from the FubuPersistence repository on GitHub.
Conclusion
I’m generally happy with this solution. So far, it’s quick in execution and we haven’t required much maintenance as we’ve progressed other than more default data. Hopefully, this solution will be applicable and reusable in future projects out of the box. I would happily recommend a similar approach to other teams.
But, but, but…
If you did read this carefully, I think you’ll find some things to take exception with:
- I’m assuming that you really are able to test functionality with bare minimum data sets to keep the setup work to a minimum and the performance at an acceptable level. This technique isn’t going to be useful for anything involving performance or load testing — but are you really all that concerned about functionality testing when you do that type of testing?
- We’re not running our application in its deployed configuration when we collapse everything down to the same AppDomain. Why I think this is a good idea, the benefits, and how we do it are a topic for another blog post. Promise.
- RavenDb is schema-less and that turns out to make a huge difference in how long it takes to spin up a new database from scratch compared to relational databases. Yes, there may be some pre-canned indices that need to get built up when you spin up the new embedded database, but with an empty database I don’t see that as a show stopper.
Other, less successful ways of controlling state I’ve used in the past
Over the years I’ve done automated testing against persisted databases with varying degrees of frustration. The worst possible thing you can do is to have everybody testing against a shared relational database in the development and testing environments. You either expect the database to be in a certain state at the start of the test, or you ran a stored procedure to set up the tables you wanted to test against. I can’t even begin to tell you how unreliable this turns out to be when more than one person is running tests at the same time and fouling up the test runs. Unfortunately, many shops still try to do this and it’s a significant hurdle to clear when doing automated testing. Yes, you can try to play tricks with transactions to isolate the test data or try to use randomized data, but I’m not a believer in either approach.
Having an isolated relational database per developer, preferably on their own development box, was a marked improvement, but it adds a great deal of overhead to your project automation. Realistically, you need a developer to be able to build out the latest database on the fly from the latest source on their own box. That’s not a deal breaker with modern database migration tools, but it’s still a significant about of work for your team. The bigger problem to me is how you tear down the existing state in a relational database to put it into a known state before running an automated test. You’ve got a couple choices:
- Destroy the schema completely and rebuild it from scratch. Don’t laugh, I’ve seen people do this and the tests were as painfully slow as you can probably imagine. I suppose you could also script the database to rollback to a checkpoint or reattach a backed up copy of the database, but again, I’m never going to recommend that if you have other options.
- Execute a set of commands that wipes most if not all of the data in a database before each test. I’ve done this before, and while it definitely helped create a known state in the system, this strategy performed very poorly and it took quite a bit of work to maintain the “clean database” script as the project progressed. As a project grows, the runtime of your automated test runs becomes very important to keep the feedback loop useful. Slow tests hamper the usefulness of automated testing.
- Selectively clean out and write data to only the tables affected by a test. This is probably much faster performance wise, but I think it will require more coding inside of the testing code to do the one off, set up the state code.
* As an aside, I really suggest keeping the project database data definition language scripts and/or migrations in the same source control system as the code so that it’s very easy to trace the version of the code running against which version of the database schema. The harsh reality in my experience is that the same software engineering rigor we generally use for our application code (source control, unit testing, TDD, continuous integration) is very often missing in regards to the relational database DDL and environment. If you’re a database guy talking to me at a conference, you better have your stuff together on this front before you dare tell me that “my developers can’t be trusted to access my database.”
The Shade Tree Developer 
What also kind of works (but has still a lot of the flaws inherent with SQL testing) is that each test is wrapped in a distributed transaction that rolls back at the end of the test. This way you can still use regular transactions (e.g. in concert with NHibernate), but you can still reset the db to a known state.
It still requires you to build the database upfront at least once, which can already be painfully slow.
That’s one of the strategies I was referring to in the “you could do this, but I wouldn’t recommend that approach” section. I’m not a fan of that strategy and I think it’s marginally useful for kind of mid-range integration tests, but I don’t see that being useful in many scenarios. Think about doing any kind of automated web testing where you’re not able to perform the entire test in one AppDomain / distributed transaction.
Hi there all, here every person is sharing these experience,
so it’s pleasant to read this blog, and I used to pay a visit this web site every day.