Category Archives: Serenity

FubuMVC Lessons Learned — Strong Naming Woes and Workarounds

TL;DR:  .Net isn’t ready for the brave new world of componentization and smaller, more rapid updates quite yet, but I have some suggestions based on the development of FubuMVC.

To recap, I’m writing a series of posts about the FubuMVC community’s technical experiences over the past five or so years after deciding to give up on new development after the shortly forthcoming 2.0 version. So far, I’ve discussed…

  • Our usage of the Russian Doll model and where we fell in the balance between repetitive ceremonial code and using conventions for cleaner code
  • Command line bootstrapping, polyglot programming, and the value in standardizing codebase layout for the sake of tooling (I say yes, .Net community says no)
  • Lots of DevOps woes trying to develop across multiple repositories using Nuget, TeamCity, and our own Ripple tool.

…and a lot of commenters have repeatedly and quite accurately slammed the documentation (that’s shooting fish in a barrel) and got strangely upset over the fact that we used Rake for our own build automation even though FubuMVC itself had no direct coupling to Ruby.

 

Strong Naming is Hamstringing Modularity in .Net

If you follow me on Twitter, you probably know that I hate strong naming in .Net with a passion.  Some of you might be reading this and saying “it only takes a minute to set up strong naming on projects, I don’t see the big deal” and others are saying to yourselves that “gosh, I’ve never had any problem with strong naming, what’s all the teeth gnashing about?”

Consider this all too common occurrence:

  • Your application uses an OSS component called FancyLogging and you’re depending on the latest version 2.1.5.
  • You also use FancyServiceBus that depends on FancyLogging version 2.0.7.
  • You might also have a dependency on FancyIoC with in turn depends on a much older version of FancyLogging 2.0.0.

Assuming that the authors of FancyLogging are following Semantic Versioning (more on this in a later post), you should be able to happily use the latest version of FancyLogging because there are no semantically breaking changes between it and the versions that FancyServiceBus and FancyIoC were compiled against. If these assemblies are strong named however, you just set yourself up for a whole lot of assembly version conflicts because .Net matches on the entire version number. At this point, you’re going to be spending some quality time with the Fusion Log Viewer (you want this in your toolbox anyway).

Strong naming conflicts are a common issue when your dependencies improve or change rapidly, and you upgrade somewhat often, and especially when upstream dependencies like log4net, IoC containers, and Newtonsoft.Json are also used by your upstream dependencies. Right now I think this problem is felt much more by shops that depend more heavily on OSS tools that don’t originate in Redmond, but Microsoft itself is very clearly aiming for a world where .Net itself is much more modular and the new, smaller libraries will release more often. Unless the .Net community addresses the flaws in strong naming and adopts more effective idioms for Nuget packaging, my strong naming conflict versions are about to come to a mainstream .Net shop near you.

 

Strong Naming Woes and Workarounds

While I’ve never had any trouble whatsoever with using it, at one point a couple years ago assembly conflicts with Newtonsoft.Json was my single biggest problem in daily development.  Fortunately, I encounter very little trouble today due to Newtonsoft.Json’s strong naming.  Why was this one library such a huge headache and what can we learn from how myself and the .Net community as a whole alleviated most of the pain?

First off, Newtonsoft.Json was and is strong named.  It’s very commonly used in many of the other libraries that the projects I work with depend on for daily development, chief among them WebDriver and RavenDb.  I also use Newtonsoft.Json in a couple different fubumvc related projects (Bottles and Storyteller2). Newtonsoft.Json has historically been a very active project and releases often. Really due to its own success, Newtonsoft.Json became the poster child for strong naming hell.

Consider this situation as it was in the summer of 2012:

  • We depended upon WebDriver and RavenDb, both of which at that time had an external dependency upon Newtonsoft.Json.
  • WebDriver and RavenDb were both strong named themselves
  • Both WebDriver and RavenDb were releasing quite often and we frequently needed to pull in new versions of these tools to address issues and subsequent versions of these tools often changed their own dependency versions of Newtonsoft.Json
  • Our own Storyteller2 tool we used for end to end testing depended upon Newtonsoft.Json
  • Our Serenity library we used for web testing FubuMVC applications depends upon WebDriver and Storyteller2 and you guessed it, we were frequently improving Serenity itself as we went along
  • We would frequently get strong naming conflicts in our own code by installing the Newtonsoft.Json nuget to an additional project within the same solution and getting a more recent version than the other projects

I spent a lot of time that summer cursing how much time I was wasting just chasing down assembly version conflicts from Newtonsoft.Json and WebDriver and more recently from ManagedEsent. Things got much better by the end of that year though because:

  • WebDriver ilmerge’d Newtonsoft.Json so that it wasn’t exposed externally
  • WebDriver, partially at my urging, ditched strong naming — making it much easier for us
  • RavenDb ilmerge’d Newtonsoft.Json as well
  • We ilmerge’d Newtonsoft.Json into Storyteller2 and everywhere else we took that as a dependency after that
  • Newtonsoft.Json changed their versioning strategy so that they locked the assembly version but let the real Nuget version float within semantically versioned releases. Even though that does a lot to eliminate binding conflicts, I still dislike that strategy because it’s a potentially confusing lie to consumers. The very fact that this is the recommended approach by the Nuget team themselves as the least bad approach is a pretty good indication to me that strong naming needs to be permanently changed inside the CLR itself.
  • ManagedEsent was killing us because certain RavenDb nugets smuggle it in as an assembly reference, conflicting with our own declared dependency on ManagedEsent from within the LightningQueues library. Again, we beat this with ilmerge’ing our dependency on ManageEsent into LightningQueues and problem solved.
  • With Ripple, we were able to enforce solution wide dependency versioning, meaning that when we installed a Nuget to a project in our solution with Ripple it would always try to first use the same Nuget version as the other projects in the solution.  That made a lot of headaches go away fast. The same solution wide versioning applied to Nuget updates with Ripple.

 

My advice for library publishers and consumers

I definitely feel that much of the following list is a series of compromises and workarounds, but such is life:

  • Be cautious consuming any strong named library that revisions often
  • Don’t apply strong naming at all to your published assemblies unless you have to
  • Don’t bundle in secondary assemblies into your Nuget packages that you don’t control — i.e., the loose ManagedEsent assembly in RavenDb packages problem or this issue in GitHub for Ripple
  • Prefer libraries that aren’t strong named if possible (e.g., why we choose NLog over log4net now)
  • Privately ilmerge your dependencies into your libraries if the consumers when possible, which I’ll freely admit is a compromise that can easily cause you other problems later and some clumsiness in your build scripts.  Do make sure that your unit and integration tests run against the ilmerge’d copy of your assembly in continuous integration builds for best results
  • Do the Newtonsoft.Json versioning trick where the assembly version doesn’t change across releases — even though I hate this idea on principle

 

Rip out the Strong Naming?

We never actually did this (yet), but it’s apparently very possible to rip strong naming out of .Net assemblies using a tool like Mono.Cecil. We wanted to steal an idea from Sebastien Lambla to build a feature into Ripple where it would remove the signing out of assemblies as part of the Ripple package restore feature. If I do stay involved in .Net development and the fine folks in Redmond don’t fix strong naming in the next release of .Net, I’ll go back and finally build that feature into Ripple.

My Approach to Strong Naming 

We never signed any of the FubuMVC related assemblies. FubuMVC itself was never a problem because it’s a high level dependency that was only used by the kind of OSS friendly shops that generally don’t care about strong naming. StructureMap on the other hand is a foundational type of library that’s much more frequently used in a larger variety of shops and it had been signed in the releases from (I think) 2.0 in 2007 until the previous 2.6.4 release in 2012. I still decided to tear out the strong naming as part of the big StructureMap 3.0 release with the thinking that I’d support a parallel signed release at some point if there was any demand for strong naming — preferably if the people making those demands for strong naming would be kind enough to submit the pull request for the fancier build automation to support that. I can’t tell you yet if this will work out, and judging from other projects, it won’t.

 

What about Security!  Surely you need Strong Naming!

For all of you saying “I need strong naming for security!”, just remember that many OSS projects commit their keys into source control.  I think that if you really want to certify that a signed assembly represents exactly the code you think it is, you probably need to compile it yourself from a forked and tagged repository that you control.  I think that the signed assemblies as security feature of .Net is very analogous to the checked exceptions feature in the Java language. I.e., something that its proponents think is very important, a source of extra work on the part of users, and a feature that isn’t felt to be important by any other development community.

To balance out my comments here, last week there was a thread on a GitHub issue for OctoKit about whether or not they should sign their released assemblies that generated a lot more pro-assembly signing sentiment than you’ll find here.  Most of the pro-strong naming comments seem to be more about giving users what they want rather than a discussion of whether or not strong naming adds any real value but hey, you’re supposed to make your users happy.

 

What about automatic redirects?

But Jeremy, doesn’t Nuget write the assembly redirects for you?  Our experience was that Nuget didn’t really get the assembly redirects right as often as not and it still required manual intervention– especially in cases where we were using config files that varied from the App.config/Web.config norm.  There is some new automatic redirect functionality in .Net 4.5.1 that should help, but I still think that plenty of issues will leak through and this is just a temporary bandaid until the CLR team makes a more permanent fix. I think what I’m saying about the automatic redirects is that I grew up in Missouri and you’ll just have to show me that it’s going to work.

 

My wish for .Net vNext 

I would like to see the CLR team build Semantic Versioning directly into the CLR assembly binding so that the strong named binding isn’t quite so finicky such that the CLR can happily load version 3.0.3 whenever that’s the present version even though the declared version in other assemblies is 3.0.1 rather than matching on exactly version 3.0.1 only. I’d like to see assembly redirect declarations in config files go away entirely. I think the attempts to build automatic redirects into VS2013 are a decent temporary patch, but the real answer is going to have to come at the CLR level.

 

 

Next time…. 

So the DevOps topics of strong naming, versioning, and continuous integration across multiple repositories is taking a lot more verbiage to cover than I anticipated and long time readers of mine know that I don’t really do “short” very well.  In following posts I’ll talk about why I think Semantic Versioning is so important, more about how Ripple solved some of our Nuget problems, recommendations for how to improve Nuget, and a specific post on doing branch per feature across multiple repositories.

Clean Database per Automated Test Run? Yes, please.

TL;DR We’re able to utilize RavenDb‘s support for embedded databases, some IoC trickery, and our FubuMVC.RavenDb library to make automated testing far simpler by quickly spinning up a brand new database for each individual automated test to have complete control over the state of our system.  Oh, and removing ASP.Net and relational databases out of the equation makes automated functional testing far easier too.

Known inputs and expected outcomes is the mantra of successful automated testing.  This is generally pretty simple with unit tests and more granular integration tests, but sooner or later you’re going to want to exercise your application stack with a persistent database.  You cannot sustain your sanity, much less be successful, while doing automated testing if you cannot easily put your system in a known state before you try to exercise the system.  Stateful elements of your application architecture includes things like queues, the file system, and in memory caches, but for this post I’m only concerned with controlling the state of the application database.

On my last several projects we’ve used some sort of common test setup action to roll back our database to a near empty state before a test adds the exact data to the database that it needs as part of the test execution (the “arrange” part of arrange, act, and assert). You can read more about the ugly stuff I’ve tried in the past at the bottom of this post, but I think we’ve finally arrived at a solution for this problem that I think is succeeding.

Our Solution

First, we’re using RavenDb as a schema-less document database.  We also use StructureMap to compose the services in our system, and RavenDb’s IDocumentStore is built and scoped as a singleton.  In functional testing scenarios, we run our entire application (FubuMVC website hosted with an embedded web server, RavenDb, our backend service) in the same AppDomain as our testing harness, so it’s very simple for us to directly alter the state of the application.  Before each test, we:

  1. Eject and dispose any preexisting instance of IDocumentStore from our main StructureMap container
  2. Replace the default registration of IDocumentStore with a new, completely empty instance of RavenDb’s EmbeddedDocumentStore
  3. Write a little bit of initial state into the new database (a couple pre-canned logins and tenants).
  4. Continue to the rest of the test that will generally start by adding test specific data using our normal repository classes helpfully composed by StructureMap to use the new embedded database

I’m very happy with this solution for a couple different reasons.  First, it’s lightning fast compared with other mechanics I’ve used and describe at the bottom of this post.  Secondly, using a schema-less database means that we don’t have much maintenance work to do to keep this database cleansing mechanism up to date with new additions to our persistent domain model and event store — and I think this is a significant source of friction when testing against relational databases.

Show me some code!

I won’t get into too much detail, but we use StoryTeller2 as our test harness for functional testing.  The “arrange” part of any of our functional tests gets expressed like this taken from one of our tests for our multi-tenancy support:

----------------------------------------------
|If the system state is |

|The users are                                |
|Username |Password |Clients                  |
|User1    |Password1|ClientA, ClientB, ClientC|
|User2    |Password2|ClientA, ClientB         |
----------------------------------------------

In the test expressed above, the only state in the system is exactly what I put into the “arrange” section of the test itself.  The “If the system state is” DSL is implemented by a Fixture class that runs this little bit of code in its setup:

Code Snippet
  1.         public override void SetUp(ITestContext context)
  2.         {
  3.             // There’s a bit more than this going on here, but the service below
  4.             // is part of our FubuPersistence library as a testing hook to
  5.             // wipe the slate clean in a running application
  6.             _reset = Retrieve<ICompleteReset>();
  7.             _reset.ResetState();
  8.         }

As long as my team is using our “If the system state is” fixture to setup the testing state, the application database will be set back to a known state before every single test run — making the automated tests far more reliable than other mechanisms I’ve used in the past.

The ICompleteReset interface originates from the FubuPersistence project that was designed in no small part to make it simpler to completely wipe out the state of your running application.  The ResetState() method looks like this:

Code Snippet
  1.         public void ResetState()
  2.         {
  3.             // Shutdown any type of background process in the application
  4.             // that is stateful or polling before resetting the database
  5.             _serviceResets.Each(x => {
  6.                 trace(“Stopping services with {0}”, x.GetType().Name);
  7.                 x.Stop();
  8.             });
  9.             // The call to replace the database
  10.             trace(“Clearing persisted state”);
  11.             _persistence.ClearPersistedState();
  12.             // Load any basic state that has to exist for all tests.  
  13.             // I’m thinking that this is nothing but a couple default
  14.             // login credentials and maybe some static lookup list
  15.             // data
  16.             trace(“Loading initial data”);
  17.             _initialState.Load();
  18.             // Restart any and all background processes to run against the newly
  19.             // created database
  20.             _serviceResets.Each(x => {
  21.                 trace(“Starting services with {0}”, x.GetType().Name);
  22.                 x.Start();
  23.             });
  24.         }

The method _persistence.ClearPersistedState() called above to rollback all persistence is implemented by our RavenDbPersistedState class.  That method does this:

Code Snippet
  1.         public void ClearPersistedState()
  2.         {
  3.             // _container is the main StructureMap IoC container for the
  4.             // running application.  The line below will
  5.             // eject any existing IDocumentStore from the container
  6.             // and dispose it
  7.             _container.Model.For<IDocumentStore>().Default.EjectObject();
  8.             // RavenDbSettings is another class from FubuPersistence
  9.             // that just controls the very intial creation of a
  10.             // RavenDb IDocumentStore object.  In this case, we’re
  11.             // overriding the normal project configuration from
  12.             // the App.config with instructions to use an
  13.             // EmbeddedDocumentStore running completely
  14.             // in memory.
  15.             _container.Inject(new RavenDbSettings
  16.             {
  17.                 RunInMemory = true
  18.             });
  19.         }

The code above doesn’t necessarily create a new database, but we’ve set ourselves up to use a brand new embedded, in memory database whenever something does request a running database from the StructureMap container.  I’m not going to show this code for the sake of brevity, but I think it’s important to note that the RavenDb database construction will use your normal mechanisms for bootstrapping and configuring an IDocumentStore including all the hundred RavenDb switches and pre-canned indices.

All the code shown here is from the FubuPersistence repository on GitHub.

Conclusion

I’m generally happy with this solution.  So far, it’s quick in execution and we haven’t required much maintenance as we’ve progressed other than more default data.  Hopefully, this solution will be applicable and reusable in future projects out of the box.  I would happily recommend a similar approach to other teams.

But, but, but…

If you did read this carefully, I think you’ll find some things to take exception with:

  1. I’m assuming that you really are able to test functionality with bare minimum data sets to keep the setup work to a minimum and the performance at an acceptable level.  This technique isn’t going to be useful for anything involving performance or load testing — but are you really all that concerned about functionality testing when you do that type of testing?
  2. We’re not running our application in its deployed configuration when we collapse everything down to the same AppDomain.  Why I think this is a good idea, the benefits, and how we do it are a topic for another blog post.  Promise.
  3. RavenDb is schema-less and that turns out to make a huge difference in how long it takes to spin up a new database from scratch compared to relational databases.  Yes, there may be some pre-canned indices that need to get built up when you spin up the new embedded database, but with an empty database I don’t see that as a show stopper.

Other, less successful ways of controlling state I’ve used in the past

Over the years I’ve done automated testing against persisted databases with varying degrees of frustration.  The worst possible thing you can do is to have everybody testing against a shared relational database in the development and testing environments.   You either expect the database to be in a certain state at the start of the test, or you ran a stored procedure to set up the tables you wanted to test against.  I can’t even begin to tell you how unreliable this turns out to be when more than one person is running tests at the same time and fouling up the test runs.  Unfortunately, many shops still try to do this and it’s a significant hurdle to clear when doing automated testing.  Yes, you can try to play tricks with transactions to isolate the test data or try to use randomized data, but I’m not a believer in either approach.

Having an isolated relational database per developer, preferably on their own development box, was a marked improvement, but it adds a great deal of overhead to your project automation.  Realistically, you need a developer to be able to build out the latest database on the fly from the latest source on their own box.  That’s not a deal breaker with modern database migration tools, but it’s still a significant about of work for your team.  The bigger problem to me is how you tear down the existing state in a relational database to put it into a known state before running an automated test.  You’ve got a couple choices:

  1. Destroy the schema completely and rebuild it from scratch.  Don’t laugh, I’ve seen people do this and the tests were as painfully slow as you can probably imagine.  I suppose you could also script the database to rollback to a checkpoint or reattach a backed up copy of the database, but again, I’m never going to recommend that if you have other options.
  2. Execute a set of commands that wipes most if not all of the data in a database before each test.  I’ve done this before, and while it definitely helped create a known state in the system, this strategy performed very poorly and it took quite a bit of work to maintain the “clean database” script as the project progressed.  As a project grows, the runtime of your automated test runs becomes very important to keep the feedback loop useful.  Slow tests hamper the usefulness of automated testing.
  3. Selectively clean out and write data to only the tables affected by a test.  This is probably much faster performance wise, but I think it will require more coding inside of the testing code to do the one off, set up the state code.

* As an aside, I really suggest keeping the project database data definition language scripts and/or migrations in the same source control system as the code so that it’s very easy to trace the version of the code running against which version of the database schema.  The harsh reality in my experience is that the same software engineering rigor we generally use for our application code (source control, unit testing, TDD, continuous integration) is very often missing in regards to the relational database DDL and environment. If you’re a database guy talking to me at a conference, you better have your stuff together on this front before you dare tell me that “my developers can’t be trusted to access my database.”