Transaction Scoping in FubuMVC with RavenDb and StructureMap

TL;DR – We built unit of work and transactional boundary mechanics into our RavenDb integration with FubuMVC (with StructureMap supplying the object scoping) that I’m still happy with and I’d consider using the same conceptual design again in the future.

Why write this now long after giving up on FubuMVC as an OSS project? We still use FubuMVC very heavily at work, I’m committed to supporting StructureMap as long as it’s relevant, and RavenDb is certainly going strong. I’ve seen other people trying to accomplish the same kind of IoC integration and object scoping that I’m showing here on other frameworks and this might be useful to those folks. Mostly though, this post might benefit some of our internal teams and almost every enterprise software project involves some kind of transaction management.

Transaction Boundaries

If you’re working on any kind of software system that persists state, you probably need to be consciously aware of the ACID rules (AtomicityConsistencyIsolationDurability) to determine where your transaction boundaries should be. Roughly speaking, you want to prevent any chance of a system getting into an invalid state because some database changes succeeded while other database changes within the same logical operation fail. The canonical example is transfering money between two different bank accounts. For that use case, you’re probably making at least three different changes to the underlying database:

  1. Subtract the amount of the transfer from the first account
  2. Add the amount of the transfer to the second account
  3. Write some sort of log record for the transfer

All three database operations above need to succeed together, or all three should be rolled back to prevent your system from getting into an invalid state. Failing to manage your transaction boundaries could result in the bank customer either losing some of their funds or the bank magically giving more money to the consumer. Following the ACID rules, we would make sure that all three operations are within the same database transaction so that they all have to succeed or fail together.

Using event sourcing and eventual consistency flips this on its head a little bit. In that case you might just capture the log record, then rebuild the “read side” views for the two accounts asynchronously to reflect the transfer. Needless to say, that can easily generate a different set of problems when there’s any kind of lag in the “eventual.”

The Unit of Work Pattern

Historically, one of the easier ways to collect and organize database operations into transactions has been the Unit of Work pattern that’s now largely built into most every persistence framework. In RavenDb, you use the IDocumentSession service as the logical unit of work. Using RavenDb, we can just collect all the documents that need to be persisted within a single transaction by calling the Delete() or Store() methods. When we’re ready to commit the entire unit of work as a transaction, we just call the SaveChanges() method to execute all our batched up persistence operations.

One of the big advantages to using the unit of work pattern is the ability to easily coordinate transaction boundaries across multiple unrelated classes, services, or functions — as long as each is working against the same unit of work object. The challenge then is to control the object lifecycle of the RavenDb IDocumentSession object in the lifecycle of a logical operation. While you can certainly use some sort of transaction controller (little ‘c’) to manage the lifecyle of an IDocumentSession object and coordinate its usage within the actual workers, we built IDocumentSession lifecycle handling directly into our FubuMVC.RavenDb library so that you can treat an HTTP POST, PUT, or DELETE in FubuMVC or the handling of a single message in the FubuTransportation service bus as a logical unit of work and simply let the framework itself do all your transaction boundary kind of bookkeeping for you.

IDocumentSession Lifecycle Management 

First off, FubuMVC utilizes the StructureMap Nested Container feature to create an isolated object scope per HTTP request — meaning that any type registered with the default lifecyle that is resolved by StructureMap within a request (or message) will be the same object instance. In the case of IDocumentSession, this scoping means that every concrete type built by StructureMap will have the same shared instance of IDocumentSession.

To make it concrete, let’s say that you have these three classes that all take in an IDocumentSession as a constructor argument:

    public class RavenUsingEndpoint
    {
        private readonly IDocumentSession _session;
        private readonly Worker1 _worker1;
        private readonly Lazy<Worker2> _worker2;

        public RavenUsingEndpoint(IDocumentSession session, Worker1 worker1, Lazy<Worker2> worker2)
        {
            _session = session;
            _worker1 = worker1;
            _worker2 = worker2;
        }

        public string post_session_is_the_same()
        {
            _session.ShouldBeTheSameAs(_worker1.Session);
            _session.ShouldBeTheSameAs(_worker2.Value.Session);

            return "All good!";
        }
    }

    public class Worker1
    {
        public IDocumentSession Session { get; set; }

        public Worker1(IDocumentSession session)
        {
            Session = session;
        }
    }

    public class Worker2 : Worker1
    {
        public Worker2(IDocumentSession session) : base(session)
        {
        }
    }

When a FubuMVC request executes that uses the RavenUsingEndpoint.post_session_is_the_same() method, FubuMVC is creating a RavenUsingEndpoint object, a Worker1 object, a Lazy<Worker2> object, and one single IDocumentSession object that is injected both into RavenUsingEndpoint and Worker1. Additionally, if the Lazy<Worker2> is evaluated, a Worker2 object will be created from the active nested container and the exact same IDocumentSession will again be injected into Worker2. With this type of scoping, we can use RavenUsingEndpoint, Worker1, and Worker2 in the same logical transaction without worrying about how we’re passing around IDocumentSession or dealing with any sort of less efficient transaction scoping tied to the thread.

TransactionalBehavior

So step 1, and I’d argue the hardest step, was making sure that the same IDocumentSession object was used by each handler in the request. Step 2 is to execute the transaction.

Utilizing FubuMVC’s Russian Doll model, we insert a new “behavior” into the request handling chain for simple transaction management:

    public class TransactionalBehavior : WrappingBehavior
    {
        private readonly ISessionBoundary _session;

        // ISessionBoundary is a FubuMVC.RavenDb service
        // that just tracks the current IDocumentSession
        public TransactionalBehavior(ISessionBoundary session)
        {
            _session = session;
        }

        protected override void invoke(Action action)
        {
            // The 'action' below is just executing
            // the handlers nested inside this behavior
            action();

            // Saves all changes *if* an IDocumentSession
            // object was created for this request
            _session.SaveChanges();
        }
    }

The sequence of events inside a FubuMVC request using the RavenDb unit of work mechanics would be to:

  1. Determine which chain of behaviors should be executed for the incoming url (routing)
  2. Create a new StructureMap nested container
  3. Using the new nested container, build the entire “chain” of nested behaviors
  4. The TransactionalBehavior starts by delegating to the inner behaviors, one of which would be…
  5. A behavior that invokes the RavenUsingEndpoint.post_session_is_the_same() method
  6. The TransactionBehavior object, assuming there are no exceptions, executes all the queued up persistence actions as a single transaction
  7. At the end of the request, FubuMVC will dispose the nested container for the request, which will also call Dispose() on any IDisposable object created by the nested container during the request, including the IDocumentSession object.

How does the TransactionalBehavior get applied?

In the early days we got slap happy with conventions and the simple inclusion of the FubuMVC.RavenDb library in the applications bin path was enough to add the TransactionBehavior to any route that responded to the PUT, POST, or DELETE http verbs. For FubuMVC 2.0 I made this behavior “opt in” instead of being automatic, so you just had to either include a policy to specify which routes or message handling chains needed the transactional semantics:

    public class NamedEntityRegistry : FubuRegistry
    {
        public NamedEntityRegistry()
        {
            Services(x => x.ReplaceService(new RavenDbSettings { RunInMemory = true}));

            // Applies the TransactionalBehavior to all PUT, POST, DELETE routes
            Policies.Local.Add<TransactionalBehaviorPolicy>();
        }
    }

You could also use attributes for fine grained application if there was a need for that.

Explicit Transaction Boundaries

I fully realize that many people will dislike the API usage with the nested closure below, but it’s worked out well for us in my opinion — especially when it’s rarely used as the exception case rather than the normal every day usage.

All the above stuff is great as long as your within the context of an HTTP request or a FubuTransportation message that represents a single logical transaction, but that leaves plenty of circumstances where you need to explicitly control transaction boundaries. To that end, we’ve used the ITransaction service in FubuPersistence to mark a logical transaction boundary by again executing a logical transaction within the scope of a completely isolated nested container for the operation.

Let’s say you have a simple transaction worker class like this one:

    public class ExplicitWorker
    {
        private readonly IDocumentSession _session;

        public ExplicitWorker(IDocumentSession session)
        {
            _session = session;
        }

        public void DoSomeWork()
        {
            // do something that results in writes
            // to IDocumentSession
        }
    }

To execute the ExplicitWorker.DoSomeWork() method in its own isolated transaction, I would invoke the ITransaction service like this:

// Just spinning up an empty FubuMVC application
// FubuMVC.RavenDb is picked up automatically by the assembly
// being in the bin path
using (var runtime = FubuApplication.DefaultPolicies().StructureMap().Bootstrap())
{
    // Pulling a new instance of the ITransaction service
    // from the main application container
    var transaction = runtime.Factory.Get<ITransaction>();
            
    transaction.Execute<ExplicitWorker>(x => x.DoSomeWork());
}

At runtime, the sequence of actions is:

  1. Create a new StructureMap nested container
  2. Resolve a new instance of ExplicitWorker from that nested container
  3. Execute the Action<ExplicitWorker> passed into the ITransaction against the newly created ExplicitWorker object to execute the logical unit of work
  4. Assuming that the action didn’t throw an exception, submit all changes to RavenDb as a single transaction
  5. Dispose the nested container

Use this strategy when you need to execute transactions outside the scope of FubuMVC HTTP requests or FubuTransportation message handling, or anytime when it’s appropriate to execute multiple transactions for a single HTTP request or FT message.

StructureMap 3.0 is very nearly done (no, seriously)

StructureMap 3.0, the next version of the original IoC/DI Container for .Net is almost done and now is a great time to speak up about any improvements and/or changes you’d like to have in SM3.  You can see a list of previous updates (and a shameful pattern of stopping and starting on my part) here.  To be honest, my primary goal at this moment — and why I’m able to work on this during day job hours this week — is to improve the performance of our FubuMVC or FubuTransportation applications with a secondary goal is to improve StructureMap’s diagnostic ability to explain what’s happening when things go wrong.

Big Changes and Improvements:

  • The exception messages provide contextual information about what StructureMap was trying to do when things went wrong
  • The nested container implementation is vastly improved, much faster, and doesn’t have the massive singleton behavior bug from 2.6.*
  • All old [Obsolete] 2.5 registration syntax has been removed, and there’s been a major effort to enforce consistency throughout the registration API’s
  • The original StructureMap.dll has been broken up into a couple pieces.  The main assembly will be targeting PCL compliance thanks to the diligent efforts of Frank Quednau, and that means that Xml configuration and anything to do with ASP.Net has been devolved into separate assemblies and eventually into different Nuget packages.  This means that StructureMap will theoretically support WP8 and other versions of .Net for the very first time.  God help me.
  • The strong naming has been removed.  My thought is to distribute separate Nuget packages with unsigned versions for sane folks and signed versions for enterprise-y folks
  • Lifecycle (scope) can be set individually on each Instance (stupid limitation left over from the very early days)
  • Constructor selection can be specified per Instance
  • Improved diagnostics, both at runtime and for the container configuration (still in progress)
  • Improved runtime performance, especially for deep object graphs with inline dependencies (i.e., FubuMVC behavior chains)
  • The interception model has been completely redesigned
  • The ancient attribute model for StructureMap configuration has been mostly removed
  • The “Profile” model has been much improved

What’s Next?

You can take the pre-release builds of StructureMap 3.0 out for a spin at any time from the fubu TeamCity Nuget feed at http://build.fubu-project.org/guestAuth/app/nuget/v1/FeedService.svc.  A public push could come as early as February 1st, 2014, but I’m not pushing to the public Nuget feed until the stuff in the next paragraph is done.  My thought is that the initial release will be the core StructureMap package, StructureMap.AutoMocking, and StructureMap.AutoFactory.  The new Xml configuration package and a new ASP.Net support package will come later when and if there’s a demand for that.

The issue list is getting shorter and more specific, so I’m hopeful that development is almost to a close.  I’m adding a lot of new explanatory acceptance tests as I write the new documentation (with FubuDocs!).  Frank is going to push through the PCL compliance and that’ll inevitably lead to some new complexity in how we build and create the Nuget’s in our CI builds.

I’m also going to take the new bits out for a spin with a new FubuMVC application and use that to test out what the new exception messages and diagnostics look like.  The forthcoming FubuMVC.StructureMap3 package will embed new diagnostic capabilities.

Early next week, I’m going to try to use StructureMap 3 in a bigger application at Extend Health with an eye toward measuring the new performance versus 2.6.3.

Kicking off StructureMap 3

Actually, I started working hard on StructureMap 3.0 in the summer of 2010 but got badly derailed by other projects and a nasty bout of burnout.  I’m writing this post because I would dearly love to get community input and contributions and I’ve got folks contacting me that are chomping at the bit to start working on this. 

StructureMap was originally written in the summer of 2003 and revamped in the spring of 2004 for its very first release in June of that year.  Over the years it has had some significant rework (the 2.5 and 2.6 releases were both large changes), but at this point I firmly believe that the current 2.6.* internal structure is not worth improving.  Yes Virginia, I am opting to gut some of the internals of StructureMap in order to fix the most egregious problems and limitations of the current architecture and build a container that is good enough to last until we all give up on this silly .Net thing.  I’d also like to tear out any feature that I think is obsolete or just plain ugly to use and make StructureMap much leaner.

Nothing here is set in stone and feedback is very welcome.

 

My thoughts for 3.0:

My personal drivers for doing StructureMap3 are mostly to kill the nested container problems and get StructureMap ready to better handle multi-tenancy scenarios in a high volume FubuMVC application. I think that better Profile’s and/or the child container feature below would make multi-tenancy easier to achieve without killing the server’s memory usage.  Well, and I would like to make StructureMap easier to use for other peopleWinking smile  Making StructureMap the most used container in .Net or competing with the other hundred container tools to do every possible crazy scenario that folks can come up with is not on my agenda.

  • Remove the [Obsolete] methods
  • Better exception messages.  The error messages and the stacktraces really took a step backwards when I replaced the old Reflection.Emit code with dynamically generated Expression’s in the 2.6 release.  At a bare minimum, the stacktrace and exception messages need to be much cleaner and more accurately present what has gone wrong.
  • Better configuration diagnostics.  Completely taking a page out of FubuMVC and Bottles, I would like a StructureMap container to be able to tell you why and how it was configured the way it is.  Why did it select this constructure, why is this the default, where did this type come from.
  • Configuration model.  Today there is the configuration model (PluginGraph) and a runtime model (PipelineGraph and InstanceManager).  I would like to eliminate the separate models and make the configuration model much easier to consume by users.  From the lessons we learned with FubuMVC, I think the key to making the convention model far better is a very good semantic model that can be easily altered and read by both conventions and explicit configuration.  I think this is going to be the biggest change in the internals.
  • Far better convention support.  See the above feature.  Think of policies like “set the value of each constructor argument named ‘connectionString’ to this” or “make any Instance the singleton lifecycle where the concrete class name ends with ‘Cache’.”  We can do that kind of thing today with FubuMVC’s BehaviorGraph model.  I’d like to do the same with StructureMap.
  • Profiles.  I think we just flat our redo Profile’s from scratch and completely redesign that functionality from all new requirements. 
  • Runtime flexibility.  I would like to be able to allow users to register policies that could “teach” StructureMap how to resolve a requested type that it doesn’t know anything about.  I think we’d convert some of the hard coded rules in current StructureMap to this new pluggable strategy.  Think things like “this is a concrete type and I can resolve all of its dependencies, so I’ll just do it” or “this type closes an open generic type that I do know about” or “the name of this class ends in ‘Settings’ so I’ll use FubuCore’s ISettingsProvider to resolve it”
  • Better Lifecycle support.  A longtime limitation in StructureMap is that lifecycle can only be configured by the requested type, i.e., all instances of ISomething have to be the same lifecycle.  I’d like to eliminate that limitation.
  • Better support for modular configuration.  We already have the Registry model and I think it has worked out very well.  Most of the other IoC containers have implemented something similar by this point.  I’d like to extend the model to allow you to specify ordering rules between Registry classes and dependencies (hence, FubuCore’s dependency analysis functionality).  I would also like to add semantics to only add configuration if it is missing or conditional configuration.
  • Pluggable strategies for selecting constructor functions.  I don’t care for this one bit, but at least a couple prominent .Net OSS frameworks need this.
  • Nested containers.  I love this feature and its usability.  FubuMVC depends very heavily on this feature in StructureMap.  Its implementation, however, is horrific and there’s a nasty outstanding bug that I felt was too difficult to fix in 2.6.*.  I think we rewrite the nested container feature so that we have proper separation in scoping between the parent and nested container and avoid the need to do any copying/shuffling of the underlying configuration structure.
  • Child containers.  Not quite the same thing as nested containers.  This would be the ability to quickly clone an existing container and override the parent’s configuration.   
  • Eliminate the Xml configuration.  I have already ripped the Xml configuration support out of the core assembly in StructureMap 3.  I wouldn’t mind coming back and adding a subset of the existing Xml configuration back as an addon assembly and nuget.
  • Eliminate the old attribute configuration.  I had left this in there for years, but I’d never recommend to anyone that they use it.  I would like to consider just using the convention support to work against a subset of the same CLR attributes that MEF uses.
  • Full, living documentation.  I rewrote the documentation for the 2.5 release, but it wasn’t usable enough and quickly got out of date when 2.6 was released.  For 3.0 I’d like to use Sphinx for the documentation generation and host on http://readthedocs.org/ and make the documentation publish with pushes to Git.  Another heavy lesson learned is that the we need to strive to make the documentation organized around the tasks that a user would do instead of organized around StructureMap jargon.
  • Recipes.  This is where I really need community help the most.  I’d like to have some examples of integrating StructureMap into common .Net tools and frameworks.  I’m at a disadvantage because I’ve become very disconnected from mainstream .Net.  I have not used Entity Framework, WCF, Silverlight, Workflow Foundation, MEF, and barely used WPF, ASP.Net MVC, Prism, or WebForms.  I just don’t have enough visibility into those tools to help much.
  • Backwards Compatibility.  With a few exceptions, I think the registry DSL in StructureMap has settled into something usable.  I’d like to remove all the [Obsolete] methods and change a few things that seem to be confusing to use, but otherwise make it as easy as possible to upgrade from 2.6.* to 3.0.
  • No Silverlight support.  I have no intention of supporting Silverlight or any other mobile variant of .Net at this time.  I’m open to this happening later and I’m contemplating at least a version of StructureMap that is usable in the client profile.  This is an important decision to make soon-ish because I would like for StructureMap 3 to take a dependency on our FubuCore library and I don’t really want to care about the size of the assembly right now.
  • Fubu project portfolio.  I would like to fold StructureMap under the Fubu project family.  Part of that is branding, but it’s also community, the convenience of the GitHub organization model, and the common infrastructure that’s starting to grow up around documentation, Ripple, and the build support.
  • Use FubuCore.  There is quite a bit of overlap between the FubuCore library and what’s in the current codebase that I’d like to eliminate.  I’d also like to use FubuCore’s dependency graph support, the ObjectConverter, and integrate the SettingsProvider service out of the box for externalized configuration in StructureMap (here’s an explanation of an earlier version that’s still relevant).

 

Things I don’t plan on changing

  • The interception model.  By and large I think it’s been good enough for anything I’ve ever needed to do with it
  • The basic Registry DSL
  • Any of the methods on IContainer
  • Still don’t plan on adding AOP, but I’d like to have addon Nuget libraries for integrating existing AOP solutions with StructureMap someday

 

 

 

So why do this at all?

A couple months back I expressed some admiration for where one of the other IoC containers was going and that I was perfectly willing to forgo trying to compete with that particular tool (don’t even ask which one because I took a harder look at it and changed my mind).  He asked me why I didn’t just contribute the stuff I needed from StructureMap that was missing in that other container.  Fair question, but no, I’m not going to do that.  Why?  Because one of the best learning experiences you can possibly have as a developer is to take a hard problem that you’ve already solved and reflect on how you could solve that same problem in a much better way with all the things you’ve learned.  I’ve worked on and off on StructureMap for 8-9 years. I’ve rewritten some of the same subsystems in StructureMap a couple different times and even got a conference talk out of my experience called The Joys and Pains of a Long Lived Codebase at QCon – but I still think I will learn a great deal by going through with one last version of StructureMap.