Skip to content

Some Thoughts on Collective Ownership and Knowledge

This is a mild rewrite of an old blog post of mine from 2005 that I think is still relevant today. While this was originally about pair programming, I’ve tried to rewrite this to be more about working collaboratively in general. 

My shop is having some constructive internal discussions about our development process and practices. While I don’t think that Pair Programming is likely to be a big part of our daily routine, I think we could still use a bigger dose of collective ownership in our various code base’s and less silo-ing of developers. Since I’m a typical introverted developer who doesn’t handle hours of pair programming and I don’t think my attitude is uncommon at all, we need to look for ways to get the same type of shared understanding that you would get from 100% paired programming.

One of my suggestions is at a minimum to stop assigning coding stories to a single developer. Even if they aren’t pair programming all the time, they can still collaborate on the approach and split tasks so they can code in parallel. On one hand this should increase our team’s shared understanding of the codebase, and on the other hand it’ll definitely help us work discrete stories serially to completion instead of having so many half-done stories laying around at any one time.

 

“This is the Way We Do It”

My favorite metaphor for software design these days is a fishing tackle box.  I want a place for everything and everything in its place.  When I need a top water lure, I know exactly where to look.  I put data access code here, business logic there, and hook the two things up like this.  When I’m creating new code I want to organize it along a predictable structure that anyone else on the project will instantly recognize.  When other coders are making changes I want them to follow the same basic organization so I can find and understand their code later.

Each application is a little bit different so the application’s design is always going to vary.  In any agile project you should hit an inflection point in the team’s velocity that I think of as the “This is the Way We Do It” moment.  Things become smoother.  There are fewer surprises.  Story estimates become more consistent and accurate.  When any pair starts a new story, they understand the general pattern of the system structure and can make the mechanical implementation with minimal fuss.

You really want to get to this point as soon as you can.  There are two separate issues to address before you can reach this inflection point:

  1. Determining the pattern and architecture for the system under development
  2. Socializing the design throughout the development team

To the first point, pairing allows you bring to bear the knowledge and experience of everybody on the team to the work at hand.  It’s just not possible for any one developer to understand every technology and design pattern in the world. By having every developer active in the project design, you can often work out a workable approach faster than a solo architect ever could.  On the one project I’ve done with theoretical 100% pairing, we had a couple of developers with a lot of heavy client experience and me with more backend and web development experience.  By pairing together with our disparate knowledge we could rapidly create a workable general design strategy for the system as a whole by bringing a wider skill set to any coding task.

Right now I’m working some with a system that’s being built by two different teams that work in two very different ways with very different levels of understanding of some of the core architecture. That’s really not an ideal situation and it’s causing plenty of grumbling. There’s a couple specific subsystems that generate complaints about their usability. In one case it just needs to be a full redesign, but that really needs to take place with more than one or two people involved. In the other case, the problems might be from the team that didn’t build the subsystem not understanding how it works and why it was built that way — or it might be because the remaining developer who worked on it doesn’t completely understand the problems that the other team is having. Regardless of what the actual problem is, more active collaboration between the teams might help that subsystem be more usable.

If you’re a senior developer or the technical lead, one of your responsibilities is fostering an understanding of the technical direction to the other developers.  Nothing else I’ve ever done as a lead (design sessions, documentation, presentations, “do this,” etc.) beats working shoulder to shoulder with other developers as a mechanism for creating a shared understanding of the project strategy.  By making every developer be involved or at least exposed to the thinking behind the design, they’ll have much more contextual information about the design.

We have a great policy of doing internal brown bag presentations at our development office and it’s giving our guys a lot more experience in presentation skills and opportunities to learn about new tools and techniques along the way. While I’d like to see that continue and I like learning about programming languages we don’t yet use, I also think our teams should probably speak about their own work much more often to try to create better shared understanding of the architectures and code structures that they use every day.

Part of any explanation or knowledge sharing about your system’s architecture, structure, and build processes needs to include a discussion of why you’ve chosen or arrived at that architecture. One unpleasant fact I’ve discovered over and over again is that the more detailed instructions you give to another developer, the worse the code is that comes back to you.  I simply can’t do the thinking for someone else, especially if I’m trying to do all the thinking upfront independently of the feedback you’d naturally get from working through the problem at hand.  If the developer doing the work understands the “why” of your design or instructions, they’ll often do a better job and make improvements as they go — and that’s especially important if you gave out instructions that turn out to be the wrong approach and they should really do something different altogether.

For example, I’ve spent the last couple years learning where some of the more rarely used cooking utensils and gadgets in our kitchen go. Instead of trying to memorize where each thing I find unloading the dishwasher goes, my wife has explained her rationale for how the kitchen is organized and I’m somewhat able to get things put up in the right place now. Organizing code isn’t all that different.

 

Improving Our Code vs. Defensiveness about My Code

Don’t for one second discount the psychological advantages of pair programming or even just a more collaborative coding process.  Formal or even just peer code reviews can be nasty affairs.  They can often amount to a divide between the prosecution and the accused.  In my admittedly limited experience, they’ve been largely blown off or devolve into meaningless compliance checks with coding style standards.  Even worse is the fact that they are generally used as a gating process just prior to the next stage of the waterfall, eliminating the usefulness because it’s too late to make any kind of big change.

The collective ownership achieved with pair programming can turn this situation on its head.  My peers and I can now start to talk about how to improve our code instead of being defensive or sensitive to criticism about my code.  Since we’ve all got visibility now into the majority of the code, we can have informed conversations about the technical direction overall.  The in depth code review happens in real time, so problems are caught sooner.  Add in the shifting of different coders through different areas of the code and you end up with more eyes on any important piece of code. The ability to be self-critical about existing code, without feeling defensive, helps to continuously improve the system design and code.  I think this is one of the primary ways in which agile development can lead to a better, more pleasant workplace.

Adventures in Custom Testing Infrastructure

tl;dr: Sometimes the overhead of writing custom testing infrastructure can lead to easier development

 

Quick Feedback Cycles are Key

It’d be nice if someday I could write all my code perfectly in both structure and function the first time through, but for now I have to rely on feedback mechanisms to tell me when the code isn’t working correctly. That being said, I feel the most productive when I have the tightest feedback cycle between making a change in code and knowing how it’s actually working — and by “quick” I mean both the time it takes for me to setup the feedback cycle and how long the feedback cycle itself takes.

While I definitely like using quick twitch feedback tools like REPL’s or auto-reloading/refreshing web tools like our own fubu run or Mimosa.js’s “watch” command, my primary feedback mechanism for code centric tasks is usually automated tests. That being said, it helps when the tests are mechanically easy to write and run quickly enough that you can get into a nice “red/green/refactor” cycle. For whatever reasons, I’ve hit several problem domains in the last couple years where it was laborious in my time to set up the preconditions and testing inputs and also to measure and assert on the expected outcomes.

 

Maybe Invest in Some Custom Testing Infrastructure?

In some cases I knew right away that testing a feature was going to be a problem, so I started by asking myself “how do I wish I could express the test setup and assertions.” If it seems feasible, I’ll write custom ObjectMother if that’s possible or Test Data Builder‘s for the data setup in more complex cases. I’ve occasionally resorted to building little interpreters that read text and create data structures or files (I do this more often for hierarchical data than anything else I think) or perform assertions on the final state.

You can see an example of this in my old Storyteller2 codebase. Storyteller is a tool for automated acceptance tests and includes a tree view pane in the UI with the inevitable hierarchy of tests organized by suites in an n-deep hierarchy like:

Top Level Suite
  - Suite 1
    -Suite 2
    -Suite 3
      - Test 1
      - Test 2

In the course of building the Storyteller client, I needed to write a series of tests on the tree view state that had to start with a known hierarchy of suites and test files as inputs. After performing actions like filtering or receiving state updates within the UI, I needed to assert on the expected display in this test explorer pane (which tests and suites were visible and were they marked as running, failed, successful, or unknown).

First, to deal with the setup of the hierarchical data I created a little custom class that read flat text data and turned that into the desired hierarchy:

            hierarchy =
                StoryTeller.Testing.DataMother.BuildHierarchy(
                    @"
t1,Success
t2,Failure
t3,Success
s1/t4,Success
s1/t5,Success
s1/t6,Failure
s1/s2/t7,Success
s1/s2/t7,Success
");

Then in the “assertion” part of the test I created a custom specification class that could again read its expectations expressed as flat text and assert that the resulting tree view exactly matched the specified state:

        [Test]
        public void the_child_nodes_are_constructed_with_the_empty_suite()
        {
            var spec =
                new TreeNodeSpecification(
                    @"
suite:Empty
suite:s1
test:s1/t4
test:s1/t5
test:s1/t6
test:t1
test:t2
test:t3
");

            spec.AssertMatch(view.TestNode);
        }

As I recall, writing the simple text parsing classes just to make the expression of the automated tests made it pretty easy to add new behavior quickly. In this case, the time investment upfront for the custom testing infrastructure paid off.

 

FubuMVC’s View Engine Support

A couple months ago I finally got to carve off some time to finally go overhaul the view engine support code in FubuMVC. My main goals were to cut the unnecessarily complex internal code down to something more manageable as a precursor to optimizing both runtime performance and FubuMVC’s time to initialize an application. Since I was about to start monkeying around quite a bit with the internals of code that many of our users depend on, it’s a good thing that we had an existing suite of integration tests that acted as acceptance tests (think layouts, partials, HTML helpers, and our conventional attachment of views to routes) so that in theory I could safely make the restructuring changes without breaking existing behavior.

Going in though, I knew that there was some significant drawbacks to using our existing mechanism for testing the view engine support and I wasn’t looking forward to the inevitable test failures or formulating new integration tests.

 

Problems with the Existing Test Suite

In order to write end to end tests against the view engine support we had been effectively writing little mini FubuMVC applications inside our integration test libraries. Quite naturally, that often meant adding several view files and folders to simulate all the different permutations for layout rendering, using partials, sharing views from external Bottles (a superset of Area’s for you ASP.Net MVC folks), and view profiles (mobile vs. desktop for example). In the test fixtures we would spin up a FubuMVC application with Katana, run HTTP requests, and make assertions against the content that should or should not be present in the HTTP response body.

It wasn’t terrible, but it came with a serious drawbacks:

  1. It wasn’t complete and I’d need to add additional tests
  2. It was expensive in mechanical effort to create those little mini FubuMVC applications that had to be spread over so many different files and even folders
  3. Understanding the tests when something went wrong could be difficult because the expression of the test was effectively split over so many files

 

The New Approach

Before going too far into the code changes against the view engine support, I built a new test harness that would allow me to express in one testing class file:

  1. What all the views and layouts were in the entire system including the content of the views
  2. What the views were in external Bottles loaded into the application
  3. If necessary, configure a complete FubuMVC application if the defaults weren’t sufficient for the test
  4. Declare what content should and should not be rendered when certain routes were executed

The end result was a base class I called ViewIntegrationContext. Mechanically, I made TestFixture classes deriving from this abstract class. In the constructor function of the test fixture classes I would specify the location, content, and view model of any number of Spark or Razor views. When the test fixture class was first executed, it would:

  1. Create a brand new folder using a guid as the name to host the new “application” to avoid collisions with existing test runs (while the new test harness does try to clean up after itself, I’ve learned not to be very trusting of the file system during automated tests)
  2. Write out the Spark and Razor files based on the data specified in the constructor function to the new application folder
  3. Optionally load content Bottles and FubuMVC configurations inside the test harness (ignore that for now if you would, but it was a huge win for me)
  4. Load a new FubuMVC application in memory with the root directory pointing to our new folder for just this test

For each test, the ViewIntegrationContext object uses FubuMVC 2.0′s brand new in memory test harness (somewhat inspired by PlaySpecification from Scala) to execute a “Scenario” where I could declaratively specify what url to render and assert what content should or should not be present in the HTML output.

To make this concrete, the very simplest test to check that FubuMVC really can render a Spark view looks like this:

    [TestFixture]
    public class Simple_rendering : ViewIntegrationContext
    {
        public Simple_rendering()
        {
            SparkView<BreatheViewModel>("Breathe")
                .Write(@"
<p>This is real output</p>
<h2>${Model.Text}</h2>");
        }

        [Test]
        public void can_render()
        {
            Scenario.Get.Input(new AirInputModel{TakeABreath = true});
            Scenario.ContentShouldContain("<h2>Breathe in!</h2>");
        }
    }

    public class AirEndpoint
    {
        public AirViewModel TakeABreath(AirRequest request)
        {
            return new AirViewModel { Text = "Take a {0} breath?".ToFormat(request.Type) };
        }

        public BreatheViewModel get_breathe_TakeABreath(AirInputModel model)
        {
            var result = model.TakeABreath
                ? new BreatheViewModel { Text = "Breathe in!" }
                : new BreatheViewModel { Text = "Exhale!" };

            return result;
        }
    }

    public class AirRequest
    {
        public AirRequest()
        {
            Type = "deep";
        }

        public string Type { get; set; }
    }

    public class AirInputModel
    {
        public bool TakeABreath { get; set; }
    }

    public class AirViewModel
    {
        public string Text { get; set; }
    }

    public class BreatheViewModel : AirViewModel
    {

    }

 

So did this payoff? Heck yeah it did, especially for scenarios where I needed to build out multiple views and layouts. The biggest win for me was that the tests were completely self-contained instead of spread out over so many files and folders. Even better yet, the new in memory Scenario support in FubuMVC made the actual tests very declarative with decently descriptive failure messages.

 

It’s Not All Rainbows and Unicorns

I cherry picked some examples that I felt went well, but there have been some other times when I’ve gone down a rabbit hole of building custom testing infrastructure only to see it be a giant boondoggle. There’s a definite bit of overhead to writing this kind of tooling and you always have to consider whether you’ll save time in the whole compared to writing more crude or repetitive testing code. While I tend to be aggressive about building custom test harnesses, you might accurately call it a speculative exercise and hold off until you feel some pain in your testing.

Moreover, any kind of custom test harness where you decouple the expression of the test (inputs, actions, and assertions) from the actual code that’s being exercised obfuscates your traceability back to the actual code. I’ve seen plenty of cases where the “goodness” of making the expression of the test prettier and more declarative was more than offset by how hard it was to debug test failures because of the extra mental overhead of connecting the meaning of the test to the code that should be implementing it. It’s for that reason that I’ve never been a big fan of most Behavior Driven Development tools for testing that isn’t customer facing.

 

 

 

Why I hate the word “Pragmatic” and other rants

I told myself that when I started my new blog that I’d try not to rant as much as I did in the CodeBetter days, but oh well, here we go anyway:

Software is a young profession and judging from the rate of change and improvement over my 16 years as a software developer, we’re still learning a lot of new lessons — or relearning old lessons for ourselves that some other group of developers already learned years ago. The great part about that ever changing landscape is that constantly learning new tools and techniques is so much more fun than my first job as an engineer where I mostly certified piping system designs with the “boiler code” that dated back to the 1800′s. The downside is that I think that developers as a whole are terrifyingly bad at sharing information and debating or reasoning about the merits of various alternatives — especially if we feel threatened by other developers promoting tools or techniques that we don’t yet know or use.

I want to ban these words and phrases from technical discussions

I’m betting that many of you are familiar with Godwin’s Law:

“As an online discussion grows longer, the probability of a comparison involving Nazis or Hitler approaches 1.”

From my perspective, the importance of Godwin’s Law in practice is that conversations effectively lose all usefulness when highly charged words and hyperbolic comparisons start getting tossed around.  Much like George Carlin’s Seven Dirty Words, I’d like to create a new list of words and phrases that should be banned in technical conversations at least until we learn to use them a little more wisely:

  • Pragmatic — If you follow me on Twitter you surely know that I despise the word pragmatic as it’s usually wielded as a magical talisman by inarticulate developers the world over to win debates and justify whatever random decision they’ve made to other developers. More on this in the next section.
  • Fetish — As in “I don’t get your fetish for static/dynamic typed languages.” What you’re doing is shutting your mind off to the merits of different ideas than you’re own.
  • Zealot — see the above.
  • Dogmatic — Dogmatic is generally used as a pejorative term, but sometimes it’s not. Some techniques are most effective when they’re applied consistently. Again, I think developers mostly use this word to denigrate or delegitimize the differing opinions of other developers.
  • Academic — Denigrating some technique as being more theory than something that’s practical to use. You might be right and my experience in a couple different fields is that there is some real distance between theory and reality, but using the word “academic” to describe a technique you don’t use is a sign of a closed mind. Sample usage: “OWIN is a kernel of a good idea wrapped in a lot of academic masturbation.” I’m admittedly guilty of using this quite frequently in reference to enthusiastic proponents of functional programming.
  • “Why don’t/didn’t you just” — I’m bad about using this on other developers. Sample usage: “instead of unraveling the Gordian Knot painstakingly by hand, why didn’t you just cut through it with the sword laying over there that no one told you about?.” The cruel reality of software development is that frequently the simplest solution involving the least work is not the obvious solution.
  • “It’s obvious” — My ex-wife absolutely hated it when her professors in college used the phrase “it should be obvious to the most casual observer” when their point wasn’t clear at all. It’s just a way of either putting down your co-workers or using loaded language to win an argument. Contributed by my colleague Matt Smith.
  • “Git r’ Done” — Used non-ironically it’s just a cutesy synonym for “being pragmatic.” I should probably ban “git ‘r done” because in my circles it’s code for bad developers that don’t care about any kind of quality as long as the immediate work gets done sorta, kinda on time and mostly “works.”
  • “Use the right tool for the job” — This is an empty calories kind of statement and a cliche. I usually interpret this as “I just want to be left alone to use what I already know and I’m not interested in continuing this conversation”
  • “We just have to get this out the door” — I think there’s a real tension and a necessary balance between making decisions tactically for immediate problems and thinking strategically for the long term. While being an architect astronaut is a sign that you’re too far into the strategic side of things, using the phrase “we just have to get this out the door” is often an indication that your team might be veering way too far in favor of short sighted tactical decisions while missing opportunities to improve their productivity by making changes in process, tools, or a redesign of a bad subsystem. My fear — and my experience backs this up — is that teams that use this phrase and moreover believe it, have shut off their minds and just started to bang harder and faster on their keyboard. My favorite analogy for this is always the saying “I’m too busy chopping trees to stop and sharpen my axe!
  • Shiny Object — I’m guilty of this one. As in, “you just want to use XYZ because it’s the new shiny object”
  • Cargo Cult — All the analogies to cargo cults and teams just going through the motions of doing Agile development were kind of funny a couple years ago, but I think the term “cargo cult” is over used mostly as a way to put down other people at this point. If you’ve been living under a rock and haven’t heard this term, I liked James Shore’s Cargo Cult Agile from back in the day.

If you use any of these words and phrases to another developer, you might have wasted a chance to learn or teach something new.

Good Decision Making, Modeling, and being Pragmatic

Taking the definition from Merriam Webster, being pragmatic is:

dealing with the problems that exist in a specific situation in a reasonable and logical way instead of depending on ideas and theories

I think that developers often conflate being pragmatic with doing things well. Being reasonable and logical sounds like a great way to be successful, but you can be pragmatic until the cows come home and still make bad decisions all day long if you’re using bad information or not accounting for other factors. Making good decisions isn’t so much about being pragmatic, dogmatic or zealous so much as it is about constructing the right mental model that represents more of the important factors that go into the decision.

As I alluded to earlier, my educational and early professional background is engineering. My primary job was often about creating models to accurately simulate the real world in order to validate decisions or optimize designs. Needless to say, the better and more accurate your models are, the better your real results.

I thoroughly enjoyed a special projects class I took my last year of school where we designed thermodynamic systems based on optimizing either cost or capacity. To do the optimization, you had to model the various factors (length of a connecting pipe, depreciation, rate of heat loss, etc.) with equations that could be solved with optimization software to find local minimums or maximums.

 

Since it’s just math, two teams working on the same project should always arrive at the same designs and decisions, right? Right?

What we found in the course of doing these projects was (unsurprisingly) that:

  • The calculated optimal design could be much different if you added or ignored certain factors. Maybe my team accounted for the pressure drop across a mile of pipeline while the other team assumed that it was constant. Maybe the other team made different assumptions than we did.
  • The calculated optimal design could also be much different for shorter timeframes than it was for longer timeframes.

To make this concrete in terms of software development, take the decision about whether or not to write unit tests for any given piece of code:

  • Writing the extra code to automate unit tests takes more time than just writing the production code. A big point against the unit tests.
  • Only end to end or integration tests would be able to definitively determine problems in the code in question. Don’t write the unit test.
  • An end to end test would be very expensive to write. Write unit tests for what you can easily unit test.
  • Writing automated unit tests might force changes to the way you structure your code — which might be a forcing function in favor of better factored code which is good or additional complexity that you don’t want
  • Without unit tests you might not catch problems until later when the problem will be harder to find and fix. A point in favor of unit tests.
  • Using a debugger is an inefficient way to solve problems in code and foregoing unit testing increases the risk of long debugging sessions. Writing automated unit tests frequently reduces the amount of time spent using a debugger.
  • Automated tests of any kind definitely save time in regression tests. A point in favor of writing the tests in the longer term.
  • The existence of automated tests can lead to better reversibility so that you can make changes safely later. A point that might be very important or not at all.

If you base your decision about whether or not to write unit tests on only one of these bullet points and not a combination of many or all of them, your decision making isn’t as likely to optimal in the end. Also, your decision making should probably come to different results on a one week project versus a multi-year project.

Again, making better decisions isn’t just about being “pragmatic,” it’s about creating a mental model that more completely and accurately accounts for the positives and negatives of any given decision and also being cognizant of the timeframe in which you need to optimize. If you make the decision about whether or not  Making decisions pragmatically but with incomplete information or understanding of your problem is still bad.

Going back to the point about being too focused on making incremental deadlines and being too focused on the tactical, I’ve far too frequently seen teams forgo solving some sort of technical debt issue that is slowing them down because they had to make their next iteration. In the very short term, they might be right — but if the time saved over X number of iterations is greater than the time it takes to solve and fix the technical debt issue then you need to stop and fix that damn technical debt problem.

To make that last paragraph more concrete with a real life example from my shop, let’s say that your team is working with a system where each and every new vertical feature requires changes to code in six different git repositories. Would you continue to pound away on new features to maintain your current velocity, or stop for a little bit to combine all the repositories into a single logical repository where it’s much easier to make changes in one place and very likely increase your velocity afterward?

Lullaby Language 

I asked my Twitter followers for more examples of words and phrases we should ban and a large number of replies contained the word “just.”  As in the time a business analyst told me that if we were to use our brand new website application as the front end for a system built by another branch of our company that neither of us had other seen it would “just be some refactoring.”  Using the word “just” is a perfect example of Lullaby Language that we should probably avoid using because it gives us and our listeners a potentially false sense of security.

Another loaded word is “should,” as in “the new build should work” with an unspoken admission that they hadn’t tested the exact scenario in question. I had a very enjoyable co-worker years ago that would react comically to statements like that with “you said ‘should!’” (for my current colleagues, this is the guy that got me started on the finger on the nose gesture you’ve all seen by now).

Other examples from the peanut gallery were:

  • “should be a quick fix/change” — putting pressure on you to turn it around quickly with no real forethought about how big it really is
  • “this is too simple/small for testing/versioning” – famous last words…
  • “The deadline is already set. Can we make it?” — I really enjoyed Jez Humble’s talk at NDC London last year about how we should concentrate more on delivering value instead of just trying to hit deadlines

 

Non Technical Folks and Estimation

I got a lot of other complaints on Twitter that mostly boiled down to project managers and other non-technical folks setting deadlines, time limits, or estimates for developers. I’m a big believer in the idea that developers should do most of the estimation work — and even then, those estimates should be treated as unproven working theories. The only time I think a project manager can get away with doing estimation is when there’s plenty of historical data for the project on similar stories and the composition of the team is stable.

Like many developers in Austin of my generation, I started my development career at a large captive IT shop that was famous for generating truly awesome horror stories. The very last thing I did there was to help kick off a new project to replace an existing web application that our business wasn’t happy with and we felt would be easier to rewrite than improve (it build dynamic HTML inside of VB6 COM objects hosted in ASP classic). Since that very year I had had my bonus slashed because of how far off my estimate was for a previous project, I was appropriately paranoid about getting it right this time around. To that end, I spent a lot of time using use case points (this was so long ago that RUP was still cool) to both define the scope and a first estimate. Because I’ve told this story so many times, I still remember the basic details of the proposed system:

  1. A small new database schema to be what we’d now call the read side model — maybe 2 primary master/detail tables plus the normal contingent of lookup data tables
  2. A pretty complicated web screen with a lot of different user interactions, and this was way before things like jQuery or Angular.js
  3. About a dozen integration points to legacy systems that we had had trouble with in previous projects

When we finally got a project manager I tried to present my use case’s and estimates to the new project manager but I was basically told “don’t worry about it hon, I’ll make the project schedule.”

The next day I got the email with the new Excel schedule attached. As memory serves me, it was an old fashioned waterfall schedule that roughly went:

  • 40 days for logical database design
  • 20 days for physical database design
  • 10 days for the integration work
  • 10 days for the web application

I was apoplectic because it was pretty well exactly opposite of what I came up with in every way and pushed all the technical risk to the very end of the project — but I quit the very next week anyway so it was okay;) The PM wasn’t amused when I drew two boxes and one line on her whiteboard and said “there, the logical database design is done!”

 

 

Final Thoughts on Nuget and Some Initial Impressions on the new KVM

There’s an index now for the FubuMVC Lessons Learned series of blogposts. I fully intend to keep going with more content in this series some day, but this post is going to wrap up my thoughts on DevOps kind of topics like Nuget, Ripple, continuous integration, multi-repository development, and build management. If you read this and say why is Jeremy being so negative about Nuget?, I’m writing this from three different perspectives:

  1. As a consumer of Nuget for complicated dependency trees
  2. As an author of value added tooling (Ripple) on top of Nuget
  3. As just an armchair software architect who enjoys looking at tools like Nuget and thinking about how I’d build that code differently if it was mine

Some of the recommendations I’m making here are either already on the Nuget team’s stated vNext roadmap or something I know they’ve thought about themselves.

My Thoughts on Project K

I’ve been piddling around with the content on this post for so long that in the meantime Microsoft finally publicly announced their Project K work (ASP.Net vNext) to the unwashed masses who aren’t some sort of ASP Insider or MVP. While I’m mostly positive about their general direction with Project K, I really wish the ASP.Net team had been much more transparent with their technical direction. Since most of Project K feels like catching up with other development platforms more than anything original or innovative, I don’t know what they buy for themselves by being so opaque.

The Project K runtime looks like it would have improved the FubuMVC team’s development experience over the current .Net framework and tooling:

  • The K runtime does not include strong naming, which was an almost unending source of unnecessary aggravation for me over the past 3 years or so. Bravo.
  • It looks like the K tooling has some level of support for our Ripple tool’s floating dependency concept.
  • Eliminating the csproj files should go a long way toward reducing the csproj merge hell problem, but I bet that the project.json file still ends up being the worst source of merge conflicts even so. Hopefully they carefully consider how they’re going to integrate Nuget into K development to avoid so much of the friction we had before we wrote Ripple.
  • Again, by eliminating the heavyweight csproj file baggage, it’s going to be much easier to write project and item generation tools like “rails new” or our own “fubu new.” I had to spend an inordinate amount of time last year building out our FubuCsProjFile library for csproj/sln file manipulation and project templating.
  • I’m still going to claim that FubuMVC with Bottles had the best modularity strategy of any .Net web framework, but much of what we did would probably be unnecessary or at least much easier if I’m correctly understanding what the Roslyn compiler is capable of. If it’s really more efficient to forgo distributing assemblies in favor of just letting Roslyn build everything into memory, then all that work we did to smuggle web content through assemblies for “feature bottles” is now unnecessary.
  • In my last post I talked about the auto-reloading/auto-refreshing development web server we built for FubuMVC development. While it sounds like it’s not usable yet, the ASP.Net team looks to be building something similar, but using the Roslyn compiler to recompile on code file changes should be a faster feedback loop than we have with our “fubu run” tool. I’ll be interested to see if they can create an experience comparable to Golang or Node.js in this regard.

Overall, I think that Microsoft has probably rode the Visual Studio.Net horse for too long.  Project K starts the work of making .Net development much more productive with lighter weight tools and better command line friendliness that can only help the community over time.

I’ve been asked a couple times on Twitter if I would consider restarting FubuMVC work after the Project K runtime is usable. My answer is an emphatic no, and moreover, I would have ditched many of our technical efforts around FubuMVC much earlier if I’d known about Project K earlier. If anything, I’d like to start over from scratch when Project K stabilizes a bit, but this time with much less ambitious goals.

 

Suggestions for Nuget in .Net Classic

  • Optionally remove strong naming during package restore. Who knows how long it’ll take to move all server side development to the new K runtime. In the meantime, strong naming is still a borderline disaster. Maybe the Nuget team should consider a function where the Nuget package restore functionality could either strip out all the strong naming of the downloaded nugets or strong name nuget assemblies on the  fly for hosting technologies that require naming. We discussed this functionality quite a bit for Ripple but it’s never gotten done.
  • Make the Nuget server API less chatty and ditch oData. I’m not the world’s foremost expert on software performance, but the very first thing I learned about the subject was to minimize the number of network round trips in your software. The Nuget client makes far too many network round trips because it seems to be treating every single dependency as a separate workflow instead of treating the entire dependency graph as one logical operation. I strongly recommend (and I’ve said this to them in person) that they add documented API alternatives where you can batch up your requirements in one request — and I think they need to go beyond oData to get this done.
  • Make the Nuget server API a standard. The Nuget server API wasn’t great to work with. We usually had to resort to using Fiddler to figure out what the built in clients were doing in order to accomplish things with Ripple. A packaged assembly to be a client to the Nuget server API would be a great way to promote value added tools on top of Nuget.
  • Decouple Nuget.Core from Visual Studio.Net. This is a big one. I think it was a huge mistake to depend on Visual Studio automation to modify the csproj files to make the assembly references. We first used crude Xml manipulation then later the FubuCsProjFile to manipulate csproj files in Ripple. Since we had no hard coupling to Visual Studio, we were able to provide valuable command line support for installing and updating Nuget packages that enabled our continuation integration across repositories approach.
  • Better Command Line Support. Just like Ripple, you should should be able to install, update, and remove nuget dependencies from the projects in your repository without Visual Studio being involved at all.
  • Modularize Nuget.Core. We found the Nuget.Core codebase to be poorly factored. Ripple would have been much easier to build if the Nuget.Core code had been reasonably modular. At a minimum, I’d recommend splitting the Nuget.Core code up in such a way that you could easily reuse the logic for applying a Nuget package to a csproj file all by itself. I’d again recommend pulling out the interaction with the Nuget server API’s. Making Nuget.Core more modular would open up opportunities for community built additions to the Nuget ecosystem.
  • Support Private Dependencies (Somehow). Several folks have mentioned to me on Twitter how Node.js’s NPM is able to isolate the dependencies of your dependencies.   Maybe Nuget could support something like ilrepack to inline assemblies that your dependencies depend on but your application itself doesn’t need otherwise.
  • Ripple Fix. Maybe it’s better now, but what we found in the early days of Nuget is that things could easily go off the rails. One of the features I’m most proud of in Ripple was our “ripple fix” command. What this did was check the declared dependencies of every single project in your repository and make everything right. If assembly references were missing for some reason, add them. If a dependency is missing, go install it. Whatever it takes, just make things work. And no, it wasn’t perfect but it still made using Nuget more reliable for us.
  • (Possibly) Adopt the Actor Model for Nuget Internals. I think that in order for batched Nuget operations across a complicated tree more efficient, Nuget needs to do a far better job of using parallelization to improve performance. Unfortunately, all the operations you do (find the latest of package A, download package B, etc.) are interrelated. The current architecture of Ripple is roughly an old-fashioned pipes and filters architecture. We’ve long considered using some kind of Actor model to better coordinate all the parallel actions and latch the code from performing unnecessary work.

 

 

A Better Development Web Server for .Net with FubuMVC 2.0

tl;dr: FubuMVC 2.0 includes an improved command line development web server for a better development time experience

First, a quick caveat. All the code in this post is part of the forthcoming FubuMVC 2.0 release and has not yet been publicly released and might not be for a couple more months. All the source code shown and referenced here is in the master branch of FubuMVC itself.

 

What’s the Problem?

If you ask me what I think are the best software development practices to come out of the past couple decades, I’d rattle off some things  like TDD/BDD, Continuous Integration, Continuous Delivery, and Iterative Development. If you take a step back and think about these “best practices” you’ll see that there’s a common thread of attempting to create more useful and definitely more rapid feedback cycles. The cruel truth about slinging code around and envisioning new software systems out of whole cloth is that it’s so very easy to be wrong — and that’s why software professionals have spent so much time and energy finding more ways to know when they’re efforts aren’t working and making it easier and less risky to introduce improvements.

Now, apply this idea of rapid feedback cycles to day to day web development. What I really want to do is to shorten the time between saving changes to any part of my web application, be it C# code or CSS/LESS or any kind of JavaScript, and seeing the impact of that change. Granted, I’d strongly prefer to use quick twitch unit tests to remove most of the potential problems in either the server side or client side code in isolation, but there are still plenty of issues where you’re going to need to do some manual testing with the complete application stack.

Enter “fubu run –watched”

Several web development frameworks include development servers that can automatically reload the application and sometimes even refresh a running browser when files in the application are changed. The Play framework in Java/Scala has the Play Console, we’re starting to use Mimosa.js and its watched server mode for pure client side development, and FubuMVC has our “fubu run” command line server that was originally inspired by PlayConsole.

While we’ve had this functionality for quite a while, I just pushed some big improvements yesterday that fixes our auto-refresh infrastructure. The new architecture looks like this:

Slide1

Installing the fubu gem* places the fubu.exe on your box’s PATH, so you can just go straight to the directory that contains your FubuMVC application and type:

fubu run -o --watched

The “o” flag just directs the command to open your default browser to the Url where the application is going to be hosted. While the fubu run process always auto-reloads the application on recompilation, the “watched” flag adds some additional mechanics to refresh the current page in your browser whenever certain files change in your application.

So how does it work?

First off, fubu run has to create a separate AppDomain to host your FubuMVC application. If you’ve done .Net development for any length of time you know that there is no way to unload and replace a loaded AppDomain. By running a separate AppDomain we’re able to tear down and recreate new AppDomain’s when you recompile your application. The other reason to use a separate AppDomain is to make the application run just like it would in a production server, and that means making the new AppDomain be based on the directory of the application instead of wherever the fubu.exe happens to be and making the new AppDomain use the correct web.config file for the application.

The FubuMVC community has some special sauce in our Bottles framework called the RemoteServiceRunner that makes it easier to setup and coordinate multiple AppDomain’s in code (I’ll happily blog about later if anyone wants me to).  As shown in this code,  fubu run loads the second AppDomain based on the right location and copies over any missing assemblies to the application bin path for FubuMVC, Katana, and their dependencies.

The next step is to bootstrap your FubuMVC application in the second AppDomain. As of FubuMVC 1.0, the idiomatic way to describe your application’s bootstrapping is with an implementation of the IApplicationSource interface. If there’s only a single concrete class implementing this interface in all of your application binaries, fubu run is smart enough to use that to bootstrap your application exactly the way it would be (with some development time differences I’ll discuss below). The simplest possible implementation might look like this class:

    public class SimpleApplicationSource : IApplicationSource
    {
        public FubuApplication BuildApplication()
        {
            return FubuApplication
                .DefaultPolicies()
                .StructureMap();
        }
    }

Part of our Bottles infrastructure is an EventAggregator class that was specifically created in order to easily send messages bidirectionally between AppDomain’s opened by our RemoveServiceRunner class. fubu run uses this EventAggregator to send and receive messages to the new AppDomain to start an embedded Katana web server and bootstrap a new FubuMVC application using the IApplicationSource class for your application. Likewise, fubu run waits to hear messages back from the 2nd AppDomain about whether or not the application bootstrapping was successful.

Watching for Changes

If in the “–watched” mode, fubu run starts up a class called FubuMvcApplicationFileWatcher to watch for changes to certain files inside the application directory and call back to an observer interface when file changes trigger certain logical actions:

    public interface IApplicationObserver
    {
        // Refresh the browser
        void RefreshContent();

        // Tear down and reload the entire AppDomain
        void RecycleAppDomain();

        // Restart the FubuMVC application
        // without restarting the 
        // AppDomain
        void RecycleApplication();
    }

To make this concrete, changes in:

  • .spark, .css, .js, or .cshtml files will trigger a refresh of the browser
  • web.config, .dll, or .exe files will cause a full recycling of the AppDomain
  • other *.config file changes will trigger an application recycle

 

Automatically Refreshing the Browser with WebSockets

The last piece of the puzzle is how we’re able to refresh the browser when the server content changes. In the currently released version of the fubu.exe tool I tried to use WebDriver to launch the browser in such a way that it would be easy to control the browser from fubu run without any impact on the html markup and the application itself. Let’s just say that didn’t work very well at all because of how easily WebDriver gets out of sync with rapid browser updates in real life.

For the FubuMVC 2.0 work, I went down a different path and used web sockets to send messages from the original fubu run process directly to the browser. My immediate and obvious goal was to pull this off without forcing any changes whatsoever onto the application’s HTML markup to support the auto-reloading. Instead, I used this work as an opportunity to revamp FubuMVC’s support for using OWIN middleware to use a new bit of custom middleware to squirt in a little bit of HTML markup into an HTML page’s <head> element after FubuMVC had rendered a page, but before the content is sent to the browser. While I’ll leave a discussion for how and why FubuMVC exposes OWIN middleware configuration much differently than other .Net frameworks for another day, it’s good enough to know that FubuMVC is only adding the auto-reloading content when the application detects that it is running inside of fubu run –watched.

On the client side, we just inject this wee bit of javascript code to listen at a supplied web sockets address (%WEB_SOCKET_ADDRESS%) for a message telling the page to refresh:

        var start = function () {
            var wsImpl = window.WebSocket || window.MozWebSocket;

            // create a new websocket and connect
            window.ws = new wsImpl('ws://localhost:%WEB_SOCKET_ADDRESS%');

            // when data is comming from the server, this metod is called
            ws.onmessage = function (evt) {
                location.reload();
            };

        }

	window.addEventListener('load', function(e){
	  start();
	});

Inside the fubu run process I used the lightweight Fleck library to run a web sockets server used to send messages to the browser. The code that uses Fleck is in the BrowserDriver class.

 

Roslyn for the Next Level of Awesomeness…

…or really just achieving parity with other web development platforms that already have a great auto-reload capability.

At this point, fubu run does not try to do the compilation for you based on file changes to *.cs files. I’m still not sure if I think that’s a good idea to do later or not. I’m perfectly okay with the “make changes, hit CTRL-SHIFT-B” workflow triggering a re-compilation which then in turn triggers fubu run to recycle the AppDomain and reload the application. Several folks have kicked around the idea of using the new, much faster Roslyn compiler behind the scenes to do the compilation and try to achieve a more rapid feedback cycle like you’d expect from a Node.js based solution. I think this would be a lot more appealing to me personally as my teams at work continue to break away from using Visual Studio.Net in favor of lightweight editors like Sublime.

 

Getting the Edge Nugets and Fubu.Exe

The edge nugets for FubuMVC 2 are on MyGet at https://www.myget.org/feed/Packages/fubumvc-edge. The fubu.exe gem can be manually downloaded from the artifacts of our TeamCity CI build.

 

 

* In an earlier post I discussed why we used Ruby Gems to distribute .Net executables instead of using Nuget. Long story short, Nuget by itself isn’t all that great for command line tools and Chocolately isn’t cross platform like Gems.

Software Development Horror Stories and Scar Tissue

My best friend (also a developer) and I meet for lunch almost every Friday. As experienced developers often do, we got to swapping horror stories this past Friday. I had so much fun telling some of my favorite stories that I thought I’d repeat some of them here just to break my blogging drought. As I started doing just that, it occurred to me that there’s a deeper theme here about how we process our bad experiences, what we learn from them, and maybe being a little more nuanced instead of making a knee jerk “guard rail to guard rail” decision about whatever we deem to be the cause of those bad experiences.

Just think, how many times have you said or heard another developer say some form of the following two statements:

  1. I’ll never use [IoC containers/NoSql/Design Patterns/mock objects] ever again because this one time at band camp I saw some people on a project (not me, of course) try to use that and it was horrible! Never again!
  2. We threw the baby out with the bathwater on [UML/Pair Programming/Stored Procedures/something that's valuable sometimes but not always]

UML here is the obvious example of a technique that was horribly abused in the past and now largely discredited. While I lived through the age of absurd UML overuse and I’ve made fun of UML only architects, I still think that some knowledge of UML can be very useful to a developer. Specifically, I’ll still sketch class diagrams to understand a subsystem or a sequence diagram to think through a tricky interaction between classes or subsystems. With multi core processors and the push for so much more parallelization, I think that activity diagrams come back as a way to think through the timing and coordination between tasks running in different threads.

The Centralized Architecture Team

Not long after I started my current job we had a huge phone call to discuss some potential changes to our development methods and projects. Someone suggested that we form a dedicated, centralized architecture team and I shot it down vehemently and colorfully. Why you ask? Because…

The most useless I’ve ever felt in my career was the 18 months I spent as the junior most member of a centralized architecture team in a large enterprise IT organization. My shop at the time had decided to double down on formal waterfall processes with very well defined divisions of specialized responsibility. Architects made high level designs, technical leads made lower level designs, and the developers coded what they were told. As you’ve probably guessed, we architects were not to be doing any actual coding. As you’ve probably also guessed, it didn’t work at all and most teams faked their compliance with the official process in order to get things done.

I firmly believe that at the core of all successful software development is a healthy set of feedback mechanisms to nudge a team toward better approaches. A non-coding architect who doesn’t stick around for the actual coding doesn’t have any real feedback mechanism to act as a corrective function on his or her grandiose technical visions.

I witnessed this first hand when I was asked to come behind one of my architect peers on what should have been a relatively simple ETL project. My peer had decided that the project should consist of a brand new VB6 application with lots of elaborate plugin points and Xml configuration to accommodate later features. The team who actually had to deliver the project wanted to just use the old Sql Server DTS infrastructure to configure the ETL exchanges and be done with it in a matter of days — but my peer had told them that they couldn’t do that because of corporate standards and they’d have to build the new infrastructure he’d designed on paper.

When I took over as the “architect,” the team was struggling to build this new ETL framework from scratch. I was able to clear the usage of DTS with one quick phone call and sheepishly told the project manager that he had my official blessing to do the project in the way that he and his team already knew was the easiest way forward. I felt like an ass. The team had the project well in hand and would have already been done if not for the “guidance” of my architecture team.

I’ve had other bad experiences with centralized architecture teams over the years in several other large companies. In each negative case I think there was a common theme — the centralized architecture team was too far removed from the actual day to day work (and many of the members of those teams knew that full well and worked in constant frustration) . There’s some obvious potential goodness in sharing technical knowledge and strategy across projects, but my opinion is that this is best done with a virtual team of technical leaders from the various project teams in an organization sharing solutions and problems rather than a dedicated team off by themselves, brown bag presentations, and just rotating developers between teams once in a while.

I ended up replacing my non-coding architect peer on a couple later projects and every single time I walked into a situation where the new team hated the very idea of an “architect” because of their scar tissue. Good times.

 

Web Services Everywhere!

The undisputed technical leader of the same centralized architecture team had a grandiose SOA vision of exposing all of our business processes as loosely coupled SOAP web services. I was dubious of some of the technical details at the time (see Lipstick on a Pig), but there was a definite kernel of solid thinking behind the strategy and I didn’t have any choice but to get on board the SOA train anyway.

A couple months in, our SOA visionary gave a presentation to the rest of the team upon the completion of the first strategic building block web service that we should be using for all new development from then on. The new standard data service exposed an HTTP endpoint that accepted SOAP compliant Xml with a single body element that would contain a SQL string. The web service would execute the SQL command against the “encapsulated” database and return a SOAP compliant Xml response representing the data or status returned by the SQL statement in the request Xml.

That’s right, instead of making our applications couple themselves to well understood, relatively efficient technologies like JDBC or ODBC and an application database, we were logically going to follow the approach of using one big damn database that would only be accessed by Xml over HTTP and each application would still be tightly coupled to the structure of that database (kind of what I call the “pond scum application” anti-pattern where little applications hang off a huge shared database).

My older, more experienced colleagues mostly nodded sagaciously murmuring that this was a good approach (I’m sure now that they all thought that the approach sounded nuts but were intimidated by our visionary and didn’t call out the quite naked emperor in the room). I sputtered and tried to explain later to my boss that this was the worst possible approach we could possibly take, but I was told to change my attitude and get with the program.

My lesson learned from that experience? The most experienced person in the room isn’t automatically right and the new shiny development idea isn’t magically going to make things better until you really understand it. And yes, dear colleagues reading this, I do realize that now *I’m* the most experienced person in the room and I’m not always automatically right when we disagree.

Would I use SOA after I saw it done in the dumbest way possible? Yes, in certain situations like this one – but I still contend that it’s better to focus on building well factored code that’s easy to extend without the overhead of distributed services (easier testing, easier debugging, just flat out less code) when you can get away with it.

 

Database Batch Programming Woes 

Most of my horror stories come from my very first “real” IT job. My original team was hired en masse to be the support team for a new set of supply chain automation applications built by a very large consulting company (I’ll give you one guess who I’m mocking here). Just before the consultants rolled off the project I was enlisted to help their technical lead try to speed up a critical database batch script that was too slow and getting much slower a mere 6 weeks after the project was put into production. So what was wrong that made my database guru friend sputter when I told this story? As best I recall:

  1. The 3rd party ERP system that fed the batch process had some custom Java code with a nasty N+1 problem where they were doing a logical database join in memory
  2. The batch process ran several times a day and started by first inserting data into a table from the result of a cross join query (still the only time I’ve ever seen that used in the wild) of every possible part our factories used, all of our offsite logistics centers, and each factory line where the parts would be needed — regardless of whether or not that permutation made any sense whatsoever.
  3. That giant table built from the cross joins was never archived or cleaned up, and the very expensive full table scan queries in the next stage of the batch process against that table were becoming exponentially slower just a couple months into the life of the system.
  4. Somewhere in this batch process was a big query where two tables had to be joined by calculated values, one created by string concatenation of 3 fields in table #1 and a matching string created by concatenating 5 different fields in table #2 to make the correct inner join (I’m not making this up).

Long story short, within about 6 months my new team wrote a new replacement system from scratch with two orders of magnitude throughput over the original consultant’s code that didn’t suffer from degrading performance over time by simply being a bit smarter about our database code. Our team got a bit of acclaim and some internal awards because of how happy the business was with our new system (as a first time technical lead I made my own share of WTF’s on that project too, but they were less damaging). Lessons learned?

  • Success is easiest when you start with a low bar
  • Building a quality system is so much easier to do well when you have some sort of prior art to learn from — even if that prior art is poorly done
  • Don’t be egregiously stupid about how you do your database access
  • Being smart isn’t perfectly helpful without a modicum of technical knowledge or experience (the very young consultants responsible for the horrible database designs were all very sharp, but very green).

I think you can interpret that second lesson in a couple useful ways. One, it might help to lighten up on whatever team went before you. Two, don’t be so hard on yourself when you’re having to go back over and replace some kind of software design you did previously with something much better. Doing it wrong first might have taught you how to it better the second time around.

I left that company a couple years later to go be a fancy pants traveling consultant and I very distinctly remember seeing that consulting company’s huge ad with Tiger Woods plastered all over O’Hare airport in Chicago talking about how they’d helped my former employer with their supply chain automation and grumbling how *I’d* built that, not them.

 

 

FubuMVC Lessons Learned: Semantic Versioning

TL;DR: I think that all .Net Nuget components should try to adopt Semantic Versioning, but it’s not all unicorns and rainbows

This series has a new home page at FubuMVC Lessons Learned. This post continues my discussion on componentized development in .Net that I started here and here. I’m going to write at least one or two more posts next week to seal this topic off.

 

Semantic Versioning

Here’s the deal, one of the huge advantages of Nuget (or any other package manager) is the ability to more easily package up new releases as an OSS publisher and apply updates as a consumer. Great, but if things are changing and evolving a lot more frequently than when we got a new version of .Net every couple years, how are we gonna keep all of these different components working together with our application when it’s so much more common to have conflicting dependency versions that may or may not be actually compatible? I strongly believe that the .Net community should go all in and SemVer all the things!

If you’ve never heard of it, Semantic Versioning (SemVer) is a standard for versioning  software libraries and tools that attempts to formalize rules for how to increment software versions based on compatibility and functionality. To summarize, a semantic version looks like MAJOR.MINOR.PATCH.BUILD, with the build number being optional. To simplify the SemVer rules:

  • Anything version below 1.0 means that anything goes and you have no reason to expect any public API to be stable (making it to FubuMVC 1.0 was a very big deal to me last year).
  • The major version should be incremented any time breaking changes are introduced
  • The minor version should be incremented with any purely additive functionality
  • The patch version should be incremented for backward compatible fixes

At the time I write this post, the current released version of StructureMap is 3.0.3.116. I introduced breaking changes to the public API in the initial 3.0 release, so StructureMap had to get a full point major release version which also resets the minor and patch versions. Since the 3.0 release I’ve had to make 3 different bug fix releases, but haven’t added any new functionality, so we’re up to 3.03. The last number (116) is the auto incrementing build number for traceability back to the exact revision in source control. If you adopted the original 3.0.0 version of StructureMap, you should be able to drop in any version less than some future 4.0 version and be confident that your code will still work assuming that StructureMap is following SemVer correctly.

Bundler and gems has specific support for expressing dependency constraints with SemVer like the following from FubuMVC’s Gemfile:

gem "rake", "~>10.0"
gem "fuburake", "~>1.2"

The version constraint ~>1.2 is the equivalent in Nuget to using the dependency constraint [1.2, 2.0). I’d love to see Nuget adopt something like this operator for shorthand SemVer constraints, and then ask the .Net community to adopt this type of dependency versioning as a common idiom. I think I’d even like to see Nuget get some kind of mode where SemVer constraints are applied by default, such that Nuget warns you when you’re trying to install version 3.0 of StructureMap when one of your other dependencies declares a dependency on StructureMap 2.6.* because Nuget should be able to recognize that the latest version of StructureMap has potentially breaking changes.

 

SemVer and Backward Compatibility

On the publishing side of things, adopting SemVer is a way to be serious about communicating compatibility about older versions to our users. Adopting SemVer also makes you much more cognizant of the breaking changes you introduce into code.

In FubuMVC, adopting SemVer made us much more cautious about changing any kind of change to public API’s. The downside I think is that proposed improvements tend to collect for a lot longer before you bite the bullet and make a brand new major release. The 1.0 release was a huge source of stress for me because I was trying really hard to clean up all the public API’s before we locked them in and we ended up introducing a lot of changes in a short time that while I still think were very beneficial, made many of our users reticent to update. It also made us think very hard about spinning out many ancillary functions that were still evolving into separate libraries and repositories so that we could lock down the core of FubuMVC to 1.0 and continue to allow the downstream libraries to evolve faster with a pre 1.0 version. We did go too far and I’m reversing some of that in FubuMVC 2.0, but I still think that was a valuable exercise over all and I’d recommend that any large sprawling framework at least consider doing that.

Looking back, I think that community participation in FubuMVC clearly tapered off after the 1.0 release when so many of our users just stopped keeping up with the latest version and I’m still not sure what we could have done to improve that situation. If I had to do it all over again, I’m not sure I would have put so many eggs into the big SemVer 1.0 basket and just spent time doing documentation and quick starts.

The 2.0 release of FubuMVC is introducing fewer changes, but I’m still endeavoring to eliminate as much technical debt as possible and simplify the usage of major subsystems like content negotiation, authorization, and framework configuration — and that inevitably means that yep, there’s going to be quite a few breaking changes all at once. I’m not entirely sure if making so many breaking changes all at one time is a great idea, but neither was dribbling out occasional breaking changes before the 1.0 version.

The easiest thing is to just be perfect upfront, but since you’ll never actually pull that off in your framework, you have to figure out some sort of strategy for how you’ll introduce breaking changes.

Another way to put this is that you can either hold the line on backward compatibility and continue to annoy your users with all the ill informed early decisions over time or really piss off your users one time by fixing your usability issues. Pick your poison I guess. Or know your users. Do they want improvements more or do they value stability over everything else?

 

Next time:

I started this series of posts about .Net componentization with the promise to the Nuget team that I’d write about what Ripple added to Nuget and my recommendations for Nuget itself, so that will be next.  I’ve also got a much shorter post coming up on doing branch per feature with Nuget across multiple repositories.

Until next time, have a great weekend one and all….

 

 

 

Follow

Get every new post delivered to your Inbox.

Join 26 other followers