The Fundamentals of Continuous Software Design

Continuing my recent theme of remember why we originally thought Agile was a good thing before it devolved into whatever it is now.

I had the opportunity over the weekend to speak online as part of CouchCon Live. My topic was to revisit some of the principles of designing software inside of an adaptive Agile Software process in a talk entitled “The Fundamentals of Continuous Software Design.”

The video has posted on YouTube, and the slides are available on SlideShare.

I went back through the Agile greatest hits with:

  • Do the Simplest Thing that Could Possibly Work
  • The Last Responsible Moment
  • Reversibility in Software Architecture
  • Designing for Testability
  • How the full development team should be involved throughout
  • And why I think contemporary Scrum is the Scrappy Doo of Agile Software Development


Remembering Why Agile was a Big Deal

Earlier this year I recorded a podcast for Software Engineering Radio with Jeff Doolittle on Agile versus Waterfall Software Development where we discussed the vital differences between Agile development and traditional waterfall, and why those differences still matter. This post is the long-awaited companion piece I couldn’t manage to finish before the podcast posted. I might write a follow up post on some software engineering practices like continuous design later so that I can strictly focus here on softer process related ideas.

I started my software development career writing Shadow IT applications and automation for my engineering group. No process, no real practices either, and lots of code. As you’d probably guess, I later chafed very badly at the formal waterfall processes and organizational model of my first “real” software development job for plenty of reasons I’ll detail later in this post.

During that first real IT job, I started to read and learn about alternative iterative development processes like the Rational Unified Process (RUP), but I was mostly inspired by the brand new, shiny Extreme Programming (XP) method. After tilting the windmill a bit at my waterfall shop to try to move us away from the waterfall to RUP (Agile would have been a bridge too far), I jumped to a consulting company that was an influential, early adopter of XP. I’ve worked almost exclusively with Agile processes and inside more or less Agile organizational models ever since — until recently. In the past couple years I’ve been involved with a long running project in a classic waterfall shop which has absolutely reinforced my belief in the philosophical approaches to software development that came out of Agile Software (or Lean Programming). I also think some of those ideas are somewhat lost in contemporary Scrum’s monomaniacal focus on project management, so here’s a long blog post talking about what I think really was vital in the shift from waterfall to agile development.

First, what I believe in

A consistent theme in all of these topics is trying to minimize the amount of context switching throughout a projectand anybody’s average day. I think that Agile processes made a huge improvement in that regard over the older waterfall models, and that by itself is enough to justify staking waterfall through the heart permanently in my book.

Self-contained, multi-disciplinary teams

My strong preference and a constant recommendation to our clients is to favor self-contained, multi-disciplinary teams centered around specific projects or products. What I mean here is that the project is done by a team who is completely dedicated to working on just that project. That team should ideally have every possible skillset it needs to execute the project so that there’s reduced need to coordinate with external teams, so it’s whatever mix you need of developers, testers, analysts all working together on a common goal.

In the later sections on what I think is wrong with the waterfall model, I bring up the fallacy of local optimization. In terms of a software project, we need to focus on optimizing the entire process of delivering working software, not just on what it takes to code a feature, or write tests, or quickly checking off a process quality gate. A self-contained team is hopefully all focused on delivering just one project or sprint, so there should be more opportunity to optimize the delivery. This generally means things like developers purposely thinking about how to support the testers on their team or using practices like Executable Specifications for requirements that shortens the development and testing time overall, even if it’s more work for the original analysts upfront.

A lot of the overhead of software projects is communication between project team members. To be effective, you need to improve the quality of that communication so that team members have a shared understanding of the project. I also think you’re a lot less brittle as a project if you have fewer people to communication with. In a classic waterfall shop, you may need to be involving a lot of folks from external projects who are simultaneously working on several other initiatives and have a different schedule than your project’s schedule. That tends to force communication into either occasional meetings (which impact productivity on its own) or through asynchronous communication via emails or intermediate documentation like design specifications.

Let’s step back and think about the various types of communication you use with your colleagues, and what actually works. Take a look at this diagram from Scott Ambler’s Communication on Agile Software Teams essay (originally adapted from some influential writings by Alistair Cockburn that I couldn’t quite find this morning):



In a self-contained, multi-disciplinary team, you’re much more likely to be using the more effective forms of communication in the upper, right hand of the graph. In a waterfall model where different disciplines (testers, developers, analysts) may be working in different teams and on different projects at any one time, the communication is mostly happening at the less effective, lower left of this diagram.

I think that a self-contained team suffers much less from context switching, but I’ll cover that in the section on delivering serially.

Another huge advantage to self-contained teams is the flexibility in scheduling and the ability to adapt to changing project circumstances and whatever you’re learning as you go. This is the idea of Reversibility from Extreme Programming:

“If you can easily change your decisions, this means it’s less important to get them right – which makes your life much simpler. ” — Martin Fowler

In a self-contained team, your reversibility is naturally higher, and you’re more likely able to adapt and incorporate learning throughout the project. If you’re dependent upon external teams or can’t easily change approved specification documents, you have much lower reversibility.

If you’re interested in Reversibility, I gave a technically focused talk on that at Agile Vancouver several years ago

I think everything in this section still applies to teams that are focused on a product or family of products.

Looking over my history, I’ve written about this topic quite a bit over the years:

  1. On Software Teams
  2. Call me a Utopian, but I want my teams flat and my team members broad
  3. Self Organizing Teams are Superior to Command n’ Control Teams
  4. Once Upon a Team
  5. The Anti Team
  6. On Process and Practices
  7. Want productivity? Try some team continuity (and a side of empowerment too) – I miss this team.
  8. The Will to be Good
  9. Learning Lessons — Can You Make Mistakes at Work?
  10. Indelible Proof of a Healthy Team

Deliver serially over working in parallel

A huge shift in thinking behind Agile Software Development is simply the idea that the only thing that matters is delivering working software. Not design specifications, not intermediate documents, not process checkpoints passed, but actual working software deployed to production.

In practice, this means that Agile teams seek to deliver completely working features or “vertical slices” of functionality at one time. In this model a team strives for the continuous delivery model of constantly making little releases of working software.

Contrast this idea to the old “software as construction” metaphor from my waterfall youth where we generally developed by:

  1. Get the business requirements
  2. Do a high level architecture document
  3. Maybe do a lower level design specification (or do a prototype first and pretend you wrote the spec first)
  4. Design the database model for the new system
  5. Code the data layer
  6. Code the business layer
  7. Code any user interface
  8. Rework 4-6 because you inevitably missed something or realized something new as you went
  9. Declare the system “code complete”
  10. Start formal testing of then entire system
  11. Fix lots of bugs on 4-6
  12. User acceptance testing (hopefully)
  13. Release to production

The obvious problems in this cycle is that you deliver no actual value until the very, very end of the project. You’re also struggling quite a bit in the later parts of the project because you’re needing to re-visit work that was done much earlier and you often struggle with the context switching that entails.

In contrast, delivering in vertical slices allows you to:

  • Minimize context switching because you’re finishing work much closer to when it’s started. With a multi-disciplinary team, every body is focused on a small subset of features at any one time which tends to improve the communication between disciplines.
  • Actually deliver something much earlier to production and start accruing some business payoff. Moreover, you should be working in the order of business priority, so the most important things are worked on and completed first. Which also serves to fail softer compared to a waterfall project cycle.
  • Fail softer by delivering part of a project on time if even if you’re not able to complete all the planned features by the theoretical end date — as apposed to failing completely to deliver anything on time in a waterfall model.

In the Extreme Programming days we talked a lot about the concept of done, done, done as opposed to being theoretically “code complete.”


Rev’ing up feedback loops

After coming back to waterfall development the past couple years, the most frustrating thing to me is how slow the feedback cycles are between doing or deciding something and knowing whether or not anything you did was really correct. It also turns out that having a roomful of people staring at a design specification document in a formal review doesn’t do a great job at spotting a lot of problems that present themselves later in the project when actual code is being written.

Any iterative process helps tighten feedback cycles and enables you to fix issues earlier. What Agile brought to the table was an emphasis on better, faster, more fine-grained feedback cycles through project automation and practices like continuous integration and test driven development.

More on the engineering practices in later posts. Maybe. It literally took me 5 years to go from an initial draft to publishing this post so don’t hold your breathe.


What I think is wrong with classic waterfall development

Potentially a lot. Your mileage may vary from mine (my experiences with formal waterfall processes has been very negative) and I’m sure some of you are succeeding just fine with waterfall processes of one sort or another.

At its most extreme, I’ve observed these traits in shops with formal waterfall methods, all of which I think work against successful delivery and why I think these traits are problematic.

Over-specialization of personnel

I’m not even just talking about developers vs testers. I mean having some developers or groups who govern the central batch scheduling infrastructure, others that own integrations, a front end team maybe, and obviously the centralized database team. Having folks specialized in their roles like this means that it takes more people involved in a project in order to have all the necessary skillset, and that means having a lot more effort to communicate and collaborate between people who aren’t in the same teams or even in the same organizations. That’s a lot of potential project overhead, and it makes your project less flexible as you’re bound by the constraints of your external dependencies.

The other problem with over-specialization is the fallacy of local optimization problem, because many folks only have purview over part of a project and don’t necessarily see or understand the whole project.

Formal, intermediate documents

I’m not here to start yet another argument over how much technical documentation is enough. What I will argue about is a misplaced focus on formal, intermediate documents as a quality or process gate. Especially if those intermediate documents are really meant to serve as the primary communication between architects, analysts, and developers. One, because that’s a deeply inefficient way to communicate. Two, because those documents are always wrong because they’re written too early. Three because it’s just a lot of overhead to go through authoring those documents to get through a formal process gate that could be better spent on getting real feedback about the intended system or project.

Slow feedback cycles

Easily the thing I hate the most about “true” waterfall projects is the length of time between getting adequate feedback between the early design and requirements documents and an actually working system getting tested or going through some user acceptance testing from real users. This is an especially pernicious issue if you hew to the idea that formal testing can only start after all designed features are complete.

The killer problem in larger waterfall projects over my career is that you’re always trying to remember how some piece of code you wrote 3-6 months ago works when problems finally surface from real testing or usage.



I’d absolutely choose some sort of disciplined Agile process with solid engineering practices over any kind of formal waterfall process any day of the week. I think waterfall processes do a wretched job managing project risks by the slow, ineffective feedback cycles and waste a lot of energy on unevenly useful intermediate documentation.

Agile hasn’t always been pretty for me though, see The Surprisingly Valuable and Lasting Lessons I Learned from a Horrible Project about an absolutely miserable XP project.

Ironically, the most successful project I’ve ever worked on from a business perspective was technically a waterfall project (a certain 20-something first time technical lead basically ignored the official process), but process wasn’t really an issue because:

  • We had a very good relationship with the business partners including near constant feedback about what we were building. That project is still the best collaboration I’ve ever experienced with the actual business experts
  • There was a very obvious problem to solve for the business that was likely to pay off quickly
  • Our senior management did a tremendous job being a “shit umbrella” to keep the rest of the organization from interfering with us
  • It was a short project

And projects like that just don’t come around very often, so I wouldn’t read much into the process being the deciding factor in its success.








Context is Important

I’ve been thinking recently about how Agile changed software development in important ways and getting back to some of the important lessons I felt that we as an industry learned from Extreme Programming back in the day. Depending on my time and ambition level, I might try to spit out some essays on team organization, feedback, engineering practices, and continuous delivery. It should be implied anyway, but I need to be clear that all the contents of this blog are my opinions alone and do not reflect that of my employer (Calavista). I.e., I’m the only one to blame;)

I’m hopefully near the end of what’s been a very frustrating project. There’s more to the story of course, but I ascribe many of the challenges we have faced on the decision to execute the project with a true waterfall lifecycle. Being forced back to waterfall development has naturally led me to reflect on what I think are the crucial differences between that model and Agile development techniques and remember just why Agile development was such a revolutionary movement (before it was all watered down by Scrum, but that’s a subject for another day). That reflection includes feedback cycles, team and organization mechanics, project management, and reversibility — but for now, let’s just talk about context.

Picking up the definition of “context” from

the set of circumstances or facts that surround a particular event, situation, etc.

In terms of software development, understanding the “context” of your week means knowing how your work fits into the greater tableau of the system and ecosystem. It also means understanding the “why” behind the technical decisions that have been made so far and the architectural approach.

It turns out that understanding the context around the software system you are building is very important. Thank you, Captain Obvious right?


Now, let’s try to use the word “context” in some sentences about software development:

  • As an architect, I make more effective decisions when I understand the context of how our work fits into the greater enterprise ecosystem. One of the things I think I’ll be doing in the next couple weeks is to give a little presentation to our client on how our dev lead and I think the architecture or our project should be streamlined in the future by consolidating responsibilities, simplifying the topology, and reducing integration chattiness. Why didn’t I and others just make better decisions upfront you ask? Conway’s Law* is a partial explanation, and our future recommendations assume that the organization in question will eliminate that artificial barrier in the future. Mostly though, we made decisions based very early on a limited view of a specific initiative in isolation. Now that we know much more about other system interactions and related projects going on at our client, we see a path for a much simpler and more reliable architecture — but we could only get to that point by seeing the bigger picture rather than dealing with “build this console app to move data from point A to point B.”
  • As any kind of technical leader, I can better set up other developers for success by explaining the context around their tasks. I learned a brutal lesson early as a technical lead. The more specific instructions I gave to the other developers working on our project, the worse the code they wrote came back. When I just had a discussion with developers about what we were trying to achieve and how their tasks tied into the greater workflow of the project, the results were very obviously better. It turns out that developers can much better make decisions when they understand the “why” of the work that they’re doing.
  • As an OSS author and maintainer, I do a much better job helping users with questions when I understand what the users are trying to accomplish rather than just trying to answer specific API questions. It’s time consuming to ask a lot of questions, but the results are usually better in the end. It does lead to a lot of “why would you possibly need to do that?” questions on my part before I understand what their scenario really is. It’s not uncommon to give much easier solutions to their real issues once I know the context behind the original question. That really depends on catching me at a moment when I’m not too busy or stressed out by other things and the other person has enough patience to go back and forth, but hey, it’s an imperfect world.
  • As a developer, I’m much faster fixing bugs and addressing questions when the context of the problem is fresh in my mind. I distinctly remember making the argument back in the early to mid-00’s that developers would be able to fix bugs much faster in an Agile process where testing happens simultaneously and the code is always fresh in your mind versus waterfall development where you’re frequently needing to address code you haven’t worked with in several months. After going back to waterfall development the past 12 months, I can attest to this argument being very much a true fact.
  • As a tester, I can more effectively troubleshoot problems and write actionable defect reports when I understand the environmental context of the functionality I’m testing. Things inevitably go wrong, and good testers help solve those problems when they can investigate the failures, find the relevant logs, and give developers the information they need in order to further diagnose or fix the problems.
  • As an architect or any kind of technical leader making technical recommendations and guidance to others, I should explain the “why” and context behind any guidance. This is absolutely vital because my experience has been that folks will eventually face scenarios where your guidance or advice does not apply very well or might even be flat out harmful. You can’t just say “always do this or that.” You need to explain why you gave the recommendations to hopefully let folks know when they should break out and look for other solutions rather than force fitting a “standard architecture.” My strong advice is to always include some context about why any particular reference architecture came about so that developers can be more effective rather than blindly following along.
  • As a developer (really as any possible role), I am less effective when I have to deal with a lot of context switching during the work day. I’ve long been in roles where I need to try to answer questions from other folks and I’ve always prided myself on my ability to quickly shift contexts and help folks out, but it definitely wears on me and I’m not as sharp as I am when it’s questions about something I’m actively working on at the time. Personally, I see my productivity lately going down the drain as I’m having to do an unusual amount of context switching between different projects.

Context and the Agile vs Waterfall Argument

I’m both a long time advocate of Agile methods and a long time critic of waterfall methods, and the issue of context and context shifting really drives home the differences to me. I feel like there’s both much more sharing of project context in Agile teams and less context shifting than you frequently see in waterfall organizations.

In an Agile team there’s simply much more face to face interaction (or at least should be) between different disciplines as opposed to waterfall style toss-the-documents-over-the-wall communication. You would hope that would lead to much more shared understanding of why decisions are made and understanding of the context around the team’s work.

In a typical Agile shop, you would try to create a self-contained, multi-disciplinary team that is completely focused on one stream of work. Developers, analysts, scrum masters, testers, and whatnot all working together and bearing down on a single backlog.

In an extreme waterfall shop**, a project team may really be a larger, virtual team of individuals who are all simultaneously working on other unrelated work streams. After all, waterfall theory says that the analysts should be able to hand over the requirements document and then move on to another project. Testers shouldn’t even come into play until all the coding is done, so they’re not even around while the requirements are being formulated and the initial coding is happening. The architect, if you have one, might not be fully engaged after the initial technical specification was handed off (dunno if any organization still tries to do that, but I experienced that in my first role as an architect).

The folks in a waterfall shop are probably doing a lot more harmful context shifting than their counterparts in an Agile shop. The waterfall folks may also be getting much less information about their project because it’s constrained through documentation rather than through higher bandwidth, interactive discussions between disciplines. If you’ve never encountered this before, I highly recommend taking a read through Alistair Cockburn’s paper on Communicating, Cooperative Teams.


* Just a reminder that Conway’s Law is neither good, bad, or an anti-pattern. It simple is. Better shops will be aware of Conway’s Law and purposely and intentionally organize themselves to support their desired enterprise architecture.

** My description of a waterfall shop may sound like a straw man argument, but it really exists even today and I’ve run into this model in multiple big shops now.