TL;DR: I don’t believe that AI is going to lead to a “Waterfall Renaissance, ” and I see SDD as valuable, but we’re leaning into it as “BDD in the AI Era”.
In my live stream yesterday (at about the 45 minute mark if you’re curious) Vedran Zakanj wrote this in the chat I think just to make me go off on a rant:
”I’m seeing a lot of people describing their AI workflows as a Waterfall renaissance”
And yeah, I’m seeing plenty of that too, so let’s talk about that, and I’m going to assume he was referring to “Spec Driven Development” (SDD):
Spec-Driven Development (SDD) is a spec-first approach. Teams define common guardrails, requirements, constraints, acceptance criteria, and edge cases up front, then use AI to generate code, tests, and supporting artifacts from that shared context.
https://developer.microsoft.com/blog/spec-driven-development-ai-native-engineering
I think the most common usage of SDD is “just” writing much more detailed and structured Markdown files that are hopefully reviewed in advance of feeding them into AI agents. Since I’m heavily invested in the Event Sourcing and CQRS worlds, I see plenty of folks experimenting or investing in Domain Specific Languages or “Intermediate Languages” that attempt to specify the events, commands, and views of an event sourced system with enough detail that some kind of AI agent build out the code with minimal or even no human oversight.
Besides the format of the specifications (Markdown, DSL, YAML/XML), the next question is the role or lifecycle of “The Model.” Is the higher level model considered to be the source of truth and the code is only meant to be generated from the model? In that case, you might try to constantly revise the model based on new information, but the assumption is that it’s relatively cheap to just generate all new code from scratch using AI if necessary.
Or instead, do you treat the initial specifications as detailed prompts to generate code with AI agents, but teams or other AI agents will work directly with the code after that. At this point there’s a little bit of fuzziness and variability about whether teams want to maintain the model to keep it synchronized with the code, or let the two things vary as the coded system advances past the original model. Even with a more lax relationship between “the model” and the code, you may want AI agents to be able to modify the code based on incremental changes to “the model.”
Excitingly for an old XP guy like me, SDD has created a lot more interest in what I’ve always like to call “Executable Specifications” – human readable specifications that also double as executable tests that start as acceptance requirements but helpfully live on as regression tests (more or less Behavior Driven Development applied outside/in).
I should also say that for visual modeling techniques like Event Modeling or Event Storming, the “model” is also an analytical tool to help you discover, refine, and review system requirements.
For the Critter Stack, we’re certainly interested in SDD support within our stack, but my bias leans very heavily toward incremental approaches and I’m very dubious about many of our competitor’s “incremental language” approaches. In many cases I’ve felt like you could write simpler C#/F# code inside of the Critter Stack programming model that was considerably simpler syntax than other tool’s modeling language. I also want absolutely nothing to do with ever trying to maintain an interactive modeling user interface because that sounds like an endless time sink to me.
Our approach so far is:
- Support imports from the Event Modeling.AI platform and Emlang files for folks that want the fancier UI tools
- Render Event Model visualizations projected from our own internal DSL and at least stubs of what will become the real C# or F# types (by “internal” I mean that you’re using a fluent interface inside of your C# or F# code that can utilize actual .NET types instead of having to use an entirely separate language).
- Heavily invest into “Executable Specifications” through our Bobcat tool that is heavily optimized for both human readable specification renderings for easy review and for AI driven development through its emphasis on textual output and diagnostics meant for fast iteration by AI agents. Again, this is just my bias, but I feel like the Bobcat specifications are potentially much more valuable for AI driven development than the modeling.
- Scaffold matching command handlers or HTTP endpoints code with the Critter Stack from Event Model definitions. We also have our forthcoming Stoat tool to serve as a RAG tool to coordinate that.
- AI Skills to glue everything together between the tools, how to use the Critter Stack tools, and all the various command line diagnostics and the Bobcat test output we built specifically to optimize all the new fangled AI centric development.
- Our CritterWatch tool (and this part is supported in the un-licensed read only mode by the way) can create Event Model visualizations of your systems as it actually is by deriving the Event Model from a combination of code signatures and observed behavior at runtime on top of any Event Model definitions embedded into your codebase.
I’m dubious about the purported value of Event Modeling or any of the new DSL or Intermediate Language models that Critter Stack competitors are trying out, and still think that the BDD-inspired part of SDD will be crucial.
I do think that Event Modeling is important as what I’ll describe later as a new “UML as Sketch” approach, but I’m extremely dubious about its value for generating application code beyond easy, happy path scaffolding.
Again, this is a result of some very old bias of mine that have carried forward from the “before times” to our current AI era. As the great philosophers The Bare Naked Ladies told us, “it’s all been done.”
What came before that might still be relevant for SDD
Let’s hop back into the wayback machine and talk about software development in the early aughts when I was a 20 something developer. At the time, I was transitioning from an old fashioned waterfall shop that was just about to self-immolate chasing CMMi certification and where we all had to read the Fish Book and Who Moved My Cheese to a different shop that was one of the pioneers of the very idealistic Extreme Programming movement.
If you ever had to suffer through the Who Moved My Cheese book in corporate America way back then, I highly recommend this podcast ripping it to shreds.
At that time I think most people at the time were rightfully frustrated with the state of software development, often quoted the Standish Chaos Report on rampant software project failures (that I had completely forgotten about until just now), and looking for ways to make things better.
There were two camps that I remember (besides the holdouts still clinging to strict Waterfall / CMMi nonsense):
- Agile Software Development — which I would personally characterize as trying to be more successful by increasing the frequency and quality of the feedback cycles within software projects (TDD, CI, later BDD), leaning into working iteratively and incrementally, and purposely choosing to work adaptively over time, and plan and architect continuously. We also talked a lot about “shifting testing left” in the life cycle, but I’d lump that with the general emphasis on feedback cycles.
- Model Driven Development — a set of competing approaches where software would be built by doing much more formal modeling upfront and code would mostly be generated from the models. The folks behind this also invested quite a bit in bi-directional mapping so there was at least some ability to evolve a model and the resulting code, but always try to keep the two things in sync. I think the general idea was that you could get more productivity by moving to higher level languages, notations, and even visual models rather than getting your hands dirty with grubby coding work.
As another way of looking at that philosophical split from back then that I think directly applies to SDD today, let’s consider the role of Unified Modeling Language (UML) in software development back then. Quoting heavily from Martin Fowler’s Uml Mode post, UML usage fell into three buckets:
- UML as Sketch: Used informally and selectively as an aid to communication, design discussions, or documentation rather than a complete specification. Details can be omitted, and strict UML syntax rules are frequently relaxed in favor of clarity. [1, 2, 3]
- UML as Blueprint: Used thoroughly and precisely to design a system before coding begins—often acting as a detailed engineering handoff for programmers to build. It aims for completeness and strict accuracy relative to the planned code. [1, 2]
- UML as Programming Language: Used with the intent of generating executable code directly from complete, platform-independent models (often tied to Model-Driven Architecture or MDSD). Here, the UML model is the source code rather than a representation of it. [1, 2, 3, 4]
To be fair, many people in the XP world hated UML, but for those of us who found it valuable, we tended to mostly use it as “UML as Sketch.” Something you drew on a whiteboard to collaborate with other developers on ideas or even just in a notebook as you worked through possible design ideas before coding.
If you replace “UML” with “Markdown” or any of the other structured approaches that people are using for SDD specifications, we still have the same breakdown that Fowler observed decades ago. I’ve mostly been in the “UML as Sketch” camp after having tried “UML as Blueprint” as a technical lead and architect before that in the Waterfall days. I do think that “UML as Blueprint” is going to be a lot more successful in our new AI world, but only if you’re really putting significant effort into the “executable specifications” as an input into AI agents.
I come with a deep seated bias in favor of incremental approaches that stress the importance of feedback cycles and I’m deeply skeptical of any kind of heavy upfront specification process. Even if the specifications are easy to understand and review by us humans, it’s very likely that we’ll learn things as we go that will modify our understanding of what the specification should actually be.
If it’s really easy to keep the code and “The Model” synchronized down to the details of the user interface and every bit of business logic, maybe that’s a viable approach — but I’m from Missouri and you’ll have to show me. I think that that type of SDD where you try to focus so much on intermediate languages or modeling notations is likely to fail in the exact same way as the attempts at Model Driven Development way back when. Count me out for the new attempts at the new “UML as Programming Language” approaches.
Closing Thoughts
Something my uncle told me years ago has long stuck with me. He said that one of the downsides of becoming older is that you naturally tend to compare new ideas or approaches to older ideas and approaches, and sometimes that comparison can lead you to miss important differences or genuinely new elements. I’m bringing this up as a long winded way of saying that anything I said in this post might turn out to be wrong because I’m definitely guilty of being heavily influenced by past experiences and long standing bias in the realm of software development.
As an aside, if you love learning about history then you probably enjoy pondering counterfactuals like “how much better would the USA be if Abraham Lincoln had survived to lead the country through Reconstruction” or “what if XP had been the dominant strain of Agile Software Development instead of Scrum?” But I digress…
Oh, but to be honest, the singular most miserable project I’ve ever been on used a “pure” XP process in a very dogmatic way. Beyond the stupidly dogmatic approach, there were special circumstances and poisonous intra-team mechanics at play. I wrote a big essay about the “surprisingly valuable things I learned on my most miserable project” (own your process instead of it owning you) once upon a time on CodeBetter.com, but it’s long gone now. I might revisit that some day because it’s still relevant.
People are different than AI of course, but what I’d generally tell you is that ironically the more detailed technical instruction you give to other developers, the worse the code because it leads to people kind of turning off their brains instead of working adaptively when it turns out that the specification has gaps. Or as is so very frequently common, the theoretical approach doesn’t work out well in actuality because of unexpected or just not well understood constraints that become clear when you actually start coding.
