Software Development Horror Stories and Scar Tissue

My best friend (also a developer) and I meet for lunch almost every Friday. As experienced developers often do, we got to swapping horror stories this past Friday. I had so much fun telling some of my favorite stories that I thought I’d repeat some of them here just to break my blogging drought. As I started doing just that, it occurred to me that there’s a deeper theme here about how we process our bad experiences, what we learn from them, and maybe being a little more nuanced instead of making a knee jerk “guard rail to guard rail” decision about whatever we deem to be the cause of those bad experiences.

Just think, how many times have you said or heard another developer say some form of the following two statements:

I’ll never use [IoC containers/NoSql/Design Patterns/mock objects] ever again because this one time at band camp I saw some people on a project (not me, of course) try to use that and it was horrible! Never again!
We threw the baby out with the bathwater on [UML/Pair Programming/Stored Procedures/something that’s valuable sometimes but not always]

UML here is the obvious example of a technique that was horribly abused in the past and now largely discredited. While I lived through the age of absurd UML overuse and I’ve made fun of UML only architects, I still think that some knowledge of UML can be very useful to a developer. Specifically, I’ll still sketch class diagrams to understand a subsystem or a sequence diagram to think through a tricky interaction between classes or subsystems. With multi core processors and the push for so much more parallelization, I think that activity diagrams come back as a way to think through the timing and coordination between tasks running in different threads.

The Centralized Architecture Team

Not long after I started my current job we had a huge phone call to discuss some potential changes to our development methods and projects. Someone suggested that we form a dedicated, centralized architecture team and I shot it down vehemently and colorfully. Why you ask? Because…

The most useless I’ve ever felt in my career was the 18 months I spent as the junior most member of a centralized architecture team in a large enterprise IT organization. My shop at the time had decided to double down on formal waterfall processes with very well defined divisions of specialized responsibility. Architects made high level designs, technical leads made lower level designs, and the developers coded what they were told. As you’ve probably guessed, we architects were not to be doing any actual coding. As you’ve probably also guessed, it didn’t work at all and most teams faked their compliance with the official process in order to get things done.

I firmly believe that at the core of all successful software development is a healthy set of feedback mechanisms to nudge a team toward better approaches. A non-coding architect who doesn’t stick around for the actual coding doesn’t have any real feedback mechanism to act as a corrective function on his or her grandiose technical visions.

I witnessed this first hand when I was asked to come behind one of my architect peers on what should have been a relatively simple ETL project. My peer had decided that the project should consist of a brand new VB6 application with lots of elaborate plugin points and Xml configuration to accommodate later features. The team who actually had to deliver the project wanted to just use the old Sql Server DTS infrastructure to configure the ETL exchanges and be done with it in a matter of days — but my peer had told them that they couldn’t do that because of corporate standards and they’d have to build the new infrastructure he’d designed on paper.

When I took over as the “architect,” the team was struggling to build this new ETL framework from scratch. I was able to clear the usage of DTS with one quick phone call and sheepishly told the project manager that he had my official blessing to do the project in the way that he and his team already knew was the easiest way forward. I felt like an ass. The team had the project well in hand and would have already been done if not for the “guidance” of my architecture team.

I’ve had other bad experiences with centralized architecture teams over the years in several other large companies. In each negative case I think there was a common theme — the centralized architecture team was too far removed from the actual day to day work (and many of the members of those teams knew that full well and worked in constant frustration) . There’s some obvious potential goodness in sharing technical knowledge and strategy across projects, but my opinion is that this is best done with a virtual team of technical leaders from the various project teams in an organization sharing solutions and problems rather than a dedicated team off by themselves, brown bag presentations, and just rotating developers between teams once in a while.

I ended up replacing my non-coding architect peer on a couple later projects and every single time I walked into a situation where the new team hated the very idea of an “architect” because of their scar tissue. Good times.

Web Services Everywhere!

The undisputed technical leader of the same centralized architecture team had a grandiose SOA vision of exposing all of our business processes as loosely coupled SOAP web services. I was dubious of some of the technical details at the time (see Lipstick on a Pig), but there was a definite kernel of solid thinking behind the strategy and I didn’t have any choice but to get on board the SOA train anyway.

A couple months in, our SOA visionary gave a presentation to the rest of the team upon the completion of the first strategic building block web service that we should be using for all new development from then on. The new standard data service exposed an HTTP endpoint that accepted SOAP compliant Xml with a single body element that would contain a SQL string. The web service would execute the SQL command against the “encapsulated” database and return a SOAP compliant Xml response representing the data or status returned by the SQL statement in the request Xml.

That’s right, instead of making our applications couple themselves to well understood, relatively efficient technologies like JDBC or ODBC and an application database, we were logically going to follow the approach of using one big damn database that would only be accessed by Xml over HTTP and each application would still be tightly coupled to the structure of that database (kind of what I call the “pond scum application” anti-pattern where little applications hang off a huge shared database).

My older, more experienced colleagues mostly nodded sagaciously murmuring that this was a good approach (I’m sure now that they all thought that the approach sounded nuts but were intimidated by our visionary and didn’t call out the quite naked emperor in the room). I sputtered and tried to explain later to my boss that this was the worst possible approach we could possibly take, but I was told to change my attitude and get with the program.

My lesson learned from that experience? The most experienced person in the room isn’t automatically right and the new shiny development idea isn’t magically going to make things better until you really understand it. And yes, dear colleagues reading this, I do realize that now *I’m* the most experienced person in the room and I’m not always automatically right when we disagree.

Would I use SOA after I saw it done in the dumbest way possible? Yes, in certain situations like this one — but I still contend that it’s better to focus on building well factored code that’s easy to extend without the overhead of distributed services (easier testing, easier debugging, just flat out less code) when you can get away with it.

Database Batch Programming Woes

Most of my horror stories come from my very first “real” IT job. My original team was hired en masse to be the support team for a new set of supply chain automation applications built by a very large consulting company (I’ll give you one guess who I’m mocking here). Just before the consultants rolled off the project I was enlisted to help their technical lead try to speed up a critical database batch script that was too slow and getting much slower a mere 6 weeks after the project was put into production. So what was wrong that made my database guru friend sputter when I told this story? As best I recall:

The 3rd party ERP system that fed the batch process had some custom Java code with a nasty N+1 problem where they were doing a logical database join in memory
The batch process ran several times a day and started by first inserting data into a table from the result of a cross join query (still the only time I’ve ever seen that used in the wild) of every possible part our factories used, all of our offsite logistics centers, and each factory line where the parts would be needed — regardless of whether or not that permutation made any sense whatsoever.
That giant table built from the cross joins was never archived or cleaned up, and the very expensive full table scan queries in the next stage of the batch process against that table were becoming exponentially slower just a couple months into the life of the system.
Somewhere in this batch process was a big query where two tables had to be joined by calculated values, one created by string concatenation of 3 fields in table #1 and a matching string created by concatenating 5 different fields in table #2 to make the correct inner join (I’m not making this up).

Long story short, within about 6 months my new team wrote a new replacement system from scratch with two orders of magnitude throughput over the original consultant’s code that didn’t suffer from degrading performance over time by simply being a bit smarter about our database code. Our team got a bit of acclaim and some internal awards because of how happy the business was with our new system (as a first time technical lead I made my own share of WTF’s on that project too, but they were less damaging). Lessons learned?

Success is easiest when you start with a low bar
Building a quality system is so much easier to do well when you have some sort of prior art to learn from — even if that prior art is poorly done
Don’t be egregiously stupid about how you do your database access
Being smart isn’t perfectly helpful without a modicum of technical knowledge or experience (the very young consultants responsible for the horrible database designs were all very sharp, but very green).

I think you can interpret that second lesson in a couple useful ways. One, it might help to lighten up on whatever team went before you. Two, don’t be so hard on yourself when you’re having to go back over and replace some kind of software design you did previously with something much better. Doing it wrong first might have taught you how to it better the second time around.

I left that company a couple years later to go be a fancy pants traveling consultant and I very distinctly remember seeing that consulting company’s huge ad with Tiger Woods plastered all over O’Hare airport in Chicago talking about how they’d helped my former employer with their supply chain automation and grumbling how *I’d* built that, not them.