The Surprisingly Valuable and Lasting Lessons I Learned from a Horrible Project

Let’s face it; we tend to learn lessons more profoundly from bad experiences, mistakes, and very unpleasant circumstances because we just don’t want to ever go through that again. Last year I tried to write a blog post about all the positive and useful things learned from Extreme Programming (XP) earlier in my career, but I kept getting stuck on one little problem. The only truly XP as described in Chairman Beck’s Little White Book* project I worked on was a horrible experience and probably the least productive team and most unpleasant working environment I have ever been a part of in almost 20 years of software development.

* Despite my cynical turn of phrase, I thought that Beck’s original XP book was well worth the read

Why bring this up now? I know that I learned some tremendously important lessons during and after that awful project that are still valuable. Plus, after all these years, writing this has made me consider what I might have done differently.

The Lessons

• Own your development process and adapt
• People communicate and learn in different ways and it pays to understand that and adapt your approach as a technical lead
• A little team turnover is good, but too much is terrible
• Build a level of trust and defuse tension between team members early
• Intensive, detailed software processes and micromanagement down to the task level can lead to very poor technical results because it discourages developer initiative and adaptation. This burned us badly on the project I’m describing and I honestly worry about this on some of our projects at my work.

Why was it so bad?

I was the nominal technical lead of a team from a consulting company that was working at our customer site with a mixed team from our company and the customer’s own staff. We were mostly building new workflow and automated decision support tools on a very early version of .Net to replace some of their homegrown Powerbuilder and Oracle applications. I was the only member of our consultant team who had any prior experience working with .Net.

So exactly how bad was it? You know that joke about how “the beatings will continue until morale improves?” That was us. We had bi-weekly mandatory team building meetings after work that were generally pretty miserable.

As usual when consultants are parachuted in, the client staff resented us somewhat for being there — and I’m pretty sure they more than suspected that their employer intended to let them all go when our project advanced far enough. It probably didn’t help that my consultant team wasn’t terribly professional at the client site.

We had way too many strong personalities and opinions about how things should be done and endless arguments about the minutiae of how an XP project should be executed. We had absurdly long and contentious iteration kickoff and retrospective meetings. We had the dreaded embarrassment meeting at the end of each iteration where the project manager did his very best to humiliate the struggling development team.

Mostly though, I was embarrassed and frustrated by how little we seemed to be able to accomplish in the code compared to my previous (and subsequent) project experiences — even though I think it probably was one of the strongest teams I’ve ever been on in terms of the individuals involved. Some of it could be explained by having many folks that were new to the platform we were using or the XP practices we were trying to use (while I’m still a big fan of TDD, it can have a pretty nasty learning curve). Mostly though, I think that using XP in a very dogmatic way was our major undoing — in particular the requirement that all production coding had to be done as pair programming made it very difficult for me because I was by necessity working somewhat different hours than the others because I was traveling from much farther away.

I honestly don’t know how the project ended up because I was effectively kicked out by the project manager in what I felt at the time was kind of an act of mercy. I left the consulting company in question not long after, mostly because of personal issues but also because I was disillusioned with them after this experience.

I did learn some great technical lessons that might appear in another post on XP someday and I worked with some truly great individual developers — which really just made it so much more frustrating how ineffective this entire project team really was.

Regular readers of this blog know how much I hate words like zealot or dogma or pragmatism because of how developers so often use those words to deodorize the stench of sloppy self-justifications, but “dogma” and “zealot” do accurately apply here;-)

Enough of that, on to the lessons…

Own your development process

The most powerful lesson I learned on this terrible project was about proactively owning and adapting your team’s process and practices instead of just trying to muddle through when something isn’t working. For most of this project, we did our iteration estimation by having developers try to estimate in ideal hours. During iterations the developers would attempt to track how many of these nebulous “ideal hours” we had spent on each story. Needless to say, we had burned a lot of hours on arguing exactly how many “ideal hours” we were really spending and what the estimate for a user story really should be in iteration kickoff meetings. The killer for me was how the project manager would take all of these numbers to show in excruciating detail exactly how badly our struggling development team was missing our estimates on every single story in front of the client and the rest of the team.

As you can imagine, this resulted in long, nasty iteration kickoff meetings as the other developers and I tried to avoid the humiliation. Just a couple incidents I remember:

1. Being pulled out of an iteration kickoff meeting because one of the BA’s was mad about how he thought we were inflating the estimates. It ended with me screaming at our account manager that I had to do that to protect myself while pointing at the project manager (if you’ve never met me, I’m a big guy and I’d bet that a very angry me might be a little bit upsetting).
2. Going up to the project manager to show him how a big new subsystem had gone much better than expected and effectively being told that I had been sandbagging estimates. To say the least, I fumed a little bit about that one.

Fortunately, our resident older developer got up at a retrospective meeting one time and made a very simple suggestion: our stories seem to come in about three different sizes, so instead of worrying about ideal hours at all, let’s just estimate stories as small, medium, or large (shirt size estimation). My recollection is that as soon as we adopted the new estimation technique, iteration meetings became much quicker and smoother. As we eliminated the idiotic estimate tracking out of our retrospectives, our meetings became less contentious, faster, and more productive.

The big lesson I took away is to never, ever treat your software process as something that is set in stone. I also learned that software teams should be empowered to make corrections to the way they work.

I blogged about a similar theme when I was on Codebetter ages ago.

Interpersonal Tension and Lack of Trust

I guess I have to be honest here just in case it’s not obvious and admit that I still hold a pretty significant grudge toward that project manager. One of the first interactions I remember having with the project manager was him approaching me at some kind of after hours kickoff social and telling me that “he just got a bad vibe off of me.” To my detriment, I just hunkered down instead of trying to ask him what the hell he meant and finding a way to get past that. I certainly never trusted him, and he made it abundantly clear that he didn’t trust the way that I was leading the development effort.

I spent a lot of time on that project worrying about protecting myself from the project manager and his verbal abuse. The project manager spent a lot of time criticizing me for “over design” or trying to subvert the XP process. I responded by veering too far toward not doing any design at all and really stopped thinking about anything that wasn’t in the iteration plan so I would have to get more lectures about how things would work better if I just got in line and followed the XP process.

The end result of that was a lot of technical debt in our heavy client related to our navigation logic. I had known for a long time that we were writing bad spaghetti code that was going to hurt us and already had a design concept based on what’s now called the screen conductor pattern. If I were doing that project today, I would opt for the screen conductor design early on, but then I was so concerned with the charges of over design that I put it off, partly because it would have involved having to create a user story that had to be approved by the project manager. Once we finally had to admit that we were having a technical debt melt down, we played an official story to change to the architecture using the screen conductor abstraction idea. In my opinion, that work took much longer than it would have taken if we’d done it early on when it was apparent that we needed to do that.

To this day, I consider this episode of delaying the cleanup of the navigation logic one of the worst technical design decisions I’ve made – and it all could have been avoided if the project manager had trusted my judgement so that I could have acted earlier as I saw fit and if I had trusted him enough to actually communicate with him. I didn’t go to him with these concerns early because I thought he’d just mock me for wanting to build a framework.

The over design issue never really went away, not even when a very senior architect from our company was brought into check up on me and assured the PM and I that things were fine.

Oh, and the project manager in question? Last I knew he was a somewhat successful “Agile Coach” in the Chicago area. He might very well be much better for his clients today than he was for me a decade ago, but the normal cynicism toward “Agile Coach” consultants probably applies here.

Team Turnover

I think that having at least a little bit of team turnover is valuable. A reasonable trickle of new team members can add new knowledge, techniques, and perspective to a team that might be getting some “that’s the way we’ve always done it” blinders.

For example, during the project I’ve been discussing, we had a senior tester with some pretty heavy test automation skills come in for just a handful of iterations and did a lot to help us jumpstart our end to end automated testing (and taught me some things that I still use and recommend today).

On the downside, we had an absurd amount of churn in our team makeup from iteration to iteration. The client developers worked with us directly at first, then they were shunted off to the side for training (on a bunch of esoteric topics of no use to our project of course). Then our company added a second team overseas, and sent one of our core guys there to help get that started, but then we brought some of the overseas developers to our site for a while. Other developers bounced in and out of the team and my eventual replacement came in later as well. The end result was that we seemed to have an all new team makeup every single iteration and I think it hurt our ability to form any kind of chemistry, we didn’t really develop the “tribal knowledge” that’s so important for long running projects, and I felt like I was constantly spinning my wheels onboarding new developers to the team.

Different Learning and Communication Styles

To break the stream of negativity, a very useful lesson I learned on this project was to pay attention to how developers have different styles of learning and communication and to tailor your approach as a development lead to individuals.

Like many developers, I’m mostly a visual learner first, and kinesthetic learner second (not so much as my knees get worse, but I like to think through design problems while shooting baskets or walking around). One of the other senior developers on that project was almost strictly an auditory learner and just didn’t derive any value whatsoever from the boxes and arrows I kept trying to draw up on the whiteboard. I quickly learned that I just had to collaborate differently with him than I did with another senior developer who was more visual like I am.

See this article for more information on learning styles.

What would I do differently today?

I’ve been working on this post for about a week and I’ve spent quite a bit of time pondering what I could have done differently to have made that project a better experience. I know that an older, more self-confident me could probably handle that situation  far more effectively and probably get past both the interpersonal and process problems. Truth be told though, if I were in that kind of situation today I would probably just look for another job and not look back because life is just too short for another of these projects.

But back to the real exercise, what could I have done better?

• Try to come to some kind of better working relationship and understanding with the project manager. What if I had tried to understand his concerns with me earlier? Hell, what if I’d gotten into his face when he jabbed at me instead of just letting him push me around like I probably did? Tell my management or the account manager that either I had to go or the project manager? That would have probably resulted in me getting moved off or even let go, but in retrospect, that would have been much better than hanging around for 9 months of punishment.
• Deal with the accusations that I was indulging in over design early in the project
• Question our process much more early on so we could have eliminated the “embarrassment meeting” anti-pattern early
• As far as the architecture is concerned, I wish I had listened more to myself and dealt with the technical debt issue in our screen navigation early when the problem was first apparent