Thursday, July 15, 2004

Why Big Teams Don't Work

I read somewhere that Microsoft has a 24-hour system for software development with automated builds going on in the night and literally hundreds of programmers on their bigger projects.

The obvious question, then, is why isn't their software dozens of times better than everyone else's? It's not only "not as good" much of the time, or "marginally better" occasionally, but it's full of bugs, expensive, and, well, just not that great. Microsoft has more money than anyone else on the planet. They're doing what they think is best. They have not bought too many copies of The Mythical Man Month, or read them.

So what's on the other end of the spectrum? Probably shareware, which typically has a single author and is a labor of love. There's lots of really good shareware in the world, some of it, frankly, rivals Microsoft's offerings.

So why is it that small teams work better than large ones, or at least anecdotally appear to?

I have a lot of theories, and for the record, I am a proponent of small teams and have never run a software project with more than 5 engineers, including myself.

My principle theory is simply that projects involve human beings, and as a species we don't handle complexity very well. We are wired to simplify: vision simplifies what is really there, recognizing patterns; socially we simplify: you're either Good, or you're Bad, Guilty or Innocent; intellectually we simplify: the scientific method is based on reducing an experiment down to as few variables as possible so you can "control" for them and measure the one you're interested in. Humans can support up to 3 simultaneously contradictory thoughts at once, before melting down into indecision and confusion.

So what happens when you put 400 programmers on one project and try to run it?

The first thing that happens is you need at least 50 managers and project managers and coordinators and cross-functional liaisons and whatever else, just to try to "communicate" about the project. Right, like that's going to help.

The second thing that happens is nobody can really get work done any more. You try to edit a file, make some changes, add a new parameter to an API somewhere. You can't just check the file into CVS (or SourceSafe, in the Microsoft case). You have to find all the people who are using the API and let them know. You have to figure out if anybody else is modifying the same file, for different reasons. You have to stop working, and talk about working instead. God forbid it turns into 3 or 4 meetings, a few specifications, and, conservatively, about 50 email messages.

I am a programmer, among other things. I've changed API's many times. With a good search-and-replace tool and a thorough understanding of all the places an API is used, you can make a sweeping, and I mean sweeping change, without any errors, fully validated and tested, in a couple of hours.

So efficiency is part of it. Confusion, cross-currents, communication, and all the things that fall out from that: backlash, politics, sandbagging, employee turnover, morale problems, managers on a tirade trying to "fix" it; you may have seen some of this. It's preventable, believe it or not.

Just use small teams. Small teams are how the best work in the world gets done. Even if it's a "big project". How many people do you think it took to build the Golden Gate Bridge? I don't know, unfortunately, but I bet you it was at most a couple hundred. If Microsoft built the Golden Gate, do you think they'd put fewer than 2,000 people on it? Do you think it would be on schedule?

I don't mean to pick on Microsoft, though they're the biggest, most resource-heavy example of large-scale product development on Earth. They're certainly not unique when it comes to large-team molasses-style development.

Maybe this is why "outsourcing" is becoming popular. Hey, maybe if we abdicate this project completely and give it to a team in Russia (translation: 1 or 2 really smart people) it will come in on time and under budget?!

Here's a suggestion: outsource it to your 2 best engineers instead. Set them up in Marin with no phone lines. They'll get it done, I promise.


Anonymous said...

...And it's interesting how so many talented people get fed up of working in those huge teams, and strike out on their own to do amazing things in smaller ones. Not a coincidence.

--Jackie Danicki

Valli Sankar said...

Very cool post!

I guess the reason why large teams do exist is because there are people (like Gates) who are good at the business part of it... they want to do / produce more than they or a small group can.

It boils down to personalities. You are a personality type that perfers depth and perfection in what you do... there is the other type that perfers to be the generalist and hire specialists to build stuff.

As long as we have generalists, we will have large organizations and large teams to produce software.

There is nothing wrong with either approach. If we have more people like you in the world, then we can get to a state of perfection... but until then the world shall exist like this.

Really a great thought provoking post!


Anonymous said...

A small team of less than 5 can work great for a smaller software product, but could such a team produce Microsoft Word or some similarly huge application? You can just look at it empirically by considering the number of lines of code (not a great measure I know, but also not awful) for a large app, and how many lines of code a single programmer can produce per year. Delivery dates would stretch way out if you only had 5 people on a huge software project. said...

Small teams can produce beautiful things, and it is possible to have all members be excellent. The joy of working with such a team is immense.

Many projects involve such complexity, though, that this approach fails. You mention bridge design and speculate a couple of hundred people... Hoover Dam had over five thousand workers. The Manhattan Project needed a lot more than six guys. Ditto Apollo.

The greatest managers are the ones who can instill a sense of joy and esprit de corp in a large organization.

This does not diminish the accomplishments, necessity for, or joy of working with a small team. It's just important to have enough perspective to remember what big teams can accomplish - and what small teams can't.

Anonymous said...

you have always been economical in your approach to all things and relationships of quality...beautifully articulated.