Engineering Process

[I found this document in my archives from a little while back and thought it would be useful to somebody out there, so I'm posting it.]

Introduction and Philosophy

You have a small team of people who are highly motivated and work well together. The right process for this group will be one that allows and encourages the organic development process while preserving clarity of purpose and minimizing surprises.
 
The big ponderables are: “what exactly are we building?” and “when will it be done?” and, to a lesser extent, “who’s going to do what?”

Schedules and Deliverables

One of the hardest things to do in building software is to accurately schedule a release and then hit the schedule. There are various reasons for this but the primary one is simple: shooting at a moving target.

It is easy to suggest that nailing down a specification and building to the specification is how to avoid schedule slips and shooting at a moving target. However, that is unrealistic for almost any organization except those that are large enough to be able to absorb errors of large magnitude.

A better philosophy for a startup is to understand that there are two variables and that either may be changed, and that they interplay. The two are delivery date and features/functionality. This seems obvious on the surface, but it is critical to manage this willfully rather than have it sneak up on you.

Axiom: if you fix the delivery date, the functionality becomes elastic and indeterminate. If you fix the feature set, the delivery date becomes elastic and indeterminate. It is best to fix one or the other, and not to let both of them float.

Alpha / Beta / Bake / Ship: A Convergence Model

Developing and shipping software is a bit like cooking. You plan a menu, buy ingredients, and try to choose a time when dinner will be served. Some things need more preparation than others, but in order for all of it to come together at a prescribed time, you must commit sooner or later to each dish, when to start the final preparation, and how long it is going to take. The dishes, preparation, and use of burners and utensils must converge on the final time. It is as much art as it is science, but it is possible to reproduce it scientifically. Software is like cooking in the sense that the more times you’ve prepared a dish, the more accurately you can predict how and when it will be done. And the more people who are in the kitchen, kibitzing and drinking wine and chatting with you, the more inaccurate your estimate will be.

In order for software to gel into something you can give to your customers, there needs to be a period of time, often called bake time, where new development stops and bugs are fixed and real regression testing is done in earnest. The duration of this period varies with the complexity of the software, the scope of the changes that have been made, and how it is delivered (burned on CD and replicated, pushed to a web site, etc).

For convergence of a large project (i.e. a major release, not a minor feature or bug fix) it is good to separate the convergence into two phases, usually called Alpha and Beta. These imply both a milestone and a period of time. The milestone is the beginning of the period of time. A product is said to “hit Alpha” when the criteria for Alpha are met, and the Alpha period is then entered. At the end of Alpha, when the product meets the criteria for the Beta milestone, it “hits Beta” and enters the Beta period. Often there is a third, shorter period following Beta which has various names, like “golden master” phase (coming from the development of software which is burned onto a disc) or simply the “bake period”. These terms are overused and poorly understood, but a good working definition of the milestones and phases
is:

• Alpha:  All features are in and working to the point of being demonstrable. That is, you could demonstrate them to someone else. To properly meet an Alpha milestone all aspects of a feature must be in and working, not just the fun/interesting part. For example, if a feature has three separate states/interfaces based on whether a person is a site visitor, a logged-in user, or a content owner, then all three of these states must be working. Bugs are tolerated, as long as a feature can be demonstrated to work at least once. The Alpha phase is not a time to finish features, it is a time to fix bugs and re-engineer any parts of a feature that can’t be made to work reliably, to squash bugs. This is important because some bugs result from actual design flaws, not just coding errors, and the bugs can’t be truly fixed without re-engineering. The Alpha phase differs from Beta primarily in that such minor redesigns are expected, and encouraged. Do it now, rather than sweeping the bugs under the rug and hoping that the design holds. Do not introduce new features, no matter how tempting. To do so will revert the product to pre-Alpha state.
  
  
• Beta:  All features are working smoothly and there are no major bugs that might suggest design flaws or major effort to fix. The Beta phase is spent polishing user interfaces, fixing bugs, and enhancing the user experience. Do not introduce new features, no matter how tempting. To do so will revert the product to pre-Alpha state.
  
  
• Bake Period: To mark this milestone, and enter this phase, you stop changing the source code. The expectation is that you can ship what you have. You test it, and test it some more, and find bugs. Resist the temptation to fix every bug you find and spin a new release. Every change to the source code resets the bake period. Think of it this way: sooner or later you will quit changing it and you will foist it on your users. In the bake period, you are pretending that you have shipped it, and you are simulating real use. Some number of bugs will come up after you’ve shipped the product, and you may or may not choose to fix them, depending on severity. But the bake period is not effective if you continue to fix bugs. It is simply a simulation of holding your breath and hoping there aren’t any barn-burning bugs in the software. If you find a barn-burner, and you all agree that the barn really will burn down if you don’t fix it, by all means fix the bug, and reset the bake period. Don’t omit this important step of resetting the bake period. You are, once again, pretending it is live. Each change, no matter how minor, violates that.
  

For a full-blown application I would suggest the following (minimum) durations for each of these phases:

Alpha: 4 weeks
Beta: 4 weeks
Bake: 1 week
 
This means that you quit adding features 8 weeks before you ship it, and you spend 4 weeks tweaking and improving the user experience of features that you’ve chosen to implement.

Using the Model

I once took a class in Marksmanship at the college I went to (University of Wisconsin) just for the hell of it. It was a zero-credit class, and officially was part of the ROTC program, though it was a very radical/pacifist school when I went there. Go figure. Anyway, a key takeaway from that class was as follows. When you hold out your arm and point a gun at a target, you should let your arm extend as naturally as possible. If you need to adjust left or right to hit the center of the target, you don’t move your arm, you move your feet. You arrange your whole body into a shooting position, then adjust your whole body toward the target, not just your arm. This is an amazingly good metaphor for all kinds of things.

If you adopt the Alpha/Beta/Bake/Ship model for developing and shipping software, it is important to think of the whole model as your whole body (to apply the metaphor), and if an adjustment is necessary, adjust the whole model, don’t try to hack/fake/adjust one part of it, or throw the whole model out the window.

It is a good idea to spend some time specifying your own criteria for each of the milestones, so you can all agree on whether or not they’re met. I’ve offered my own definitions (“feature-complete, all features demonstrable”, etc). Your mileage may vary. It is also important to establish the durations of your phases, though they can be adjusted (see below). But decide on milestone criteria and phase duration, and apply them. The model will reward you.

Note: this model, although it seems formal and big-company-ish, is really just a simple way to force yourself to really look at your software at each phase, and apply criteria to convince yourself, through measurement, that you really are where you want to be. You can fool yourself if you want to, but software is a bit like dieting: if you never step on a scale, you can pretend to be any weight you want, but if you really want to make progress, and understand honestly where you are, step on the scale every day. It won’t change what you weigh, it will only let you know the truth of what you weigh, removing the human temptation to fool oneself.

Adjusting the Schedule

Using the model just described, and the concept of adjusting your feet, not your arm, there are a few ways to adjust without throwing out the model.

Phase Adjustment:  Adjust the amount of time spent in each of the Alpha, Beta, and Bake periods, to reflect the nature of your development, testing, and anxious users. This is an important and flexible aspect of the model. The duration of these phases is up to you. It is also
important to note that since criteria are applied to each phase/milestone, it is possible to skip a phase completely. I’ve seen it happen. Software occasionally meets both the Alpha and Beta criteria at the same time, and you can skip over the Alpha period. This might result in your being actually ahead of schedule if the remaining phases hold. This is rare, but achievable. Resist the temptation to celebrate by adding new features.

Resetting Milestones:  This is difficult, but necessary. If you add a feature, reset your process
model to pre-Alpha state. You can accelerate through the milestones/phases if the criteria are met, but it’s critical not to throw the whole model out the window just because you’ve decided to add a feature. “To add features is human; to reset the model to pre-Alpha, divine.”

Again, stick to the model and it will reward you. It is a discipline, but it is important to note the difference between a milestone and a phase. When you reset because of a new feature, you may be able to sail through the phases more quickly because the change is small, and your engineers are brilliant and got it right the first time. But it’s still good to re-apply your criteria and test and make sure you’re still good.

Using CVS

CVS is your friend, believe it or not. It records the exact state of your software and supporting files, and will give them back to you at any time of your choosing. It also tracks changes so you can assign blame, or review changes, or move backwards if need be.

The right way to think of your software is that there is “one true version” and that it is the one that is currently top-of-tree in CVS. If you embrace and use CVS it, too, will reward you.

If you modify a file, always do it by first syncing with CVS (e.g. do a “cvs update” first to make sure you’re working from the current version of all the files). Then make sure to check your changes in directly from the working copy. Do not copy files out of the CVS sandbox and then copy them back after a ‘cvs update’.  You will stomp somebody else’s changes this way.  Just work “live” on a CVS sandbox copy at all times, and commit the changes when you’re ready.

Scaffolding

It is important that anything you commit to CVS be internally and externally consistent.  The paradigm is: don’t break the build! Don’t make any changes that will cause existing features not to work, unless the changes are either commented out or conditioned on an “ifelse” that is false. That is, you can have “if ( 0 ) { your_stuff(); } else { current_stuff(); } and set the “0” to “1” while you’re working on it, but leave it at “0” when you check it in.

While you’re working on something there will be temporary states and what I think of as “scaffolding” that will be removed later, but makes the project safer and provides fewer surprises while it is being worked on. Consider your scaffolding carefully to avoid committing anything to CVS that is not production-quality, that you wouldn’t mind making its way into a
live version of the software.

Communication: What Changed?

Effective communication between Engineers and “everybody else”, especially testers, is important to good quality management. “What changed in this build?” is a familiar refrain in any software company. 

It’s hard to remember what you changed, and why, when you’re working on software. Writing release notes after the fact is unwelcome and error-prone: you just forget. There are various ways to mitigate this. The methodology that I find works best works by side-effect of using CVS
effectively.

Imagine that you are an engineer and you spend 3-4 hours working on the software, fix a couple of bugs, half-implement a new feature, add some comments to the code. You feel that it’s time to commit your changes to CVS. What do you do?

CVS Diff:  You type “cvs diff > changes.txt” and you take a look at what you changed. You look through the diffs to see exactly what you changed. Sometimes there will be something there that you forgot to remove (scaffolding). Sometimes there will be a bug-fix that you forgot about. Sometimes it will just be just as you thought. But this is an important step in software development, to diff your changes before you submit them. It’s a final check for accuracy, a memory jog, and just good practice. But keep the diffs either with copy/paste or some other method.  You’ll need them later.

Checkin comments:  Type in a few lines when you commit your changes to CVS to explain what the changes were. Keep a copy of this too. You’ll need it for the third step.

Send Email:  Send an email to your “dev” alias or make up a new one for “code changes”. Send the text of your CVS comments, and the text of your CVS diffs. Add a Cc: copy to the code-change wiki for extra credit (it will log all changes for future perusal).

The discipline of reviewing your own changes, summarizing them, and sending an email to whomever cares will make for ad-hoc release notes and will add an extra valuable step to the process. Not only will you review your changes one last time before committing them, it gives your team a chance to review your changes too, to see the magnitude of changes (if it is in the
Bake Period, for example, when you’re not supposed to be changing anything), to provide an extra set of eyes to help spot bugs/issues, and just to give people that sense of “what changed?”

Testing / Versioning

There are as many approaches to testing as there are QA teams. I won’t recommend any particular methodology. What I will recommend is strong versioning.  Make a visible version number somewhere in each build, and make sure you change it every time you change the software. Don’t be lazy. My mottos is “integers are cheap.” Use them liberally. Add one to the version number at every change. Don’t ever let two slightly different versions of the software be indistinguishable by a tester, validator, or engineer. Then make sure all bug reports accurately report the version of the software in which the bugs are found, and make sure the Engineers always mark each bug with the version in which the bug-fix is expected.

Conclusion

The building blocks of a good process boil down to these things:

Alpha/Beta/Bake/Ship:  Agree on your milestones and phases, and stick to them religiously. They don’t have to be big, dramatic things. Just be clear about what you’re trying to do, and when you’ve done it. Don’t lie to yourselves.

Trust and Use CVS:  CVS is your friend, and if you build your process around it and never deviate, you will always have repeatable, surprise-free releases.

Stick To It:  Adjust the process (move your feet), change the milestone criteria, reset the ship date, but don’t abandon the model. Make it work for you, in a way that makes sense to you, but stick to it. It will reward you with a smooth, surprise-free development/release process.

You can’t avoid bugs and bad software. No process can save you from that. But you can avoid silly mistakes, human error, and surprises, and you can feel good that you are releasing what you’ve tested, you’ve built what you thought you built, and nothing snuck in around the edges to ruin everything.