Abstraction

Ah, the Waterfall method. For 40 years software companies have attempted to implement a pure Waterfall method, without much success. Even the paper most often (inaccurately) attributed as the origin of the method (the 1970 essay “Managing the Development of Large Software Systems” by W. W. Royce”) describes the Waterfall as prone to error and cautioned against using it. And yet it remains very popular. What’s going on?

To review. The Waterfall method is a software development process in which a product is created in a series of discrete stages. For example, a set of software requirement are written, then the product is designed, then built, then tested, then installed and maintained.

On paper, it’s rational. This rationality is key to its continued popularity: the many milestones and associated deliverables (requirements documents, design documents, test plans etc.) make people comfortable. There is a clear road ahead, a discrete series of steps toward success, all of which can be measured.

Unfortunately, development just doesn’t work that way. Software is hard, and it’s messy. For example:

• Requirements are incomplete when design and development begins. Feature requirements written at the beginning of a project will always be incomplete because the design process will dig up a many “derived” requirements that were not initially obvious. It’s not possible to think through every iteration, use case, and failure cases of a complex software product in the abstract — things become clear with time.
• Development estimates are wrong. Software estimates (i.e. the amount of time a project will take) are typically performed at the beginning of a software life cycle. Unfortunately, this is when the project is least well understood. Designs aren’t fully worked out. Complexity isn’t fully appreciated. Optimistic assessments are made relating to the eventual quality of the code and the amount of time that will be required for testing and debugging. A corollary to this rule is that once an initial estimate is made, it’s rarely updated as the project wears on.
• Requirements change. Customers change their minds. They add new requirements, they modify old requirements. Often they don’t really know what they want, don’t know how to prioritize, and don’t fully appreciate the impact a change can have on the development process. These requirements changes require time to document, design, develop and test, and out of this process even more derived requirements emerge.

Truth is, I don’t believe the Waterfall method really exists. I’d bet that no software project of any meaningful size has ever been developed in discrete stages. A good product manager understands this, and implements process to accommodate randomness, to anticipate chaos. This flexibility anticipates incomplete requirements, poor development estimates, and changing feature sets.

I don’t mean that a product manager should attempt to overturn process within a company committed to the Waterfall method (by insisting that an Agile process be adopted, for example, or a rational unified process). That’s not usually realistic or possible. You can borrow techniques from other methods, however, and integrate those techniques into the existing process.

For example, I’ve had success implementing a daily “semi-Scrum” meeting with all the stakeholders working within a Waterfall-style project. The meeting fostered a common understanding of every detail affecting the project and forced an action toward resolution. It doesn’t matter what specific methods you use, as long as you can answer the following questions in detail:

• How do we decide what to do with a change request? That is, how do we determine whether or not a change request (a new feature, or a altered requirement) meets business goals and therefore should be integrated into the project, deferred, or discarded?
• How do we ensure that development estimates are accurate?
• How do we manage potential delays in the project, or make decisions in terms of trading quality, feature depth, or delivery time.

Incidentally, understanding these issues is a basic measure of an experienced product (or project) manager. It’s amazing how many people I’ve interviewed that present themselves as experienced and savvy, and can about deliverables, document templates, and milestones, but are unable to meaningfully discuss how to manage the problem of unexpected change.