From: Ken Schwaber [ken.schwaber@verizon.net]
Sent: Sunday, September 15, 2002 4:53 PM
To: Jeff Sutherland
Subject: RE: Waterfall methodology

Jeff,

Reading Shawn Presson’s comments regarding CMM, waterfall, and agile reminded me of Barry Boehm’s comments at XP/Agile Universe’02 and the comments of several CMM authors at INCOSE’02. I was reminded of the good intentions and sound footings of CMM and Waterfall, both initiated to address problems in systems development and systemic project failures rates that have only marginally improved over the decades.

I was vastly amused by Presson’s suspicion that we’re only contrasting agile and previous approaches to sell books. If I could even afford to drink Starbuck’s coffee with book royalties, I would be pleased. The contrast is painted from two perspectives: the current state of CMM and waterfall, and the theoretical contrast between CMM/waterfall and agile processes.

I was challenged at INCOSE’02; “how do you intend to prevent agile from being turned into a pile of junk, as has happened to CMM.” CMM was transformed from its principles and vision into its practices, KPA’s, certification, and institutionalization by DoD implementations, consultants, carpetbaggers and everyone else trying to make a buck. The same thing happened with structured methods, information engineering, and CASE tools that would generate code from design and design from code. All reduced to overhead and rubble. Of course the kernels of modeling, precision of expression, thinking before coding, and having a process persist, but for many the weight of the commercial implementations have done their damage.

So, some of the contrast and reaction of “agilists” to CMM and waterfall are to current implementations, not the vision nor the intentions.  All of you have to is look at CMM as implemented at most major organizations or waterfall as implemented in such methodologies as Navigator to understand the damage. I was brought into one CMM Level 3 shop because it was so badly implemented that development had literarally stopped. Problems couldn't be reduced to an adequate level of agreement and definition to proceed with coding. However, the rest of the contracst occurs because CMM, waterfall spring from a different theoretical basis than agile processes. Increased precision and definition are at their core, which is one of the alternatives for controlling a process. However, this approach only works when the definition and the problem have a mapping approaching 1:1 and the problem domain (technical, requirements, and people) is relatively simple: “It is typical to adopt the defined (theoretical) modeling approach when the underlying mechanisms by which a process operates are reasonably well understood. When the process is too complicated for the defined approach, the empirical approach is the appropriate choice.” Process Dynamics, Modeling, and Control, Ogunnaike and Ray, Oxford University Press, 1992.

The agile process is based on the empirical approach, accepting the complexity of the problem and addressing it through frequent inspection and constant adaptation. Empiricism is seen even in the Agile Manifesto, where the signatories pose agility as an empirical counterpoint to a defined approach, rather than as a new set of defined absolutes. The various agile processes implement empiricism with the underlying practices of iterative development, frequent inspection of status and increments of work, self-organizing teams, emerging architecture and requirements, and solid collaboration. You will not find many detailed task definitions, pert charts, or assignments in agile processes; by contrast, CMM and waterfall progressively implement themselves by increasing the amount of detail.

Agile is not a silver bullet. It is simply a correct approach to complex process control. It reflects a lean manufacturing approach to software development, and – as manufacturing started doing twenty years ago – eschews the Taylor model of defined process control.

I expect the success of agile processes to be challenged by the same degenerative, buck-making processes that hurt CMM and waterfall. I expect adopting agile processes to be quite difficult in many organizations because of the degree of change required; indeed, our agile processes have primarily succeeded when organizations absolutely need the projects to succeed and can’t risk failure. I expect agile to also be challenged by the abysmal state of engineering practices in our industry, such as code management, release control, testing,  and builds. Agile implementations highlight poor engineering practices immediately, whereas model and definition-based approaches let this failure slide until near the end of the project.

Shawn Presson is a kindred spirit, not a foe. We all are trying to improve the software development process.

Ken Schwaber