Thursday, March 20, 2003

Continuous Integration: A Step Beyond Daily Builds

Today I was talking to Craig Larman about his forthcoming book on a manager's guide to agile development processes. He mentioned that daily builds are for wimps and real developers do "continuous integration." The configuration management system checks for updates to code every few minutes, runs unit tests, runs exceptance tests, publishes results automatically on the web, and emails any offending developers on the need to fix their code. 100 builds a day is not uncommon.

Sounds like we all should be doing this and there are a couple of open source products to help - Anthill, Cruise Control, Krysalis Centipede, and Maven.

Continuous Integration
Martin Fowler and Matthew Foemmel, ThoughtWorks

An important part of any software development process is getting reliable builds of the software. Despite it's importance, we are often surprised when this isn't done. Here we discuss the process that Matt has put into place on a major project at ThoughtWorks, a process that is increasingly used throughout the company. It stresses a fully automated and reproducible build, including testing, that runs many times a day. This allows each developer to integrate daily thus reducing integration problems.

Tuesday, March 11, 2003

SCRUM: Another way to think about scaling a project

It is hard for people to realize how radical SCRUM is because it is such a simple process, anyone can do it. Here is an example to expand your thinking about SCRUM.

Below is data from Jones, Capers. Applied Software Measurement, Second Edition. McGraw Hill, 1997 on industry averages combined with data on one of the key papers that influenced the first SCRUM. Coplien's paper was part of an ATT Bell Labs study that investigated many large projects and analysed the development process, environment, and implementation.

James Coplien. Borland Software Craftsmanship: A New Look at Process, Quality, and Productivity. Proceedings of the 5th Annual Borland International Conference, Orlando, 1994.

Borland Quattro for Windows (BWP) Project - SCRUM-like implementation

1,000,000 lines of C++ code..........BWP..........Industry standard
Time in months.............................31..............>50
Function points per staff month.......77...............2

So in the extreme case for large projects, it is possible to scale to a 500 developer effort with only 40 developers on staff and deliver the project in almost half the time with SCRUM.

This is the goal of SCRUM and was the target when it was invented. Even if your team only gets 10% of the way towards the goal, you will smoke your competition.

SCRUM: Get Your Requirements Straight Before Coding

Here are some comments on requirements needs for a SCRUM motivated by postings in the list. Here I am talking about product companies. There are parallels for IS internal development.

The SCRUM's that I work with deal with changes to the requirements during the sprint iteration. The product manager is part of the SCRUM for this reason. The development SCRUM does not define initial requirements unless they are a product marketing SCRUM.

We have always started a development SCRUM with some working code. Thus a SCRUM is designed to improve a codebase. The functional specification provided by marketing should be based on running prototypes shown to customers or potential customers who agree that is what they want for the next release. It may be one sprint or several to get to the release.

My current customers are physicians and they don't have time to meet in the daily SCRUMs, so product marketing physicians have to be their surrogates. Physicians have taught me that a product WILL NOT BE USED, unless it carefully meets their workflow, is extremely responsive, and can be customized quickly whenever it has shortcomings. Since failure is immediate by physician refusal to use the product for more than a few days, it must be close to right the first time and fixed within a month, or you fail and have to go find another customer.

This tight discipline required by the physician environment has made me realize that all projects should be run with this discipline. If they are not, failure may take longer, but the system ultimately does not sell well or get used well. So I think my comments are relevant in other environments. You won't win big unless you do this.

In addition, if product marketing is not clear about customer (or market) needs, neither is senior management. If senior management is not behind a project, priorities will get confused. This will sap focus and resources from the SCRUM and reduce probability of success.

If product marketing can't function, development has to do it. They will not do it as well and they should not treat it as cutting code. They must get out of the office, do market research, and get into the customers offices unless they can bring the customer to them. They must do sales support and be there when the customer is buying or not buying and understand why or why not. They must give up their development job long enough to define the use cases, get a user interface the customers love, and clarify the customer workflow and business logic. When that is done, they can think about coding the product.

In a greenfield project without existing code, I have seen a development SCRUM work on a prototype for a week and define it as their code base.

The task then is to refine the code base to better meet customer need. If that is not clear, the programmers should not write a line of code. Every line of code costs money to write and more money to support. It is better for the developers to be surfing than writing code that won't be needed. If they write code that ultimately is not used, I will be paying for that code for the life of the system, which is typically longer than my professional life. If they went surfing, they would have fun, and I would have a less expensive system and fewer headaches to maintain.

When the customer need is clear you write the minimal lines of code to meet the defined need. This is the XP view and should be the SCRUM view.

That said, product marketing should be run as a SCRUM. I once led a senior management team (CEO, EVP, SVPs, and Product Marketing Directors) as a product marketing SCRUM. With one lunch meeting a week they made more product changes and inovations in 6 months than they had made in the five previous years.

This topic probably requires a book, rather than a note. So keep the comments coming.

Tuesday, March 04, 2003

Object Technology: Web Services

ACM has just shipped the premier issue of Queue, billed as not just another magazine, but one that "deals directly with the challenges ahead for software engineers and developers involved in the critical design and planning decisions most likely to affect product directions and operational practices for years to come." The first issue is on web services which is the object technology du jour, component engineering wrapped up in a new package.

I just reviewed a paper for a conference where the authors advocated defining web services using a language designed for intelligent agents. The authors presented a well reasoned and clearly articulated case. However, they did not show how to get from the stuff the average developer writes today, to an environment with interoperating, interacting, goal seeking agents play.

The first question that came to mind it, "What is a web service." Well, an interview with David Bosworth in the premier issue of Queue answers the question:

Kirk McKusick (Queue): People sure talk a lot about Web Services, but it's not clear they're all talking about the same thing. How would you define "Web Services?"
Adam Bosworth: The term Web Services refers to an architecture that allows applications to talk to each other. Period. End of statement.

So we are in the Alice in Wonderland phase of the hype cycle. Web Services means whatever I want it to mean in the current conversation, which may change in the next conversation depending on how I am feeling.

Bosworth was a senior manager at Microsoft responsible for XML industry standards definition and is now the Chief Architect and SVP of Advanced Development at BEA Systems. So his definition is as good as any.

My critique on approaches to web service architectures is that proponents should clearly define what a web service is, and make the case that definition is reasonable given the confusion that exists in the industry. They they should then clearly lay out a layered architecture like the following:

1. Define how objects are useful for creating components.
2. Describe how an enterprise component architecture can be used to expose APIs that do useful stuff.
3. Show how these API's can be exposed to the world as callable XML components, the lowest level of web service stuff.
4. Describe how distributed internet workflow can be layered on a group of heterogeneous low level web services to execute a business process that includes multiple business partners.
5. Show how a Real Web Service can be defined that executes a clearly defined workflow that completes a useful business process.
6. Describe how an external request can search for such a workflow, understands its inputs, outputs, and constraints, and contract with the workflow service provider.
7. Then tell me how to build intelligent agents that can converse with one another, and execute goal directed behavior that seeks out appropriate workflows that get useful stuff done.
8. Make sure any legacy technology can be wrapped and used in this architecture.

And don't just tell me that defining web services as intelligent agents solves the problem of "Web Services." We haven't even defined what a web service is yet.