Model Driven Development: Do it or you will be outsourced!

More and more IT jobs are being sent to India. Some of the best software technologists in the U.S. are Indian graduates of Stanford and MIT. They have built successful U.S. consulting organizations and have access to an unlimited number of Indian graduates of technology universities in India. They select only the best of Indian graduates who are better trained than the average American programmer and are employable at 1/10 the cost. Therefore, American programmers must find a way to be ten times as productive or they are history, except for managing outsourced projects and performing local maintainance. Yourdon took a lot of flak for comments like this in:

Yourden, Ed. The Decline and Fall of the American Programmer. Prentice Hall, 1993.

Yourdon wrote The Rise and Resurrection of the American Programmer in 1996 at the beginning of the dot.com era. The internet boom slowed down the rate of outsourcing until the dot.com bust. Now it is back with a vengeance.

The way out of this dilemma is Model Driven Development (MDD) and the Sep/Oct 2003 issue of IEEE Software is devoted to the topic. Steve Mellor, the model driven enthusiast among the authors of the Agile Manifesto, leads off with some recommended reading:

Mellor, Stephen et al. Model-Driven Development. IEEE Software 20:5:14-18, Sep/Oct 2003.

The current focus on Agile Development has deemphasized modeling tools. I've noticed (and this was confirmed by a lead architect for Rational recently) that the industry is moving more towards a visualization of the code base with less focus on building UML models which help to generate code.

Mellor argues that advances in knowledge, tools, and industry standards makes it possible to move up a level of abstraction and use UML, which is now computationally complete, to generate systems and to maintain the systems at the model level.

In 1994, I worked with a group of leading thinkers at the Object Management Group to start a movement towards model driven development. Initially this effort was in the Business Object Task Force and then moved to the OOAD Task Force with the adoption of UML as a standard. More recently, a Model Driven Architecture became a core program of the OMG. Much of the early history is documented in:

Sutherland, Jeff. Why I Love the OMG: The Emergence of a Business Object Component Architecture. ACM StandardView 6:1:4-13, 1998.

Rapid advances in chip technology have been driven by model driven development. The productivity increase in software development has not been signficant because we still write programs in 3rd generation languages, a technology going on 40 years old. When enough of the old way of doing things has been outsourced we may begin to see a change.

Parsers: When you need one you need one ...

It's vacation time so some interesting thinking can be done. More than just figure out how to meet my software delivery schedule.

Problem: Use a universal language to drive an electronic frequency generator used for killing pathogens in the human body to drive alternative frequency devices. The best language available is for the F100 and I want to use it to drive an FSCAN for you gadget freaks. The inventors of both devices have provided me with helpful documentation.

History: Most programmers hack up parsers without thinking. I once inherited a COBOL team that had hand crafted a multi-million line COBOL program for generating reports on a large banking software product. Average time to fix a bug in the report language parser was infinite (some could never be fixed). I took two C programmers, fed them the Dragon book, and in six weeks they replaced the parser with standard tools, eliminating hundreds of thousands of lines of code and 80% of the bugs out of the box. Average time to fix any remaining bug - less than an hour.

Question: What's the best open source parser for a small job like this?

Answer: Spirit

Caveat: If you have a better one send me a note ...

Why would you want to use Spirit?

Spirit is designed to be a practical parsing tool. At the very least, the ability to generate a fully-working parser from a formal EBNF specification inlined in C++ significantly reduces development time. While it may be practical to use a full-blown, stand-alone parser such as YACC or ANTLR when we want to develop a computer language such as C or Pascal, it is certainly overkill to bring in the big guns when we wish to write extremely small micro-parsers. At that end of the spectrum, programmers typically approach the job at hand not as a formal parsing task but rather through ad hoc hacks using primitive tools such as scanfs. True, there are tools such as regular-expression libraries (such as boost regex) or scanners (such as boost tokenizer), but these tools do not scale well when we need to write more elaborate parsers. Attempting to write even a moderately-complex parser using these tools leads to code that is hard to understand and maintain.

One of the prime objectives is to make the tool easy to use. When one thinks of a parser generator, the usual reaction is "it must be big and complex with a steep learning curve." Not so. Spirit is designed to be fully scalable. The framework is structured in layers. This permits learning on an as-needed basis, after only learning the minimal core and basic concepts.

For development simplicity and ease in deployment, the entire framework consists of only header files, with no libraries to link against or build. Just put the spirit distribution in your include path, compile and run. Code size? Very tight.

Security: Snake Oil Warning Signs

Here is a useful overview of security with pointers to the right FAQs and articles. An excerpt will give you the flavor...

Snake Oil Warning Signs: Encryption Software to Avoid

"Trust Us, We Know What We're Doing''
Perhaps the biggest warning sign of all is the ``trust us, we know what we're doing'' message that's either stated directly or implied by the vendor. If the vendor is concerned about the security of their system after describing exactly how it works, it is certainly worthless. Regardless of whether or not they tell, smart people will be able to figure it out. The bad guys after your secrets (especially if you are an especially attractive target, such as a large company, bank, etc.) are not stupid. They will figure out the flaws. If the vendor won't tell you exactly and clearly what's going on inside, you can be sure that they're hiding something, and that the only one to suffer as a result will be you, the customer.

Technobabble
If the vendor's description appears to be confusing nonsense, it may very well be so, even to an expert in the field. One sign of technobabble is a description which uses newly invented terms or trademarked terms without actually explaining how the system works. Technobabble is a good way to confuse a potential user and to mask the vendor's own lack of expertise...

Secret Algorithms
Avoid software which uses secret algorithms. This is not considered a safe means of protecting data. If the vendor isn't confident that its encryption method can withstand scrutiny, then you should be wary of trusting it.

A common excuse for not disclosing an algorithm is that ``hackers might try to crack the program's security.'' While this may be a valid concern, it should be noted that such ``hackers'' can reverse-engineer the program to see how it works anyway. This is not a problem if the algorithm is strong and the program is implemented properly.

Using a well-known trusted algorithm, providing technical notes explaining the implementation, and making the source code available are signs that a vendor is confident about its product's security. You can take the implementation apart and test it yourself. Even if the algorithm is good, a poor implementation will render a cryptography product completely useless. However, a lock that attackers can't break even when they can see its internal mechanisms is a strong lock indeed. Good cryptography is exactly this kind of lock...

Revolutionary Breakthroughs
Beware of any vendor who claims to have invented a ``new type of cryptography'' or a ``revolutionary breakthrough.'' True breakthroughs are likely to show up in research literature, and professionals in the field typically won't trust them until after years of analysis, when they're not so new anymore.

The strength of any encryption scheme is only proven by the test of time. New crypto is like new pharmaceuticals, not new cars. And in some ways it's worse: if a pharmaceutical company produces bogus drugs, people will start getting sick, but if you're using bogus crypto, you probably won't have any indication that your secrets aren't as secret as you think...

Experienced Security Experts, Rave Reviews, and Other Useless Certificates
Beware of any product that claims it was analyzed by "experienced security experts'' without providing references. Always look for the bibliography. Any cipher that they're using should appear in a number of scholarly references. If not, it's obviously not been tested well enough to prove or disprove its security...

Unbreakability
Some vendors will claim their software is "unbreakable.'' This is marketing hype, and a common sign of snake oil. No algorithm is unbreakable. Even the best algorithms are susceptible to brute-force attacks, though this can be impractical if the key is large enough...

"Military Grade''
Many crypto vendors claim their system is ``military grade.'' This is a meaningless term, since there isn't a standard that defines ``military grade,'' other than actually being used by various armed forces. Since these organizations don't reveal what crypto they use, it isn't possible to prove or disprove that something is ``military grade.''