Response to Brook's "No Silver Bullet" Essay

I turned this in as an essay for professor David Russo’s Software Planning and Management class at UT Dallas (SE 4381, Spring 2006) prior to posting it here.

I ramble at the end, but I think the essay includes a few ideas worth sharing.

In his essay “No Silver Bullet,” Frederick Brooks presents the view that software development is an unusual field because we will never discover a single technique or technology which provide massive increases in development productivity. Brooks claims that this is due to the intrinsic complexity of software development. I concur with Brooks on the general claim that there is no silver bullet that will slay the “demon” that is software development productivity, but I dispute much of his explanation of this principle as well as his reasons for arriving at it. I argue that the closest we can achieve to a silver bullet is to construct software components with reuse in mind, but that the market works contrary to the goal of improved productivity.

Before expending mental effort on whether Brook’s holy grail exists, we must ask: why would we even be looking for a silver bullet? Most fields lack one. Brook’s grounds for expecting a silver bullet in software are the great gains made in the related field of computer hardware production. This makes little sense to me. If lumber prices were at an all time low, would we expect the task of an architect to rapidly become easier because of its association? Surely not. Hardware seems to be the exception rather than the rule. Furthermore, computer hardware production is still a relatively young industry, and we cannot assume that current hardware trends will continue indefinitely.

Brooks is correct in saying that complexity is an intrinsic property of software. A piece of software is generally written to solve a specific problem. Software is written to solve hard problems—quite simply, problems which require the power of computers. Some useful pieces of generic software have indeed been written, but when software is most useful is primarily when it is written for a specific problem. No piece of software is ultimately generic, just as none is ultimately specific; rather, every piece of software is written with a given amount of specificity towards every domain it can possibly aide. Generally we find that greater specificity means greater gains in the relevant field. A handyman cannot carry a single tool for all needs—pliers may be a great general tool, but if you really want to gap a spark plug, a spark plug-gapper is the ideal tool. The same is true of software tools.

Ex vi termini, generic applications cannot natively address specific software needs. Often a generic piece of software can be configured to solve a specific problem, but the software lacks the desired function prior to this customization. The more specific the problem, the more configuration is required. But this configuration is a form of programming in itself. The question comes down to, “should the programming be on the head or the tail end of the software life cycle?” A spreadsheet can be used to implement invoices and bid sheets, but not without either the user or the software producer writing these problem-specific sheets and macros. We cannot expect to escape the general principle of needing to program towards a specific problem. But we may be able to make programming towards a specific problem easier.

Configuring a generic piece of “software”—essentially high-level programming—can produce large productivity gains over building a unit of software from scratch (whether generic or not is of little consequence). If useful pieces of configurable, generic software can be utilized, then we can greatly reduce our production costs. Writing an invoice application built on spreadsheets allows a much more rapid process than writing an invoicing application from scratch, if (and only if) the spreadsheet application allows for the specific kinds of customization the (sub-)application writer requires.

In short, the key to improving software productivity is the availability of reusable, programmable general purpose software tools. It may not be a silver bullet, but it is a bullet. By “programmable” I mean that the components should not only be able to fulfill a wide-breadth of similar needs, but also that they should not stand in the programmer’s way. We must minimize our tools’ ability to restrict our usage of them. Software should not get in the way. Part of not getting in the way means that components need to be combined easily. Ease of combination means widespread use of abstract interfaces, open standards. Software components can’t “talk” to one another unless we provide a formal means to do so.

But when combining many different components, possibly from different vendors, development complexity can increase tremendously. In-house coding and design standards may no longer be followed, and code-reading becomes more difficult. Even combining pre-built modules can become difficult when they are designed with different development paradigms in mind. As software engineers, we must work to reduce this complexity through the use of stable, encapsulating interfaces. This encapsulation need not restrict the programmer’s ability to access component internals—as, indeed, many modern programming language constructs have enforced—but simply to ensure that a component has few external requirements. As long as a component can be utilized without the programmer knowing its internal function, the most important goal of encapsulation has been met. When Nike designs a new pair of tennis shoes, Nike doesn’t have to know how to make produce rubber for the sole—the rubber company takes care of that. Leave rubber to those who have spent decades perfecting the process. The software industry has much to learn from other industries.

But our various incompatible languages and libraries are working against this goal. The general principle behind 90% of programming languages is identical—is it really that difficult to make them inter-operable? It is certainly not impossible, but doing so has not been undertaken as a priority. How many basic TCP/IP socket libraries does the world really need? One? Two? Yet hundreds have been written.

Certainly generic mechanisms are less efficient than problem-specific solutions, meaning that development productivity works contrary to product efficiency. But as computer hardware technology improves, software efficiency becomes less important. Certainly a slight reduction in program efficiency would be worthwhile if it could reduce the number of critical bugs in the application—and this is what we are finding to be true. Indeed, we are now realizing that it is productive to trade some program efficiency even for such intangibles as code-readability because of the long-term impacts of unintelligible code.

But we are quickly leaving behind the days of programming to the machine, and entering the days of programming to the idea. We need languages written to weave ideas, not to toggle bits.

Most modern programming languages place too much focus on how things should be done, when they should be focusing on what needs to be done. It is of trivial importance whether a set of boolean values is stored in integer variables or combined into a bitmask; these kinds of issues don’t matter to the modern application designer—he shouldn’t live so close the hardware. We should no longer tolerate needing six lines of code to sort a list of values. The idea can be expressed (nearly-)unambiguously in two English words: “sort list”. Modern programming languages should either match this terseness or come close.

The reign of C++ and Java is coming to a close—or, it rightfully should be—and the days of Lisp are finally upon us. Brooks says, “I believe the hard part of building software to be the specification, design, and testing of [the] conceptual construct, not the labor of representing it and testing the fidelity of the representation.” But in software, the difference between design and implementation is the ineffectiveness of the programming language. Software is built on nothing more than ideas, we simply need the appropriate language to write our ideas down in. A software engineer is like an architect with an unlimited supply of building materials and a flawless construction crew—all he needs is a sharp pencil to draw his plans. But as software engineers, we are left with only crayons. If we can use the appropriate language choice and concept-encapsulation to bring programming and idea-specification to be the same activity, then suddenly the difficult portion of building software (the former portion of Brook’s statement) becomes significantly easier (as the latter portion of the statement).

Unfortunately the free market works contrary to the two great potential software advances I have discussed: using better languages, and reusing code. The former because the market is resistant to change, despite the known rapid advances in technology. The latter because one does not simply aide the competition. When our ideas are our product, we can’t simply share ideas.

When you are making engines, you own an engine factory, have ties to steel manufacturers, wire-makers, etc. These material goods leave you with something (something large, actually) even if someone else has your engine designs. You may be able to make the engine cheaper, or make it from better materials. Your competitor even has to have the capability to make an engine (most people don’t). With software, the paramount design document is the source code. The source code is both design and implementation. If someone takes your source code, you have lost your product (excluding any legal recompenses). With the triviality of duplicating a piece of software (any computer user can do it), and the subsequent wide-spread piracy of software, we cannot make profits on unit sales alone in an unprotected market. We must protect our ideas and hence our code.

Software completely defies the standard notion of laissez faire capitalism in a fashion similar to media with the invention of mass-publishing techniques. We either need to implement new mechanisms for making money from software in a capitalistic-friendly manner, or we need to accept that the software development process will never advance as quickly as we’d like because the market will slow it down.

Software quality is poor because it doesn’t have the full force of capitalism to help it along in the same way that many other industries have benefited.

In summary, I agree with Brook’s claim that there is no silver bullet in the software industry. Nothing is going to solve all of the industry’s problems. As software designers and programmers, we search for elegant solutions to all problems, but not all real-world problems have such a solution. The greatest gains we can achieve in the current state of the software industry stem from using (and developing) programming languages which allow us to build towards ideas rather than implementation details, and from the use of generic, programmable code modules.