Note: www.cdegroot.com is in rebuild. Please accept my apologies for broken links, missing stuff, etcetera - more
  Home

Towards Organic Software

Towards Organic Software

Cees de Groot

$Revision: 1.2 $

This paper discusses why software engineering is a bad idea. It talks about some interesting developments in science and how they should affect our view towards software development.


Copyright (C)1999, Cees de Groot

Reproduction of this publication without prior permission by the author prohibited.

1. Introduction

Since 1986, people have been paying me to write software. Since 1986, I've mostly made a mess of it, and the only reason people still are paying me to write software is the fact that my mess is at worst the same, but luckily often less a mess than the mess my colleagues make. That's what makes you a good programmer: not fouling up too much.

This is an exaggeration, of course. However, this industry's state is a sad one: software is just as buggy as it used to be, and users haven't become measurably more productive. At the same time, businesses are throwing awful lots of money at solving the same problem: making information available and usable.

[Brooks95] already told us that there is no silver bullet. There is no magical method of making things work like they seem to work in "other engineering disciplines". I quote that because it implies that making software is an engineering discipline, and I don't think it is. At least, not one that is comparable with the standard engineering disciplines, which seem to have gotten their act reasonably well together (when is the last time your car crashed?).

This paper attempts to analyze why progress has been so slow, and tries to reason from first principles why current software engineering methods don't work very well.


1.1. Feedback

This is a work in progress, so feedback is very welcome. For feedback, please use the Organic Software Page on my website's Wiki or send me an email at .


2. Order, Chaos, Complexity

2.1. Order

Most of the time since Galileo and Newton, science has been working under the assumption that things behave in an orderly, predictable manner. If you want to know where and at what time Venus will rise tomorrow, you can just plug some values into some equations, press on the "Calculate" button, and read the answer. This attitude started with Newton discovering the mathematical formulae behind gravitation and other basic mechanics, and it has spread out to most other sciences as well. The assumption is that nature follows laws, you can deduce these laws from observing behavior, and then you can make predictions about what will happen by plugging values into these laws and calculating the results. Modern science has achieved most of what we know today by following this framework, so at the very least it seems a workable model of how nature operates. One of the prime reasons that governments and businesses pour enormous amounts of money in scientific research is indeed the very hope that models will result, allowing the Maecenas to use the model to make predictions - about whether the moon lander has enough fuel to get back, or whether raising interest will help to temper inflation, whatever.


2.2. Chaos[1]

However, all these models assume that systems behave in a linear manner. They assume that if you plug in more-or-less the correct initial value (say, for the gravitational pull of the moon and the weight of the moon lander), you get more-or-less the correct result (the amount of fuel needed). Add a small safety margin, and you're all set.

Edward Lorenz, a meteorologist, discovered in 1960 that this needn't be true. What he was working on was a simple model of the world's weather. He left out most of the details like snow and rain, and concentrated on the simplest of simplest: movement of air in the atmosphere. One day, he wanted to restart a simulation but as he didn't want to wait too long, he just punched in the numbers the computer printed out as the initial conditions for the next run. Coming back after a cup of coffee, he thought he'd made a mistake entering the numbers because the weather model was completely different from the first run. But no, the numbers were correct. Then, he realized that, effectively, he had been rounding off the numbers: the computer printed three decimals, but internally worked with six decimals, which meant that the second model differed not more than one part in a thousand from the original model. It seemed that his simple set of equations was so sensitive to initial conditions, that the model took off in a completely different direction.

This was the end of a meteorology dream (long-term weather prediction), and the beginning of a new science. It was one of these things for which the time was right, because numerous people from all kinds of disciplines came up with the same results: non-linear systems, often deceptively simple, could show unpredictable, chaotic behavior. Something as simple as iterating over a quadratic function and looking where the result would go resulted in something as complex as the well-known Mandelbrot fractal.

Suddenly, people realized again that the linear models used in standard scientific practice were just that: models. When calculating where Venus would rise, they were using a model to make a prediction—a very accurate prediction, but still a prediction. Furthermore, it dawned upon scientists that a lot of "noise" that they would tend to filter out of their experiments were actually the results of non-linear, chaotic behavior. The 1980's were the time that top-notch physicists forgot about expensive particle accelerators and started studying basic, "high school student" things like the behavior of a dripping water faucet.


2.3. Complexity

The Santa Fe Institute is probably the center of the world for a new discipline, called Complexity Science[2]. Founded in 1986, it is meant to be a basic research institute that studies complexity, the stuff that has made life, societies, and other complex organizations possible. The basis for complexity is a reversal of the classical science view, which is based on dissecting stuff until the basic elements are found (at the forefront, of course, is particle physics, but biology, economy, etcetera all basically do the same thing). Complexity arises in the other direction: when basic elements are added together and allowed to interact in certain ways, their interactions tends to form something that is discernible at a higher level as a pattern that cannot be explained from the elements. It's the old reductionism versus holism all over again, with the not-so-small difference that computer simulations, a relatively young entry in the scientist's tool chest, seem to prove the validity of this particular form of holism over and over again.

Let me give two well-known examples to make this point clear. First, John Conway's game of Life. It's not really a game, technically it belongs to the class of simulations called Cellular Automata, and the rules are very simple:

  1. If a cell has 2 or 3 filled neighbors (out of the nine possible neighbours), it will be filled in the next round;

  2. Otherwise, a cell will be empty in the next round.

Put up a checkerboard, put some stones on it ("filled cells") at random, and start playing. Better, get one of the zillion computer simulations - almost every programmer on Earth has implemented The Game Of Life one time or the other (the editor I'm writing this in has a built-in version!). These simple rules for individual cells can lead to amazing patterns of groups of cells (the higher level), and even some 25 years after Conway came up with it people are still amazed by this game. Whole catalogs of patterns of cells that do interesting things exist. While watching a Life simulation, you cannot help feeling that something on that screen, well, lives.

In 1986, Craig Reynolds made a computer model for the kind of movement behavior a flock of birds or a school of fish shows: intricate patterns that seem to steer a bunch of organisms as if it were a single organism avoiding obstacles or just bouncing around. As for Life, Reynolds found that only a few simple rules sufficed to model this behavior:

  1. Separation: steer to avoid crowding local flock mates;

  2. Alignment: steer towards the average heading of local flock mates;

  3. Cohesion: steer to move toward the average position of local flock mates.

The results of these three rules can be seen in every modern animation movie where flocking animals feature (be it bats in "The Hunchback of the Notre Dame" or a herd of gnus in "The Lion King"). Again, a set of simple basic rules about how elements could interact was enough to cause intricate patterns on a higher level.

Over and over again, this point has been shown. We're living examples: a reasonably simple set of rules (interactions between molecules—chemistry) was enough to trigger a sort of upwards avalanche called evolution, where chemical cross-breeding between molecules (now and then helped by mutations through cosmic radiation) and natural selection created the most complex things we know of, like human beings and whales.

Emergent, bottom-up complexity seems to be how nature puts things together. Even though on a scientific level a lot of the work is preliminary and unproven, the pattern is too clear to ignore.


2.4. Phase transitions

How do order, chaos and complexity fit together? Chris Langton, the father of Artificial Life, has come up with the idea that Complexity sits on a thin line between Order and Chaos, like the molecules between solid and fluid in a phase transition. In fact, he has shown a lot of analogies, both qualitative and quantitative, between some physics thingy called "second-order phase transitions" and the division of systems behavior into Order, Complexity, and Chaos.

By now, a lot of people are convinced that there is a driving force of nature towards complexity, just like there is a driving force towards nothing at all (worded in the Second Law of Thermodynamics). There is no rigorous proof yet, but the idea is simple: say that there's a completely orderly system of elements interacting in some way, with the possibility for the elements to adapt their behavior to one another. Now, if some external or internal influence gives a little kick to one or more elements in the system, it might give this element an edge over the others and this behavior might spread. In an economy, a company may come up with an invention like, say, the transistor or the integrated circuit. This invention gives rise to new industries, people will leave companies and setup their own, and on the whole will result in a ripple effect through the whole economy, bringing the system to a less orderly state overall. Now, imagine a system that is in a chaotic state. Here it is even clearer to see that if elements of the system start to interact in patterns, they have an edge over the rest. The fact that people organizing and specializing had an edge over the people still roaming about collecting their own food has led to the rise of societies. So, orderly systems tend to get less orderly, and chaotic systems tend to get less chaotic. Right in the middle, there's a stable situation of complex systems. Adapting systems to internal and external influences so that they become complex systems seems to be nature's way of dealing with organization.


3. Software is not Hardware...

A holy grail of software engineering seems to be to mimic other engineering disciplines, like electronics or building. Although there are lessons to be learned from these disciplines, I maintain that it is impossible to find an analogy that holds on closer inspection, and that it is therefore not wise to base software engineering discipline on analogies from other engineering disciplines.

There are a couple of aspects that are not present (or not on the forefront) in "hardware engineering" that are present in software engineering: testability, interaction complexity and malleability. I'll detail them in the next few sections.


3.1. Malleability

When the mayor opens a bridge, one thing he doesn't do a week later is to go to the bridge engineers and ask them "You know, this bridge is basically OK but could you please give it round pillars instead of square pillars? I think it looks better." No, a bridge is planned in advance, built, and unless there is something seriously wrong with it the bridge is a take-it-or-leave-it deal. That's why engineers take a couple of years to design the bridge, make scale models, do wind tunnel tests, etcetera - whatever it takes to validate the design in all its aspects is done because building the bridge is a one-shot action: it must succeed, because there's no way to beta test it.

Software, on the other hand, is pure brain stuff. The hardware is completely unimportant, and things like the language the software is expressed in are unimportant as well. Software is thoughts and concepts in some convenient notation, and this notation typically lives in formats that make it easy to modify (like in the format of text files on hard disks). Software is almost infinitely malleable. The first thing a user does when starting to work with new software, especially with new custom software, is think about what needs to be changed and often pushing the supplier to make these changes ("yesterday would be fine, thank you").

The good thing here is that software builders get a second chance. If things don't work out, they can fix it. The bad thing, of course, is that they get a second chance—endless design, prototype and test cycles aren't seen as necessary, because if the user has a problem, it can be fixed. In fact, the pace of the whole information technology industry is such that pre-production activities taking years to complete, like in bridge building, are simply unacceptable.


3.2. Testability

How do you test whether a car has certain aerodynamic qualities? Simple: you take a lump of clay, form it according to the design drawings, put it into a wind tunnel, and let sensors and computers to the calculations. The end result is a coefficient - if it's too high, you try to make little changes to get it down, and if it's way too high, you send the thing back to the drawing board.

How do you test software? Alan Turing came with a proof, more than half a century ago, that you cannot deduce whether a non-trivial program will finish its task or not without actually running the program and sitting it out.[3] Later work has shown that it is impossible to perform complete tests on software that reacts on users or networks. Software is just too complex, there are too many states and inputs, and the whole thing exhibits highly nonlinear behavior: you can check the behavior for value X, but that doesn't guarantee you that the behavior for (X plus a tiny bit) will be the same.

Furthermore, because of all this, there's no way to make a scale model of a piece of software and put it in the wind tunnel. Even with electronics engineering, you can make a model and exercise and probe it before actually starting up the chip factory, but with software there is no such thing as a model you can test - the model is the software, and vice versa.

Some schools of thought in software engineering maintain that you can test software, by testing the components. The reasoning is based on reductionism: you take the problem apart, build low-level components, and solve the problem by bringing the components together. Testing is done on the low-level components and, they say, this says something about the high-level software because in the reductionist thinking, the whole cannot be more than the parts. Software models like Life and Boids prove otherwise, I think.[4]


3.3. Interaction Complexity

A last difference is the level of interaction between components. Because software runs mostly on digital electronic computers (out of convenience, not out of necessity), software engineers see in electronics engineers their close cousins. Moreover, electronics engineering has the nice property that a lot of the work consists of plugging existing components together, not in constantly reinventing and redeveloping the components themselves. Lots of the work in electronics engineering consists of building higher-level components out of low-level components, and using these to build even higher level components, etcetera. It seems natural that software engineers drool over the idea of being able to do this themselves.

Attempts to componentize software have been made and have been partially successful. However, they have been successful at the "uninteresting" level - the level of commodity software. In the early days of operating systems, it made a real difference which operating system you used because of all the different capabilities. However, the current view is that it really doesn't matter too much what operating system is abstracting away the hardware, so the interface became less important as a competitive feature, making operating system suppliers go away as an identifiable business and hardware suppliers bundle some POSIX-compliant operating system with their hardware.[5] The same holds for things like databases, which do nothing else than implementing well-defined behavior (a relational database) and most software developers don't really care which one is storing tables and indexes from a technical viewpoint.

Typically, commodity software like operating systems and databases is the first software to componentize - hide behind a well-defined interface and abstract away implementation differences for the user. Just as an average electronics engineer doesn't really care whether a type 555 timer chip comes from Texas Instruments or NEC, an average software engineer doesn't really care whether a database comes from Oracle or Sybase. However, even for electronics the story doesn't hold for higher-level components. I think the average electronics engineer does care whether an embedded CPU comes from Zilog, Motorola or Intel. So the analogy is correct in sofar that even if software can be componentized as much as in electronics, this won't lead to an automatic Nirvana because even in electronics, only commodity components can be swapped freely, but key components cannot. Furthermore, the implicit assumption that electronics engineering is somehow easy doesn't hold as well, as we can witness in the numerous hardware bugs that are popping up now that electronics reaches previously unknown complexity levels (and starts to look more and more like pure thought matter - software).

I think, though, that the analogy is basically wrong in that it makes a lot of assumptions about similarities between electronic components interacting and software components interacting. I think there's a difference in the kind of messages that components send each other: in electronics, a lot of components are "content" with not interpreting the data. There's a whole series of components that operates on serial bit streams, bit by bit, doing useful things to them on the higher (byte) level without knowledge of that higher level. This makes interfaces simple: there's only one "brand" of electric current, it is simple to agree on voltage and current characteristics, and it is sufficient to do so. There aren't many discrete components that would really perform better at 220V than at TTL levels. And if there are, it is easy enough to build an adaptor to hide these internal differences.

Software components do normally interpret the data they act on, apart from the lowest-level components. But the low-level components of software are machine language statements (or maybe small groups of them), not the complex entities we talk about when discussing software components! A software component is very high level, and has a very specific interface, just like in electronics high-level components tend to have very specific interfaces (on the level of software components, a television set consists of a cathode ray tube, a high voltage amplifier, and a receiver. Apart from the relatively simple CRT, I think it is probably easier to design amplifier and receiver yourself than trying to bridge a Sony amplifier with a Philips receiver "component"). Because messages between components are very high-level and full of meaning and semantics, interfaces between components become highly specialized and therefore automatically hard to re-use in new, previously unthought-of situations. Furthermore, there's much to gain functionality-wise and competition-wise from altering and specializing these interfaces, so there's no internal drive for standardization. There probably never was one in the electronics supply side, but the fact that the demand side was relatively small and well-organized (only the electronics industry) made it easier for the demand side to require standardization.

In fact, the only place where I can see that components have worked in a grand way is the Unix operating system. By dumbing down all data to character streams, it became possible to componentize all kinds of specialized utilities that can operate on character streams and let the user or administrator put these together in all kinds of interesting and useful ways. I doubt, though, whether this is possible at the level of strongly typed interfaces between objects as the current mainstream idea seems to be.[6]


4. ... So stop engineering it!

In the previous section, I have tried to show why all sorts of analogies between software engineering and hardware engineering don't hold. It is well-known that basing procedures and decisions on bad analogies results in bad procedures and decisions, so I maintain that it is not wise to try to base software engineering on analogies with hardware engineering.

The problem is, this rules out 99% of the existing software engineering methodologies.

Actually, this is not really a problem because most of these methodologies don't work very well (apart probably for the couple that do work like bridge building and do start with a couple of years of design, testing and prototyping. So if you work for NASA working on STS control software, you can stop reading here). The problem is, of course, that no-one seems to come up with something that does work very well, or maybe that the ones that do come up with something don't do it from the perspective of coming up with a methodology and therefore forget to write a book about it. As I'll show later, these alternative methodologies do exist but nobody calls them that way and "serious" software shops don't use them. Let's look back at what a software system is, so that we may come up with some perspective from first principles.

Most software can, or better: should, be viewed as software systems consisting of a lot of elements in constant interaction. In modern software technique, these elements are often objects or constellations of objects, but even in earlier days people often put block diagrams on the white board: "here's the spell-checker, and this piece interprets files and converts them into the internal data structure which is managed here and interpreted there for display. Here's a piece that intercepts keystrokes and changes the internal data structure, and finally there's the stuff that converts the internal data structure to an external file." Each of these pieces of a hypothetical word processor can be reduced to other pieces, and if you break down often artificial hierarchical barriers you're left with a sea of small pieces of code and data—objects, subroutines, whatever—that interact with each other in intricate, complex ways and which can and will adapt to each other and the environment (by manual programming labor, but still).

Not coincidentally, this is the same as the definition for a complex adaptive system, which is seen as a pre-requisite for a system that is capable of exhibiting complex, emergent behavior. Where are the object models and the design diagrams for the behavior that Life shows on the screen? They aren't there, because the behavior emerged from the set of rules that John Conway made. In the systems that complexity science studies, there are no blueprints, only recipes[7]—whatever results from executing the recipe cannot be predicted without actually executing the recipe (take a cookbook and try it!), so the system's behavior is said to be emergent, rather than pre-defined.

Complex adaptive systems are said to live "on the edge of chaos". They aren't linear, but they are not chaotic either. Because they are not linear, you cannot readily predict what happens if you change a bit of the system. Because elements of the system react to each other, and because elements of the system never stabilize, the whole system is constantly changing - the elements are adapting to environmental changes as well as to their counterparts (the latter concept is called coevolution in biology and it is regarded as a major driving force for keeping evolution going on).

There's another similarity between such systems and software systems: all systems exhibiting complex adaptive behavior can be abstracted away from the substrate and be put into the form of pure information, like software. The intricate chemical interactions which have probably lead to catalytic loops and ultimately to life; the not much simpler system that is formed by a country's economy; all sorts of ecological masterpieces of balance and arms race—ultimately, they can be interpeted as software systems executing on various types of computers.

If we accept that software systems themselves fall under this classification, we can explain a lot from it. The fact that bugs occur (and are fixed, and result in other bugs); that it is hard to predict what will happen when changing a component; that master designs and detailed designs and whatever blueprints are made never quite seem to match the actual software, so that they need to be modified after the fact (of course, when this doesn't happen the designs become obsolete merely by the fact that they are stable things in a dynamic system). Furthermore, there is a lot to gain from this model: if we accept it, it means that the only way to make software survive is by constant change - by designing change into the system and allowing behavior to emerge, it is possible to make sure that software can match the pace of change that the outside worlds requires from it. Furthermore, such software would be so adaptable that instead of confronting every user with all functionality, or confronting every user with the lowest common denominator, it could offer every user with just the functionality he or she needs.[8]


5. Software Gardening

A good software engineering approach would then be to stop with engineering and start growing it. In fact, growing software has been used successfully in what is now called the Open Source movement but was formerly known as "spare time hacking". The prime example (it is getting boring) is the Linux kernel. There is no master plan, I have never seen a design document, there are just a whole bunch of pieces of the kernel competing for survival, and a guy that plays the role of "laissez-fair gardener": the garden just happens, anyone can plant seeds or even whole plants, although whatever the gardener regards as weed is mercilessly removed. The fun thing about this kind of garden is, of course, that if a piece of weed is removed, you can clone your own garden, grow the weed into a beautiful flower, and then retry. The gardener, Linus Torvalds, just defined the basic rules (some stuff about code formatting and dealing with locks, timing and interrupts) and stood back. He planted the seeds for what could become a complex adaptive system and that is what happened.

For example, he only implemented drivers for a Minix-compatible file system himself. Soon (within the year), there were the XIA file system, the Extended file-system, and not much later the Second Extended file system competing for market share. The Second Extended file system won, but as is usual in dynamic systems, this victory is only momentarily: two modern journaling file systems are already starting to enter the arena and surely one of them will be the next temporary king-of-the-hill (or not, maybe it'll share the throne, who knows). In the meantime, the Minix file system fills a niche (it has a bit less overhead so it is used for storage-tight applications like floppy disks) and the Extended and XIA file systems have been deceased.

By allowing and catering for competition, Linus has "grown" a much better file system than if he would have sat behind his computer and designed one himself. It is the natural way to get at good software, and in contrast classic engineering methods seem a bit like brute force: they get at a certain result, but often at the cost of quality and adaptability. And change is what software is all about, so the brute-forced systems simply lose because they cannot keep up with the market pace.


5.1. How to?

So how to grow software? Well, two things are clear: forget the design and instead focus on change. Grow software - confront users early and often with changes, in fact make them used to changes (personally, I think that users are far more capable in dealing with software that accommodates itself to them on a daily basis than to software that is completely redone to accommodate all the users every year). Have a master gardener that defines the rules and keeps things from going astray. Allow competing implementations (several implementations of the same stuff for different users), and make sure that users can select among them. A lot of this can be found in the Extreme Programming methodology/philosophy, although as far as I can see this system doesn't have an explicit complexity science background but rather is a collection of best practice patterns. However, it is probably about the only system that has at least the potential to be adapted to some sort of explicit complex adaptive systems-based way of building software, especially because it de-emphasizes design and puts a lot of weight with refactoring, a process that is consistent with growing software (refactoring can be seen as splitting plants, removing extraneous branches, etcetera—basic garden maintenance). It doesn't add competition between alternative implementations, and this may well be the hardest part. Unix file systems can be freely interchanged, because they all adhere to a simple and well-defined interface (this is also a problem, because lots of potentially interesting features cannot be implemented through this interface); but how to inject useful competition into the implementation of, say, an electronic mail system?


6. Conclusion

In this paper, I have tried to make clear why I think that current software engineering practice doesn't match a very natural way of looking at software systems, namely as complex adaptive systems. From first principles, I have given some guidelines how software development consistent with this view can be done, and suggested Extreme Programming as a possible starting point for such a process.

Bibliography

[Brooks95] The Mythical Man-Month: Essays on Software Engineering, Frederick Brooks, 1995, 0-201-83595-9, Addison-Wesley.

[Dawkins96] Climbing Mount Improbable, Richard Dawkins, 1996, Viking.

[Gleick88] Chaos: The Making of a Science, James Gleick, 1988, William Heinemann.

[Hofstadter79] Godel, Escher, Bach: An Eternal Golden Braid, Douglas Hofstadter, 1979, Basic Books.

[Waldrop92] Complexity: The Emerging Science at the Edge of Order and Chaos, Mitchell Waldrop, 1992, Simon & Schuster.

Notes

[1]

For an excellent introduction, see [Gleick88]

[2]

An interesting account of the initial years of the SFI with a lot of information on the science itself can be found in [Waldrop92]

[3]

I don't have the original reference, but [Hofstadter79] has a very good explanation and you should have read this book anyway.

[4]

Note that I'm not saying that unit tests are superfluous, just that you cannot extrapolate the results of unit tests to the whole software. Furthermore, some unit tests are always out of necessity testing complex behavior and test a system themselves.

[5]

Even the difference between, say, Linux and Windows NT has largely to do with perceived stability, politics, and add-ons—not with the intrinsic capabilities of the Windows or Linux kernels.

[6]

One of the component systems with the most potential, Enterprise Java Beans by Sun Microsystems, has large gaps because data storage issues—two "stories" below the level of EJB beans—bubble up and cannot be separated out.

[7]

The distinction between blueprints and recipes is stressed and explained very well by biologist Richard Dawkins in [Dawkins96]. Basically, a blueprint is a scale diagram of the end result, while a recipe is an algorithm for making the end result. DNA classifies as a recipe, and so do the rule sets for complex adaptive systems.

[8]

I've heard this concept being called "run-time design", which covers the idea although clearly the design here is not by making blueprints, but more by letting a certain behavior emerge as a result of user feedback and actions.


 
Copyright (C)2000-2011 Cees de Groot -- All rights reserved.