Approaches to Building Software
Cees de Groot
This paper examines various approaches to building software, and
tries to put each one into its right place. The central argument is
that there is no single "correct" approach.
Time and time again, flamewars rage over the Usenet concerning the
topic of whether building software is an art, a science, an engineering
discipline, or a craft. It is a very interesting topic, not in the last
place because one possible outcome of the discussion is whether people
who build software must be accredited engineers - at the moment,
various institutions in the United States are working on the subject of
certified Software Engineer, to be put at the same level as other
Engineers.
One's view towards building software largely influences one's view
towards the methodology of building software. Lightweight methodologies
seem to emphasize the craft aspect; work in logic and formal methods
emphasizes the science aspect; while heavyweight methodologies stress
the engineering side. Furthermore, it can be argued that the great
admiration for certain individuals in the Open Source movement is
explainable by the view of (mostly younger) programmers of software
building as a form of art.
In order to structure the discussion, therefore, I want to use this
paper as an inventorying excercise. I am known to hold fairly strong
opinions on what I think is "right", but for in this particular
instance I will try to stay neutral. The four aspects will be discussed
in alphabetic order - I think it is shortsighted to rank a particular one
as being "less" than any other one. People who do this don't deserve to
be part of this very important discussion about the basics of our
profession. As it happens, the order makes up for a good flow of
arguments, but this is pure coincidence.
Art in the meaning we will use here is defined
by Merriam-Webster OnLine as "the conscious use of skill and creative
imagination especially in the production of aesthetic objects".
In the definition, there are two things that set Art apart from the
others: 1) a stress of creativity, and 2) the fact that the product is
something of mainly aesthetic value.
That software can be used to create art is
evident: from the earliest days of computers on, programmers have
applied their programming skills and their creativity in order to
produce software that created "aesthetic objects". In the early days,
line printers were coerced into producing pictures built up from lots
of different characters; soon after the introduction of multi-media
capable microcomputers highly skilled programmers started bringing out
"demos", intricately crafted presentations pushing the computer to
its limits. ACM's Siggraph announcements are customarily illustrated
with both scientific and artistic imagery.
However, for the purpose of this paper I think it is more correct
to view software, or source code, as the medium, not as the tool. The
question then becomes one of a manner of building programs mainly for
the aesthetic value of the source code.
Since 1984, a yearly competition deals with exactly that: source
code mainly written for aesthetic value. The International Obfuscated C Code
Contest gives prizes to entrants who come up with novel or
interesting (or plain disgusting ways) to implement algorithms in the C
programming language. More informally, programmers often code up
algorithms in novel ways for "hack value", up to the point where simple
operations that normally run in constant time are implemented in crazy
algorithms that require polynomial execution time.
Therefore, writing software can be an art form, both with software being
the tool as well as the medium. Needless to say, the market for the
latter category will be quite small (only colleague programmers will be
able to read the code, and often you need to be quite an expert to
fully understand all the details). Furthermore, programming as an art,
thus mainly for aesthetics, suffers the same drawback as other artforms: it is
nigh impossible to make a living off it, and especially for the "art
with software as the medium" category it is probably impossible to
find a Maecenas.
Merriam-Webster OnLine defines Craft as "an
occupation or trade requiring manual dexterity or artistic skill".
There's an interesting circularity here: the definition of Art
refers to skill (something typically associated with Craft), while the
definition of Craft refers back to "artistic skill". A liberal
interpretation here would be that Art is typically associated with production
for aesthetic value, while Craft is associated with production for use
value, but they largely pull from the same set of skills (and
creativity).
My personal experience with woodworking, music and photography seems to
confirm this view. I learnt woodworking when I was boatsman for our
student rowing club, from an old master who could put a competition skiff
together from mahogany triplex plates, balsa wood, and his hands. Lots
of the tools were made by his father, and I think it was typical for
the craft aspect that design drawings were sketchy (in the literal
sense). Photography sees a similar debate as software: some see it as a
pure artform, others as a craft. Because, as a process, is applied
physics and chemistry, one could even hold the viewpoint that it is
engineering, and indeed some photographer's darkrooms and studios are
full with measuring equipment. Finally, music (in my case: piano and
trombone) is generally seen as an art form, although my mother often
wondered what Czerny Etudes have to do with art...
All three forms of expression have a similar basis: in order to master
them, you need a lot of skill. For woodworking, this means practicing
on scrap wood; for photography, it means lots of test photos; for
music, it means lots of etudes. Additionally, knowledge is important:
you need to know about properties of wood, about the differences
between various chrome emulsions, about music history for correct
interpretation. The groundwork, therefore, for Art and Craft, is the
same: practice the skill, read about it, and have a master who can
correct and guide you.
The deviation becomes apparent only when you start considering the
goal. As we've seen, for Art this goal is mostly aesthetics. Craft puts
the emphasis onto use, which constrains the amount of creativity you can
put into it. A craftsman often gets commissioned work, and you don't
have a long career if you build boats with beautiful holes in them or
take pictures of just the bride, because you think the groom is ugly
and he would spoil the aesthetic value of your photographs.
Software as a craft is practiced especially in smaller shops, which
shouldn't surprise us because this parallels other disciplines (the sole
reason that the car industry is what it is today is that Ford did away
with craftsmen). The work is typically done in the context of
light-weight methodologies, because small shops emphasize interpersonal
communication which is more efficient than lots of paperwork but doesn't
scale well to larger teams.
I don't have the exact figures, but my guess is that small teams
(lets arbitrarily say this is less than 25 people) are responsible for
more than 90% of all software written. Don't forget that software
systems within large corporations are often written by small teams
inside the enterprise; even when they are part of the larger entity,
if the system is insulated well enough from other teams' efforts, the
team can be regarded as a small software shop.
A small team typically has total commitment to the end product (the
source code), and cares less for lots of drawings, formality,
etcetera. This is very much the way a craftsman goes about - he knows
his skills, the medium he works in, and therefore can set about
executing his assignment without too much research and ritual. The fact
that he has a fairly good idea where things can go wrong helps him to go
slower at tricky points, make sure that there's a way back (for
example, a woodsworker may first fit everything together before
applying glue - if parts don't fit, she knows that you can just throw
away the single wrong part instead of half the work).
Building software is supportive of this approach, because it is so
infinitely fluid. Everything can always be changed, thrown away,
redone, until it is just right. Writing and composing share this
property, and any woodworker (let alone any sculptor) is probably very
jealous of writers, composers, and programmers!
Engineering is defined by Merriam-Webster
OnLine as "the application of science and mathematics by which the
properties of matter and the sources of energy in nature are made
useful to people".
For the purposes of this discussion, I think it is useful to
emphasize the result of engineering: it helps us scale beyond what
Craft can produce. In many cases, the application of "science and
mathematics" in engineering
didn't help humanity by producing new things, but by making the product
available to many of us. Indeed, engineers are often the ones who
translate findings of science (or "craftsmen" - the proverbial lone
inventors probably qualify for that title) into mass production
methods.
Engineers succeed in their discipline by being very rigorous. Where
a craftsman can wander about, probe, stumble, and still put a high
quality product together, an engineer does not have this luxury: a
production line cannot afford to experiment about; the thing either
works or it is a failure. Rigor and scalability come at a price,
however: lots of it is attained by replacing informal methods with
(often expensive) formal methods, and by replacing inter-person
communication with volumes of written documentation: procedures,
standards, designs, plans, etcetera.
Clearly engineering is an astounding success, as probably 90% of
anything physical you posess has been engineered, and not
crafted. There is a price to pay, of course. When an engineer makes a
mistake, it multiplies by the number of units produced. And a table
that the local carpenter built for you has a different feel than the
one you got from Ikea. Engineering emphasizes utility, which is
probably something else than "use", the word I applied to Craft. The
difference, I think, is the Alexandrian "Quality Without a Name"
It is no surprise that "software engineering" is a dream of many in
the field. It bears the promises of measurable quality, methodical
approaches, and a way out of the "software crisis". Large teams can
indeed benefit from approaches borrowed from other engineering
practices, because inter-person communication doesn't scale well, and
above a certain size procedures and written documentation become
inevitable ways for team members to coordinate.[1] Large software teams subscribe to heavyweight
methodologies, which are probably very much the same as the
methodologies applied in other engineering fields: documents are
produced, verified for correctness and quality, and passed on. Whereas
the design of small "craft" teams is hidden in the source code, the
design of large "engineering" teams is hidden in the
documentation.
I discussed engineering as a matter of scalability, but the
second well-known aspect is the matter of public safety. Licensed
engineers are educated in such a way that they are very concerned
about the effects of their actions on public safety. It may be
economic to leave shielding from wires, but public safety certainly
isn't furthered by it.
Software builders, especially those who work in public
safety-critical areas, probably can learn a lot from these
engineers. I think it is a separate track of the "is software
engineering" issue, one that is - to an extent - orthogonal to the
scalability feature of enginnering
In Merriam-Webster OnLine, Science is defined
as "the state of knowing: knowledge as distinguished from ignorance or
misunderstanding". Another meaning probably also applies: "3a:
knowledge or a system of knowledge covering general truths or the
operation of general laws especially as obtained and tested through
scientific method".
Science "feeds" Engineering, and to a lesser extend Craft and Art
as well. Just as electronics engineers don't run away with every
research paper that physicists publish, I think it is prudent for
software builders to be selective in the application of computer
science research results to the daily routine. There is a practicality
test to be made and history has shown that this filtering function can
only be executed by workers in the field, not by the scientists
themselves (often to the frustration of the latter group, who cannot
believe why people keep using "old" technologies). There are a lot of
debates going on between computer scientists and field workers, and
they do or do not influence the field (object-oriented programming
conquered the field, but functional programming still seems to have
problems catching on). This relation with science, however, doesn't
make the practice of building software a science in itself.
Software can be part of science, of
course. There is academic research, both in the field of computer
science as in the field of pure mathematics (in a sense, software is
applied mathematics and software can be described in pure mathematical
terms). Computer science research seems to be quite healthy in that it
seems to fill a bell curve: a lot of "standard" stuff in the middle -
performance tuning, tweaks on existing systems, etcetera; a bit of
completely uninteresting or plain wrong work (I'll refrain from giving
examples here), and a bit of highly interesting, innovative research
(over the years: things like OO programming, RISC chips, functional
languages, and so on).
Science goes one step further than engineering in rigor. It is
often not enough to show that something is working (by duly testing
it), but the requirement is to prove why something is working as it
works, and describe it in such a way to make the work repeatable. This
calls for extraordinary formalism, which in itself is an interesting
research area (proof-carrying code, for example). Some engineering
shops probably can benefit from these approaches, like those who help
building nuclear plants or large passenger aircraft. Proving that
something works is possible, and a scientific proof that withstands
scrutiny of peers after publication in a journal is probably the best
way to make sure that the quality is at the highest standards.
To be written after initial comments.
Stumble It!
Post to del.icio.us