Why is Software Development the Butt of All Those Jokes?

This is the first in an irregular series of posts on the subject of software development, its history, its current state, and its potential futures. Software development is a subject I've been passionate about for a long time, and one which has demanded a lot of my professional life. I'd like to present some of my experience and the conclusions I've drawn from it for others to use.

I've been a software developer for almost three decades now, and I probably know all the jokes about how bad software development is. There's some reason for the jokes. Software is too hard to develop, it takes too long, costs too much, and has too many bugs. Big systems aren't reliable, are too hard to understand. No one understands how to manage software development, how to plan it, or how to design it so that it performs the functions that are required.

Probably the best known joke about software is the old line, "If buildings and bridges were built the way we build software, the first woodpecker to come along would destroy civilization." True or not, the fact that so many software professionals repeat the joke indicates that someone believes there's truth in it. It shows a bad case of what I call "concrete envy" in the field, a feeling of inferiority to the older, notionally more mature, engineering disciplines.

Now that I've said the things that everyone knows, let's talk about the real elephant in the living room. The reason nobody knows how to manage software is that nobody really knows what software is. Computing is a bastard field, out of electronics by mathematics. Computer science as a research or educational discipline at first was organized in either Electrical Engineering or Mathematics departments, and the two had very different ideas of what "programming" was. Was it science or engineering? Mathematical formulae or electrical diagrams? Abstract symbology or concrete instructions? And then it got more complicated as new generations of computer languages came into use by programmers who had never constructed a patch block or written a program in machine code. They took the abstractions of the languages for granted as real things, not combinations of registers and arithmetic units. Several generations of languages and tools later almost no one in the field has any connection with the original concepts, which are buried under layers of other abstractions.

To most practitioners now, software is neither science nor engineering but a technical craft with its own rules and requirements, its own language and symbology. Its purpose isn't learning about nature or about what can possibly be constructed, it's about building specific devices for specific purposes. It's not done by universities or by engineering labs but by corporations that need software to operate their businesses, and need to manage the cost and schedule of projects to construct it. And the people responsible for the creation of software are often managers who were never programmers themselves and don't have any understanding of what what needs to be done, or what the result of the work will be. Sometimes (especially in open source software communities) the people responsible are also the ones who do the development, but they are less concerned by how the software is used than by how it is designed and built.

Now this is starting to sound like a common software rant, I know. But consider the implications of there being many different kinds of people with different kinds of expertise involved in the software industry. There's no agreement on what they're working on, let alone how to work on it. And then we get into the deep quagmire of requirements versus specifications: the question of how to determine what the software should do, and how to translate that into a description of how the software should work. Many's the project that's bogged down in that pit only to be dug up later by paleontologists in chunks of peat.

So here's the task I've taken on for these posts. I'm going to describe the state that software development is in today, with some attempts to understand how that state came to be. Then I'm going to prescribe what I think is a good, but certainly not the only good, treatment regime to improve matters. Along the way I'll talk about some of the prescriptions that have been handed out before, and why they did or didn't work. I promise that this will not be a completely objective analysis: I have strong opinions, and a particular viewpoint to work from, and that will color what I have to say. I hope that means that it will be colorful as well.

Some topics I will definitely touch on: the checkered history of programming languages, tools, and paradigms, the business of development methodologies, and the effect of the high-tech economy on progranmming. I welcome your comments on what I have to say, and will respond as I can.

Next up: what is software, and what does it mean to "build" it?

Wedding Photographs

Pursuant to the "Wedding Apparel - Never Worn" thread on Making Light, follow the link on the title of this post, or go to the "My Current Photo Gallery" link on the left side of this page.

There are some pictures of my older son's wedding in 2005 on the beach in Florida, and some pictured of my own wedding in funny clothes and hairstyles in 1970. Try not to think too badly of us oldsters.

Cthulhu's Wild Ride

Especially for Serge

Oh Bother, Said Pooh ...

An illustration of why Disney's desire for permanent copyright would be bad for Winnie the Pooh.

Oh bother, said Pooh, as the Great Old One Walt rose up from the unspeakable depths.

The Stairs Like Dust

The remodeling job continues, slower than hoped for, of course. We have mold problems, but since we already knew we had asbestos that needed removal, I made a deal with the remediators to do both at once. And then we found still more mold. So when I get home today, I hope that I will find that they've gotten it all and it won't cost any more than I already expect it to.

So far, though, I'm very happy with the people doing the work. They've put up plastic dust barriers to keep the debris from floating all over the house; thats sort of a necessity for Eva, with her asthma and allergies, and something I appreciate too, though I could probably survive without it. So there really isn't a lot of dust on the stairs, but I just couldn't resist the title pun.

The only real problem we've had is that the Porta-Potty deliverer left it in the wrong place, and so far hasn't come back to move it though the company has been asked twice now. We don't know how long we'll need it, but it's looking like at least another week, so convenience is an issue. They left it at the top of the drive, right by the bike path on the road. This is a busy street; it's also a major bike and pedestrian route from this whole part of town to Wilson High School, about 1/3 of a mile from us. So I would really like to have the Porta-Potty be down the driveway, under the carport and right by our side door. That way I can go out in the morning in my bathrobe, before getting dressed. It would also mean I wouldn't have to have the sneaking feeling that one of these days I'm going to be sitting in there in the morning when a gaggle of passing male teens decides to celebrate tradition by tipping over the outhouse.

The Coffee is Awfully Strong This Morning

Here Come the Dust

As if my posting to this blog weren't spotty enough, we're having a major remodel job done on our house, both bathrooms at once, and the demolition starts on Monday morning. We'll be staying in a motel for a couple of days so we'll be able to breathe. In the meantime, my schedule is going to be fairly nutty and crunchy, so posting here will be on a "as-possible" basis.

Drinking the Cthulhade

It's all Mary Dell's fault.