Why ‘software engineering’ is a misnomer

These days I’m a free agent, and I’m lucky enough to be able to choose what projects I work on, but in a past life, I was what I suppose is properly referred to as an ‘enterprise software developer’. Yes, I once functioned in an environment where terms like ‘mission-critical’, ‘project life-cycle’, ‘stakeholders’ and ‘change management’ came up quite a lot. I’m grateful for the experience I gained over 12 years of doing that, but I’m also very glad to be free of it now.

One term which was bandied around a lot in this environment was ‘software engineering‘. The implication of this term, and the expectation of many people using it, is that designing and creating software is very much like physical engineering disciplines, such as erecting a bridge or building a block of flats. When a software project starts to go off the rails, and fingers are starting to be pointed, a common question is why large physical engineering projects can come in on time an budget, when most software projects beyond a certain size don’t? Surely if the people who create software claim to be ‘engineers’, they should be able to adhere to the same standards of other engineering professions?

At face value it sounds like a fair argument, and I’ll be the first one to admit that software engineers are very often not as rigorous as others bearing the title. But there’s a good reason – the belief that software projects are akin to building bridges is completely misplaced, and it is a belief which too often leads people to think that software projects should be run using techniques such as fixed prices, up-front design, and waterfall life-cycles. This is despite decades of case studies indicating that such techniques rarely work, and are more often a collective delusion adopted because the reality – that you’re never going to be able to predict a large software project accurately up-front – is too terrifying to be contemplated. The term ‘software engineering’, while indicating some of the responsibilities of such a person, is an incomplete definition of what building software actually entails, and just reinforces these misconceptions.

So, why is making software so different to building a bridge?

  1. Software is intangible.
    You can see and touch a bridge, it has a physical presence you can measure and its purpose is absolutely obvious to any observer, as are its success and failure conditions (getting people across a gap without falling down). Even though there are parts of the bridge the user will never see, their purpose can be easily unambiguously defined – carrying utility cables/pipes, providing maintenance access, etc. Software is intangible, and even the part that is seen by those defining its purpose is often hard to evaluate until it can be seen and used directly, never mind the internal working processes. Evaluating and defining software ahead of time is like trying to describe a dream seen through a keyhole – incomplete and everyone has a slightly different impression, and regardless of how much you try to write it down, you can’t capture or communicate its essence completely to others, or guarantee that they interpreted it the same way.
    People are just naturally very bad at defining and evaluating intangible things. There’s no point pretending they can with 300-page requirements documents, you’re just wasting everyone’s time – accept that people won’t know what they like until they can play with it – iterate!
  2. Interaction complexity
    There’s only a small number of ways to use a bridge, and they’re each quite straight forward. Yes, there’s much complexity to the structure of the bridge itself, but that really doesn’t matter to a person driving or walking across it – that doesn’t need input or feedback from people except for whether they like the look of it.
    Software is basically defined by complex interaction, which is often context- and data-sensitive. The number of variations mean that even if every one is bug free (yeah, right), there’s the question of whether it’s solving the correct problem, which is a function of people and their interpretation / opinion. While building projects can be complex, those complexities are most often a function of the engineering challenge, whilst most complexity in a software project is a function of the involvement of people, which are far more difficult to pin down.
  3. Change is a constant factor
    Change happens everywhere of course – maybe the bridge builder discovers that the ground isn’t as firm as they thought, or suddenly someone wants to make an extra deck available for trains. Big disruption, and an effect on time and budget – all understandable. The problem when it comes to software is that the perception of change is very different. If suddenly you have to sink massive extra foundations or double the amount of steel you need to use, people understand the magnitude of that intuitively and accept the effect on the schedule. Changes in a software project tend to be perceived as ‘just a little tweak’ by those requesting it, which tends to mean tolerance for schedule impact is lower, and indeed the potential for interaction between changes in unexpected ways is often overlooked. Now, I’m not for a second advocating that change should not be allowed, but often the mechanisms that have been established for managing a project – up-front estimation, fixed feature sets and so on – prevent the efficient integration of change, or at least obscure its impact to a point where things become critical. It’s important to appreciate that change is more common and expected in software projects, by nature of the issues raised in the previous points, and therefore setting your expectations based on a discipline where change is less common is not a smart move.
  4. Lack of hard limits
    Building  a bridge involves respecting some hard, unchallengeable limits. As a wise man once said “You canna change the laws of physics, Cap’n!”. When you’re building software, however, there are very rarely any unbreakable limits, and even the soft limits that there are tend to be invalidated very regularly as new technology comes on stream. No-one can realistically stand up and say ‘this is impossible’ when faced with a request, because almost nothing is impossible. Hard limits in physical projects make all stakeholders ‘get real’ – you can’t use more land than you have, you can’t build a skyscraper out of paper, there are non-negotiable rules for safe load-bearing and so on. Barring some really crazy and ground-breaking structures (which BTW, rarely come in on time & budget), these limits ground everyone involved in the project to realistic expectations. In a software project, reining in those feature requests, or those ideas that the implementors have for a new tech they could use, is much more abstract affair, and everything is up for debate.

All these things put together mean that if you think you can use the same kinds of techniques to run a software project as you would use for a building project, you’re kidding yourself. They’re so very different in fundamental ways, it’s like trying to catch a shark with a shrimping net (because it’s just a fish, right?).

So please, stop with these delusions:

  1. that estimating the cost of any non-trivial software project (that isn’t just a carbon copy of something else) is any more sophisticated than mildly informed guesswork.
  2. that it’s reasonable to think that a fixed-price can be realistic for any software project longer than a couple of weeks
  3. that ‘change control’ is the about exceptions, rather than the norm

Of course, agile approaches attack these particular delusions rather well. But try to get those adopted in very large organisations where the ‘software is like a bridge’ mentality still thrives :)

  • http://jdhardy.ca Jeff Hardy

    It’s too bad that software engineering is so often conflated with project management (by just about everyone, so I’m not criticizing your take on it). However, there’s a whole engineering mindset that is forgotten that is just as applicable to software as it is to bridges.

    The chief principle is that engineering all about tradeoffs. There is rarely a ‘right’ way to solve a problem, just various options that have different advantages and disadvantages. Also, engineers also have to be willing to accept that ‘good enough’ is, in fact, good enough.

    Software engineers, IMO, should worry about understanding those tradeoffs and being able to decide which ones are applicable and which are not. Leave project management concerns to the project managers.

  • Kikito

    There’s a very good talk about the whole “engineering” thing by Glenn Vanderburg here:

    http://confreaks.net/videos/282-lsrc2010-real-software-engineering

    I found it … enlightening. Give it a look if you can.

  • Jabberwocky

    Well said.

    The problem is that this doesn’t help us escape from the need to make cost and time estimates for “software engineering” projects. If someone’s paying your company to build some software, it won’t fly to say “I don’t know how long it will take, and I’ll let you know later how much I need to charge you.”

    That being said, what you should say is something pretty close close:

    “We’ll need some time to analyze the problem, assemble the technology, build a prototype, mock-up the inteface, and nail down as best we can a common vision of what we’re trying to build. Then I can make an estimate, with a pretty wide contingency fund for error. And any changes, of which most projects have, will need to be further estimated, which in itself is paid work.”

    Tough way to do business. ;)

  • bibiteinfo

    My 4 years in professional software engineering at university were pretty much all about what you told in that article. It seems that the course were right on track :)

  • http://www.stevestreeting.com Steve

    @Jabberwocky: Indeed – and when you have a savvy customer who has a vision (rather than a ‘big ball of mud’ list of low-level requirements), and a budget, and is willing to evolve and refine collaboratively over time in order to stay within their budget and get what most fulfils the vision, everybody wins. The best outcomes I’ve had from contract projects have been with that kind of setup, and everyone goes away happy. I’ve never seen a large project which starts off with a massive list of low-level requirements and a fixed budget go anywhere but down the toilet with a terrible end relationship (and I’ve seen this from both sides of the contracting fence). I won’t touch these projects with a barge pole anymore :)

  • http://www.jade-ds.com/ Andreas Podgurski

    Very true. One thing missing for me is the fact of fast changing conditions in software development, game development hits this very hard. A project may take that long, that the conditions have changed half way through the project, like interfaces, dependencies or even syndication with external services. That’s very different to bridge building, as there the conditions change much slower, in terms of years and decades instead of months.

  • http://voidsoft.ro Marius Gheorghe

    The reality is that software is a very young industry compared to the other “engineering” disciplines. We still have some things to figure out. Right now we’re still in the process of hiting out own hands with the hammer.

  • Marko

    I read a related article a long time ago:

    http://www-plan.cs.colorado.edu/diwan/3308-07/p17-armour.pdf