These days I’m a free agent, and I’m lucky enough to be able to choose what projects I work on, but in a past life, I was what I suppose is properly referred to as an ‘enterprise software developer’. Yes, I once functioned in an environment where terms like ‘mission-critical’, ‘project life-cycle’, ‘stakeholders’ and ‘change management’ came up quite a lot. I’m grateful for the experience I gained over 12 years of doing that, but I’m also very glad to be free of it now.
One term which was bandied around a lot in this environment was ‘software engineering‘. The implication of this term, and the expectation of many people using it, is that designing and creating software is very much like physical engineering disciplines, such as erecting a bridge or building a block of flats. When a software project starts to go off the rails, and fingers are starting to be pointed, a common question is why large physical engineering projects can come in on time an budget, when most software projects beyond a certain size don’t? Surely if the people who create software claim to be ‘engineers’, they should be able to adhere to the same standards of other engineering professions?
At face value it sounds like a fair argument, and I’ll be the first one to admit that software engineers are very often not as rigorous as others bearing the title. But there’s a good reason – the belief that software projects are akin to building bridges is completely misplaced, and it is a belief which too often leads people to think that software projects should be run using techniques such as fixed prices, up-front design, and waterfall life-cycles. This is despite decades of case studies indicating that such techniques rarely work, and are more often a collective delusion adopted because the reality – that you’re never going to be able to predict a large software project accurately up-front – is too terrifying to be contemplated. The term ‘software engineering’, while indicating some of the responsibilities of such a person, is an incomplete definition of what building software actually entails, and just reinforces these misconceptions.
So, why is making software so different to building a bridge?
- Software is intangible.
You can see and touch a bridge, it has a physical presence you can measure and its purpose is absolutely obvious to any observer, as are its success and failure conditions (getting people across a gap without falling down). Even though there are parts of the bridge the user will never see, their purpose can be easily unambiguously defined – carrying utility cables/pipes, providing maintenance access, etc. Software is intangible, and even the part that is seen by those defining its purpose is often hard to evaluate until it can be seen and used directly, never mind the internal working processes. Evaluating and defining software ahead of time is like trying to describe a dream seen through a keyhole – incomplete and everyone has a slightly different impression, and regardless of how much you try to write it down, you can’t capture or communicate its essence completely to others, or guarantee that they interpreted it the same way.
People are just naturally very bad at defining and evaluating intangible things. There’s no point pretending they can with 300-page requirements documents, you’re just wasting everyone’s time – accept that people won’t know what they like until they can play with it – iterate!
- Interaction complexity
There’s only a small number of ways to use a bridge, and they’re each quite straight forward. Yes, there’s much complexity to the structure of the bridge itself, but that really doesn’t matter to a person driving or walking across it – that doesn’t need input or feedback from people except for whether they like the look of it.
Software is basically defined by complex interaction, which is often context- and data-sensitive. The number of variations mean that even if every one is bug free (yeah, right), there’s the question of whether it’s solving the correct problem, which is a function of people and their interpretation / opinion. While building projects can be complex, those complexities are most often a function of the engineering challenge, whilst most complexity in a software project is a function of the involvement of people, which are far more difficult to pin down.
- Change is a constant factor
Change happens everywhere of course – maybe the bridge builder discovers that the ground isn’t as firm as they thought, or suddenly someone wants to make an extra deck available for trains. Big disruption, and an effect on time and budget – all understandable. The problem when it comes to software is that the perception of change is very different. If suddenly you have to sink massive extra foundations or double the amount of steel you need to use, people understand the magnitude of that intuitively and accept the effect on the schedule. Changes in a software project tend to be perceived as ‘just a little tweak’ by those requesting it, which tends to mean tolerance for schedule impact is lower, and indeed the potential for interaction between changes in unexpected ways is often overlooked. Now, I’m not for a second advocating that change should not be allowed, but often the mechanisms that have been established for managing a project – up-front estimation, fixed feature sets and so on – prevent the efficient integration of change, or at least obscure its impact to a point where things become critical. It’s important to appreciate that change is more common and expected in software projects, by nature of the issues raised in the previous points, and therefore setting your expectations based on a discipline where change is less common is not a smart move.
- Lack of hard limits
Building a bridge involves respecting some hard, unchallengeable limits. As a wise man once said “You canna change the laws of physics, Cap’n!”. When you’re building software, however, there are very rarely any unbreakable limits, and even the soft limits that there are tend to be invalidated very regularly as new technology comes on stream. No-one can realistically stand up and say ‘this is impossible’ when faced with a request, because almost nothing is impossible. Hard limits in physical projects make all stakeholders ‘get real’ – you can’t use more land than you have, you can’t build a skyscraper out of paper, there are non-negotiable rules for safe load-bearing and so on. Barring some really crazy and ground-breaking structures (which BTW, rarely come in on time & budget), these limits ground everyone involved in the project to realistic expectations. In a software project, reining in those feature requests, or those ideas that the implementors have for a new tech they could use, is much more abstract affair, and everything is up for debate.
All these things put together mean that if you think you can use the same kinds of techniques to run a software project as you would use for a building project, you’re kidding yourself. They’re so very different in fundamental ways, it’s like trying to catch a shark with a shrimping net (because it’s just a fish, right?).
So please, stop with these delusions:
- that estimating the cost of any non-trivial software project (that isn’t just a carbon copy of something else) is any more sophisticated than mildly informed guesswork.
- that it’s reasonable to think that a fixed-price can be realistic for any software project longer than a couple of weeks
- that ‘change control’ is the about exceptions, rather than the norm
Of course, agile approaches attack these particular delusions rather well. But try to get those adopted in very large organisations where the ‘software is like a bridge’ mentality still thrives