Why we humans can undertake large construction or engineering projects with relative success and yet we struggle to deliver software projects without bugs and on time.
Even with excellent development processes it is still difficult to do software development right.
Engineering vs Programming
1. The software industry is young
Humans have been building house, roads and bridges for thousands of years.
Software industry is only about 50 years old. We still have a long way to go before we have the body of experience behind us that the construction and manufacturing industries have.
Today the construction industry use mostly prefabricated materials and parts. Most of these are made by machine and have been tried and tested on many other projects.
The software industry on the other hand doesn’t have the range of pre-built components that other industries have.
Software systems are fundamentally built by a process of discovery, invention, and creation of new components with the result that each new system is a custom project created from scratch. This leads us to our next point.
2. Every line of code is a potential point of failure
Because all new projects are custom built it follows that every line of code is unproven and therefore should be tested. However, in the real world, this is totally impractical.
Each line of code will have dozens, even thousands, of possible inputs, outputs, states or dependencies to deal with. It can impact, or be impacted by, other lines of code or by external factors. Even if it was possible to document every test case for a line of code you still couldn’t be sure that there wasn’t some unknown factor that could cause an error.
And testing a single line of code is only part of the challenge. No line of code exists on its own. It is part of the whole system and the whole needs to be tested to ensure that all parts of the application function correctly.
The sheer complexity of software means it is impossible to test every path so in the real world the best project teams will implement processes that are designed to increase the likelihood of the software being defect free. They will use techniques such as coding standards, unit testing, smoke testing, automated regression testing, design and code reviews etc. all of which should improve the quality of the software.
3. Lack of user input
Reasons for this can include:
- The system is being promoted by the management and so the business users have no buy-in
- The users are too busy and have “more important” things to do
- Relations between the user community and the I.T. team are poor
4. Users don’t know what they want until they see it
Even with good input from the users no amount of analysis of user requirements can take away an immutable fact that users only think that they know what they want. In truth, it’s not until they start seeing something, and using it, that they begin to really understand what they need. This is especially true when the software is being developed for a new idea or process that they haven’t used before.
Studies have shown that the average project experiences about a 25% change in requirements from the “requirements complete” stage to the first release. This is the famous “scope creep” problem that besets nearly all projects. It usually starts once the first designs begin to appear which cause the users to think more deeply about what they really want.
Agile development, have evolved to make it easier to adapt to changing requirements.
Changing requirements during the development phase is one of the great challenges facing all software developers.
5. There are no barriers to entry to become a programmer
It is relatively easy to learn how to write code but there is a huge gap between that and delivering great software.
One could possibly equate it to learning a new language. Yes you may pick up the grammar and acquire a reasonable vocabulary but that’s a whole different ball game to having a fluent two-way discussion with some native speakers.
Various studies have shown that the productivity ratio between different grades of developer can be as high as 28:1. With that in mind it you surely would only want to hire the best developers. Unfortunately this is not easy as great developers are a very rare commodity.
There is no barrier to entry into the programming world and thus it is awash with many poor programmers who adversely affect projects. In addition, even potentially good young developers will still make mistakes that a more experienced developer will have learned to avoid.
It really is worth paying more for a top-class experienced developer. They will do things quicker, and better and with less code. Your project will be delivered quicker and will have fewer defects. They will save you money now and they will also save you money through the life of the system in support and maintenance costs.
6. All software is affected by external factors
Physical structures obey physical laws and can therefore be modelled and predicted.
Software is “mindware” and therefore doesn’t obey physical laws but it usually must conform to external constraints such as hardware, integration with other software, government regulations, legacy data formats, performance criteria, scalability etc.
Understanding and catering for all of these external factors is a near impossible task. Even a seemingly simple requirement, such as supporting multiple browsers, exponentially increases the difficulty of both building and testing software. If you then add in a requirement to support multiple versions of each browser then you are again exponentially increasing the complexity and the difficulty.
7. Estimating is an art not a science
Given the premise that all new projects are custom built, that there are no pre-built components, that the project will suffer from scope creep and that the level of skill across the development team is usually varied then it is no wonder that estimating the duration of the project can never be a scientific exercise. There are too many unknowns. If you’ve never built the same software before with the same team then how can you know how long it will take?
A software application is like an iceberg – 90% of it is not visible. The major complexity in the application lies below the waterline and is invisible to the user. The next time you are wondering why software projects appear so difficult to get right you might perhaps spare a thought for the developers. They are generally working hard to deliver on time against a tidal wave of challenges and complexity.