There are many examples of disastrous software projects - why the high failure rate when compared with high-rise buildings or bridges? For most buildings, the technology is commonplace and everything has been worked out in detail before the project starts. Occasionally, a new building requires the simultaneous development of new technology, and it is usually a fiasco too - the Sydney Opera House for example. Most industrial organisations do not mix their development and manufacturing, because the timescales are very different. A new car might take three years to develop, the tooling can be built in six months and the car will be manufactured in a day. With software, we continue to assume that development of the concepts and the coding of programs can be carried out in parallel. But we need to code the programs to understand the concepts.
Development Is A Hazardous Journey
Any
new development is hazardous. If we use the analogy of a boat setting sail, it can be seen
as rather like an Odyssey through a world of monsters, but sometimes it is more like a
Greek Tragedy, where we bring our own monsters with us wherever we go.
The Lake of Low Expectations
If we try to stay in a static world and do no development, nothing much seems to go wrong, except that the fate of the maker of button up boots may befall us - we die for lack of adaptation. Our competitors power away, taking risks but earning rewards. We can laugh at their failures, but they may learn from their mistakes, whereas we learn little.
The Island of Priorities
A new development in an organisation needs a priority - without it, it will die. But priorities change, so a project that started out under full sail may cease to be strategic and wither away. The Island of Priorities is like a volcanic island that rears up out of the sea with much noise and fanfare, only to sink back out of sight a few years later. Too much fanfare can be a bad sign - great expectations can lead to early disappointment.
The Conceptual Sand Bar
An ambitious software project is dealing in concepts, and concepts can shift like sand as the tide of implementation sweeps in. The initial project description will often describe the integration of concepts as though they can be simply bolted together like girders in a bridge. If the concepts are new to the organisation, at least one person in the project team has to assemble them in their head and ensure that the intended connections are sound. The timescale of this process is difficult to predict, but the process will take months or years to complete, not days.
The Land of Teetering Rocks
The software development tools used in a project often behave like teetering rocks. They work well while the requirements are simple, and come crashing down on the unwary when the requirements become difficult as the interactions intensify. Having the tools supplier fix the bugs can be a lengthy process (they have their own hazardous journey), so either the limitations are accepted, greatly constraining what can be achieved, or other ways must be found, under pressure.
The Land of Storms
The people making up the project team, and the people with whom the team interact, can produce storms of epic proportions. If a project is easy, people become bored and human interaction problems increase. If the project is difficult, people react to the severe stress by opting for the short term solution. Someone can contribute an idea and literally fight to the death of the project for its inclusion, no matter how flawed. Tiredness and high error rates can have the project going backwards, so it is necessary to tailor the degree of difficulty of the project to the capacity of the people available. It is not usually a good idea to launch a difficult project, then try to find the people to man it.
The Combinator
When all the other hazards have been negotiated, and clear water beckons, a new hazard appears in the shape of the Combinator. When production quantities are encountered, combinatorial problems can surface and make the running time for the application unsustainable. The concept seemed fine, the implementation was flawed, but if there is no workable implementation, the concept was terrible.
Evolutionary Prototyping
We can make a simple prototype to help us understand the concepts, and then evolve it into a working system. We can make a paper plane, but it doesn't tell us much about a supersonic jet. The prototype needs to model the operating regime of the final system quite well, otherwise it will tell us little. We may need to telescope the development and testing of several generations into one project to get to the level of complexity and performance we require. This requires a flexibility from the people on the business side that is not usually possible.
What To Do?
Most of the problems are not unique to a particular project and are usually all too predictable. Just accepting there are risks, or doing a static analysis of the risk at the start, does little to lower the risk. To expect failure is the first step to avoiding it, or at least keeping it partial and containable. The project plan should include alternatives, contingencies for the high risk areas. If there is no alternative in some area, then this becomes a focus of the implementation. Some people revel in solving problems and will deliberately "wing it", but solving a problem after it occurs is costly and dumb in comparison to avoiding it.
Some areas that need careful planning:
Skill Formation
Where is the activity, what is its duration. If the result of acquiring the skill can be seen and assessed for errors after a day, allow a week. If the result can only be seen after three months, allow a year to learn it. If the time can't be allowed to learn the skill, it must be bought, but can it then be integrated or transferred or maintained.
Team Formation
Welding a cohesive team together takes time. In what other difficult endeavour would we assemble a team overnight and set out the next day. If interaction among the team members is important, some practice is necessary so the rough edges are rubbed off. The more intellectual the interaction, the more that concepts need to be transmitted by spoken word and image, the longer it will take to get a smoothly functioning team.
Interface Mismatch
There will be several areas of severe mismatch in the project, the main one being the conversion of business requirements to programs. The system builder needs a long term stable specification of requirements, difficult in today's environment. Changing a few words on the business side can dramatically alter the topology of the application, causing the developing application structure to be heavily patched or junked. This is where tools that combine the What and Why with the How, When and How Much can contribute greatly to the success of the project - they are assisting in the synthesis and can show the change in topology.
More Discipline
Other forms of engineering have a discipline that software development lacks. If you build a bridge that falls down, or design a plane that can't fly, it is usually fairly easy to work out what went wrong and learn from it. Because software is hard to visualise, people sometimes make the most dreadful decisions, then roll on to the next project with no understanding of what went wrong. One way of combatting this problem is to ensure that the plan controlling the project contains all the constraints that apply to the project - the constraints on the What and Why, as well as the When. Some people with strong egos are attracted into software development, so it is essential that the plan is strong enough to withstand considerable opposition. The plan needs to be malleable to handle the necessary changes in the project, yet its conclusions from the constraints on the project must be inescapable.
Put simply, we attempt too much with too little preparation in software projects, things that we would never do for projects in other areas with far less chance of failure. We use planning tools and methodologies intended for concrete projects - tools that simply add up durations for easily quantifiable and committed activities, and then schedule those activities without any possibility of planning whether they should occur. With software projects, what is important is how well people understand what they are doing, and how well the planning system can handle large changes in the topology of the plan. Estimation of the level of understanding and smoothness of integration of concepts is very different from counting cement trucks onto the site.
More appropriate planning tools can improve the chance of success by highlighting areas where more time and more care are initially needed. A little extra care can head off the disaster recovery mode that many software projects blunder into. Still, sitting on the cusp somewhere between low expectations and too many project failures requires a sophistication bought with bitter experience.
Project Management Planning Tools
A presentation.