Anecdotally, I had always known that Software Engineers are terrible at estimation. I had never realised exactly how bad we are.
Some rules of thumb which seem to pop up now and again, is to take your engineers best estimates, and double them. Then you’re actually in the ball park.
However Jørgensen (2009) would seem to suggest that this doesn’t go far enough. He details several experiments in his paper, but Experiment C resonates most with me. It is under a similar context that I have had to create estimates and risk assessments for previous software projects.
In this experiment, he takes 50 software engineers and randomly divides them into two groups (LESS and MORE). He then presents the requirements for a database project to both groups. Those in group LESS are asked to specify the largest project risk, give an estimate of effort required to implement the project, give a success assessment of the project, and declare how competent the feel they are to achieve the project. Those in group MORE were asked for similar outputs, with the exception of being asked to think of as many risks as possible. They were also given a small risk assessment method, being asked to think about risk severity and probability, as well as receiving a checklist of risks to consider.
The results of this experiment are rather facinating. The mean number of risks identified by both LESS and MORE were 1.0 and 4.1 respectively. While the mean effort was estimated by LESS at 316 work-hours, while those in the MORE group had a mean estimate of 200 work-hours. One would think if you identify more risks, you would estimate more effort to overcome them.
The requirements used in Experiment C were given to 4 independent software companies for actual implementation, and the mean effort required to implement the project was found to be 700 hours.
The net result of these experiments seem to suggest that software engineers are over-optimistic in their effort estimates, and over-confident in their assessments of estimation accuracy. Is it any wonder so many software projects fail under traditional project management techniques?
If there are any lessons to be learned, perhaps we need to adapt our development processes to be resilient to the fact that the estimates we produce are so poor.
- Jørgensen, M. (2009) Identification of More Risks Can Lead to Increased Over-Optimism of and Over-Confidence in Software Development Effort Estimates. Retrieved July 15, 2009 from http://simula.no/research/engineering/publications/Simula.SE.621