John Kay argues that many of the models used to assess and quantify real life projects are bogus. They are based on the assumption of complete and perfect information, but a model can never be a true representation of the world and nor should we expect it to be. The skill of a policy analyst is in identifying the models and evidence that are relevant.
A wide-spread characteristic of modern life is a phenomenon I describe as ‘bogus modelling’. Three of the best, or worst, examples of this are firstly, WebTag; a framework used for appraising transport projects, most recently the high speed rail link. Second, are public sector comparator models, which are used to assess potential private financial initiative (PFI) projects. And finally, value at risk modelling (VAR), which is used for risk management in banks. It would be an understatement to suggest that these models have not been wholly successful.
Bank risk management, based on VAR, led to the most comprehensive collapse of the banking sector that we have seen in a century. PFI projects, despite meeting the requirements of the value for money comparators, are set to cost tax payers billions in funding costs. The UK is admired around the world for the quality of its transport modelling, but is certainly not admired around the world for the quality of its transport infrastructure. Despite this, all of these models are not only still in routine use, but they are more or less compulsory.
There is a common structure to all of these bogus models. They begin by considering how you would make a particular decision if you had complete and perfect knowledge of the world, now and in the future. As a process, it is incredibly data intensive, for obvious reasons. The problem is, very little of the relevant data is actually known. The solution? Make it all up.
To get an idea of what this invented data actually includes, we can use some examples from transport modelling. In the world of WebTag, an individual’s time is given a monetary value depending on the mode of transport by which they travel. So, according to the Department for Transport, as a taxi passenger, your time is worth £44.69 per hour, whereas the taxi driver’s time is considered much less valuable, at £9.77 per hour (2002 prices). Absurd as it might seem to put such a precise value on time in the present, the model demands that this level of precision continues into the future. Growth projections make it possible to predict how valuable time will be in 2052, to the penny. This ensures that every cell in the spreadsheet can be filled and that at the end of the process, some numbers will be provided. Most objective observers would conclude that this exercise has gotten out of hand.
What is wrong with these approaches?
Firstly, the way in which lack of knowledge about the future is addressed is unrealistic by most people’s standards. It is assumed to be essentially similar to the present, except for certain mechanical projections of demand, income etc. I have no idea how you will be getting about in 2052, it might be by personal flying platform or by horse and cart, and nor would I attempt to guess. A model, however, expects that you will still be travelling in the exact same way; the only thing that will have changed is the value of your time.
Another important aspect that is dealt with inadequately is the terminal value of projects. The very first cost-benefit analysis of a UK transport project was conducted in the 1960’s to assess the potential value of the Victoria Line. The assessment period covered fifty years, so it was assumed that there would be no benefits beyond 2011 at all. In the 1960’s the Victoria Line cost £90m to build; today it would probably be closer to £10bn. Even discounting that figure back fifty years, would still indicate that it is more valuable today than it has ever been. The same is true for many of the tube lines, but for some transport projects fifty years would be far too long a period of assessment, relative to their realistic life expectancy. It is a critical issue that is essentially ignored.
The prescription of a universal template obstructs the proper use of judgement and experience. But more than that, the cost of these exercises actually gets in the way of intelligent public debate. That is why opponents of the high speed train project, instead of simply voicing their concerns, felt compelled to commission their own assessment using the same standard WebTag model. It is probably not a coincidence that their study came to the opposite conclusion. These exercises have cost millions and yet the debate remains stagnant.
Why do we engage in these exercises?
Firstly, governments are under pressure to find an objective, analytic process for decision making via a mechanism that can be universally defended. Secondly, rationality and quantification are being confused. Lord Kelvin famously said, that if “you cannot measure it… your knowledge is of a meagre and unsatisfactory kind.” Frank Knight, a Chicago School economist sarcastically responded, “…and if you can’t measure it, measure it anyway.” That is what is being done here.
There are also significant entry barriers that have been constructed to ensure the continuing employment of people associated with this process, primarily consultants, civil servants and risk managers. The group of firms that build these models is small, and the only realistic way to enter the industry is by hiring from within them. The vested interest is clear.
What should we do instead?
We need to accept that a model can never be a true representation of the world and nor should we expect it to be. A good model is a purpose- specific simplification of the world. Its usefulness is in its relevance to the problem at hand, not its comprehensiveness. The real purpose of a model is to identify the key factors that ought to be influencing an assessment. For example, in the case of high speed rail, a critical element is how valuable it would be to passengers to reduce their journey time by half an hour. Having framed the issue in this way, it then becomes an exercise in gathering evidence.
Quantification can be a helpful analytical tool, but if it is too precise it becomes meaningless. To say that a project will cost more than £1 million but less than £1 billion, is considerably more useful than giving a bogus estimate of £43 million, based on fabricated numbers.
Above all we should abandon completely the concept of a standard template that can be applied to every problem with similar characteristics. The reason that these templates remain in use is largely due to the commercial and professional interests of the people involved.
I am strongly in favour of quantification, modelling and evidence-based policy. What I am against is bogus modelling that in my view discredits all of these things. These are all tools that are essential for good policy making, but the skill of a policy analyst is in identifying the models and evidence that are relevant. We must not confuse a model with the world that it is being used to describe. It was the polish philosopher, Alfred Korzybski, who put it best when he said, “the map is not the territory”.
Please read our comments policy before posting.
Note: This article gives the views of the author, and not the position of the British Politics and Policy blog, nor of the London School of Economics.
John Kay is one of Britain’s leading economists and visiting Professor at the LSE. His interests focus on the relationships between economics and business. His career has spanned academic work and think tanks, business schools, company directorships, consultancies and investment companies. For more details of John’s biography, see the his website.
Full disclosure: I used to own a portion of a simulation software modelling company and occasionally still do some consulting work in that field.
Your arguments here are spot on. I always use the old computing adage “garbage in = garbage out” when framing a model with someone who is new to the process. Often times the models will reflect the bias of the author(s) rather than the actual behavior of the system. In many cases they are less than transparent (sometimes deliberately), but they generate a nice quantitative number that decision makers can point to when everything goes wrong. Absolved of responsibility, decision
makers will frequently accept the conclusion of the model without question.
As you point out these models that generate a simple go/no go decision are trying to model a situation which is inherently uncertain – quantifiable into a range perhaps, but nevertheless uncertain. I actually couldn’t believe your claims about the WebTag framework and actually had to go verify that they used a single number. If we actually had 50,000 goes at running to 2052 maybe we would arrive at the exact number that has been provided (but I seriously doubt it and we obviously have only one shot at getting to 2052). As you correctly point out we will not know about some of the drivers of that number for 10 or 15 years. Unfortunately in a practical model we must bound the problem, but again that is why a range of values is more appropriate.
In many cases the practical value of predictive modeling is in examining the range of possible outcomes and mitigating/avoiding the most unpleasant. One of my colleagues in the simulation software company had a terrific example of why looking at a single value can lead to erroneous conclusions. He would present a list of expected present values of different approaches to an environmental remediation project he had modelled. The present value of one project was significant, but the remaining projects had negligible or negative present values. He’d ask the audience which project the group would pick – they would inevitably pick the one with the highest present value. Then he showed the probability distribution of the present values for each approach. The one with the highest present value was extremely wide – from very negative to very positive. The other approaches had narrower probability distributions close to the breakeven point. He’d ask again and the audience would come to a very different conclusion.