So you like evidence-based policy? Here’s a couple of crucial things to remember…

by Stephen Tall on October 31, 2012

Evidence-based policy. How could anyone doubt its importance? Well, not me for a start. One of the good things about being signed-up to a liberal ideology is that a rational-sceptical outlook — a determination to look beyond gut instinct and to demand to see how policy is working in practice — is its bread-and-butter.

Sometimes a demand for evidence-based policy is equated with an ideological vacuum, that the ‘what matters is what works mantra’ requires the suspension of your philosophical self. I don’t buy that interpretation.

To me, evidence-based policy is at heart a very liberal process. First, it demands that you measure your intended outcomes against your actual results. That’s a pretty good discipline. Secondly, what is termed ‘evidence’ (ie, real-life situations) will need adapting to local circumstance to be effective — in other words, it requires trust in people to work out how to adapt what’s worked elsewhere to their own context.

Both these points are made very effectively in an article by Jeremy Hardie and Nancy Cartwright on the LSE’s British Policy and Politics Blog:

There are too many examples where doing here what worked there, failed because of a failure to see what was relevant to success beyond the fact that the intervention had worked somewhere else.

Small class sizes worked to raise reading scores in Tennessee, but the same policy failed in California. As often, with hindsight the explanation is simple. If, as they did in California, you introduce smaller classes on a large scale very quickly, and of course you have to have lots more teachers, you end up with a lower average quality of teacher, because you have to recruit less experienced or well trained people. And of course you have to have many more class rooms, so you take space for reading classes from other activities which themselves contribute to the flourishing of pupils and hence to their reading ability. These and other changes mean that the potential of lower classes never gets realised. Even if you are confident that smaller class sizes played a determining causal role in Tennessee, you have to think whether it can do so in California. And that means thinking about what helping factors are needed and whether they will be present. The true fact that it worked in Tennessee does not of itself tell you how to think about that. It does not tell you what helping factors have to be present to allow smaller classes to play their causal role. Let alone how you find out whether those factors will be present. …

Here is another example. Suppose we think that CCTVs in car parks reduce car theft. You have to think how they do that if you are to use them properly. If it is by deterrence, you make them visible. If by giving the police a chance to catch people red-handed, you make them invisible. And you have to have enough police, near, to make arrest likely. And you may not have to worry if CCTV evidence is valid in court if the causal role is based on deterrence. Similarly with speed cameras.

If this all sounds down to earth, so it is. Exactly what evidence you need, of what, is the key question that has to be asked for making real evidence based social policy interventions. And the answer, given the complexity of multidimensional social problems, will very likely be quite specific to the particular context. It is unlikely that you can limit yourself to establishing that it worked, on average, there.

You can read their post in full here.