Stephen Tall

• Tweet
• Sharebar

• Tweet

If you’ve any interest at all in evidence-based public policy, then do listen to Ben Goldacre’s Bad Evidence, available for the next 6 days on BBC iPlayer here. It’s a fascinating exploration of the issues around using randomised control trials (RCTs) to work out whether the policies we hope will work actually do work in reality.

The medical profession uses RCTs all the time: there is little controversy which surrounds it because it’s become accepted practice, a necessary way of gathering the information clinicians need to ensure their prescriptions are doing us good (and, more importantly, not doing us harm). Yet suggest the use of RCTs in social policy and people come over all queasy. Ironically, if it’s not a matter of life or death then ‘experimentation’ becomes, apparently, a matter of life or death.

Governments have been notoriously bad in the past at actively using robust evidence to inform policy, still less to recognise the need for additional evidence, or proper evaluation of existing policies. Most politicians either don’t understand RCTs, or if they do are too nervous or impatient (or both) to find out if their policies are delivering the promised goods.

Take the Educational Maintenance Allowance: originally the EMA was trialled in a couple of areas (a decade ago), but before even its first pilot evaluation had reported Gordon Brown decided to roll it out nationwide, thus making the original pilots useless. Later evaluation by the IFS suggested the EMA carried a significant ‘dead-weight cost’ (ie, paying young adults to attend school/college who would have attended anyway), but that it was probably justified based on long-term benefits anyway. But by that time, of course, there was no control group to compare the results against. And in the end of course it was ditched by the Coalition anyway in favour of a more targeted (ie, cheaper) bursary scheme.

Liberals should welcome RCTs. Not only because they help promote the kind of rational-sceptical approach to policy-making that’s our natural bent anyway; but also because they encourage devolved, local experimentation, and the piloting of new ideas. Critics call it a ‘postcode lottery’. In reality it’s a way of testing ideas in a smaller setting, checking their impact, adapting based on experience, and allowing for local adaptation. As I wrote a couple of months ago:

Evidence-based policy is not about experts telling professionals, whether in teaching or anything else, what they must do. It is about arming them with the knowledge they need to be evidence-literate. And that evidence-literacy is crucial in working out what will work best in a particular context — because evidence can only tell us that something has worked there, not that it will work here.

Evidence is not a replacement for democratic accountability — unless your concept of democracy is implementing policy and ignoring its impact. It is about giving decision-makers — whether they’re professionals or politicians — the information they need to make informed choices, and how best to put their values into action.

Declaration of interest: one of those interviewed by Ben Goldacre is Kevan Collins, chief executive of the Education Endowment Foundation where I work. Oh, and as Ben remarked on Twitter during the broadcast:

The Education Endowment Fund is AMAZING btw bit.ly/zVS7wo They spend 10% of money on RCTs to appraise impact bbc.in/WiK6f3

— ben goldacre (@bengoldacre) January 1, 2013