A little bit about my job. A whole lot more about the importance of evidence-based policy and RCTs.

by Stephen Tall on November 23, 2012

A rare work-focused post follows…

It’s been almost a year since I started a new job, working at the Education Endowment Foundation. It’s a new grant-making charity, a rare combination of a start-up with money (you can normally be one or the other). The fact that we have a grant of £125m from the Department for Education has enabled us to get on with things in double-quick time.

Every project that’s funded has a sound evidential basis in educational research. More importantly, the EEF’s funding has an explicit aim of testing the projects: every single one will be independently evaluated, almost all through randomised control trials (RCTs) so that we can find out if they really have made a difference to children’s attainment.

Everyone has a view on education based on their own experience. And charity is an emotive and deeply personal topic. As a result, educational charities too often follow their hearts not their heads.

The EEF is (unapologetically) at the opposite end of the spectrum. We don’t fund projects based on our own beliefs or values. We do follow the evidence of what appears to work and fund projects to find out if they stand up to robust evaluation.

Matthew Green published an interesting post on his Thinking Liberal blog questioning the notion of ‘evidence-based policy’. Where I agree with him is that too often the catechism of evidence-based policy is used glibly. I also agree that there is a risk of ‘science-ism’, of folk putting too much trust in one single RCT to prove or disprove a policy. (At the EEF we use a cumulative approach which recognises trials can produce conflicting evidence.)

Where I disagree with him is the idea that a commitment to evidence-based policy is of little real-world value. In education, most interventions are benign — in other words, most things work to some extent: but there is a big difference in the range. Introducing (for example) school uniforms or marginally lowering class sizes have a very small, positive impact; training teachers to give more effective feedback to their pupils or meta-cognition strategies (thinking about how you learn) has a much more significant impact. This is the kind of information that — in the real world — schools want to know when deciding how to allocate their resources.

I disagree, too, with Simon Titley’s argument here that evidence-based policy is a technocratic device for abdicating responsibility for making political choices. Of course it can be used in that way — just as ‘value judgements’ can become dogmatic if not used intelligently.

The point, surely, of evidence-based policy is to enable us to work out how most effectively to apply our own values? If the evidence told me the most effective way to boost pupil attainment was to subject pupils to day-long rote-learning in a hot-house I would ignore that evidence on the basis it would probably make my child miserable. But if the evidence told me that a particular learning style had proved especially effective then I would want to find out more.

Evidence-based policy is not about experts telling professionals, whether in teaching or anything else, what they must do. It is about arming them with the knowledge they need to be evidence-literate. And that evidence-literacy is crucial in working out what will work best in a particular context — because evidence can only tell us that something has worked there, not that it will work here.

Evidence is not a replacement for democratic accountability — unless your concept of democracy is implementing policy and ignoring its impact. It is about giving decision-makers — whether they’re professionals or politicians — the information they need to make informed choices, and how best to put their values into action.