A little bit about my job. A whole lot more about the importance of evidence-based policy and RCTs.

by Stephen Tall on November 23, 2012

A rare work-focused post follows…

It’s been almost a year since I started a new job, working at the Education Endowment Foundation. It’s a new grant-making charity, a rare combination of a start-up with money (you can normally be one or the other). The fact that we have a grant of £125m from the Department for Education has enabled us to get on with things in double-quick time.

Every project that’s funded has a sound evidential basis in educational research. More importantly, the EEF’s funding has an explicit aim of testing the projects: every single one will be independently evaluated, almost all through randomised control trials (RCTs) so that we can find out if they really have made a difference to children’s attainment.

Everyone has a view on education based on their own experience. And charity is an emotive and deeply personal topic. As a result, educational charities too often follow their hearts not their heads.

The EEF is (unapologetically) at the opposite end of the spectrum. We don’t fund projects based on our own beliefs or values. We do follow the evidence of what appears to work and fund projects to find out if they stand up to robust evaluation.

Matthew Green published an interesting post on his Thinking Liberal blog questioning the notion of ‘evidence-based policy’. Where I agree with him is that too often the catechism of evidence-based policy is used glibly. I also agree that there is a risk of ‘science-ism’, of folk putting too much trust in one single RCT to prove or disprove a policy. (At the EEF we use a cumulative approach which recognises trials can produce conflicting evidence.)

Where I disagree with him is the idea that a commitment to evidence-based policy is of little real-world value. In education, most interventions are benign — in other words, most things work to some extent: but there is a big difference in the range. Introducing (for example) school uniforms or marginally lowering class sizes have a very small, positive impact; training teachers to give more effective feedback to their pupils or meta-cognition strategies (thinking about how you learn) has a much more significant impact. This is the kind of information that — in the real world — schools want to know when deciding how to allocate their resources.

I disagree, too, with Simon Titley’s argument here that evidence-based policy is a technocratic device for abdicating responsibility for making political choices. Of course it can be used in that way — just as ‘value judgements’ can become dogmatic if not used intelligently.

The point, surely, of evidence-based policy is to enable us to work out how most effectively to apply our own values? If the evidence told me the most effective way to boost pupil attainment was to subject pupils to day-long rote-learning in a hot-house I would ignore that evidence on the basis it would probably make my child miserable. But if the evidence told me that a particular learning style had proved especially effective then I would want to find out more.

Evidence-based policy is not about experts telling professionals, whether in teaching or anything else, what they must do. It is about arming them with the knowledge they need to be evidence-literate. And that evidence-literacy is crucial in working out what will work best in a particular context — because evidence can only tell us that something has worked there, not that it will work here.

Evidence is not a replacement for democratic accountability — unless your concept of democracy is implementing policy and ignoring its impact. It is about giving decision-makers — whether they’re professionals or politicians — the information they need to make informed choices, and how best to put their values into action.

11 comments

New from me > A little bit about my job. A whole lot more about the importance of evidence-based policy and RCTs. http://t.co/NWaz2zOc

by Stephen Tall on November 23, 2012 at 3:59 pm. Reply #

A little bit about my job. A whole lot more about the importance of evidence-based policy and RCTs. http://t.co/vzayRTlU

by BexhillBattleLibDems on November 23, 2012 at 6:32 pm. Reply #

New from me > A little bit about my job. A whole lot more about the importance of evidence-based policy and RCTs. http://t.co/vbtzSVwn

by Stephen Tall on November 23, 2012 at 6:59 pm. Reply #

Really good post by @stephentall (who knows a thing or two about the subject) on evidence-based policy http://t.co/U8NzPECu

by Nick Thornsby on November 23, 2012 at 7:02 pm. Reply #

New from me > A little bit about my job. A whole lot more about the importance of evidence-based policy and RCTs. http://t.co/vbtzSVwn

by Gary Peile on November 23, 2012 at 7:19 pm. Reply #

New from me > A little bit about my job. A whole lot more about the importance of evidence-based policy and RCTs. http://t.co/vbtzSVwn

by Gary Peile on November 23, 2012 at 7:19 pm. Reply #

New from me > A little bit about my job. A whole lot more about the importance of evidence-based policy and RCTs. http://t.co/vbtzSVwn

by Gary Peile on November 23, 2012 at 7:19 pm. Reply #

Really good post by @stephentall (who knows a thing or two about the subject) on evidence-based policy http://t.co/U8NzPECu

by Stephen Tall on November 23, 2012 at 8:03 pm. Reply #

New from me > A little bit about my job. A whole lot more about the importance of evidence-based policy and RCTs. http://t.co/H69ozI6n

by Stephen Tall on November 23, 2012 at 9:59 pm. Reply #

Thank you for the kind words about my blog post. I’m not sure that we are disagreeing about all that much, if anything. There are a lot of areas where evidence based policy is extremely helpful. But a lot of places where it isn’t. In education, for example, for specific teaching methods evidence based policy is clearly useful – so long as it doesn’t lead to a one-size-fits all attitude. It will guide you to what helps most pupils, but rarely all. On the other hand I think it is of little help in broader issues of school governance, for example (academy vs LA controlled, etc). First because of difficulties in finding the right quality of evidence – and second because it is too broad a question, too far up the decision tree. Most questions of political interest are in this latter category – an appeal to “evidence based” policy is usually just a device for saying no to any given reform: this is what I mean by the “dark side”. Or as Simon Titley argues an excuse for New Labour type bureaucrats to take decisions away from ordinary people.

There is a different question about different types of evidence which needs to be talked about. I think that carefully analysed case studies would be perfectly valid evidence for more political decisions – but they would not satisfy a fetish for statistics among some social scientists, and not provide neat, one-dimensional answers.

by Matthew Green on November 24, 2012 at 4:22 pm. Reply #

“educational charities too often follow their hearts not their heads.” – Are you saying that the heart is appealing, but the head is right?

Statements such as: “The evidence shows that the policy was effective.” are too general to be evaluated. The devil is in the detail – “what evidence, exactly?” For instance a survey response can be merely a collection of opinions, a written examination result does not necessarily represent an expertise (in cooking for instance).

Of course the people designing the RCTs take all this into account, but I believe they can be only more or less successful.

So, the evidence must be looked at case by case, but I would also argue that there are some subjects, like students emotional development, which would be particularly tricky to develop RCT’s for.

I’m sure you know the old joke of the man looking for his keys under the street light, even though he dropped them in the alley – because there was more light there!

It is more important to promote the aspects of education which we value most, than to promote those we can measure most effectively – they are not always the same. And see how easy value judgments can creep into experimental design.

Sometime the heart simply knows best!

by Richard G on November 27, 2012 at 5:03 pm. Reply #

Leave your comment

Required.

Required. Not published.

If you have one.



You might also likeclose
Plugin from the creators of Brindes :: More at Plulz Wordpress Plugins