A liberal approach to evidence-based policy-making

by Stephen Tall on July 31, 2015

“It is a fundamental principle of good public services that decisions are made on the basis of strong evidence and what we know works.”
Danny Alexander and Oliver Letwin, launching the Government’s What Works Network of evidence centres in 2013.

Suddenly politicians are interested in evidence-based policy. About time, too. The fields of public health and social policy are strewn with warnings of the dangers of ignoring evidence:

• an estimated 10,000 infant deaths in the UK could have been prevented if official guidance had followed the available evidence and parents had been advised not to place their babies to sleep on their front1)[1] From the mid-1950s until the early 1990s, parents were advised to place infants to sleep on their front – even though by 1970 clinical research showed that this was likely to be harmful, with a statistically significantly increased risk of Sudden Infant Death Syndrome (SIDS, commonly known as ‘cot death’). See Ruth Gilbert, Georgia Salanti, Melissa Harden and Sarah See, ‘Infant sleeping position and the sudden infant death syndrome: systematic review of observational studies and historical review of recommendations from 1940 to 2002’ (International Journal of Epidemiology, Oxford University Press, 2005). [Accessed 13/5/2014];

• the ‘Scared Straight’ programme – designed to deter young people from criminal behaviour by bringing them into contact with adult inmates to make them aware of the grim realities of life in prison – is still championed by British police forces2)For instance, Peter Neyroud, a former Chief Constable of the Thames Valley Police Force, notes in a blog-post, ‘Transforming policing through science‘ (5 March 2013): ‘One British Chief proudly told me about his local “scared straight” initiative and, sadly, remained proud of it even after I had given a copy of the systematic review which should have told him he was making things worse.’ [Accessed 13/5/2014] despite strong evidence that it leads to higher offending rates.3)The ‘Scared Straight’ programme developed in the USA in the 1970s to deter juvenile delinquents and at-risk children from criminal behaviour. Early studies, including in the UK, showed astonishingly high success rates, as much as 94 per cent. However, none of these evaluations had a control group showing what would have happened to these participants if they had not participated in the programme. When tested through Randomised Controlled Trials it was discovered ‘Scared Straight’ led to higher rates of offending behaviour. See Laura Haynes, Owain Service, Ben Goldacre and David Torgerson: ‘Test, Learn, Adapt: Developing Public Policy with Randomised Controlled Trials’ (Cabinet Office – Behavioural Insights Team, 2012), p.17. [Accessed 13/5/2014]

Such salutary lessons should encourage government to place a premium on commissioning and using high-quality evidence in order to inform its own policy-making. However, a recent National Audit Office report on the quality of almost 6,000 government evaluations found the strength of evidence to be in inverse proportion to the claims made for the effectiveness of policies: the most positive claims were based on the weakest research.4)In a review of almost 6,000 analytical outputs published on 17 main departmental websites between 2006 and 2012, the National Audit Office ‘found some evidence that evaluation reports that are weaker in identifying causality tend to be more positive in assessing what the intervention achieved.’ Report by the National Audit Office, ‘Evaluation in government‘ (National Audit Office, December 2013), p.8 [Accessed 13/5/2014]

Civil servants might argue they take their cue from their political masters. Research by the Institute for Government has highlighted how some politicians may prefer to avoid the ‘inconvenient truths’ of evidence to avoid unpopularity and be unwilling to commit to multi-year evaluations which don’t fit the electoral timetable.5)The Institute for Government ran a series of seminars on evidence-based policy in 2012. On avoiding unpopularity, it recorded: ‘one politician pointed out “problems came when evaluations recommended policies you thought would mean you lose your job”. There were areas, such as hospital closures where robust evidence pointed very strongly in one direction – towards amalgamation of services – but local opinion was hostile and tactical oppositions could make life impossible for government ministers who supported unpopular decisions. In those cases evidence did very little to sway public opinion.’ On evaluations not fitting electoral cycles, it recorded a politician saying: ‘“I sat down with a research body the other day and they set out what they were planning to do. I said that ‘do you realise that by the time you reach your conclusions, it will be far too late to be of use to anybody … and I’ll be the fisheries minister by then’”.’ Quoted in Jill Rutter, ‘Evidence and Evaluation in Policy-Making: a problem of supply or demand?’ (Institute for Government, September 2012), pp.17-19 [Accessed 13/5/2014]

I do not lightly dismiss such difficulties. However, there are two things which government can do easily and can do now. First, it can look at the evidence that already exists for pointers as to which policies are most likely to work effectively.6)This is the role of the six centres of evidence comprising the What Works Network in the following areas: Health and social care – National Institute for Health and Care Excellence (NICE); Improving education outcomes for school-aged children – Sutton Trust/Educational Endowment Foundation); Crime reduction – College of Policing; Effective early intervention – Early Intervention Foundation; Fostering local economic growth – Centre for local economic growth, hosted by LSE, Arup, Centre for Cities; and Promoting active and independent ageing – Big Lottery Fund’s Centre for Ageing Better. Secondly, it can ensure when trying out new ideas the pilots are set up in a way that allows their effects to be appropriately evaluated. As Tim Harford points out:

While randomised trials are not going to tell us when to raise interest rates or get out of Afghanistan, there are many policies that could and should be tested with properly controlled trials. Is Jamie Oliver right to emphasise healthy school meals? Run a trial. Should young offenders be sent to boot camp, or to meet victims of crime? Run a trial. What can we do to persuade households to use less electricity? Run a trial.

Such talk makes some liberals queasy. They fear evidence will trump our values, that we will end up led blindly wherever the ‘outcome measures’ take us. I understand the concern, so here are three points I believe are crucial for a liberal approach to evidence-based policy-making:

1. Evidence matters: in particular it matters when tackling those problems in society which are most intractable.
2. Evidence alone is not sufficient: it is the means by which we work out how best to realise and develop our liberal values.
3. Evidence and localism go hand-in-hand: we need continuous experimentation and evaluation to find out what works within different contexts.

Let’s examine each of these points in turn.

1. Evidence matters

Good intentions aren’t enough. There is no point funding a ‘back to work’ programme with the best of motives if it does not increase the number of people who gain employment as a result. Indeed, there is a significant opportunity cost if the money could have been spent on a more effective programme instead. That is why testing such policies through Randomised Controlled Trials – the invention of John Stuart Mill7)‘An RCT [Randomised Controlled Trial] is a study design based on John Stuart Mill’s method of difference for making causal inferences’. Nancy Cartwright and Jeremy Hardie: ‘Evidence-Based Policy: a practical guide to doing it better’ (OUP USA, 2012), p.33 – is so important in establishing whether what we hoped to see happening is what is actually happening.

There is no alternative to robust evaluation if we are serious about finding out what works in ensuring (in the words of the Liberal Democrat Preamble) that ‘no one shall be enslaved by poverty, ignorance or conformity’.

In his chapter in The Orange Book, ‘Reclaiming Liberalism’ , David Laws identified four mutually integral components, economic, social, personal and political liberalism. The first three are a useful framework for examining examples of why evidence matters in addressing the Preamble’s three ‘Giant Evils’: poverty, ignorance and conformity.

An example of how evidence-based economic liberalism can address poverty

Immigration is seen by voters as the second most important issue facing Britain today8)According to the Economist/Ipsos MORI April 2014 Issues Index, 34% of the UK public views immigration/race relations as the most important issue facing the country. The economy was ranked top, with 36%, unemployment third (31%), and the NHS trailed in fourth (25%). [Accessed 13/5/2014], with many fearing the ready supply of ‘cheap, foreign labour’ results in fewer jobs and lower wages for natives (as well as pressures on public services and housing). This view has, for instance, been voiced by the editor of The Spectator, Fraser Nelson, who argues mass immigration in recent years ‘broke the link between more jobs and less dole’. Yet the evidence is clear this is not the case, with economist Jonathan Portes noting: ‘the ‘lump of labour’ fallacy is just that: … more jobs for immigrants doesn’t mean fewer for natives’. And though it may drive down wages for the low-skilled ‘the effect is small compared to that of other factors (technological change, the national minimum wage and so on)’.

The reality is that the free movement of labour is crucial to the UK’s prosperity and, therefore, our ability to invest in public services and afford decent social security provision for those who need it. Migrants plug gaps in our labour market. They are also net tax contributors yet low-demand consumers of public services, helping to defuse the demographic time-bomb of our ageing population. As the Office for Budget Responsibility has highlighted, if all migration ended tomorrow, the UK’s average annual growth rate would fall to 2 per cent and the national debt would spiral to 120 per cent of GDP by the middle of the century.

An example of how evidence-based social liberalism can address ignorance

Education, the antidote to ignorance, is the most powerful lever of social mobility available to us, and the best exit route from low-skill, low-wage jobs. That makes it all the more depressing that the attainment gap which divides pupils from rich and poor backgrounds is such a stubborn feature of the domestic landscape.9)The OECD, an international economic organisation of 34 countries, recently reported that, in England, ‘the impact of pupils’ socio-economic background is significantly higher than the OECD average’. Emily Knowles and Helen Evans: ‘PISA 2009: How does the social attainment gap in England compare with countries internationally?‘ (Department for Education, April 2012) [Accessed 13/5/2014] Tackling educational inequality extends well beyond the scope of schools, of course. But schools can be part of the answer. Research by the Education Endowment Foundation has shown there are 428 secondary schools, one-in-seven across England, where the GCSE performance of pupils eligible for free school meals exceeds the national average for all pupils.10)Kevan Collins: ‘Closing the gap – follow the evidence’ (presentation to Ofsted South East leadership conference, 7 March 2014), slide 38/59 [Accessed 13/5/2014]

The political debate can be stereotypically sterile. The ‘right’ – in thrall to private schools it wrongly believes to be superior11)As Professor Dylan Wiliam of the Institute of Education noted in an article, ‘Why London’s schools are the best in England‘: ‘OECD data shows that the average achievement in private schools in the UK is the same as that in state schools, once social class is taken into account, even though the average class size is 13 in the private sector and 25 in the state sector.’ (The Guardian, 7 August 2012). [Accessed 13/5/2014] – obsesses about school structures. The ‘left’ – convinced government can fix problems from the centre – too easily assumes school improvement is a simple function of public money spent.12)In England, revenue spending on schools increased by 47 per cent in real terms from 2001 to 2011. Yet when Robert Coe, Professor of Education at Durham University, reviewed England’s performance in the international surveys of attainment, he observed: “the pattern of results from the different international surveys is actually fairly consistent: not much change between 1995 and 2011”. (‘Improving Education: a triumph of hope over experience’ (Inaugural lecture at Durham University’s Centre for Evaluation and Monitoring, 18 June 2013) [Accessed 13/5/2014] Yet the evidence is clear that what makes most difference to overall attainment and narrowing the gap is improving the quality of teaching and learning in schools. As the OECD’s Andreas Schleicher has remarked: “The quality of an education system cannot exceed the quality of its teachers.” This means focusing our energies on crucial areas like high-quality initial teacher training and continuing professional development.13)See, for example, the proposals in James Kempton’s ‘To teach, to learn’ (CentreForum, November 2013) [Accessed 13/5/2014]

An example of how evidence-based personal liberalism can address conformity

The ‘war on drugs’ is one manifestation of successive governments’ willingness to ignore evidence while legislating against citizens’ lifestyle choices. Labour home secretary Alan Johnson sacked the chairman of the Advisory Council on the Misuse of Drugs in 2009 for daring to publish a research briefing on the relative harm of different drugs which showed alcohol to be more dangerous than cannabis, LSD and ecstasy. His Conservative successor, Theresa May, is currently seeking to criminalise khat, even though the ACMD stated this would be ‘inappropriate and disproportionate’. Here is one measure of the (in)effectiveness of such evidence-free policy-making: it is officially estimated 36.5 per cent of adults in the UK – around 12 million people – have taken an illicit drug in their lifetime.

To their credit, the Liberal Democrats have not succumbed to this peer pressure. Home office minister Norman Baker has made clear his own opposition to the khat ban , while Deputy Prime Minister Nick Clegg has urged a major review of Britain’s narcotics laws: “If you were waging any other war where you have 2,000 fatalities a year, your enemies are making billions in profits, constantly throwing new weapons at you and targeting more young people — you’d have to say you are losing and it’s time to do something different.”

2. Evidence alone is not sufficient

Evidence matters in navigating the best route to your destination, but it is not an end in itself. Your values will be fundamental in determining the kind of country you want to live in. Let us turn to David Laws’ fourth liberal tenet – political liberalism – to illustrate this point.

Across the western world, nations are worried about declining voter turn-out. In the UK, 78 per cent of the electorate cast their votes at the 1992 election; by 2001, just nine years later, this had plummeted to 59 per cent, and it has since recovered only a little (to 65 per cent in 2010).

I believe in representative democracy and would like to see many more of my fellow citizens exercise their right to vote. I might therefore look for evidence of where voter turnout is high and alight on Australia as a shining beacon: more than 90 per cent of its citizens consistently turn-out to vote. But I would quickly realise that such success has been achieved by making voting compulsory: the ultimate punishment for non-compliance is jail. As I believe voting is a right, not an obligation, the evidence that compulsory voting would boost turnout is not sufficient to persuade me that the UK should copy Australia.

3. Evidence and localism go hand-in-hand

Evidence guarantees nothing: it can tell you what worked there, then, for those people. That does not automatically mean it will work here, now, as you would implement it, for these people. Government often expects too much from evidence, assuming that successful trials can simply be expanded or transplanted and deliver the same results.

For example, California attempted in the 1990s to re-create the success that had been achieved by Tennessee in boosting reading scores for its pupils, which the evaluation had attributed to lower class sizes. Yet when California moved to smaller class sizes it made no difference. Its decision to hire lots more teachers very quickly meant it had to recruit less experienced, less well-trained people: the average quality of its teachers dropped.14)Nancy Cartwright and Jeremy Hardie: ‘Evidence-Based Policy: a practical guide to doing it better’ (OUP USA, 2012), pp.3-4

This is why ongoing experimentation and evaluation at the local level is crucial: making good use of the evidence of what has worked there, but adapting where necessary according to our own contexts until we find what works here. This is why I believe that evidence is an empowering tool.

Local authorities should, for instance, have the power to pilot alternative ways of levying taxes to fund the services they provide: why shouldn’t Brighton and Hove City Council be able to introduce a Land Value Tax and abolish its Council Tax and Business Rates?15)Caroline Lucas, Green MP for Brighton Pavilion, proposed in Parliament in 2013 ‘a new tax on the value of land to be gradually introduced over a period of 10 years as a replacement for Business Rates and Council Tax’ [Accessed 13/5/2014] And while it is the democratic right of government ministers to require changes to the running of our public services, front-line professionals should also have the right first to ask of them, ‘Show me your evidence’.

Conclusion

There are no silver bullets in public policy-making. As H.L. Mencken observed, ‘For every complex problem there is an answer that is clear, simple, and wrong.’ However, evidence – used intelligently, applied locally – offers us the best guide we have in developing liberal answers that can tackle effectively the messy, difficult problems we face.

(I’ve written this in a personal capacity, but obviously it’s informed by my day-job at the Education Endowment Foundation, where we fund trials in schools of evidence-based ideas designed to increase the attainment of disadvantaged pupils. This was originally submitted to CentreForum in May 2014 for its essay competition, The Challenges Facing Contemporary Liberalism. However, as that’s died a death I’m publishing it here. An adapted version has been published in One Hundred Days for Early Action (PDF) by the Early Action Task Force in April.)

Enjoy reading this? Please like and share:

References   [ + ]

1. [1] From the mid-1950s until the early 1990s, parents were advised to place infants to sleep on their front – even though by 1970 clinical research showed that this was likely to be harmful, with a statistically significantly increased risk of Sudden Infant Death Syndrome (SIDS, commonly known as ‘cot death’). See Ruth Gilbert, Georgia Salanti, Melissa Harden and Sarah See, ‘Infant sleeping position and the sudden infant death syndrome: systematic review of observational studies and historical review of recommendations from 1940 to 2002’ (International Journal of Epidemiology, Oxford University Press, 2005). [Accessed 13/5/2014]
2. For instance, Peter Neyroud, a former Chief Constable of the Thames Valley Police Force, notes in a blog-post, ‘Transforming policing through science‘ (5 March 2013): ‘One British Chief proudly told me about his local “scared straight” initiative and, sadly, remained proud of it even after I had given a copy of the systematic review which should have told him he was making things worse.’ [Accessed 13/5/2014]
3. The ‘Scared Straight’ programme developed in the USA in the 1970s to deter juvenile delinquents and at-risk children from criminal behaviour. Early studies, including in the UK, showed astonishingly high success rates, as much as 94 per cent. However, none of these evaluations had a control group showing what would have happened to these participants if they had not participated in the programme. When tested through Randomised Controlled Trials it was discovered ‘Scared Straight’ led to higher rates of offending behaviour. See Laura Haynes, Owain Service, Ben Goldacre and David Torgerson: ‘Test, Learn, Adapt: Developing Public Policy with Randomised Controlled Trials’ (Cabinet Office – Behavioural Insights Team, 2012), p.17. [Accessed 13/5/2014]
4. In a review of almost 6,000 analytical outputs published on 17 main departmental websites between 2006 and 2012, the National Audit Office ‘found some evidence that evaluation reports that are weaker in identifying causality tend to be more positive in assessing what the intervention achieved.’ Report by the National Audit Office, ‘Evaluation in government‘ (National Audit Office, December 2013), p.8 [Accessed 13/5/2014]
5. The Institute for Government ran a series of seminars on evidence-based policy in 2012. On avoiding unpopularity, it recorded: ‘one politician pointed out “problems came when evaluations recommended policies you thought would mean you lose your job”. There were areas, such as hospital closures where robust evidence pointed very strongly in one direction – towards amalgamation of services – but local opinion was hostile and tactical oppositions could make life impossible for government ministers who supported unpopular decisions. In those cases evidence did very little to sway public opinion.’ On evaluations not fitting electoral cycles, it recorded a politician saying: ‘“I sat down with a research body the other day and they set out what they were planning to do. I said that ‘do you realise that by the time you reach your conclusions, it will be far too late to be of use to anybody … and I’ll be the fisheries minister by then’”.’ Quoted in Jill Rutter, ‘Evidence and Evaluation in Policy-Making: a problem of supply or demand?’ (Institute for Government, September 2012), pp.17-19 [Accessed 13/5/2014]
6. This is the role of the six centres of evidence comprising the What Works Network in the following areas: Health and social care – National Institute for Health and Care Excellence (NICE); Improving education outcomes for school-aged children – Sutton Trust/Educational Endowment Foundation); Crime reduction – College of Policing; Effective early intervention – Early Intervention Foundation; Fostering local economic growth – Centre for local economic growth, hosted by LSE, Arup, Centre for Cities; and Promoting active and independent ageing – Big Lottery Fund’s Centre for Ageing Better.
7. ‘An RCT [Randomised Controlled Trial] is a study design based on John Stuart Mill’s method of difference for making causal inferences’. Nancy Cartwright and Jeremy Hardie: ‘Evidence-Based Policy: a practical guide to doing it better’ (OUP USA, 2012), p.33
8. According to the Economist/Ipsos MORI April 2014 Issues Index, 34% of the UK public views immigration/race relations as the most important issue facing the country. The economy was ranked top, with 36%, unemployment third (31%), and the NHS trailed in fourth (25%). [Accessed 13/5/2014]
9. The OECD, an international economic organisation of 34 countries, recently reported that, in England, ‘the impact of pupils’ socio-economic background is significantly higher than the OECD average’. Emily Knowles and Helen Evans: ‘PISA 2009: How does the social attainment gap in England compare with countries internationally?‘ (Department for Education, April 2012) [Accessed 13/5/2014]
10. Kevan Collins: ‘Closing the gap – follow the evidence’ (presentation to Ofsted South East leadership conference, 7 March 2014), slide 38/59 [Accessed 13/5/2014]
11. As Professor Dylan Wiliam of the Institute of Education noted in an article, ‘Why London’s schools are the best in England‘: ‘OECD data shows that the average achievement in private schools in the UK is the same as that in state schools, once social class is taken into account, even though the average class size is 13 in the private sector and 25 in the state sector.’ (The Guardian, 7 August 2012). [Accessed 13/5/2014]
12. In England, revenue spending on schools increased by 47 per cent in real terms from 2001 to 2011. Yet when Robert Coe, Professor of Education at Durham University, reviewed England’s performance in the international surveys of attainment, he observed: “the pattern of results from the different international surveys is actually fairly consistent: not much change between 1995 and 2011”. (‘Improving Education: a triumph of hope over experience’ (Inaugural lecture at Durham University’s Centre for Evaluation and Monitoring, 18 June 2013) [Accessed 13/5/2014]
13. See, for example, the proposals in James Kempton’s ‘To teach, to learn’ (CentreForum, November 2013) [Accessed 13/5/2014]
14. Nancy Cartwright and Jeremy Hardie: ‘Evidence-Based Policy: a practical guide to doing it better’ (OUP USA, 2012), pp.3-4
15. Caroline Lucas, Green MP for Brighton Pavilion, proposed in Parliament in 2013 ‘a new tax on the value of land to be gradually introduced over a period of 10 years as a replacement for Business Rates and Council Tax’ [Accessed 13/5/2014]

One comment

[…] the end of last month Stephen Tall posted an essay entitled A liberal approach to evidence-based policy making on his blog. The essay had originally been submitted for last year’s CentreForum essay […]

by Changing behaviour | Alex's Archives on August 27, 2015 at 11:00 pm. Reply #

Leave your comment

Required.

Required. Not published.

If you have one.