As the Obama administration’s evidence agenda rolls forward it is beginning to stir countervailing views on the appropriate use of evidence. What counts as evidence in the field of social policy? How should it be used? How will the challenges of replicating evidence-based programs be overcome?
This article asks those questions of three different experts in the field of evidence-based policy, each with a distinct view on the evidence debate: Jon Baron from the Coalition for Evidence-Based Policy, Lisbeth Schorr of the Center for the Study of Social Policy, and David Muhlhausen of the Heritage Foundation.
Rise of the Randomized Controlled Trial (RCT)
If you were looking for someone outside government whose thinking on evidence seems to have gained significant traction with executive branch officials in both the Bush and Obama administrations, it would be difficult to find a better organization than the Coalition for Evidence-Based Policy, headed by Jon Baron.
The nonpartisan Coalition’s views are so well respected by the current administration that the group’s work was prominently cited in a 2012 White House memo to federal agencies on the use of evidence in their budget requests. The reference was to the use of administrative data for low-cost, rigorous evaluations, something that received a leg up early this year when OMB released a new memo endorsing its use.
Baron’s organization is known for advocating the use of randomized controlled trials (RCTs), where feasible, to evaluate the effectiveness of social policies. In RCTs, program participants are randomly assigned to one of two groups: a treatment group that receives the tested program services, or another group, called the control group, that receives standard services. Randomly assigning participants to one of the two groups ensures that they are comparable in both observable characteristics, such as age, income level, and educational background, and unobservable characteristics, such as motivation and family support. At the end of the study, comparisons of outcomes between the groups are used to determine whether the tested program has produced results.
RCTs are sometimes called the “gold standard” for evaluating program effectiveness — although, as we shall see, there is some disagreement about this. Baron believes that large, well-conducted RCTs are needed because the results of other, more preliminary studies — including small, short-term RCTs and many non-randomized forms of evaluation — are too often not confirmed when they are subjected to more definitive RCT evaluations.