Addressing the Research “Replication Crisis”: Evidence-based Policy’s Hidden Vulnerability

One of the central tenets of evidence-based policy is that programs and policies that are deemed “evidence-based” should be replicable in other settings.

This is not always easy — even interventions backed by solid evidence can be hard to replicate. There are lots of potential reasons for this, including differences in the target populations, different settings, poor implementation, and insufficient fidelity to the original model, to name a few.

But the challenge becomes even more difficult if the original research is suspect. As it turns out, this is a potential problem for all kinds of research, much of which is in the throes of a full-blown replication crisis. Why does that matter? If the research is shaky, then the entire edifice of evidence-based policy comes crashing down.

The Replication Crisis in Research

To be clear, this particular replication crisis is not about replicating an evidence-based program that is based on strong research. That is an implementation problem, which is its own challenge. No, this replication crisis is about problems with the underlying research itself.

The research replication crisis is a problem, not just for basic science, but for the social sciences, too. For example, one recent attempt to repeat the findings of 100 peer-reviewed psychology studies found that only 39 could be reproduced. Empirical economics has faced its own similar replication crisis. Stanford professor John Ioannidis, a long-time critic of research more generally, has argued that the primary reasons are poorly-designed studies and researcher bias.

How do we solve these problems? One way is to review the studies to determine their rigor. That is the central mission of evidence clearinghouses, several of which are run by the federal government. (Other reviewers like the Campbell Collaboration are privately run.) Unfortunately, some of the clearinghouses have been experiencing challenges of late.

Their job could be made much easier, however, if the researchers themselves implemented best practices that would help show that their research is credible. Such practices could promote both greater transparency and wider replication, a relatively rare undertaking.

What Can Be Done?

Enter MDRC, one of the nation’s best evaluators. One of their researchers, Rachel Rosen, wrote an interesting and somewhat timely blog post earlier this month that describes what her organization is doing to deal with this.

Continue reading

Posted in Uncategorized

Senator Murray, Leading Democratic Appropriator, Demands Answers from HHS on Evidence

Senator Patty Murray (D-WA) sent a letter this week to the acting head of the Department of Health and Human Services demanding answers on its continued support for evidence-based policy.

Murray is a strong evidence proponent and well placed to put pressure on HHS.  She is the leading Democrat on both the Senate Health, Education, Labor and Pensions Committee and the appropriations subcommittee that oversees HHS’s annual budget.

She has used those positions to insert evidence provisions into a variety of laws and appropriations bills over the past few years. She was also House Speaker Paul Ryan’s leading Democratic cosponsor on the bipartisan legislation that created the Commission on Evidence-Based Policymaking.

In her January 16 letter to HHS, Murray cited SAMHSA’s recent decision to freeze its evidence clearinghouse, the National Registry of Evidence-based Programs and Practices (NREPP), and the CDC’s reported ban on certain words, including the term “evidence-based,” in its budget.

She also cited concerns with recent developments for evidence-based teen pregnancy prevention programs. According to her letter:

HHS staff has stated they are “hopeful that Congress will eliminate the Teen Pregnancy Prevention Program,” and HHS has recently invested new money in “sexual risk avoidance programs.” These continued actions and statements make clear the Department does not value the work of this high quality evidence-based program that aims to assess the needs of diverse communities and populations.

Murray concluded her letter by asking for a response from HHS by January 31 that answered her questions about NREPP, the CDC, and the teen pregnancy program. She also asked HHS to describe plans for any new evidence-based policies, programs, or registries.

It is unclear how, or whether, HHS will respond, but Murray’s position as the leading Democrat overseeing its annual budget gives her enormous influence.

Posted in Children and Families, Evidence

SAMHSA Issues Statement on Evidence, But Still No Word on Its Clearinghouse

Perhaps to mollify critics of its decision to stop work on its evidence clearinghouse, the Substance Abuse and Mental Health Services Administration (SAMHSA) today announced two new sources of technical assistance to help communities that are grappling with opioids and mental health problems. Combined, the assistance amounts to nearly $30 million in TA.

“Ensuring that communities across the nation have access to evidence-based services is critical to advancing the behavioral health of the nation,” said Dr. Elinore F. McCance-Katz, SAMHSA’s Assistant Secretary, in the announcement. “I am committed to ensuring that the implementation of evidence-based practices is a top priority.”

Still No Word on NREPP

Despite the assurance, however, today’s announced efforts are not a replacement for the agency’s clearinghouse, the National Registry of Evidence-based Programs and Practices (NREPP). SAMHSA is required to maintain a web-based version of the clearinghouse by law. Technically, it appears to be staying in compliance by keeping the current version online, although the web site has reportedly not been updated since last September.

Clearinghouses differ from technical assistance because they are intended to provide a rigorous third-party assessment of the often complex studies that support the purported effectiveness of various treatment programs. Such ratings, if done well, can help practitioners select an intervention that is backed by rigorous evidence, not one backed by the claims of vendors.

Clearinghouses have been criticized in the past, however, including by this comprehensive overview by Bridgespan in 2015. More recently, the What Works Clearinghouse at the U.S. Department of Education has drawn criticisms for the quality of its reviews and its slow pace.

In earlier announcements, SAMHSA’s administrator said she would assign oversight of the clearinghouse to the agency’s newly created Policy Lab, which launched earlier this month. However, the Lab is reportedly still staffing up, which suggests it may not be ready to update NREPP immediately.

Blowback on Capitol Hill

On Capitol Hill, five Senate Democrats sent McCance-Katz a letter saying they were concerned that NREPP had been discontinued before a replacement was fully implemented, according to a story in The Hill.

“We are concerned that freezing NREPP means individuals and communities that may benefit from these new programs will not be able to learn about them or access them, and that the freeze also may hamper the work being done around the country to develop important interventions at a time when we are in the midst of a heroin, fentanyl and opioid epidemic and we need them more than ever,” they said in the letter.

While maintaining strong support for evidence in general, McCance-Katz was sharply critical of NREPP in a statement issued last week, calling many of its program ratings questionable or “entirely irrelevant” to the disorders they were claimed to treat.

“We at SAMHSA should not be encouraging providers to use NREPP to obtain EBPs, given the flawed nature of this system,” she said in the statement.

Those sentiments were backed by a separate journal article published last summer that sharply criticized the clearinghouse for questionable reviews and failing to address possible conflicts of interest.


Posted in Health

Trump Administration’s First Work-based Medicaid Waiver May Produce Little New Evidence

A new waiver policy intended to promote work by Medicaid recipients drew substantial criticism when it was announced by the Trump administration earlier this week. But will it generate any valuable new evidence of what works?

It is too early to know for sure, but the first approved plan submitted by the state of Kentucky does not seem especially promising.

The Kentucky Plan

The Kentucky proposal, called Kentucky Health, would impose work requirements on adults aged 19-64, with exemptions for certain groups such as pregnant women and the disabled. Failure to fulfill the proposed work requirements would result in a loss of coverage. The proposal also features rising premiums, as well as a substance use disorder program open to all Medicaid beneficiaries (Kentucky has been very significantly affected by the growing opioids crisis).

CMS approved the plan on January 12, authorizing it for five years through September 2023. The demonstration project is expected to start in July, according to state officials, if it is not halted by a lawsuit before then.  If fully implemented, it is expected to reduce the number of people  enrolled in the state Medicaid program by about 100,000, according to state estimates, saving an estimated $2.4 billion over five years, most of it in federal funds.

Continue reading

Posted in Children and Families, Health

House Committees Explore Possible Welfare Changes

Despite a weekend retreat between President Trump and GOP congressional leaders that seemingly put partisan welfare changes on the backburner, several House leaders are looking at ways to push forward anyway.

According to Reuters:

House Agriculture Committee Chairman Mike Conaway, a Texas Republican, said the 2018 farm bill will including tighter work requirements for the Supplemental Nutrition Assistance Program (SNAP), better known as food stamps. He gave no details.

House tax committee Chairman Kevin Brady, also a Texas Republican, said his panel is exploring ways to get workers who are “trapped in the welfare system … off the sidelines.”

Representative Bradley Byrne, an Alabama Republican, said safety net programs are under review by a bipartisan group he co-chairs called the Opportunity Action Caucus. He said he hoped the group this month would propose ways for “able-bodied people who are presently on one or more of our welfare programs, to go out and get the skills and training they need to get a job.”

The Reuters article also cites yesterday’s decision by the Trump administration to approve work-focused waivers in the Medicaid program as evidence that Republicans are pushing forward on the issue. It also cites positive polling:

[P]olling in mid-2017 by the Kaiser Family Foundation found 70 percent of respondents favored work requirements for Medicaid for non-disabled adults. At the same time, 74 percent said they had a favorable view of Medicaid as it is now.

In Wisconsin, House Speaker Paul Ryan said he does not see his chamber taking on more ambitious entitlement changes. He emphasized the need for a “bipartisan consensus.”

The Reuters article does not cite any similar activity in the Senate, where Senate Majority Leader Mitch McConnell has been less enthusiastic than Ryan about entitlement changes.

Nevertheless, if the House does something bipartisan, it may provide the basis for inclusion of some of the provisions in a broader deal later in the year. This is a relatively common occurrence in Congress, particularly if the changes do not prompt major push-back from the narrowly-divided Senate.

Posted in Children and Families

Trump Administration Terminates SAMHSA Evidence Clearinghouse Amid Questions About Its Objectivity

A Trump administration decision to terminate the contract of one of the federal government’s top evidence clearinghouses, the National Registry of Evidence-based Programs and Practices (NREPP), has raised questions about its ongoing commitment to evidence.  But the decision may have actually reflected the opposite — a stronger commitment to evidence amid concerns about the clearinghouse’s credibility and that of the medical research industry as a whole.

The clearinghouse, operated by the Substance Abuse and Mental Health Services Administration (SAMHSA), reviews studies of evidence-based mental health and substance use programs, including those related to opioids, an administration priority.

The contractor overseeing the clearinghouse received word of its termination from SAMHSA over the Christmas holidays. However, the decision may have been made much earlier in the year, according to a story yesterday in the Washington Post:

Agency officials froze the existing website in September, and no new postings have been added, according to mental health advocacy groups. As a result, about 90 new programs that have been reviewed and rated for their scientific merits since September are not available to the public, they said.

SAMHSA has been largely silent on the matter, saying only that the duties of the clearinghouse will now be shifted to a new Policy Lab that was created by federal law and that was launched earlier this month. [Edit: SAMHSA has issued a statement. See below.]

In describing the Lab’s duties, Elinore McCance-Katz, the agency’s new administrator, told Congress last year that:

[T]he primary focus of the Policy Lab is to “periodically review programs and activities” relating to the diagnosis, prevention, treatment and recovery from mental illness and substance use disorders. The lab will also identify programs or activities “that are duplicative and are not evidence-based, effective or efficient,” she said.

The Policy Lab will also play a role in awarding grants to state and local governments, educational institutions and nonprofits to develop evidence-based interventions, she said.

Continue reading

Posted in Health

Trump Administration’s Plan B for Evidence-based Welfare Reform? Medicaid Waivers

Just days after a weekend retreat at Camp David, where President Trump and the Republican congressional leadership agreed to set aside House Speaker Paul Ryan’s ideas for welfare reform, a Plan B appears to be emerging.

The goal?  Encourage more states to adopt work requirements, a longtime Republican objective.  The principal strategy?  Medicaid waivers, a tool intended to build the evidence base of what works.

Medicaid is the largest source of welfare-related spending for the nation’s poor, dwarfing similar federal programs like TANF. Total Medicaid spending exceeds $500 billion per year. It provides health insurance to more than one in five Americans.

In a letter sent today to state Medicaid directors, CMS said that states could pursue work requirements for Medicaid recipients using Section 1115 waiver authority.  Such waivers are intended to support experiments, pilots, and demonstration projects of various kinds under Medicaid and CHIP.  They must be budget-neutral and include evaluations.

Such waivers are intended to build out the evidence base. It is unclear what evidence-based strategies the states might use to promote work by Medicaid recipients.

At least 34 states are currently using the waivers for various other purposes, including expanding coverage and modifying provider payments. They are usually focused on controlling health care costs while improving access and quality. Several states have used them to implement their Medicaid expansions under the Affordable Care Act.

The Obama administration consistently rejected state requests to use them for work requirements, according to the Washington Post. That policy has now been reversed by the Trump administration.

In its letter to state Medicaid directors, the administration said it would also support state efforts to align SNAP or TANF work-related requirements with the Medicaid  program.

The change could have an immediate impact. According to the New York Times, at least 10 states have expressed interest in using them for work requirements: Arizona, Arkansas, Indiana, Kansas, Kentucky, Maine, New Hampshire, North Carolina, Utah and Wisconsin.

CMS could approve the first waiver for Kentucky as soon as Friday.

“This is going to go to court the minute the first approval comes out,” Matt Salo, executive director of the National Association of Medicaid Directors, told the Post.

Earlier this week, President Trump had indicated during a press event at Camp David that he wanted to work on a bipartisan basis with Democrats on welfare reform. The new policy seems contrary to those sentiments, but it may also increase pressure on both sides to take up the issue this year.

Previous bipartisan welfare reform efforts had included significant evidence-based provisions, including consideration of a possible welfare-focused evidence clearinghouse.


Posted in Children and Families, Health

Is a Bipartisan Evidence-based Welfare Bill Still Possible This Year?

Early news reports from a weekend retreat at Camp David between President Trump, Sen. Mitch McConnell, and House Speaker Paul Ryan suggest that a Ryan-backed plan to revamp the nation’s welfare laws along partisan lines is now dead.

Little noticed in the reports, however, was the fact that Trump left the door open to working with Democrats on a more bipartisan plan — something that is not completely inconceivable, despite the poisonous partisanship that has marked Washington of late.

Ryan had headed to the weekend retreat hoping to win over the president and McConnell. The president was initially supportive, but McConnell opposed the plan as undoable in the Senate, where Republicans hold a narrow 51-vote majority, and Trump eventually sided with McConnell.

According to Politico:

The increasingly dire political environment for the GOP is one of the reasons senior Republicans have leaned on Ryan to scale back his entitlement reform ambitions. A majority of Ryan’s more conservative conference are eager to cut spending and would surely embrace his ideas. But doing so could put the two dozen House Republicans in Hillary Clinton-carried districts in an even more tenuous position.

Two Republican sources said Ryan has narrowed his entitlement push to welfare programs only, like food stamps and housing for the poor. He’ll likely push for work requirements for adults who do not have disabilities and frame this issue as one that helps — not hurts — the poor by breaking the “cycle of poverty” and helping the unemployed get jobs. Republicans would likely be able to target such programs through powerful budget reconciliation procedures that prevent the use of the filibuster in the Senate.

While Trump and White House officials have signaled a desire to take up the matter, it’s unclear if they will be able to convince a more cautious McConnell to get on board.

After this weekend’s meeting in Camp David, the answer was apparently no, at least not as Ryan had framed it. But Trump suggested that he was still open to a bipartisan plan during a Saturday press event.

“It’s a subject that’s very dear to our heart,” Trump said. “We’ll try and do something in a bipartisan way, otherwise we’ll be holding it for a bit later.”

“But we’ll be looking to do that very much in a bipartisan way,” he added.

Trump’s Saturday statements reflected McConnell’s views, which he expressed last month:

“I’ve been here a while, and the only time we’ve been able to do that is on a bipartisan basis,” McConnell told reporters about welfare reform before leaving Washington for the Christmas recess. “It was Ronald Reagan and Tip O’Neill, who raised the age of Social Security and that was before I got here, so it’s been a while.”

“The sensitivity of entitlements is such that you almost have to have a bipartisan agreement in order to achieve a result,” he added.

Continue reading

Posted in Children and Families

GAO Finds Federal Use of Evaluation Appears Unchanged Since 2013

A GAO report released in late September has found that that fewer than half (40 percent) of surveyed federal managers say that an evaluation had been completed within the past 5 years of any program, operation, or project they were involved in. The figure is statistically unchanged from 2013, the last time GAO surveyed federal managers on the topic.

The apparently unchanged use of evaluation comes despite years of promotion under the Obama administration, public scorecards released by outside groups like Results for America, and actions by Congress, such as the creation of the Commission on Evidence-based Policymaking. According to the report:

For several years, OMB has encouraged agencies to use program evaluations and other forms of evidence to learn what works and what does not, and how to improve results. Yet, agencies appear not to have expanded their capacity to conduct or use evaluation in decision making since 2013.

Continue reading

Posted in Evidence