Evidence-based Child Welfare Legislation Enacted

Evidence-based child welfare prevention programs will receive funding under a new law that was included in a large funding bill that Congress passed in the early morning hours of February 9.

The legislation, called the Family First Prevention Services Act, will fund evidence-based preventive mental health, substance abuse, and in-home parenting programs for children at risk of entering the child welfare system. Starting October 1, 2019, the legislation will allow Title IV-E entitlement money to be used for these services for up to 12 months. (See the relevant provisions of the bill or the full bill text for details.)

The Family First legislation was nearly enacted in late 2016, but pulled at the last moment after objections by Sen. Richard Burr. Several states had also objected to the bill’s funding of congregate care.

A report released by RAND last year suggested that prevention programs like those funded by the bill could achieve better child outcomes and save money.

What Counts As Evidence-Based?

To qualify for federal funding, the services must be trauma-informed and meet the legislation’s standards for programs that are promising, supported, or well-supported by evidence. At least half of the federal share of this funding must be spent on services that meet the highest (well-supported) evidence standard.

The Department of Health and Human Services is directed to issue guidance on the provisions later this year — by October 1, 2018 — a year before they become effective. The guidance must also include a pre-approved list of services that satisfy the bill’s evidence requirements. After that, HHS is directed to update the list as often as it deems necessary. The bill also authorizes the creation of a clearinghouse to rate the research on these services, either by the department or through a grant or contract.

Many of these programs have already been rated by clearinghouses such as the California Evidence-Based Clearinghouse and SAMHSA’s National Registry of Effective Prevention Programs (NREPP), although the latter clearinghouse has been frozen since last September. The new law’s evidence standards are reportedly very similar to those used by the California clearinghouse.

Several other resources are also available that could help child-serving agencies implement such evidence-based programs, including:


Posted in Children and Families

Home Visiting Extension Authorizes Pay-for-Outcomes Transactions

Legislation that would extend the federal home visiting program for another five years, which was included in a final budget bill that was passed by Congress earlier today, also includes provisions that would authorize the use of pay-for-outcomes transactions in the program.

The extension of the Maternal, Infant, & Early Childhood Home Visiting (MIECHV) program was included in the final version of the bill. The pay-for-outcomes provisions are in Section 50605 of the MIECHV section of the law.

The provisions are optional, but if they are used they must meet the following requirements:

(4) Pay for outcomes initiative.–The term `pay for outcomes initiative’ means a performance-based grant, contract, cooperative agreement, or other agreement awarded by a public entity in which a commitment is made to pay for improved outcomes achieved as a result of the intervention that result in social benefit and direct cost savings or cost avoidance to the public sector. Such an initiative shall include–

(A) a feasibility study that describes how the proposed intervention is based on evidence of effectiveness;

(B) a rigorous, third-party evaluation that uses experimental or quasi-experimental design or other research methodologies that allow for the strongest possible causal inferences to determine whether the initiative has met its proposed outcomes as a result of the intervention;

(C) an annual, publicly available report on the progress of the initiative; and

(D) a requirement that payments are made to the recipient of a grant, contract, or cooperative agreement only when agreed upon outcomes are achieved, except that this requirement shall not apply with respect to payments to a third party conducting the evaluation described in subparagraph (B).

The provisions would reauthorize MIECHV through 2022. The program technically expired late last year, which has reportedly caused some states to freeze new enrollments. The program is one of a handful of federal initiatives that are evidence-based.

Posted in Children and Families, Home Visiting

Does Congress Need an Evidence Office?

Does Congress need an office to help it sort through the ever-growing body of research and analysis so that it can make its decisions more evidence-based?  That was one of the intriguing ideas quietly floated by the Bipartisan Policy Center in a report it released last year.

Could such an office be created? Maybe. (The relevant language from the BPC report is pasted at the bottom of this story). But even if not, the ideas may show up on Capitol Hill in other ways.

Such an office could certainly be useful. As rigorous evidence steadily grows across the range of policy issues of interest to Congress, it is impossible for any one person to keep up, much less have the expertise needed to figure out which studies are sound and which are not. An independent office that helped Congress keep tabs on the research could be a valuable resource.

Continue reading

Posted in Evidence, Politics

Evidence Commission’s Bill Still Held Up In the Senate

A House hearing held yesterday on data issues — including recommendations of the Commission on Evidence-Based Policymaking — highlighted tensions between data and privacy concerns that may be holding up legislation (HR 4174 and S 2046) that would implement some of the commission’s recommendations.

The legislation has been framed as a “downpayment” on the recommendations, which were released in September. The House passed its version of the bill relatively soon afterward, on November 15, probably owing to the strength of its principal House sponsor, Speaker Paul Ryan. The Senate has not yet moved its version of the bill.

A good summary of the legislation is available from the Bipartisan Policy Center, which has been taking an active supporting role. The bill omits what was probably the commission’s central recommendation, however — the creation of a National Secure Data Service that would act as a liaison for researchers seeking access to federally-held administrative data sets.

An Education Week story on the bill suggested that it may be getting held up for the same reasons that the Family Educational Rights and Privacy Act (FERPA), the primary federal education data privacy law, has not been reauthorized. Those disagreements may also be holding up separate legislation that would reauthorize IES, the research agency at the Department of Education that houses the What Works Clearinghouse. So the evidence commission’s bill is not alone.

The commission stressed the importance of privacy concerns in its recommendations last fall. They drew support from some data privacy advocates, including the Electronic Privacy Information Center at a September 7 roll out event. The organization also issued a statement on January 29, in advance of the House hearing.

Will the Senate pass the evidence commission’s bill? It is difficult to know. The upper chamber tends to move bills like this more quietly, often by consensus and voice vote. It is hard to know in advance when or if that might happen. (And this assumes a hold has not been — or will not be — put on it by an undisclosed senator, which happens from time to time).

A video of yesterday’s hearing, held by the House Committee on Education and the Workforce, is below. The witness list and written testimony are here.



Posted in Evidence

Policy Drivers: Putting the Politics Back in Implementation

Implementation of evidence-based programs and practices is a topic that has received increased attention recently — and rightly so as more laws, grants, and other policies are beginning to be adopted with evidence requirements.

What does it take to implement (i.e., successfully replicate or scale) an evidence-based policy or program? As more organizations have begun to explore the topic, we are beginning to find out.

Thus far, however, a related topic seems to be largely missing from the conversation — politics. Politics is centrally important to implementation for at least two reasons. First, it creates the legal authority, funding, and other capacities that are needed to implement a program effectively. Second, politics does not stop when a law has been enacted. It continues throughout the implementation process.

Continue reading

Posted in Implementation, Politics

Canary in a Coal Mine? SAMHSA’s Clearinghouse Signals Larger Threat to Evidence-based Policy

It started as a simple story. Once again the Trump administration had demonstrated its reputed disdain for facts and evidence. This time it had revoked the contract of one of the federal government’s top evidence clearinghouses — one that reviewed studies of mental health and drug treatment programs to determine their effectiveness.

The decision, which came quietly during the Christmas holidays, seemed to further prove that the administration cared nothing about facts, nothing about evidence, and little about evidence-based approaches to opioids, which it had elevated to a White House-level priority.

And then the story began to fall apart. An independent review of the clearinghouse had revealed substantial problems with its ratings, including significant potential conflicts of interest. The newly-appointed SAMHSA director, who terminated its contract, echoed those criticisms in a strident public statement.

But underneath the charges and counter-charges, there was a quieter, lurking story — one of widespread problems and alleged corruption in medical research. It is an important story, one that could be a harbinger of growing threats to the evidence-based movement as a whole.

Continue reading

Posted in Evidence, Health

Addressing the Research “Replication Crisis”: Evidence-based Policy’s Hidden Vulnerability

One of the central tenets of evidence-based policy is that programs and policies that are deemed “evidence-based” should be replicable in other settings.

This is not always easy — even interventions backed by solid evidence can be hard to replicate. There are lots of potential reasons for this, including differences in the target populations, different settings, poor implementation, and insufficient fidelity to the original model, to name a few.

But the challenge becomes even more difficult if the original research is suspect. As it turns out, this is a potential problem for all kinds of research, much of which is in the throes of a full-blown replication crisis. Why does that matter? If the research is shaky, then the entire edifice of evidence-based policy comes crashing down.

The Replication Crisis in Research

To be clear, this particular replication crisis is not about replicating an evidence-based program that is based on strong research. That is an implementation problem, which is its own challenge. No, this replication crisis is about problems with the underlying research itself.

The research replication crisis is a problem, not just for basic science, but for the social sciences, too. For example, one recent attempt to repeat the findings of 100 peer-reviewed psychology studies found that only 39 could be reproduced. Empirical economics has faced its own similar replication crisis. Stanford professor John Ioannidis, a long-time critic of research more generally, has argued that the primary reasons are poorly-designed studies and researcher bias.

How do we solve these problems? One way is to review the studies to determine their rigor. That is the central mission of evidence clearinghouses, several of which are run by the federal government. (Other reviewers like the Campbell Collaboration are privately run.) Unfortunately, some of the clearinghouses have been experiencing challenges of late.

Their job could be made much easier, however, if the researchers themselves implemented best practices that would help show that their research is credible. Such practices could promote both greater transparency and wider replication, a relatively rare undertaking.

What Can Be Done?

Enter MDRC, one of the nation’s best evaluators. One of their researchers, Rachel Rosen, wrote an interesting and somewhat timely blog post earlier this month that describes what her organization is doing to deal with this.

Continue reading

Posted in Uncategorized

Senator Murray, Leading Democratic Appropriator, Demands Answers from HHS on Evidence

Senator Patty Murray (D-WA) sent a letter this week to the acting head of the Department of Health and Human Services demanding answers on its continued support for evidence-based policy.

Murray is a strong evidence proponent and well placed to put pressure on HHS.  She is the leading Democrat on both the Senate Health, Education, Labor and Pensions Committee and the appropriations subcommittee that oversees HHS’s annual budget.

She has used those positions to insert evidence provisions into a variety of laws and appropriations bills over the past few years. She was also House Speaker Paul Ryan’s leading Democratic cosponsor on the bipartisan legislation that created the Commission on Evidence-Based Policymaking.

In her January 16 letter to HHS, Murray cited SAMHSA’s recent decision to freeze its evidence clearinghouse, the National Registry of Evidence-based Programs and Practices (NREPP), and the CDC’s reported ban on certain words, including the term “evidence-based,” in its budget.

She also cited concerns with recent developments for evidence-based teen pregnancy prevention programs. According to her letter:

HHS staff has stated they are “hopeful that Congress will eliminate the Teen Pregnancy Prevention Program,” and HHS has recently invested new money in “sexual risk avoidance programs.” These continued actions and statements make clear the Department does not value the work of this high quality evidence-based program that aims to assess the needs of diverse communities and populations.

Murray concluded her letter by asking for a response from HHS by January 31 that answered her questions about NREPP, the CDC, and the teen pregnancy program. She also asked HHS to describe plans for any new evidence-based policies, programs, or registries.

It is unclear how, or whether, HHS will respond, but Murray’s position as the leading Democrat overseeing its annual budget gives her enormous influence.

Posted in Children and Families, Evidence

SAMHSA Issues Statement on Evidence, But Still No Word on Its Clearinghouse

Perhaps to mollify critics of its decision to stop work on its evidence clearinghouse, the Substance Abuse and Mental Health Services Administration (SAMHSA) today announced two new sources of technical assistance to help communities that are grappling with opioids and mental health problems. Combined, the assistance amounts to nearly $30 million in TA.

“Ensuring that communities across the nation have access to evidence-based services is critical to advancing the behavioral health of the nation,” said Dr. Elinore F. McCance-Katz, SAMHSA’s Assistant Secretary, in the announcement. “I am committed to ensuring that the implementation of evidence-based practices is a top priority.”

Still No Word on NREPP

Despite the assurance, however, today’s announced efforts are not a replacement for the agency’s clearinghouse, the National Registry of Evidence-based Programs and Practices (NREPP). SAMHSA is required to maintain a web-based version of the clearinghouse by law. Technically, it appears to be staying in compliance by keeping the current version online, although the web site has reportedly not been updated since last September.

Clearinghouses differ from technical assistance because they are intended to provide a rigorous third-party assessment of the often complex studies that support the purported effectiveness of various treatment programs. Such ratings, if done well, can help practitioners select an intervention that is backed by rigorous evidence, not one backed by the claims of vendors.

Clearinghouses have been criticized in the past, however, including by this comprehensive overview by Bridgespan in 2015. More recently, the What Works Clearinghouse at the U.S. Department of Education has drawn criticisms for the quality of its reviews and its slow pace.

In earlier announcements, SAMHSA’s administrator said she would assign oversight of the clearinghouse to the agency’s newly created Policy Lab, which launched earlier this month. However, the Lab is reportedly still staffing up, which suggests it may not be ready to update NREPP immediately.

Blowback on Capitol Hill

On Capitol Hill, five Senate Democrats sent McCance-Katz a letter saying they were concerned that NREPP had been discontinued before a replacement was fully implemented, according to a story in The Hill.

“We are concerned that freezing NREPP means individuals and communities that may benefit from these new programs will not be able to learn about them or access them, and that the freeze also may hamper the work being done around the country to develop important interventions at a time when we are in the midst of a heroin, fentanyl and opioid epidemic and we need them more than ever,” they said in the letter.

While maintaining strong support for evidence in general, McCance-Katz was sharply critical of NREPP in a statement issued last week, calling many of its program ratings questionable or “entirely irrelevant” to the disorders they were claimed to treat.

“We at SAMHSA should not be encouraging providers to use NREPP to obtain EBPs, given the flawed nature of this system,” she said in the statement.

Those sentiments were backed by a separate journal article published last summer that sharply criticized the clearinghouse for questionable reviews and failing to address possible conflicts of interest.


Posted in Health