Evidence Commission’s Proposed Data Service May Be Built in Stages

Posted on September 13, 2017 by Patrick Lester

Additional details are beginning to emerge about next steps following the release last week of the final report of the Commission on Evidence-Based Policymaking.

A hearing by the House Committee on Oversight and Reform, originally scheduled for September 12, has been postponed, possibly until September 27. The hearing was reportedly delayed because of hurricanes and resulting flight cancellations that prevented members of Congress from returning to Washington.

Despite the delay, the House is still expected to push forward a bill to implement some of the commission’s recommendations. At this time, it appears that this “downpayment” legislation will be attached to the Open Government Data Act (HR 1770, S 760), which commands bipartisan support in both chambers.

However, it also appears that the commission’s central recommendation — the creation of a National Secure Data Service to act as a liaison to pre-approved researchers who want access to federal data — will need to wait until a second bill moves in Congress later this year or next. Until that time, the Center for Administrative Records Research & Applications at the Census Bureau may be doing some of the initial work preparing for the new service.

During a call sponsored by Results for America earlier today, Katharine Abraham, the chair of the commission, affirmed the central importance of the proposed Data Service. Abraham and Ron Haskins, the co-chair, are meeting with stakeholders to support the commission’s recommendations.

Their work is being supported by the Bipartisan Policy Center.  Results for America and others are also drumming up support on Capitol Hill.

Related

Posted in Evidence

Congress Expected to Act on Evidence Commission Recommendations

Posted on September 7, 2017 by Patrick Lester

The House and Senate are reportedly readying legislation that will adopt at least some of the recommendations of the Commission on Evidence-Based Policymaking, which released its final report earlier today.

House Speaker Paul Ryan and Sen. Patty Murray, who sponsored legislation creating the commission, both indicated at today’s event that further legislation is being drafted to implement some of the commission’s recommendations. The House Committee on Oversight and Reform has scheduled a hearing next Tuesday, September 12.  Legislation could move as early as the next two weeks.

The commission’s report includes 22 recommendations (see below) to increase evaluator access to data while simultaneously protecting privacy. The commission’s recommendations were approved unanimously by all 15 commissioners. Its members were appointed by congressional and administration leaders of both political parties, with five commissioners serving as experts on privacy.

The commission’s efforts on privacy drew particular praise at today’s event. Marc Rotenberg, president of the Electronic Privacy Information Center, complimented the commissioners for the report’s “depth on privacy issues.”

The centerpiece of the commission’s recommendations is a proposal to create a new office that would act as a liaison to pre-approved researchers, helping them gain access to high-quality data sets. The office, called the National Secure Data Service, would be housed at the Commerce Department alongside the Census Bureau.

At least two bills appear likely to flow from the recommendations. The first, expected to move as early as this month, would include at least some of the commission’s least controversial items. More controversial proposals, which could involve changes to data and privacy laws, would be addressed in other legislation later this year or next.

At this point, it is unclear which recommendations will be included in the first bill. The proposed National Secure Data Service seems to be a high-priority item, since it would allow the service to staff up while Congress considers additional measures. However, it is unclear if the office will make it into the bill that is expected to move this month.

Other items are likely to be fleshed out in coming months. At today’s announcement, the commission co-chairs also announced that, although the commission itself has expired with the completion of its report, the Bipartisan Policy Center will be providing staffing support for further work.

Continue reading →

Posted in Evidence

Conservatives Push Back on Liberal-leaning Academic Research

Posted on September 5, 2017 by Patrick Lester

Lurking below the bipartisan harmony on evidence-based policy is an important conflict. How much of the evidence is driven by left-leaning political agendas?

The division surfaced in interviews with prominent conservatives last year:

“A lot of the people doing these studies are perceived to be—and in fact are—left of center in their values and opinions,” said Robert Doar, the Morgridge Fellow in Poverty Studies at the American Enterprise Institute. “Conservatives are justified in being a little cautious about this because we want to make sure the researchers don’t have their thumb on the scale,” he said.

The issue has surfaced again more recently in an article for the right-leaning City Journal. In it, two conservative academics take issue with the consistent liberalism of their academic colleagues in the criminal justice field.

Evidence of the liberal tilt in criminology is widespread. Surveys show a 30:1 ratio of liberals to conservatives within the field, a spread comparable with that in other social sciences … Led by the work of Jonathan Haidt, a growing number of scholars now acknowledge that a lack of ideological diversity in the social sciences skews research in favor of leftist claims, which become the guiding principles of many fields, challenged only at the risk of harming one’s career. Liberal assumptions go unchecked and tendentious claims of evidence become fact, while countervailing evidence doesn’t get published or faces much more rigorous scrutiny than the assertions that it challenges …

Unfortunately, criminology has had a long history of suppressing evidence for expressly political reasons. For most of its history, the discipline has overtly censored research, for instance, on biological, genetic, and neurological factors that scientists have shown to be associated with antisocial traits and behavioral problems. Even today, despite lots of hard scientific evidence—such as that 50 percent of the variance in antisocial behavior is attributable to genetic factors, or neuroimaging studies that show systemic structural and functional brain differences between offenders and non-offenders—those who pursue this line of research get branded as racists or even eugenicists. We have personally experienced hostile receptions when presenting our work in these areas at professional conferences and have been excoriated in the anonymous-review process when attempting to publish our papers.

How serious and widespread is this concern? If Doar and others are to be believed, it appears substantial. And the ideological differences are not limited to academic circles.

Earlier this year, Pew published survey results showing that Republicans have become sharply more skeptical of the positive impact of colleges in the past two years. According to Pew, “58% of Republicans and Republican-leaning independents say colleges and universities have a negative effect on the way things are going in the country, while just 36% say their effect is positive.”

FT_17.07.20_CollegesSince2015

Related

Posted in Criminal Justice, Evidence

Medicare Value-based Care Program Penalizes Physicians Serving High-risk Patients

Posted on August 5, 2017 by Patrick Lester

A new paper in the Journal of the American Medical Association (JAMA) has indicated that value-based payments to physicians in the Medicare program penalize those who serve socially or medically high-risk patients.

The study investigated first year results for the Physician Value-Based Payment Modifier (PVBM) Program. It found that practices that served high-risk patients had lower quality in general. Those serving socially high-risk patients had lower costs, while those serving medically high-risk had higher costs.

In each case, these practices received fewer bonuses and faced greater financial penalties.

“As value-based payment programs continue to increase in size and scope, practices that disproportionately serve high-risk patients may be at particular risk of receiving financial penalties,” wrote the study’s authors.

The study underlines a central difficulty facing value-based care and similar outcomes-driven payment models. Outcomes are driven not just by the quality of care, but also the characteristics of the population served. Without sufficient focus on value-added, programs working with disadvantaged populations may be penalized.

Related

Posted in Health

Using Active Contract Management to Improve Program Outcomes

Posted on July 25, 2017 by Patrick Lester

Jeffrey Liebman, a former Obama administration official who is now at the Harvard Kennedy School of Government, has authored a paper that brings together some of the disparate strings of evidence-based policy and performance management.

The paper, Using Data to Make More Rapid Progress in Addressing Difficult U.S. Social Problems, includes a fairly explicit criticism of evidence-based policy as it is commonly  understood. But it also proposes a practical solution to some of its limitations that is based on his work at Harvard’s Government Performance Lab. Continue reading →

Posted in Performance Management

Study Suggests Central Role of Leadership in Evidence-based Change

Posted on July 25, 2017 by Patrick Lester

A study of research use in schools published in the American Educational Research Journal suggests that involved leadership is centrally important to the success of evidence-driven improvement programs.

The study followed 23 school administrators across six school districts that were implementing research-backed school improvement programs. It found that more than half showed no improvement over the 18 month study period. Districts that worked with  technical assistance organizations were more likely to show growth than the others, but alone it was not sufficient to fundamentally alter district practices.

By contrast, districts were more likely to improve when leadership was directly involved. According to a review in Education Week:

In districts that successfully improved based on research, superintendents and central office staff reflected on their own practices rather than focusing only on school staff. “It’s not just any training,” Honig said. “Some bosses do professional development, but it’s not from a teaching-and-learning stance. We see again and again how powerful it is when a superintendent says, ‘Hey, this is hard, but I’m learning it with you.’ It’s such a strong signal that it sends, that the district is focused on continuous learning.”

Districts were also more likely to improve when central office staff modeled research-backed practices and connected them to specific goals. “One thing that helps people cross over is really personal feedback,” Honig said. “They all get the ideas, but it wasn’t until [research partners] observed [principals] leading and said, ‘Here is what you are doing, but here is what the research says,’ that we saw change. Districts often don’t make that kind of investment in their leaders; they are only just starting to make that kind of investment in their teachers.”

Posted in Education

Wrong: Why Experts Keep Failing Us

Posted on July 15, 2017 by Patrick Lester

Evidence-based policy rests, in part, on the assumption that the effectiveness of certain programs can be determined through rigorous research. But what if the research — not just on social programs, but science in general — is usually wrong?

That is the provocative question asked by David Freedman in his book: Wrong: Why Experts Keep Failing Us — And How to Know When Not to Trust Them.

Freedman’s book, published in 2010, predates the more recent debates over Fake News that helped contribute to the recent decline in trust in the media and most of our societal institutions (as measured by Gallup). Freedman’s motivations were not ideological, however, but to help improve our understanding of the flaws inherent in expert opinion so that we can come closer to the truth.

Freedman’s story begins with John Ioannidis, a doctor and researcher at Johns Hopkins University and the National Institutes of Health who spent time in the 1990s reading medical journals to see how patients fared with certain treatments. According to Freedman:

In examining hundreds of these studies, Ioannidis did indeed spot a pattern — a disturbing one. When a study was published, often it was only a matter of months, and at most a few years, before other studies came out to either fully refute the findings or declare that the results were “exaggerated” in the sense that later papers revealed significantly lesser benefits to the treatment under study. Results that held up were outweighed two-to-one by results labeled “never mind.”

… They exhibited the sort of wrongness rate you would associate more with fad-diet tips, celebrity gossip, or political punditry than with state-of-the-art medical research.

When Freedman asked him about these findings, Ioannidis responded:

“The facts suggest that for many, if not the majority, of fields, the majority of published studies are likely to be wrong,” he says. Probably, he adds, “the vast majority.”

What are the sources of this wrongness and what can we do about it?  Freedman spends the entire book attempting to answer those questions.

Continue reading →

Posted in Evidence

Trump Budget Uses Research Evidence to Justify Spending Cuts

Posted on May 23, 2017 by Patrick Lester

President Trump’s full budget proposal for the coming federal fiscal year (FY 2018), which was released earlier today, has shown that evidence can be used not just to justify increased funding for programs that work, but also to cut funding for those that the administration says do not.

During the Obama administration, evidence was typically used for continuous improvement purposes or to shift funding from low performing programs to others that seemed more promising. At the time, conservative analysts criticized the administration for failing to use evidence to reduce the size of government.

For better or for worse, this can’t be said of the new Trump budget, which recommends substantial cuts in Medicaid, SNAP, and several other federal safety net programs. Many of the proposed cuts, which are included in a list of major savings and reforms, are justified by citing what the administration says is insufficient evidence of impact.

Continue reading →

Posted in Evidence

Congress Set to Defund Social Innovation Fund

Posted on May 1, 2017 by Patrick Lester

An omnibus appropriations package that Congress is slated to vote on later this week contains no funding for the Social Innovation Fund.  Barring any last minute changes, this may be the end of the road for this signature evidence-based program.

According to a Politico story about the overall package, it appears likely to be enacted as-is:

The bill clocks in at more than 1,600 pages and Congress must pass it before Friday evening to avert a shutdown, though the bill is likely to pass easily because it contains key boosts to defense and domestic programs viewed by leaders in both parties as vastly preferable to another stopgap measure.

SIF has achieved a number of successes since it was created in 2009. It has also served as a major platform for federal support for pay for success. In the end, however, it was probably too closely associated with President Obama to survive in a Washington controlled by Republicans.

Barring any unforeseen circumstances, the program would likely be forced into wind down mode.  The fate of existing SIF-funded projects, all of which had at least some funding from other sources, is less clear.

Other evidence-based initiatives in the appropriations bill fared better:


Trump Administration on Evidence

Despite the disappointing news on SIF, the Trump administration has taken some other positive steps on evidence-based policy. In an April 12 memo from OMB, the White House reaffirmed its commitment to the use of evidence in the budget. Key language from the memo is below:

Build and use a portfolio of evidence to improve effectiveness.  Agencies should propose strategies to use limited resources as smartly as possible by asking: what works, for whom, and under what conditions; whether programs are being implemented effectively; and how programs can be improved to produce better results. Evidence may include results from program monitoring and evaluations, performance measures, statistics, and other forms of research and analysis.

The administration’s commitment to evidence may be made more clear in its full budget request for the next fiscal year, which may be coming soon.


Update (5/5/17):  The omnibus appropriations bill has been passed and signed into law. The Senate passed it on May 4, sending it on to the president, who signed it into law on May 5.

Posted in Evidence, Social Innovation Fund