GAO Finds Federal Use of Evaluation Appears Unchanged Since 2013

A GAO report released in late September has found that that fewer than half (40 percent) of surveyed federal managers say that an evaluation had been completed within the past 5 years of any program, operation, or project they were involved in. The figure is statistically unchanged from 2013, the last time GAO surveyed federal managers on the topic.

The apparently unchanged use of evaluation comes despite years of promotion under the Obama administration, public scorecards released by outside groups like Results for America, and actions by Congress, such as the creation of the Commission on Evidence-based Policymaking. According to the report:

For several years, OMB has encouraged agencies to use program evaluations and other forms of evidence to learn what works and what does not, and how to improve results. Yet, agencies appear not to have expanded their capacity to conduct or use evaluation in decision making since 2013.

Continue reading

Posted in Evidence

Evidence Commission Recommendations Generate Pushback in House Hearing

A hearing today by the House Committee on Oversight and Reform appeared to generate more questions than support for the recommendations of the Commission on Evidence-Based Policymaking.

The hearing’s four witnesses, all of whom served on the commission (including the chair and co-chair, Katharine Abraham and Ron Haskins), used their time to explain its 22 recommendations. However, the central purpose and value of those recommendations — to help build a national evidence base on federal programs and interventions to determine which work and which do not — did not seem to be clear to most of the members of the House committee.

Several Democrats used the occasion to criticize the Trump administration for policies that they felt were insufficiently evidence-based, including those on global warming and teen pregnancy prevention.  Republicans seemed skeptical that a new office — the proposed National Secure Data Service — was needed at the Commerce Department, expressing a traditional Republican small-government point of view.

The central bipartisan point, that evidence could enable federal agencies to achieve better outcomes at lower cost, seemed lost on most of the members of Congress who were present.

Fortunately, backing from the commission’s main supporters in Congress — House Speaker Paul Ryan and Sen. Patty Murray — is still likely to be strong enough to ensure that some form of “downpayment” legislation moves in the weeks ahead.

Related

Posted in Evidence

What Does the Federal Evidence Infrastructure Look Like?

What is needed at the federal level to expand the use of evidence?  Where are we?  What else do we need to do? Earlier this month, the Forum for Youth Investment released a very good report that answers these questions.

The report, Managing for Success: Strengthening the Federal Infrastructure for Evidence-based Policymaking, does an excellent job describing the existing infrastructure. It breaks this information out across five types of evidence (data, statistics, evaluation, performance metrics, and social and behavioral sciences) and six infrastructure components (White House, interagency, and agency offices plus policies, guidebooks, and assistance to state and local governments).

Compiling all of this information in one place — mostly in the appendices — was a useful endeavor all by itself. The paper also includes multiple recommendations for improvement — recommendations that take on added relevance because they appear to be rooted (at least in part) in interviews, recommendations, and feedback from federal officials.

The paper was released at the same time as the final report of the Commission on Evidence-Based Policymaking, although the Forum’s paper seems broader than the Commission’s work, which focused primarily on making data more accessible to researchers.

What were some of the Forum paper’s more notable recommendations? One suggested that evidence is too often viewed in terms of what works and what does not, with associated implications for program budgets. Instead, the paper recommends greater use of budget-neutral strategies.

According to the paper:

Perhaps more problematically, such use creates a backlash against evidence itself. There are cases in which Congress has responded to a negative evaluation of a popular program by passing legislation prohibiting any future evaluations of it. As one federal official noted, “we have a history in which negative results on an evaluation can mean that the money goes away. But the problem you are trying to solve has not gone away. The ‘fear factor’ is real for programs—if anyone fears that the information will be used to kill their program, the learning won’t happen.” …

Fortunately, there are two approaches to using evidence that are revenue neutral and therefore have the potential to be scalable.

The first is shifting funding from programs that evidence suggests do not work to programs that evidence suggests do work, for the same population and issue area …

The second approach is to use evidence to spend the exact same amount of money on the exact same program but to encourage or require changes to the program that will make it more efficient and effective.

This approach, which the paper suggests should be used by both OMB and Congress in budget development, seems to mirror what the Obama administration did with evidence. That approach was criticized by some on the right, however, for not cutting back programs that are not effective. So far, the Trump administration seems to be taking a harder line.

The other recommendations range from technical to familiar (for example, the paper echoes previous recommendations to create chief evaluation officers in federal agencies and set aside 1% of program funds for evaluations). Nevertheless, the paper serves as a good companion piece to the Evidence Commission’s work.

Posted in Evidence

Evidence Commission’s Proposed Data Service May Be Built in Stages

Additional details are beginning to emerge about next steps following the release last week of the final report of the Commission on Evidence-Based Policymaking.

A hearing by the House Committee on Oversight and Reform, originally scheduled for September 12, has been postponed, possibly until September 27. The hearing was reportedly delayed because of hurricanes and resulting flight cancellations that prevented members of Congress from returning to Washington.

Despite the delay, the House is still expected to push forward a bill to implement some of the commission’s recommendations. At this time, it appears that this “downpayment” legislation will be attached to the Open Government Data Act (HR 1770, S 760), which commands bipartisan support in both chambers.

However, it also appears that the commission’s central recommendation — the creation of a National Secure Data Service to act as a liaison to pre-approved researchers who want access to federal data — will need to wait until a second bill moves in Congress later this year or next. Until that time, the Center for Administrative Records Research & Applications at the Census Bureau may be doing some of the initial work preparing for the new service.

During a call sponsored by Results for America earlier today, Katharine Abraham, the chair of the commission, affirmed the central importance of the proposed Data Service. Abraham and Ron Haskins, the co-chair, are meeting with stakeholders to support the commission’s recommendations.

Their work is being supported by the Bipartisan Policy Center.  Results for America and others are also drumming up support on Capitol Hill.

Related

Posted in Evidence

Congress Expected to Act on Evidence Commission Recommendations

The House and Senate are reportedly readying legislation that will adopt at least some of the recommendations of the Commission on Evidence-Based Policymaking, which released its final report earlier today.

House Speaker Paul Ryan and Sen. Patty Murray, who sponsored legislation creating the commission, both indicated at today’s event that further legislation is being drafted to implement some of the commission’s recommendations. The House Committee on Oversight and Reform has scheduled a hearing next Tuesday, September 12.  Legislation could move as early as the next two weeks.

The commission’s report includes 22 recommendations (see below) to increase evaluator access to data while simultaneously protecting privacy. The commission’s recommendations were approved unanimously by all 15 commissioners. Its members were appointed by congressional and administration leaders of both political parties, with five commissioners serving as experts on privacy.

The commission’s efforts on privacy drew particular praise at today’s event. Marc Rotenberg, president of the Electronic Privacy Information Center, complimented the commissioners for the report’s “depth on privacy issues.”

The centerpiece of the commission’s recommendations is a proposal to create a new office that would act as a liaison to pre-approved researchers, helping them gain access to high-quality data sets. The office, called the National Secure Data Service, would be housed at the Commerce Department alongside the Census Bureau.

At least two bills appear likely to flow from the recommendations. The first, expected to move as early as this month, would include at least some of the commission’s least controversial items. More controversial proposals, which could involve changes to data and privacy laws, would be addressed in other legislation later this year or next.

At this point, it is unclear which recommendations will be included in the first bill. The proposed National Secure Data Service seems to be a high-priority item, since it would allow the service to staff up while Congress considers additional measures. However, it is unclear if the office will make it into the bill that is expected to move this month.

Other items are likely to be fleshed out in coming months. At today’s announcement, the commission co-chairs also announced that, although the commission itself has expired with the completion of its report, the Bipartisan Policy Center will be providing staffing support for further work.

Continue reading

Posted in Evidence

Conservatives Push Back on Liberal-leaning Academic Research

Lurking below the bipartisan harmony on evidence-based policy is an important conflict. How much of the evidence is driven by left-leaning political agendas?

The division surfaced in interviews with prominent conservatives last year:

“A lot of the people doing these studies are perceived to be—and in fact are—left of center in their values and opinions,” said Robert Doar, the Morgridge Fellow in Poverty Studies at the American Enterprise Institute. “Conservatives are justified in being a little cautious about this because we want to make sure the researchers don’t have their thumb on the scale,” he said.

The issue has surfaced again more recently in an article for the right-leaning City Journal. In it, two conservative academics take issue with the consistent liberalism of their academic colleagues in the criminal justice field.

Evidence of the liberal tilt in criminology is widespread. Surveys show a 30:1 ratio of liberals to conservatives within the field, a spread comparable with that in other social sciences … Led by the work of Jonathan Haidt, a growing number of scholars now acknowledge that a lack of ideological diversity in the social sciences skews research in favor of leftist claims, which become the guiding principles of many fields, challenged only at the risk of harming one’s career. Liberal assumptions go unchecked and tendentious claims of evidence become fact, while countervailing evidence doesn’t get published or faces much more rigorous scrutiny than the assertions that it challenges …

Unfortunately, criminology has had a long history of suppressing evidence for expressly political reasons. For most of its history, the discipline has overtly censored research, for instance, on biological, genetic, and neurological factors that scientists have shown to be associated with antisocial traits and behavioral problems. Even today, despite lots of hard scientific evidence—such as that 50 percent of the variance in antisocial behavior is attributable to genetic factors, or neuroimaging studies that show systemic structural and functional brain differences between offenders and non-offenders—those who pursue this line of research get branded as racists or even eugenicists. We have personally experienced hostile receptions when presenting our work in these areas at professional conferences and have been excoriated in the anonymous-review process when attempting to publish our papers.

How serious and widespread is this concern? If Doar and others are to be believed, it appears substantial. And the ideological differences are not limited to academic circles.

Earlier this year, Pew published survey results showing that Republicans have become sharply more skeptical of the positive impact of colleges in the past two years. According to Pew, “58% of Republicans and Republican-leaning independents say colleges and universities have a negative effect on the way things are going in the country, while just 36% say their effect is positive.”

FT_17.07.20_CollegesSince2015

Related

Posted in Criminal Justice, Evidence

Medicare Value-based Care Program Penalizes Physicians Serving High-risk Patients

A new paper in the Journal of the American Medical Association (JAMA) has indicated that value-based payments to physicians in the Medicare program penalize those who serve socially or medically high-risk patients.

The study investigated first year results for the Physician Value-Based Payment Modifier (PVBM) Program. It found that practices that served high-risk patients had lower quality in general. Those serving socially high-risk patients had lower costs, while those serving medically high-risk had higher costs.

In each case, these practices received fewer bonuses and faced greater financial penalties.

“As value-based payment programs continue to increase in size and scope, practices that disproportionately serve high-risk patients may be at particular risk of receiving financial penalties,” wrote the study’s authors.

The study underlines a central difficulty facing value-based care and similar outcomes-driven payment models. Outcomes are driven not just by the quality of care, but also the characteristics of the population served. Without sufficient focus on value-added, programs working with disadvantaged populations may be penalized.

Related

Posted in Health

Using Active Contract Management to Improve Program Outcomes

Jeffrey Liebman, a former Obama administration official who is now at the Harvard Kennedy School of Government, has authored a paper that brings together some of the disparate strings of evidence-based policy and performance management.

The paper, Using Data to Make More Rapid Progress in Addressing Difficult U.S. Social Problems, includes a fairly explicit criticism of evidence-based policy as it is commonly  understood. But it also proposes a practical solution to some of its limitations that is based on his work at Harvard’s Government Performance Lab. Continue reading

Posted in Performance Management

Study Suggests Central Role of Leadership in Evidence-based Change

A study of research use in schools published in the American Educational Research Journal suggests that involved leadership is centrally important to the success of evidence-driven improvement programs.

The study followed 23 school administrators across six school districts that were implementing research-backed school improvement programs. It found that more than half showed no improvement over the 18 month study period. Districts that worked with  technical assistance organizations were more likely to show growth than the others, but alone it was not sufficient to fundamentally alter district practices.

By contrast, districts were more likely to improve when leadership was directly involved. According to a review in Education Week:

In districts that successfully improved based on research, superintendents and central office staff reflected on their own practices rather than focusing only on school staff. “It’s not just any training,” Honig said. “Some bosses do professional development, but it’s not from a teaching-and-learning stance. We see again and again how powerful it is when a superintendent says, ‘Hey, this is hard, but I’m learning it with you.’ It’s such a strong signal that it sends, that the district is focused on continuous learning.”

Districts were also more likely to improve when central office staff modeled research-backed practices and connected them to specific goals. “One thing that helps people cross over is really personal feedback,” Honig said. “They all get the ideas, but it wasn’t until [research partners] observed [principals] leading and said, ‘Here is what you are doing, but here is what the research says,’ that we saw change. Districts often don’t make that kind of investment in their leaders; they are only just starting to make that kind of investment in their teachers.”

Posted in Education