Evidence-Based Policymaking: Selected Agencies Coordinate Activities, but Could Enhance Collaboration

Congress and OMB have taken steps intended to strengthen federal evidence-building activities. In September 2017, a federal commission found that agencies had uneven capacity to support, or did not fully coordinate, a full range of evidence-building activities.

GAO was asked to examine the coordination of federal evidence-building activities. This report (1) describes selected agencies' actions that align with direction from Congress and OMB to strengthen evidence-building activities and (2) examines the extent to which selected agencies' processes for coordinating those activities reflect leading practices for collaboration.

To address these objectives, GAO reviewed documents and interviewed officials about federal evidence-building activities at five selected agencies. GAO selected these agencies based on the greater number of experiences they had in comparison to other agencies incorporating these activities into the design and implementation of certain programs. GAO assessed their coordination of these activities against four leading practices for collaboration identified in GAO's past work.

Federal decision makers need evidence about whether federal programs and activities achieve intended results as they set priorities and consider how to make progress toward national objectives. The five agencies GAO reviewed took actions that align with direction from Congress and the Office of Management and Budget (OMB) to strengthen their evidence-building activities. The five agencies are: the Departments of Education, Health and Human Services (HHS), and Labor (DOL); the Corporation for National and Community Service (CNCS); and the U.S. Agency for International Development. For example, based on a statutory requirement, a majority of grant funding for HHS's Maternal, Infant, and Early Childhood Home Visiting program is to be used for home visiting models with sufficient evidence of their effectiveness. Consistent with this requirement, HHS annually assesses evidence, such as the results of program evaluations, to identify effective home visiting models that grantees can implement.

Evidence-building can involve assessing existing evidence, identifying any new evidence needs, and prioritizing when to fulfill those needs. These efforts are fragmented within each of the five agencies—that is, each has multiple organizational units with responsibilities for evidence-building. For example, DOL has established separate units responsible for different sources of evidence—evaluations, performance information, and statistics. Effective collaboration can help agencies manage this fragmentation, and lead to improved results.

; (3) clarifying roles and responsibilities ; and (4) documenting that information in written guidance . However, agencies' processes for determining which new evidence to generate, when, and how (i.e., prioritizing new evidence) did not always reflect the leading practices (see figure).

GAO is making a total of seven recommendations to DOL, CNCS, and HHS to better reflect leading collaboration practices in their evidence prioritization processes. DOL concurred, CNCS neither agreed nor disagreed, and HHS did not concur with the recommendations. CNCS and HHS stated, but did not provide information to support, that each had already taken relevant actions. GAO continues to believe the recommendations are valid, as discussed in the report.

Federal decision makers need evidence about whether federal programs  and activities achieve intended results as they set priorities and consider  how to make progress toward national objectives. The Office of  Management and Budget (OMB) defines evidence as “the available body  of facts or information indicating whether a belief or proposition is true or  valid.” OMB’s guidance further states that evidence may come from a  variety of sources, including descriptive statistics, performance  measurement, policy analysis, program evaluations, and other research.

To ensure that decision makers have the evidence they need, agencies  undertake a range of activities. Evidence-building activities involve  assessing existing evidence and identifying any need for additional  evidence; determining which new evidence to generate, when, and how  (i.e., prioritizing new evidence); generating that evidence; and using  evidence in decision-making. Congress and OMB have taken actions  intended to strengthen federal evidence-building activities. For example,  the Government Performance and Results Act of 1993 (GPRA), as  updated and expanded by the GPRA Modernization Act of 2010,  established a government-wide framework for generating and using  performance information.

In March 2016, Congress passed, and the President signed, legislation  establishing the Commission on Evidence-Based Policymaking to study  the availability and use of evidence in government. In its final report,  issued in September 2017, the commission found that within federal  agencies, multiple entities (i.e., component agencies or offices) had  responsibilities for generating different sources of evidence. However,  the commission found that federal agencies’ capacities to generate a full  range of evidence were uneven. The commission further found that where  capacity existed, it was often poorly coordinated. This included  coordination within an agency—across its different evidence-building  entities. In total, the commission made 22 recommendations aimed at  strengthening federal evidence-building activities.

Subsequently, the Foundations for Evidence-Based Policymaking Act of  2018 (Evidence Act), enacted in January 2019, created a framework  intended to take a more comprehensive and integrated approach to  federal evidence-building activities. According to OMB, the Evidence Act  addressed about half of the commission’s recommendations, advancing  data and evidence-building functions in the federal government. For  example, in line with the commission’s findings and recommendations, 24  major federal agencies are to designate an Evaluation Officer, who has  responsibilities for coordinating evidence-building activities required by  the Evidence Act with other relevant agency officials.

You asked us to examine the coordination of federal evidence-building  activities. In response to that request, this report (1) describes activities  selected agencies took that aligned with congressional and OMB direction  to strengthen evidence-building, and (2) examines the extent to which  selected agencies’ processes for assessing and prioritizing evidence  needs reflect leading practices for collaboration.

To address both objectives, we analyzed agency documents about  federal evidence-building activities and interviewed relevant staff at OMB  and officials at five selected agencies: the Departments of Education,  Health and Human Services, and Labor; the Corporation for National and  Community Service; and the U.S. Agency for International Development.  We selected these five agencies based on the greater number of  experiences they had in comparison to other agencies’ incorporation of  evidence-building activities into the design and implementation of certain  programs. These experiences included evidence-based approaches,  such as pay for success projects, performance partnerships, and tiered  evidence grants.

For the first objective, we reviewed information from the five selected  agencies and identified examples of evidence-building activities within  each agency since 2010. We then determined where these examples  illustrated actions that aligned with evidence-building statutory  requirements and directions from OMB, including guidance,  memorandums, and activities outlined in the President’s Management  Agenda.

For the second objective, we evaluated processes each selected agency  established to take a coordinated approach to assessing and prioritizing  evidence needs across the agency. We compared these processes to  leading practices for collaboration identified in our prior work. For this  report, we focused on a subset of four collaboration practices: defining a leadership model;  involving all relevant participants;  clarifying roles and responsibilities of those involved; and ensuring processes are documented and explained through written  guidance.

We selected these four collaboration practices because our past work on  evidence-building activities, such as analysis of performance information  and program evaluations, has similarly identified them as key approaches  related to evidence building. Appendix I provides additional details  about our objectives, scope, and methodology.

We conducted this performance audit from April 2018 to December 2019  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

According to OMB guidance, evidence can consist of quantitative or  qualitative information and may be derived from a variety of sources.  Those sources include foundational fact-finding (e.g., aggregate  indicators, exploratory studies, descriptive statistics, and other research),  performance measurement, policy analysis, and program evaluation.  OMB recommends that agencies build a portfolio of high-quality, credible  sources of evidence—rather than a single source—to support decision- making. Further, since different sources of evidence have varying  degrees of credibility, the use of evidence in decision-making requires an  understanding of what conclusions can—and cannot—be drawn from the  information.

Evidence-building can be viewed as a cycle of activities that can help  decision makers obtain the evidence they need to address policy  questions or identify the questions they should address. As illustrated in  figure 1, the following four activities comprise the evidence-building cycle: assessing existing evidence to determine its sufficiency and if  additional evidence is needed to further understand results and inform  decision-making; prioritizing among the identified needs which new evidence to  generate, when, and how; generating new evidence, by collecting, analyzing, and synthesizing  sources of data and research results; and using that evidence to support learning and decision-making  processes.

Our prior work highlights long-standing challenges agencies continue to  face in generating some sources of evidence—developing performance  measures for federal programs and conducting evaluations of their  programs. Our work also identified variations in the use of evidence for  decision-making by agency leaders and managers.

Fragmentation of Federal Evidence-Building Activities

Fragmentation refers to those circumstances  in which more than one federal agency (or  organization within an agency) is involved in  the same activity and opportunities exist to  improve implementation of that activity.

The Commission on Evidence-Based Policymaking found that evidence- building activities are fragmented in the federal government. For example,  it found that within agencies, many organizations have evidence-building  responsibilities, including statistical agencies and programs, evaluation  and policy research offices, performance management offices, policy  analysis offices, and program administrators. In addition, the commission  highlighted challenges the federal government faces in fully addressing  cross-cutting research and policy questions when evidence-building  activities span multiple agencies. The commission’s final report noted that  this fragmentation (see sidebar) can lead to duplication of effort or missed  opportunities for collaboration. The commission’s report stated that  when activities are fragmented within an agency or across the federal  government, they should be coordinated to improve the capacity to fully  address a specific research or policy question.

Similarly, our past work highlights the importance of coordination and  collaboration to reduce or better manage fragmentation, overlap, and  duplication. We found that uncoordinated or poorly coordinated efforts  can waste scarce funds and limit their effectiveness. Even when efforts  are coordinated, enhancements to those efforts can lead to improvements  in effectiveness. As noted earlier, our work also identified leading  practices that can help agencies enhance and sustain their  implementation of collaborative efforts.

Efforts to Improve Federal Evidence-Building

Congress and OMB have taken actions to strengthen federal evidence- building activities and improve coordination of those activities during the  last decade. Figure 2 provides a timeline of selected actions. Appendix II  provides additional detail regarding the selected actions.

Selected Agencies Have Taken Actions that Align with Congressional and OMB Direction to Strengthen Evidence- Building

Selected Agencies Implemented Evidence- Based Approaches in Response to Congressional Direction

Tiered evidence grants seek to  incorporate evidence of effectiveness  into grant making. Federal agencies  establish tiers of grant funding based on  the level of evidence grantees provide  on their approaches to deliver social,  educational, health, or other services.  Grantees generally are required to  evaluate their service models as a  condition for the receipt of grant funds. spent on home visiting models with sufficient evidence of their  effectiveness.

To support this requirement, the program incorporated activities across  each element of the evidence-building cycle. For example, through its  Home Visiting Evidence of Effectiveness review, HHS annually assesses  existing evidence about the effectiveness of new and existing home  visiting models to identify those that meet criteria for inclusion in the  program. The most recent review, in October 2018, identified 20 models  that met HHS’s criteria for an evidence-based early childhood home  visiting model. Of those, HHS determined that 18 models were eligible for  MIECHV grantees to select for implementation. In addition, based on  statutory requirements, officials prioritized the generation of new evidence  to assess the program’s results in certain areas, including child health  and development, and child maltreatment. The program generated this  evidence through program evaluations assessing both program  implementation and results. For example, an impact evaluation of four  home visiting models published in January 2019 found that these models  may reduce household aggression. Because child abuse has been  shown to be associated with negative long-term outcomes, reducing  household aggression could benefit children as they grow older.

In another example of the use of tiered evidence, the Department of  Labor’s (DOL) Workforce Innovation Fund, established in 2011, intends to  generate long-term improvements in the performance of the public  workforce system. The fund established and funded projects in three  different tiers:  1.  those that proposed new and untested approaches, with little or no  evidence of effectiveness;  2.  those with promising approaches that were tested and existing  evidence suggested could be effective; and  3.  those that adapted proven approaches, supported by ample and  robust evidence.

To further build DOL’s base of evidence on the effectiveness of evidence- based approaches, it required grantees to plan for third-party evaluations  of their programs.

During the first grant round in 2012, the Workforce Innovation Fund  awarded 26 grants, including one for approximately $1.4 million in tier one  funding to the Pasco-Hernando Workforce Board in Florida. This grant  supported making one-stop services, such as employment workshops  and workforce program orientations, more accessible to job seekers by  providing online access. In addition, the grant supported offering virtual  case management and business services through a call-in Employment  Support Center to individuals who found it difficult to access these  services in person. According to a 2016 case study of this project  conducted by DOL, users of the online one-stop accessed services nearly  twice as much during this 3-year grant period when compared to the prior  3-year period. In addition, the case study found there was a 53 percent  increase in job placements during this 3-year grant period.

Selected Agencies Have Taken Evidence-Building Actions that Align with OMB Direction for Cross- Agency Priority Goal Implementation

The selected agencies’ evidence-building activities also aligned with  implementation actions outlined by OMB for selected cross-agency  priority (CAP) goals. As required by the GPRA Modernization Act of  2010, at least every 4 years, OMB is to coordinate with other agencies to  develop and implement CAP goals. Two current CAP goals, established  in March 2018 in the President’s Management Agenda, place a particular  focus on evidence-building activities.

Leveraging data as a strategic asset. OMB and agency efforts to  implement this goal included developing a long-term, enterprise-wide  federal data strategy to better govern and leverage the federal  government’s data. Published in June 2019, this strategy established 10  principles and 40 practices intended to leverage the value of federal data  assets while protecting security, privacy, and confidentiality. Officials at  each of the five selected agencies described actions taken by their  agencies that aligned with the federal data strategy’s principles and  practices.

Federal Evidence Clearinghouses  According to the Office of Management and  Budget (OMB), evidence or “what works"  clearinghouses are repositories that  synthesize evaluation findings in ways that  make research more useful to decision  makers, researchers, and service  organizations. These repositories provide  tools for understanding what service models  are ready for replication or expansion and  disseminating results. grade. Officials told us in September 2018 that preliminary evidence  suggested the model could help close the literacy gap for the target  population. In addition, officials told us they intended to disseminate the  final results to stakeholders to help inform their decision-making about the  approach. To do so, Education officials developed a communication plan  to share this evidence via the OELA website, its Facebook account, the  National Clearinghouse for English Language Acquisition (see sidebar),  and a listserv of more than 10,000 recipients, among other means. As of  September 2019, this study had not been completed. Therefore  Education has not implemented its communication plan.

Results-oriented accountability for grants. One of the four strategies  for this CAP goal focuses on the achievement of grant program goals and  objectives. In October 2019, OMB staff told us that the strategy aims to  hold grant recipients accountable for promising performance practices  that support the achievement of those goals and objectives while  streamlining compliance requirements for those grant programs that  demonstrate results. According to the September 2019 quarterly update  for this goal, initial efforts for this strategy involved developing  performance management processes to help grant-making entities  improve their ability to monitor, and ultimately improve, the performance  of grantees. The update stated that OMB and the Chief Financial Officers  Council completed efforts in fiscal year 2019 that included soliciting  information from agencies on their current grants performance  management practices and identifying emerging and innovative  performance practices. Subsequent efforts for this goal involved hosting  monthly grants practitioner sessions (called Innovation Exchange  Sessions) to share new ideas and approaches to grants management,  which began in May 2019. The September 2019 session focused on data- driven decision-making for grants.

We identified actions that each of the selected agencies took, aligned with  the intent of this CAP goal, to better assess the performance of their grant  programs. Officials at each agency told us that they took steps to further  incorporate evidence-building requirements into their grant programs.  They told us they did this based in part on their experiences in  implementing the evidence-based approaches, such as the tiered  evidence grants described earlier in this report.

For example, officials at the Corporation for National and Community  Service (CNCS) described their incorporation of evidence-building  requirements into the agency’s AmeriCorps State and National program.  Agency officials told us that grantees have been required to evaluate their  programs since 2005. In recent years, CNCS embedded the evidence  generated by these evaluations into their grant-making activities. For  instance, its grant announcement for 2019 stated that AmeriCorps State  and National applications would be scored, in part, based on the reported  empirical evidence supporting the applicants’ proposed projects. In  addition, the announcement required applicants proposing projects in the  education focus area to choose one of 13 models that had previously  demonstrated effectiveness. According to CNCS officials, this was based  on evidence generated in previous projects supported by AmeriCorps  State and National grants or CNCS’s Social Innovation Fund.

Selected Agencies’ Component Organizations and Programs Developed Learning Agendas Aligned with OMB Guidance

Although the Evidence Act’s requirements apply to the agency-wide level,  OMB’s guidance strongly encourages lower-level organizations within  agencies to develop and implement their own learning agendas (see side  bar). We found instances where officials developed learning agendas at  lower organizational levels within several of the selected agencies prior to  the issuance of the June 2019 OMB guidance. These learning agendas  covered individual component agencies, bureaus, offices, and programs.

Learning Agendas  According to Office of Management and  Budget (OMB) guidance for implementing the  Evidence Act, a learning agenda is to define  and prioritize relevant questions and identify  strategies for building evidence to answer  them. In developing a learning agenda, an  agency should involve key leaders and  stakeholders, to help (1) meet their evidence  needs for decision-making and (2) coordinate  evidence-building activities across the  agency.

For example, from September 2016 to June 2017, the U.S. Agency for  International Development (USAID) conducted a landscape analysis of  learning agendas, in which officials identified 15 documented, office-,  bureau-, or initiative-wide learning agenda processes at different stages  of development within USAID. This included an office-wide learning  agenda developed by the Center of Excellence on Democracy, Human  Rights, and Governance (DRG). According to USAID, DRG seeks to  elevate and integrate democracy, human rights, and governance issues  within USAID’s overall development portfolio. According to DRG’s 2017  learning agenda, its development was informed by ongoing DRG  research and evaluation efforts, and consultations with a range of internal  stakeholders, including USAID staff from other bureaus and missions.  The learning agenda included a set of 11 questions across five thematic  areas, as illustrated in figure 3.

DRG outlined steps it planned to take throughout 2017 to address each  question, such as assessing existing evidence, identifying any gaps, and  conducting new research and evaluation activities to fill those gaps. For  example, DRG commissioned a study to help answer a question about  the effects of human rights awareness campaigns. The study, published  in September 2017, synthesized the results of a literature review to  identify (1) characteristics of effective campaigns, and (2) typical causes  of unintended negative consequences of human rights awareness  campaigns and ways to avoid them.

Selected Agencies Established Processes to Coordinate Fragmented Evidence-Building Activities, but Processes to Prioritize New Evidence Did Not Always Reflect Leading Practices

Selected Agencies Established Processes to Coordinate Fragmented Evidence-Building Activities

We found that evidence-building activities are fragmented within each of  the five selected agencies and occur at multiple levels and entities within  and across the agencies. As illustrated in figure 4, this fragmented  approach to evidence-building includes separate component agencies or  offices with responsibilities for building specific sources of evidence, such  as performance information, evaluations, and statistical data.

For example, at the Department of Labor (DOL), different organizations at  the department level are responsible for certain evidence-building  activities. This includes the Bureau of Labor Statistics (collecting  statistical data), Office of the Chief Evaluation Officer (conducting  program evaluations) and Performance Management Center (developing  performance information).

In addition, some evidence-building activities are dispersed throughout  agencies and occur at multiple organizational levels (see figure 5).

For example, at the Department of Health and Human Services (HHS),  evidence-building activities are generally managed at the component  agency level (referred to as divisions). The divisions manage their own  offices and programs, which include evidence-building responsibilities.  For instance, within the Administration for Children and Families (ACF),  an operating division within HHS—the Office of Planning, Research, and  Evaluation—is responsible for ACF-related evidence-building activities.  These activities include program evaluations, research syntheses,  descriptive and exploratory studies, data analyses, and performance  management activities.

Officials at the selected agencies said that evidence-building activities are  fragmented and occur at lower levels for a variety of reasons. First, this  approach helps ensure that decision makers at different levels within the organization have the evidence they need to inform decisions. Second,  officials stated that many times these evidence-building activities have  been undertaken in response to direction from Congress—for example,  through provisions in laws or related committee reports directed at a  component agency or program. Third, agency officials said they have  undertaken these activities based on OMB direction, such as  memorandums or budget guidance. This has encouraged agencies to  take actions at different organizational levels.

However, each of the selected agencies had established processes for  coordinating their evidence-building activities. For example, officials at  each agency established one or more processes intended to regularly  coordinate the assessment and prioritization of evidence needs across  the agency, as described later in this report.

Agency officials also described other efforts to coordinate evidence- building activities, but these efforts were either ad hoc (i.e., they did not  occur regularly) or not comprehensive in nature (i.e., they did not focus  broadly across different sources of evidence or did not cover the entire  agency). For example, in August 2017, the Corporation for National and  Community Service (CNCS) published the results of an assessment of  existing evidence—results from research and evaluation activities  conducted between fiscal years 2015 and 2016—in its State of the  Evidence report. However, CNCS has not conducted a similar analysis or  issued a similar report since that time. Moreover, the assessment did not  cover all of the agency’s activities. While the report included evidence  related to its programs, CNCS did not assess evidence related to other  activities, such as internal management functions including information  technology or human capital management.

We identified instances in which effective coordination helped selected  agencies better manage their fragmented evidence-building activities. For  example, the U.S. Agency for International Development (USAID)  developed an agency-wide Private Sector Engagement learning agenda,  published in May 2019. This learning agenda is intended to guide and  coordinate crosscutting efforts to develop evidence of effective  approaches for engaging the private sector to help partner countries meet  development goals and ultimately move beyond the need for foreign  assistance. This learning agenda includes establishing performance  measures to monitor progress on engagement with the private sector,  and further evaluate the results of its activities. The coordinated evidence- building approach established by this learning agenda can help USAID  better focus limited resources on building new evidence in this  crosscutting area for use across the agency, thereby reducing any  unwarranted overlap or duplication of effort.

Effectively-coordinated processes can help agencies ensure they are  comprehensively and systematically looking across their organizations to  leverage their existing evidence and focus limited resources on building  new evidence. They can also help agencies manage their fragmented  evidence-building activities to improve effectiveness and reduce the  potential for any unwarranted overlapping or duplicative efforts. Such  processes can help ensure agencies are well positioned to meet  forthcoming Evidence Act requirements related to assessing and  prioritizing evidence across the entire agency.

Selected Agencies Use Similar Approaches to Assess Evidence Needs That Reflect Leading Practices for Collaboration

Selected Agencies Established Similar Approaches to Assess Existing Evidence

Each of the five selected agencies established a similar approach for  assessing existing evidence and identifying gaps or other evidence needs  across the agency. Agency officials said that these approaches  responded to OMB guidance for agencies to conduct annual strategic  reviews. Specifically, in its guidance for implementing the GPRA  Modernization Act of 2010, OMB established an annual process in which  each agency is to review progress in achieving strategic objectives— goals that reflect the outcome or impact the agency is seeking to  achieve—established in its strategic plan. According to OMB’s  guidance, as a part of those reviews, the assessment of existing evidence  should inform agency decisions about where to focus limited available  resources to build new evidence to fulfill any identified needs.

OMB’s guidance encourages agencies to leverage existing decision- making processes, such as the budget development process, to  implement these reviews. Each of the five selected agencies conducts  strategic reviews and associated evidence assessments in similar ways,  through a variety of existing decision-making processes:

CNCS and HHS use their budget formulation processes;

Education incorporates strategic objective reviews into existing  quarterly reviews of progress in meeting goals;

DOL uses a stand-alone strategic review process; and

USAID leverages an existing review process conducted at lower  levels (i.e., its missions).

Officials at selected agencies identified instances in which they used their  agency strategic reviews to (1) assess a variety of existing sources of  evidence—a portfolio of evidence—to determine progress toward a  strategic objective, and (2) identify the need for additional evidence, as  illustrated by the following examples.

Assessing a portfolio of evidence. DOL’s guidance for its strategic  review process directs its component agencies to assess a variety of  evidence sources to determine results and risks or challenges that  may affect future outcomes. This includes performance information,  program evaluations, risk assessments, and findings from reports by  us and the department’s Office of Inspector General (OIG), among  other sources. In its fiscal year 2018 Annual Performance Report,  DOL identified different sources of evidence to demonstrate the  effectiveness of some of its programs, and challenges related to  others, for its strategic objective to create customer-focused workforce  solutions for American workers. For example, it cited statistics and  performance data to provide context and some quantitative results  related to this objective. It also shared the results from several  program evaluations, including a 2017 impact evaluation that  suggested DOL’s Adult and Dislocated Worker programs were  effective at increasing participants’ earnings and employment.

DOL’s performance report also highlighted that its OIG identified  aspects of several programs that support this objective as Top  Management and Performance Challenges for Fiscal Year 2018.  One of those challenges related to maintaining the integrity of Foreign  Labor Certification Programs. DOL’s performance report stated that  balancing the quality review of applications with employers’ needs for  timely processing has been a challenge for years. Based on the  totality of evidence, DOL identified this strategic objective as a focus  area for improvement for fiscal year 2018.

Identifying evidence needs. In its Strategic Plan for Fiscal Years  2018-22, Education established a strategic objective to increase high- quality education options and empower students and parents to  choose an option that meets their needs. To implement this strategic  objective, the strategic plan states that the department will encourage  state and local education agencies to expand school choice by  administering programs that increase education options, such as the  Charter Schools Program (CSP). One of the performance measures  Education uses to assess the program and progress on this strategic  objective is the aggregate number of charter schools that are open,  operating, and supported by CSP.

Education officials told us that they identified limitations with this measure  through the department’s strategic review process, and the need for  additional evidence. As an aggregate count, the measure did not allow  the department to accurately identify underlying changes in individual  charter schools served by the program or the results and activities of  CSP. For example, Education officials set a goal to increase the number  of CSP-supported charter schools by 150 for the 2017-2018 school  year. However, Education reported a decrease of four charter schools  for this time period. To better understand CSP’s performance, Education  officials told us they needed additional evidence to assess other aspects  of the program’s performance.

Education officials identified additional sources of evidence within the  department that they could use to understand the program’s performance.  These included statistics from Education’s National Center for Education  Statistics (NCES) on the total number of charter schools that opened and  closed over the same time period, and annual performance reports from  grantees. According to information on Performance.gov, these additional  sources of information showed that, in the 2017-2018 school year, 134  new charter schools supported by CSP opened, and 101 charter schools  expanded under a CSP grant.

These actions illustrate an instance of effective coordination of evidence- building activities to manage fragmentation and reduce the risk of  duplication. Education officials looked across the agency and leveraged  existing evidence generated by different organizational units—CSP and  NCES—to better understand program performance. Had this not  occurred, CSP might have collected data that duplicated what was  already generated by NCES.

Selected Agencies’ Processes to Assess Existing Evidence Reflect Leading Practices for Collaboration

Agencies’ assessments of the sufficiency of their existing evidence— conducted via processes for their strategic reviews—reflect the four  leading collaboration practices. Although OMB’s guidance provides  flexibility in how the reviews are conducted, it also sets specific  expectations for who should lead the process, who should participate in  the process, and the types of roles and responsibilities for these  individuals. Table 1 provides illustrative examples of the selected  agencies’ evidence assessment processes that reflect leading practices  for collaboration.

Some Agency Processes to Prioritize New Evidence to Generate Reflect Leading Practices for Collaboration

Selected Agencies Established a Range of Processes to Coordinate Evidence Prioritization

Unlike the similar processes they use for assessing existing evidence and  identifying needs, the five selected agencies use a variety of processes to  prioritize new evidence to generate. Agency officials told us that much of  this prioritization takes place at lower organizational levels. For example,  at HHS, the department’s component agencies—11 operating divisions  and 14 staff divisions—generally lead their own evidence-building  processes, through which they prioritize which evidence to generate.

Officials from HHS’s Office of the Assistant Secretary for Planning and  Evaluation told us that this decentralized model is due to the size and  complexity of the department, and that it respects the unique needs of the  divisions. According to these officials, a 2017 review by this office found  variation in the processes that the components use for this purpose. HHS  officials said that most components prioritize their evidence needs  through their budget formulation processes.

Officials at each of the selected agencies identified one or more  processes intended to coordinate the prioritization of evidence needs  across the entire organization. Table 2 describes these processes.

We identified instances in which officials used these processes to more  effectively focus limited resources to build new evidence through  coordination across the agency. For example, CNCS officials described  an instance in which agency leadership used the agency’s budget  formulation process to prioritize evidence-building activities to address  knowledge gaps about the AmeriCorps National Civilian Community  Corps (NCCC) program. According to CNCS officials, through the  agency’s evidence assessment processes, they found that the agency did  not have evidence to fully assess the impact of NCCC programs on  members and communities.

Moreover, existing evidence showed that NCCC had experienced a  decline in the number of qualified applicants and the retention of its  members since 2014. To better understand the performance and results  of this program, CNCS officials told us that agency leadership approved  funding in fiscal years 2018 and 2019 for NCCC to undertake a multi-year  impact evaluation. This evaluation, which is being conducted in  conjunction with CNCS’s Office of Research and Evaluation and an  independent contractor, is expected to examine the member retention,  leadership development, and community impact of NCCC programming.

Officials at each of the selected agencies told us that they were  considering how best to meet Evidence Act requirements to take a  systematic and coordinated approach to prioritizing evidence-building  activities, such as through learning agendas. For example, as described  in table 3, Education created a new body in March 2019—the Evidence  Leadership Group—to coordinate its evidence-building activities.

Education officials told us that in establishing this new group, they took  into consideration our leading practices for collaboration.

Evidence Prioritization Processes at Four Agencies Reflect Leading Practices for Collaboration to Varying Extents

As described in table 3, all five selected agencies identified one or more  leadership models for their evidence prioritization processes.

We found that all five of the selected agencies involved at least some  relevant participants in their evidence prioritization processes, as  summarized in table 4. Our past work related to evidence-building  activities identified a wide range of relevant participants to involve.  Within agencies, these participants include agency leadership, program  staff, and those with functional management responsibilities including  budget, human capital, and information technology. External stakeholders  include Congress, other federal agencies, state and local governments,  grant recipients, and regulated entities.

The five selected agencies include a range of relevant internal  participants, although the evidence prioritization process at CNCS does  not always include key internal stakeholders. CNCS’s budget hearings  involve discussions about prioritizing evidence, but primarily focus on  budget formulation decisions. Therefore, agency leaders and budget  officials are consistently involved in the hearings, but others, such as the  Director of the Office of Research and Evaluation, are not. Involving all  key internal stakeholders helps ensure that those involved in a  collaborative effort can commit resources, make decisions, and share  their knowledge, skills, and abilities. This can also help ensure that the  evidence that will be subsequently generated will be useful to decision  makers across the organization.

Education and USAID established expectations to seek input from  external stakeholders in their evidence prioritization processes.  Education’s charter for its recently-established Evidence Leadership  Group states that the group is to engage a wide array of external  stakeholders in its work. Similarly, for the evidence prioritization activities  that occur through USAID’s program cycle and learning agendas, related  guidance sets expectations to involve or obtain the perspectives of  external stakeholders. As USAID developed its Self-Reliance learning  agenda, it sought input from external stakeholders including officials from  other federal agencies, organizations that implement USAID programs,  and experts in international development, among others.

Three of the selected agencies, however, do not always have  mechanisms in place to involve, or consider the evidence needs of, a  range of external stakeholders in their evidence prioritization processes.  Officials at CNCS, HHS, and DOL told us that, because they consider  their prioritization processes to cover internal management purposes and  decisions, including external stakeholders is not appropriate. Officials at  these three agencies described ways in which they sought input on  evidence needs from some stakeholders, such as from interactions with  grant recipients and external researchers. However, these agencies have  not developed an approach to collect and consider input on evidence  needs from all relevant stakeholders to inform their prioritization  processes.

Our past work highlights the importance of engaging key external  stakeholders, especially Congress, to better understand and meet their  evidence needs. Engaging external stakeholders can also create a  shared understanding of competing demands facing the agency and  ensure that their efforts and resources are targeted at the highest  priorities across the agency. Moreover, through this engagement,  agencies may find that external stakeholders have, or are aware of,  existing evidence that helps the agency meet its needs or provide a fuller  picture of performance. Involving a full range of relevant stakeholders in  the process for prioritizing new evidence to generate would help each of the selected agencies ensure it is meeting the evidence needs of decision  makers within and external to the agency.

Four of the selected agencies—Education, HHS, DOL, and USAID—fully  define roles and responsibilities for those involved in their evidence  prioritization processes, while the process at CNCS partially reflects this  practice, as described in table 5.

CNCS officials said that the primary focus of the agency’s process is  budget formulation. Therefore, roles and responsibilities are generally  related to that purpose instead of the evidence prioritization activities that  also take place during that process.

Clearly defining roles and responsibilities can ensure all participants are  aware of and agree upon (1) who will have what responsibilities, (2) how  they will organize their joint and individual evidence-building efforts, and  (3) how they will make decisions.

As described in table 6, Education and USAID’s processes reflect this  practice, while those at CNCS, DOL, and HHS reflect it in part.

Officials at CNCS, HHS, and DOL gave different reasons for why their  written guidance and agreements related to evidence prioritization  processes do not fully reflect this leading practice.

CNCS’s and HHS’s written guidance primarily focuses on their budget  formulation processes, since this is where their evidence prioritization  activities take place. Thus, these guidance documents contain  information on leadership, participants, and roles and responsibilities  related to budget formulation activities, but not all relevant details  related to evidence prioritization.

Officials at DOL stated that they do not want to take a “one-size-fits- all” approach to developing learning agendas within the department.  They told us they had not developed specific written guidance for that  process to provide flexibility to component agencies to develop  processes that work best for them in developing their learning  agendas.

As we have previously found, documenting a clear and compelling  rationale to work together—and how that work will be done and by  whom—is a key factor in successful collaboration. By incorporating this  leading practice into their existing guidance, CNCS, HHS, and DOL would  have greater assurance that they are effectively collaborating to prioritize  evidence needs.

Decision makers need evidence to help them address pressing  governance challenges faced by the federal government. Agencies  undertake a range of efforts at different organizational levels to build  evidence to meet their own decision-making needs, as well as those of  others, such as Congress. However, these evidence-building activities  are fragmented within agencies. Through a more comprehensive and  coordinated framework, Evidence Act implementation provides  opportunities to improve the effectiveness of federal evidence-building  activities.

The five selected agencies have taken steps to improve the coordination  of evidence-building activities across their organizations, with Education’s  and USAID’s evidence-building activities reflecting the leading practices  for collaboration. CNCS, DOL, and HHS would have greater assurance  that they are comprehensively considering evidence needs across their  individual organizations by fully incorporating leading collaboration  practices into their agency-wide efforts to prioritize new evidence to  generate. These actions could also help ensure these agencies are better  managing fragmented evidence-building activities and more effectively  focusing their limited resources to generate evidence to meet decision  makers’ needs. In addition, improved coordination could reduce the  potential for any unwarranted overlap and duplication in their efforts, and  better position the agencies to meet the Evidence Act’s requirements and  related implementation actions outlined in OMB’s guidance.

We are making a total of seven recommendations, including three to  CNCS, two to HHS, and two to DOL. Specifically:  The Chief Executive Officer of CNCS should develop an approach to  ensure that all relevant participants are involved in the agency-wide  process for prioritizing evidence needs. (Recommendation 1)

The Chief Executive Officer of CNCS should define roles and  responsibilities for all relevant participants involved in the agency-wide  process for prioritizing evidence needs. (Recommendation 2)

The Chief Executive Officer of CNCS should revise written guidance for  the agency-wide process for prioritizing evidence needs to ensure it  identifies all relevant participants and their respective roles and  responsibilities. (Recommendation 3)

The Secretary of Health and Human Services should develop an  approach to ensure that all relevant participants are involved in the  department-wide process for prioritizing evidence needs.  (Recommendation 4)

The Secretary of Health and Human Services should revise written  guidance for the department-wide process for prioritizing evidence needs  to ensure it identifies all relevant participants and their respective roles  and responsibilities. (Recommendation 5)

The Secretary of Labor should develop an approach to ensure that all  relevant participants are involved in the department-wide process for  prioritizing evidence needs. (Recommendation 6)

The Secretary of Labor should revise written guidance for the  department-wide process for prioritizing evidence needs to ensure it  identifies all relevant participants and their respective roles and  responsibilities. (Recommendation 7)

We provided a draft of this product for comment to OMB and the five  selected agencies—CNCS, Education, HHS, DOL, and USAID. CNCS,  Education, HHS, DOL and USAID provided written comments, which are  summarized below and reproduced in appendixes V, VI, VII, VIII, and IX,  respectively. In addition, CNCS, Education, HHS, USAID, and OMB  provided technical comments, which we incorporated as appropriate.

In its written comments, CNCS neither agreed nor disagreed with the  three recommendations we directed to it. The agency stated that it  believes the planned actions included in its Strategic Evidence Plan,  published in September 2019, address those recommendations. The plan  includes a goal to strengthen how the agency prioritizes and uses  evidence, and outlines various actions intended to achieve that goal. The  plan does not include sufficient details to enable us to assess the extent  to which its implementation would fully address the issues identified in our  review and covered by our recommendations.

Education stated in its written comments that the department is  committed to maximizing the performance of its programs, and it views  building, using, and disseminating evidence as critical to those efforts.  Education also outlined planned and proposed actions that it believes  would further its evidence-building activities.

In its written comments, HHS did not concur with the two  recommendations we directed to it. In response to both  recommendations, HHS stated that the department had developed an  approach for including all relevant participants in its process for  prioritizing evidence needs. However, according to an HHS official in  November 2019, HHS had not yet finalized the approach, and therefore  was unable to provide any additional information about it. Thus we could  not assess the extent to which HHS’s stated actions would address our  recommendations.

DOL agreed with the two recommendations we directed to it, and in its  written comments described an action it plans to take to address them.  We will monitor DOL’s action, which we believe would likely address our  recommendations, if effectively implemented.

USAID, in its written comments, reiterated the agency’s commitment to a  comprehensive and integrated approach for its evidence-building  activities. In the draft of this report we sent to USAID for its review in  October 2019, we included a recommendation to USAID that it ensure  that all relevant participants are involved in agency-wide processes for  prioritizing evidence needs. USAID subsequently provided documentation  that it had not provided previously that showed the agency had taken  various steps to seek the input of a range of external stakeholders. We  determined that these actions addressed our draft recommendation.  Thus, we removed the draft recommendation from our report.

We are sending copies of this report to the appropriate congressional  committees, the Director of the Office of Management and Budget, the  Chief Executive Officer of the Corporation for National and Community  Service, the Secretary of the Department of Education, the Secretary of  the Department of Health and Human Services, the Secretary of the  Department of Labor, the Administrator of the U.S. Agency for  International Development, and other interested parties. In addition, the  report is available at no charge on the GAO website at  http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-6806 or sagerm@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who made key contributions to this report  are listed in appendix X.

Appendix I: Objectives, Scope, and Methodology

This report responds to a request that we review the coordination of  federal evidence-building activities. This report (1) describes activities  selected agencies have taken that align with congressional and Office of  Management and Budget (OMB) direction to strengthen evidence- building, and (2) examines the extent to which selected agencies’  processes for assessing and prioritizing evidence needs reflect leading  practices for collaboration.

To address both objectives, we analyzed agency documents about  federal evidence-building activities and interviewed relevant staff at OMB  and officials at five selected agencies: the Departments of Education,  Health and Human Services, and Labor; the Corporation for National and  Community Service; and the U.S. Agency for International Development.

We selected these five agencies based on their experiences incorporating  evidence-building activities into program design and implementation.  These experiences include evidence-based approaches such as pay for  success projects, performance partnerships, and tiered evidence grants.  At the time we made our selection, these five agencies had designed or  implemented evidence-based approaches to a greater extent than other  agencies we identified.

The agencies we selected vary in size—as measured by budget authority  and employees—and organizational structure (see table 7).

For the first objective, we reviewed information from the five selected  agencies and identified examples of evidence-building activities within  each agency since 2010. We then determined if these examples  illustrated actions that aligned with evidence-building statutory  requirements and directions from OMB including guidance,  memorandums, and activities outlined in the President’s Management  Agenda. To do so, we reviewed relevant laws and OMB guidance.

For the second objective, we evaluated processes each selected agency  had established to take a coordinated approach to assessing and  prioritizing evidence needs across the agency. We compared these  processes to four selected leading practices for collaboration identified in  our prior work (see table 8).

We selected these four collaboration practices because our past work on  evidence-building activities, such as analysis of performance information  and program evaluations, has similarly identified them as key approaches  related to evidence-building. Table 9 illustrates this alignment for selected  past reports.

We conducted this performance audit from April 2018 to December 2019  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

Appendix II: Selected Actions Taken by Congress and OMB to Strengthen Federal Evidence-Building Activities and Improve Coordination

The Office of Management and Budget (OMB) has issued several  memorandums and other key policy documents that encourage agencies  to take actions to strengthen their capacity to build evidence. For  example, in a July 2013 memorandum, OMB encouraged agencies to  identify proposals for building evidence in their budget requests. Such  proposals could be used to improve existing programs or inform decisions  about new programs. The OMB guidance highlighted several evidence- based approaches for agencies to consider, including pay for success,  performance partnerships, and tiered evidence grants, described further  in the text box below.

Examples of Evidence-Based Program Approaches Identified in Office of  Management and Budget (OMB) Guidance  Pay for success. Pay for success is a contracting mechanism under which final  payment is contingent upon achieving specific outcomes. The government specifies  performance outcomes in pay for success contracts and generally includes a  requirement that contractors assess program outcomes or impacts through an  independent evaluation. The evaluators may also generate and analyze performance  data to inform program management and improvement during implementation.

Performance partnerships. Performance partnerships allow federal agencies to  provide grant recipients flexibility in how they use funding across two or more programs  along with additional flexibilities. In exchange, the recipient commits to improve and  assess progress toward agreed-upon outcomes by developing and using evidence.

Tiered evidence grants. Tiered evidence grants seek to incorporate evidence of  effectiveness into grant making. Federal agencies establish tiers of grant funding based  on the level of evidence grantees provide on their approaches to deliver social,  educational, health, or other services. The grant generally requires grantees to evaluate  their service models as a condition for the receipt of grant funds.

In addition, Congress passed laws aimed at strengthening and better  coordinating evidence-building activities, which OMB reinforced through  related guidance to implement those laws.

GPRA Modernization Act (GPRAMA). GPRAMA established a  framework aimed at taking a more crosscutting and integrated approach  to improve government performance. Requirements included in that  framework, such as cross-agency priority (CAP) goals and strategic  reviews, were intended to strengthen evidence-building activities and  improve coordination.

CAP goals. At least every 4 years, OMB is to coordinate with other  agencies to develop and implement CAP goals. These goals are to  address issues in a limited number of policy areas requiring action  across multiple agencies, or management improvements that are  needed across the government. The President’s Management  Agenda, released in March 2018, established the third set of CAP  goals since GPRAMA was enacted. Implementation of each CAP  goal can involve evidence-building activities; however, two goals in  particular are to focus on them, as described further in the text box.

Cross-Agency Priority (CAP) Goals Focused on Evidence-Building  Leveraging data as a strategic asset. The President’s Management Agenda  highlights several root causes for the challenges the federal government faces. One  root cause is that agencies do not consistently apply data-driven decision-making  practices. This agenda states that agencies need to make smarter use of data and  evidence to orient decisions and accountability around service and results. The  administration established this CAP goal to improve the use of data in decision-  making to increase the federal government’s effectiveness.

Results-oriented accountability for grants. According to the June 2019 update for  this goal, the federal government uses grants to invest approximately $700 billion  each year in mission-critical needs. However, the report states that grant managers  report spending 40 percent of their time using antiquated processes to monitor  compliance instead of analyzing data to improve results. The administration  established this CAP goal to maximize the value of grant funding by applying a risk- based, data-driven framework that balances compliance requirements with  demonstrating successful results.

A strategic objective is a type of goal that  reflects the outcome or impact the agency is  seeking to achieve. The agency is to identify  the strategies—the portfolio of organizations,  regulations, tax expenditures, programs,  policies, and other activities—within and  external to the agency that contribute to each  strategic objective. As a set, the agency’s  strategic objectives are to encompass all of its  activities.

Strategic reviews. In its guidance for implementing GPRAMA, OMB  established an annual process in which each agency is to review  progress in achieving the strategic objectives established in its  strategic plans (see sidebar). To do so, OMB’s guidance directs  agencies to assess existing sources of evidence to understand the  progress made toward each strategic objective and identify where  additional evidence is needed to determine effectiveness. In addition,  OMB’s guidance states that another purpose of strategic reviews is to  strengthen collaboration. It notes that the reviews can do so by  identifying and addressing crosscutting challenges and fragmentation.

The Foreign Aid Transparency and Accountability Act of 2016  (FATAA). Among other things, FATAA requires the President to establish  guidelines for establishing measurable goals, performance metrics, and  monitoring and evaluation plans for federal foreign assistance. In  January 2018, OMB issued guidelines for federal agencies that  administer foreign assistance—which includes the Departments of Labor  and Health and Human Services, and the U.S. Agency for International  Development. Among other things, the guidelines provide direction on  strengthening evidence-building activities, such as establishing annual  monitoring and evaluation plans, and disseminating findings and lessons  learned. Agencies were directed to align their monitoring and evaluation  policies with the guidelines by January 2019.

The Foundations for Evidence-Based Policymaking Act of 2018  (Evidence Act). In June and July 2019, OMB released its initial guidance  on implementing the Evidence Act. Among other things, this guidance  provides direction to agencies on developing evidence-building plans,  also known as learning agendas (see text box below). According to  OMB, these plans will serve as the driving force for other evidence- building activities required by the Evidence Act.

Prior to the enactment of the Foundations for Evidence-Based Policymaking Act  (Evidence Act), both the Office of Management and Budget (OMB) and the  Commission on Evidence-Based Policymaking highlighted and recommended the use  of learning agendas by federal agencies to strengthen and coordinate their evidence- building activities.

According to OMB’s guidance for implementing the Evidence Act, a learning agenda is  to define and prioritize relevant questions and identify strategies for building evidence  to answer them. A federal agency developing a learning agenda should involve key  leaders and stakeholders to help (1) meet their evidence needs for decision-making,  and (2) coordinate evidence-building activities across an agency.

OMB’s guidance stated that the Evidence Act emphasizes the need for  collaboration and coordination of agency staff and activities to achieve  successful implementation. The guidance provides time frames for a  phased approach to implement several Evidence Act requirements. For  example, although learning agendas are not required to be published until  February 2022, OMB’s guidance includes several interim milestones and  deliverables to build toward the final published version.

OMB embedded portions of Evidence Act implementation guidance in its 2019 update to  Cir. No. A-11. In it, OMB noted that many of the Evidence Act’s provisions support the  Federal Performance Framework for Improving Program and Service Delivery (Part 6 of  the Circular), which provides guidance for implementing GPRAMA and other related laws  and policies. OMB Cir. No. A-11, at § 200.2 (2019).

Appendix III: Examples of Evidence-Building Approaches at Five Selected Agencies

Evidence-Building Approaches Used by Selected Agencies

We identified 20 examples of the five selected agencies’ incorporating  evidence-based approaches in their program design and  implementation. Table 10 describes each of these examples.

Selected Agencies’ Use of Evidence- Based Approaches Aligned with OMB Direction

OMB’s July 2013 memorandum stated that agencies’ use of evidence- based approaches could help strengthen agencies’ abilities to improve  program performance by using experimentation and innovation to test  new approaches for service delivery. In addition, it noted that these  approaches can be used to (1) generate new knowledge, and (2) apply  existing evidence about approaches found to be effective.

Generate new knowledge. OMB guidance notes that new knowledge  can be used to improve existing programs or inform decisions about new  ones. For example, Education designed the First in the World program to  generate evidence about effective strategies for improving college  completion rates for underrepresented, underprepared, or low-income  students. Program officials told us that, prior to the issuance of the 2014  grant solicitation for the program’s first year, Education had limited  evidence of effective approaches. As noted in the solicitation, Education  sought to expand its evidence base about effective approaches through  the first round of grant awards.

Using a tiered evidence approach, the program awarded grants to  institutions of higher education to implement and evaluate the  effectiveness of approaches, such as coaching or advisement services,  intended to increase the number of these students who complete  postsecondary education. The first round awarded grant funds to projects  in a single evidence tier to test and evaluate the effectiveness of  approaches. Education officials told us that after the program’s first year,  they conducted a literature review to identify approaches that were  supported by some evidence of their effectiveness. Using this evidence,  Education created a second tier for the 2015 grant awards, for which  grantees could receive increased funding by implementing one of the  program designs identified in the literature review.

Officials told us they intend to publish the final results of First in the World  grant recipient evaluations in Education’s What Works Clearinghouse.  Evaluation results will not be available until after the completion of the  grant periods, the first of which ended in September 2019. However,  Education officials told us that the evidence they have generated to date  has improved their understanding of services that could potentially help  at-risk students complete post-secondary education.

Apply effective approaches. To meet increased demand for services in  a constrained resource environment, OMB’s guidance encourages  agencies to allocate resources to programs and approaches backed by  strong evidence of effectiveness. In addition, OMB’s guidance  encourages agencies to “scale up” effective program approaches by  expanding them to a larger or different group of recipients. For example,  USAID created the Development Innovation Ventures program in 2010 as  a tiered evidence grant competition to create a portfolio of innovative  approaches to reducing global poverty. This program provides funding in  three tiers, with greater funding provided to those approaches with  greater evidence of effectiveness. These three tiers (which USAID  referred to as stages) were as follows:  1.  Proof of concept. The program provided smaller grants to test the  viability of an innovative approach;  2.  Testing and positioning for scale. Grantees determined, through  rigorous assessments, whether their approach could achieve greater  results and also be implemented successfully at a larger scale; and  3.  Scaling. The program funded the expanded implementation of an  effective approach within one country or replicated that approach in  another country.

For example, from 2013 to 2015, the Development Innovation Ventures  program awarded stage two funding to a nonprofit organization in India.  The organization designed a methodology to help primary school  students improve reading skills by grouping students according to skill  level, instead of age or grade and tailoring lessons to their learning level.  Evidence generated through randomized control trials showed that the  approach was effective. Based on that evidence, in 2017, the program  awarded stage three funding to replicate the approach in Zambia.

Appendix IV: Additional Examples of Selected Agencies’ Coordination of Evidence-Building Activities

Earlier in this report, we discussed agency-wide evidence assessment  and prioritization processes established by the five selected agencies. In  addition to those processes, officials described other actions they have  taken to coordinate fragmented evidence-building activities across  organizational levels (see table 11). Some of these actions were ad hoc  (i.e., they did not occur regularly) or not comprehensive in nature (i.e.,  they did not focus broadly across different sources of evidence or did not  cover the entire agency).

Appendix V: Comments from the Corporation for National and Community Service

Appendix VI: Comments from the Department of Education

Appendix VII: Comments from the Department of Health and Human Services

Appendix VIII: Comments from the Department of Labor

Appendix IX: Comments from the U.S. Agency for International Development

Appendix X: GAO Contact and Staff Acknowledgments

In addition to the above contact, Benjamin T. Licht (Assistant Director),  Daniel Webb (Analyst-in-Charge), Amanda Prichard, Kelly Turner, and  Brian Wanlass made significant contributions to this report. Valerie  Caracelli, Jacqueline Chapin, Ann Czapiewski, Steven Putansu, and  Andrew J. Stephens also made key contributions.

Agencies can bring multiple sources of information together to determine whether federal programs are working as intended — a practice known as “evidence-building.” Agencies assess existing evidence, determine whether new evidence is needed, and set priorities to get decision makers the evidence they need.

Collaboration within an agency can help ensure that evidence-building efforts are effective. While the agencies we reviewed have processes to coordinate evidence-building, we recommended that the Corporation for National and Community Service and the Departments of Labor and Health and Human Services collaborate better on setting priorities.