Foreign Assistance: Federal Monitoring and Evaluation Guidelines Incorporate Most but Not All Leading Practices

The Trump Administration requested $28.5 billion in foreign assistance in fiscal year 2019, to be administered by at least 22 federal agencies. Almost 95 percent of this assistance is administered by six agencies—the Departments of Agriculture (USDA), Defense (DOD), State (State), Health and Human Services (HHS), the Millennium Challenge Corporation (MCC), and the U.S. Agency for International Development (USAID). FATAA required the President to set forth guidelines for M&E of U.S. foreign assistance. In January 2018, OMB issued the required guidelines for federal agencies. FATAA also contained a provision for GAO to analyze the guidelines established by OMB; and assess the implementation of the guidelines by the agencies.

In this report, GAO examined the extent to which (1) OMB's M&E Guidelines incorporate GAO leading practices, and (2) agencies incorporate the OMB Guidelines in their M&E policies and plans. GAO assessed the OMB Guidelines against GAO's 28 leading practices identified in GAO-16-861R . GAO also assessed the six agencies' foreign assistance M&E policies against the Guidelines and interviewed OMB and relevant agency officials in Washington, DC.

The Office of Management and Budget's (OMB) foreign assistance Guidelines incorporate most of GAO's leading practices for monitoring and evaluation (M&E), but gaps exist (see figure).

Monitoring : The Guidelines define monitoring as the continuous tracking of program or project data to determine whether desired results are as expected during implementation. The Guidelines do not require GAO's leading practices on risk assessments, staff qualifications, and program close-out procedures.

Evaluation : The Guidelines define evaluation as the systematic collection and analysis of program or project outcomes for making judgments and informing decisions. They do not require GAO's leading practices on developing staff skills and following up on recommendations.

OMB officials indicated the Guidelines are focused on elements required in the Foreign Aid Transparency and Accountability Act of 2016 (FATAA), but noted that agencies can add additional requirements to their own M&E policies. FATAA requires the President to set forth guidelines “according to best practices of monitoring and evaluation.” OMB staff acknowledged that GAO's leading practices are important, but stated that there is no singular established standard for best monitoring practices. Nevertheless, all of GAO's leading practices can help agencies address impediments, effectively manage foreign assistance, and meet their goals.

When assessing agencies' M&E policies against OMB Guidelines, GAO found that agencies incorporated most of the requirements. However, for monitoring, one of the six agencies GAO reviewed—DOD—did not include the requirements to establish agencies' roles and responsibilities and ensure verifiable data for monitoring activities. For evaluation, agencies required most Guideline requirements, but not all. For example, DOD, HHS, and USDA did not require conducting impact evaluations for pilot programs or projects. Without a clear requirement to do such evaluations, agencies risk duplicating or scaling up programs without fully understanding the factors that could lead to their success or failure. Agencies GAO reviewed have plans or mechanisms in place to oversee the implementation of their M&E policies. For example, State developed a guidance document to operationalize and oversee its M&E policy to ensure the implementation of the Guidelines.

GAO is making recommendations to OMB, DOD, State, and USDA. OMB did not agree with the recommendation to update the Guidelines, but GAO maintains that doing so can help to emphasize the importance of the M&E practices we identified. DOD, State, and USDA agreed with GAO's recommendations.

The Trump Administration has requested $28.5 billion in foreign  assistance in fiscal year 2019, to be administered by at least 22 federal  agencies. Foreign assistance is used to improve the lives and health of  millions living in poverty, support democracy, enhance global security,  and achieve other U.S. foreign policy goals. Managing these funds  effectively requires reliable monitoring and evaluation (M&E) systems.  Almost 95 percent of foreign assistance is administered by six agencies— the Departments of Agriculture (USDA), Defense (DOD), Health and  Human Services (HHS), and State (State); the Millennium Challenge  Corporation (MCC); and the U.S. Agency for International Development  (USAID).

Enacted in July 2016, the Foreign Aid Transparency and Accountability  Act of 2016 (FATAA) required the President to set forth guidelines for  establishing measurable goals, performance metrics, and M&E plans for  U.S. foreign assistance within 18 months. In January 2018, the Office of  Management and Budget (OMB) issued the required guidelines for  federal agencies on developing these M&E policies (Guidelines).  Agencies were required to align their M&E policies with the Guidelines by  January 2019.

FATAA also contained a provision for GAO to (1) analyze the guidelines  established by OMB; and (2) assess the implementation of the guidelines  by the agencies, bureaus, and offices that implement U.S. foreign  assistance as outlined in the President’s budget request within 18 months  of OMB issuing the guidelines.

In this report, we examine the extent to which (1) OMB’s M&E Guidelines  incorporate GAO leading practices, and (2) agencies incorporate the  OMB Guidelines in their M&E policies and plans. To examine the extent  to which OMB Guidelines incorporate GAO’s leading practices, we  assessed the Guidelines against the 28 leading practices—14 for  monitoring and 14 for evaluation—identified in GAO-16-861R. In 2016,  we developed this list of leading practices for monitoring and evaluating  foreign assistance programs. For monitoring, we identified 14 leading  practices primarily from our review of Standards for Internal Control in the  Federal Government; the GPRA Modernization Act of 2010; and  Uniform Administrative Requirements, Cost Principles, and Audit  Requirements for Federal Awards.

For evaluation, we identified 14 leading practices from our review of the  American Evaluation Association’s (AEA) An Evaluation Roadmap for a  More Effective Government (AEA Roadmap). AEA published the AEA  Roadmap to guide the development and implementation of federal  agency evaluation programs and policies. The AEA Roadmap offers a set  of general principles intended to facilitate the integration of evaluation  activities with program management. In addition to reviewing OMB  guidelines, we interviewed relevant OMB staff and agency officials in  Washington, D.C., involved in developing the OMB Guidelines.

To examine the extent to which agencies are implementing OMB’s  Guidelines, we focused on the six agencies that reported obligating the  most foreign assistance: USDA, DOD, HHS, State, MCC, and USAID. We  collected these agencies’ foreign assistance M&E policies and assessed  them against OMB’s Guidelines for M&E. For HHS, we reviewed the M&E  policy for the U.S. President’s Emergency Plan for AIDS Relief  (PEPFAR). State administers PEPFAR funds and HHS implements  many of its programs. For USDA, we reviewed the Foreign Agricultural  Service’s (FAS) M&E policy. In addition to reviewing policies, for all six  agencies, we reviewed agency-wide implementation plans and  interviewed relevant agency officials involved in developing and  implementing these policies and plans in Washington, D.C. Appendix I  contains additional information on our scope and methodology.

We conducted this performance audit from July 2018 to July 2019 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

In January 2018, OMB released (M-18-04) Monitoring and Evaluation  Guidelines for Federal Departments and Agencies that Administer United  States Foreign Assistance (the “Guidelines”) in response to the 2016  FATAA legislation. (See appendix III for additional information on the  requirements in the legislation). The Guidelines provide direction to  federal departments and agencies that administer foreign assistance on  monitoring the use of resources, evaluating the outcomes and impacts of  the foreign assistance projects and programs, and applying the findings  and conclusions of such evaluations to proposed project and program  design. The goals of the Guidelines are to set forth key principles to guide  each agency and to specify requirements, where appropriate, that  agencies must cover in their own policies on M&E of foreign assistance.

The Guidelines define monitoring and evaluation as follows:

Monitoring is the ongoing and systematic tracking of data and  information relevant to policies, strategies, programs, projects, and/or  activities and is used to determine whether desired results are  occurring as expected during program, project, or activity  implementation. Monitoring often relies on indicators, quantifiable  measures of a characteristic or condition of people, institutions,  systems, or processes that may change over time.

Evaluation is the systematic collection and analysis of information  about the characteristics and outcomes of the program, including  projects conducted under such program, as a basis for making  judgments and evaluations regarding the program; improving program  effectiveness; and informing decisions about current and future  programming.

Table 1 lists OMB’s M&E requirements and key excerpts of the  descriptions as noted in the OMB M-18-04.

In 2016, we reported on leading practices for foreign assistance program  M&E. We identified 28 leading practices—14 for monitoring and 14 for  evaluation. Table 2 lists and defines these monitoring practices.

Table 3 lists the evaluation practice and corresponding definition.

OMB’s Foreign Assistance Monitoring and Evaluation Guidelines Incorporate Most but Not All of GAO’s Leading Practices

Based on our review, the Guidelines incorporate most of GAO’s leading  practices for monitoring and evaluation. However, they do not incorporate  practices on developing monitoring plans that are based on risks,  ensuring that staff are appropriately qualified to conduct monitoring,  establish procedures to close out programs, developing staff skills for  evaluation, and following up on evaluation recommendations. OMB  indicated that it intended the Guidelines to focus on elements required by  the FATAA legislation. Nevertheless, incorporating these leading  practices in the Guidelines can help ensure that all agencies address  impediments, effectively manage foreign assistance, and meet their  assistance goals.

The Guidelines Incorporate Most of the GAO’s Leading Practices for Monitoring, but Do Not Include Risk Assessments, Staff Qualifications, or Close-Out Procedures

Based on our review, OMB incorporates 11 of 14 GAO’s leading  practices. Figure 1 shows our assessment of the Guidelines with regard  to monitoring foreign assistance.

The OMB Guidelines do not incorporate practices on developing  monitoring plans that are based on risks, ensuring that staff are  appropriately qualified to conduct monitoring, and establishing close-out  procedures for projects and programs.

Developing monitoring plans based on an assessment of risk.

The Guidelines do not incorporate GAO’s leading practice of  developing monitoring plans based on as assessment of risks related  to achieving the defined objectives. Identifying and assessing risks  can help agencies determine if impediments exist that they might  need to mitigate in order to manage their foreign assistance more  effectively. Additionally, determining which activities warrant greater  oversight and which require less can also help agencies ensure the  appropriate allocation of foreign assistance.

Ensuring Staff qualifications for monitoring. The Guidelines do not  incorporate GAO’s leading practice for agencies to ensure that staff  members responsible for monitoring programs or projects have the  relevant knowledge, skills, and training. By having qualified staff for  monitoring programs or projects, agencies can help ensure they meet  their foreign assistance goals. By hiring qualified staff and providing  them the right training, tools, structure, incentives, and  responsibilities, agencies can make operational success possible.

Establishing close out procedures for projects and programs.

The Guidelines do not incorporate GAO’s leading practice for  agencies to establish program closeout procedures for all required  work and administrative actions completed by the implementing  partner. By establishing such procedures, agencies can help ensure  their foreign assistance is less susceptible to fraud, waste, and  mismanagement; addresses increases to potential costs in fees for  maintaining foreign assistance; and increases their ability to redirect  foreign assistance to other projects.

The Guidelines Incorporate Most of the GAO’s Leading Practices for Evaluation, but Do Not Include Developing Staff Skills and Following Up on Recommendations

Based on our review, OMB incorporates 12 of 14 GAO’s leading  practices. Figure 2 shows our assessment of the Guidelines with regard  to evaluating foreign assistance.

The OMB Guidelines do not incorporate some practices such as  developing staff skills for evaluation and following up on evaluation  recommendations.

Developing staff skills regarding evaluating. The Guidelines do not  incorporate GAO’s leading practice for agencies to establish  requirements that the staff responsible for overseeing and using  evaluations should continually undertake the relevant education,  training, or supervised practice needed to learn new concepts,  techniques, and skills. By having their staff continually undertake such  education, training, or supervised practice, agencies can benefit more  fully from program evaluations.

Following up on recommendations. The Guidelines do not  incorporate GAO’s leading practice for agencies to determine whether  management or programs have accepted the recommendations made  in evaluation reports and taken the actions needed to address them.  By developing mechanisms to track recommendations, agencies can  better address inefficient, mismanaged, or costly programs or  projects.

OMB Notes the Guidelines for Monitoring and Evaluation Include Elements Required in the FATAA Legislation

The FATAA requires the President to set forth guidelines “according to  best practices of monitoring and evaluation” but does not define these  best practices. Specifically, FATAA states, “the President shall set forth  guidelines, according to best practices of monitoring and evaluation  studies and analyses, for the establishment of measurable goals,  performance metrics, and monitoring and evaluation plans that agencies  can apply with reasonable consistency to covered United States foreign  assistance.”

OMB staff told us that the Guidelines were intended to focus on elements  required by the FATAA legislation but noted that agencies are free to add  additional requirements to their own M&E policies. However, we have  previously reported that while some of these agencies have incorporated  these leading practices, others have not. Furthermore, agencies that  have incorporated these practices would not necessarily continue to  include them if they are not required in the Guidelines.

Regarding leading practices, officials noted that while these practices are  important, there is no singular established standard for best monitoring  practices. Nevertheless, both OMB’s circulars and recent legislation note  the importance of leading practices for M&E. For example, Circular A-123  notes that management should identify internal and external risks that  may prevent the organization from meeting its objectives. Additionally,  the Foundations for Evidence-Based Policymaking Act of 2018 requires  OPM, in consultation with the OMB, to identify skills and competencies  needed for program evaluation, establish a new occupational series or  update an existing one for program evaluation, and establish a new  career path for program evaluation.

Most Agencies Have Incorporated OMB’s Guideline Requirements in Their Policies, and All Have Taken Initial Steps to Implement Them

Based on our review, most agencies incorporated all of OMB’s Guidelines  for monitoring in their policies. However, DOD did not include the  requirement to establish roles and responsibilities among agencies that  participate in funding transfers or ensure that verifiable, reliable, and  timely information is collected and available to monitoring personnel. We  also found that agencies incorporate most of OMB’s Guideline  requirements for evaluation in their policies, but some did not include the  requirement to conduct impact evaluation on all pilot programs. Without  incorporating these Guideline requirements, agencies risk losing  accountability over their funding and monitoring and evaluating activities.  They also risk replicating programs without fully understanding their  effectiveness. We also found that all of the agencies we reviewed have  taken initial steps to implement their M&E policies.

Most Agencies Have Incorporated OMB’s Guideline Requirements for Monitoring

Based on our review of agency monitoring policies, all the agencies  except DOD incorporated relevant Guideline requirements.

All six agencies we reviewed incorporated the requirement to establish  monitoring policies that apply to their major foreign assistance programs.  For example, State, USAID, and MCC have agency-wide policies for  foreign assistance M&E. USDA and HHS have policies relevant to their  major foreign assistance programs—for USDA, the Foreign Agriculture  Service’s food aid programs, and for HHS, the President’s Emergency  Plan for AIDS Relief (PEPFAR). All of the agencies with relevant  monitoring policies—DOD, HHS, MCC, State, USAID, and USDA— incorporate the requirement to develop, collect, analyze, and report data  on performance indicators. These policies help ensure the measurement  of project implementation and progress, and promote the timely analysis  and reporting of results that could identify any needed corrections.

DOD did not incorporate Guideline requirements to establish  agencies’ roles and responsibilities and ensure verifiable data  for monitoring activities.

Establishing agencies’ roles and responsibilities when funds  are transferred. DOD did not include the Guideline requirement  for agencies to establish roles and responsibilities in funding  transfers. Without defined roles and responsibilities, agencies risk  losing accountability over funding and monitoring activities. In  addition, agencies could miss opportunities to collaborate and  leverage interagency efforts to facilitate decision-making and  address barriers across agency boundaries.

Ensuring verifiable, reliable, and timely data. DOD did not  include the Guideline requirement for agencies to ensure they  collect and provide verifiable, reliable, and timely data to  monitoring personnel. Without ensuring that such data are  available to monitoring personnel, agencies risk employing  inappropriate methods, continuing ineffective programs or  projects, and making uninformed decisions.

DOD officials told us these practices are currently not required because  they are still in the process of fully aligning their policy with the  Guidelines. Officials explained that working on prioritizing and directing  resources towards M&E efforts has been a challenge. Officials noted they  expect to update the policy to include these requirements in the future,  but they have no specific timelines in place.

Agencies Incorporate Most but Not All of OMB’s Guideline Requirements on Evaluation

The agencies we reviewed incorporated nearly all relevant Guideline  requirements on evaluation. Three of the six agencies—DOD, HHS and  USDA—did not include a requirement to conduct impact evaluations on  all pilot programs or projects. Figure 4 shows our assessment of  agencies’ evaluation policies against the Guidelines.

All the agencies we reviewed have established project-specific evaluation  plans. For example, HHS implements PEPFAR’s evaluation plan which  indicates specific requirements for describing the evaluation component,  strategy, or intervention, the reason for the evaluation, the type of  evaluation, the key evaluation questions, the data sources, the methods  by question, and the dissemination and utilization plan. All the agencies  we reviewed also had policies on distributing their evaluation reports  internally and publicly reporting them. For example, State and USAID  have a web-based, customized Evaluation Registry system that they  jointly maintain for bureaus and independent offices to record and track  planned, ongoing, and completed evaluations.

Conduct impact evaluations for pilot programs or projects. DOD,  HHS, and USDA did not include the Guideline requirement for  agencies to conduct impact evaluations for pilot programs or to  conduct only a performance evaluation and to provide a justification  for not conducting an impact evaluation. Without a requirement to  conduct impact evaluations of pilot programs, agencies risk  duplicating or scaling up programs without fully understanding the  factors that could lead to their success or failure.

DOD. DOD officials told us they do not require this practice  because they are still in the process of fully aligning their policy  with the Guidelines. According to DOD, it has determined that  impact evaluations are impractical and inappropriate for the  planned evaluations; instead, it plans to conduct only performance  evaluations and provide justifications for not conducting impact  evaluations, as required by the OMB Guidelines. DOD plans to  address the evaluation methodology of pilot programs in future  updates, according to officials. However, DOD has no specific  timelines in place for these updates.

HHS. PEPFAR’s M&E documents indicate that PEPFAR teams  are encouraged but not required to evaluate all current pilot  programs to see which should be taken to scale for specific  populations. Officials from HHS and the Office of the U.S. Global  Aids Coordinator noted that they conduct their own evaluation of  pilot programs and use routine program data to inform scaling of  programs. However, PEPFAR policies do not specifically require  that such evaluations be like the impact evaluations described in  the Guidelines.

USDA. FAS’s M&E documents indicate that when selecting  projects to undergo impact evaluation the agency will consider  pilot projects. USDA officials told us they have no requirement to  conduct impact evaluations on all pilot projects because impact  evaluations may be cost prohibitive and project lifecycles are short  (i.e., 3 to 5 years). Officials further noted that implementing  partners can conduct an impact evaluation on pilot programs, but  are not required to do so. Although the Guideline requirement  indicates that agencies can forgo impact evaluations, they must  provide a justification in their M&E policy. USDA officials have not  provided such a justification provided in their M&E policy.

Establish agencies’ roles and responsibilities for evaluation  activities when funds are transferred. DOD did not include the  Guideline requirement for agencies to define roles and responsibilities  when there are funding transfers between or among U.S. government  agencies to ensure accountability for evaluation activities. Without  defined roles and responsibilities, agencies risk losing accountability  over funding and evaluation activities. In addition, they could miss  opportunities to collaborate and leverage interagency efforts to  facilitate decision-making and address barriers across agency  boundaries.

Evaluate all programs at least once whose dollar value equals or  exceeds that of a median sized program within the agency. DOD  did not include the Guideline requirement for agencies to evaluate all  programs, at least once during their existence, whose dollar value  equals or exceeds that of a median sized program in the agency.  Without a mechanism to evaluate all these types of programs,  agencies risk continuing inefficient, mismanaged, or costly projects.

DOD officials told us they do not currently require these practices  because they are still in the process of fully aligning their policy with the  Guidelines. They noted that they expect to update the policy to include  these requirements, but they have no specific timelines in place.

Agencies Have Taken Initial Steps to Implement Their M&E Policies

Since the six agencies we reviewed recently updated their M&E policies  to align with the OMB Guidelines, many existing assistance projects and  programs may not be governed by these requirements. Nonetheless, the  agencies we reviewed have taken initial steps to help ensure  implementation of agency M&E policies. In interviews, agencies provided  us with the following examples of such steps.

State. State developed a guidance document and tool-kit to  operationalize and oversee its M&E policy to ensure it implements the  Guidelines. According to State officials, they provide classroom  training on the M&E policy and are piloting a revised online and  classroom evaluation courses for staff. Officials also noted that they  have dedicated staff to assist bureaus in implementing the Guidelines,  among other agency policies.

USAID. USAID has an approval process to ensure key deliverables  include Activity plans that meet Guideline requirements. Additionally,  USAID’s policy requirements indicate that each mission program  office must identify a point of contact for monitoring and evaluation to  ensure that USAID and its partners are complying with the agencies  policies and foreign assistance M&E guidelines.

MCC. MCC also has an approval process through their Department of  Policy and Evaluation to ensure implementation of the Guidelines. As  part of the process, the MCC Board of Directors or the appropriate  partner country must approve initial M&E plans.

HHS. Within HHS, the Centers for Disease Control and Prevention  (CDC) are responsible for implementing the monitoring and evaluation  guidance for their PEPFAR programs. CDC officials told us that they  have existing mechanisms and supervisory structures in place to  ensure that the Guidelines’ requirements are met in PEPFAR  programs.

USDA. USDA officials told us that the current M&E policy applies only  to food assistance programs within FAS and not for other USDA  programs. Officials explained they are trying to develop a structure  that allows FAS to ensure all USDA components are implementing the  OMB Guidelines.

DOD. DOD developed guidance for fiscal year 2020 on implementing  its M&E policy. DOD officials we spoke to noted they are working on  identifying resources, skills, and capabilities to fully implement DOD’s  M&E policy.

OMB’s Guidelines set forth key principles to guide agencies and to  specify requirements, where appropriate, which they must cover in their  own policies on M&E of foreign assistance. However, they do not include  key leading practices for M&E that GAO identified for ensuring agencies  meet their foreign assistance goals and objectives. While OMB allows  agencies discretion to include these or other best practices, it is unknown  if the agencies will do so. By ensuring that OMB’s government-wide  Guidelines include these best practices, agencies can help address  impediments, effectively manage foreign assistance, and meet their  goals. Although all agencies we reviewed developed or updated their  M&E policies to align with the Guidelines, not all of them include  important requirements. DOD, HHS, and USDA did not include the  requirement for agencies to conduct impact evaluations for pilot programs  or to conduct performance evaluations and provide a justification for not  doing an impact evaluation. Without a requirement to conduct impact  evaluations of pilot programs, agencies risk duplicating or scaling up  programs without fully understanding the causes that could lead to their  success or failure.

We are making the following seven recommendations, including one to  OMB, four to DOD, one to State, and one to USDA.

The Director of the Office of Management of Budget should update the  Guidelines to include GAO’s leading practices of developing monitoring  plans that are based on risks, ensuring that monitoring staff have  appropriate qualifications, establishing procedures to close-out programs,  developing staff skills regarding evaluations, and establishing  mechanisms for following up on evaluation recommendations.  (Recommendation 1)

The Secretary of Defense should update the Department’s monitoring  and evaluation policies to define roles and responsibilities among  agencies that participate in interagency funding transfers.  (Recommendation 2)

The Secretary of Defense should update the Department’s monitoring  and evaluation policies to ensure verifiable, reliable, and timely data are  available to monitoring personnel. (Recommendation 3)

The Secretary of Defense should update the Department’s monitoring  and evaluation policies to ensure that it evaluates all programs, at least  once in their lifetimes, whose dollar value equals or exceeds that of the  median program in the agency. (Recommendation 4)

The Secretary of Defense should update the Department’s monitoring  and evaluation policies to require the agency to conduct impact  evaluations on all pilot programs before replicating or expanding, or  conduct performance evaluations for those programs and provide a  justification for not conducting an impact evaluation. (Recommendation 5)

The Department of State’s U.S. Global AIDS Coordinator, in collaboration  with HHS and other implementing agencies, should update the PEPFAR  monitoring and evaluation policies to require these agencies to conduct  impact evaluations on all pilot programs before replicating or expanding,  or conduct performance evaluations for those programs and provide a  justification for not conducting an impact evaluation. (Recommendation 6)

The Secretary of Agriculture, in collaboration with the Foreign Agriculture  Service, should update their monitoring and evaluation policies to require  USDA to conduct impact evaluations on all pilot programs before  replicating or expanding, or conduct performance evaluations for those  programs and provide a justification for not conducting an impact  evaluation. (Recommendation 7)

We provided a draft of this product to the DOD, HHS, MCC, OMB, State,  USDA, and USAID for comment. OMB commented on the draft report in  an email from the staff responsible for economic policy, federal financial  management, and international affairs. In the email, OMB disagreed with  the recommendation to revise the Guidelines. It emphasized that an  interagency group had developed the Guidelines and had consulted a  number of expert sources on monitoring and evaluation policies and  practices, including GAO’s leading practices. OMB also developed the  guidelines to achieve the objectives contained in the Foreign Aid  Transparency and Accountability Act of 2016 within the context of other  existing OMB guidance. OMB suggested that it would be more effective to  remind agencies that, in addition to the Guidelines specified in M-18-04,  they should follow all guidance OMB had issued affecting monitoring and  evaluation activities. This guidance includes policies for closeout  procedures in the Uniform Guidance, for the Enterprise Risk Management  and Internal Control in A-123, and for the Foundations for Evidence- Based Policymaking Act on using evaluation information and monitoring  and evaluation staff skills and qualifications.

We acknowledge that relevant monitoring and evaluation guidance is  available to agencies in other forms beyond the Guidelines. However, we  believe it is important for OMB to incorporate this guidance into its  Guidelines, if only by reference, in order to emphasize the importance of  these practices in the context of monitoring and evaluation of foreign  assistance. This step would help ensure that OMB had integrated this  guidance into the management of foreign assistance programs as  appropriate.

DOD concurred with our recommendations and indicated that it would  address many of them in the next iteration of its M&E policy for security  assistance (see appendix IV for written comments). DOD noted that two  of our recommendations had limited applicability to DOD for security  assistance, but described how it would implement them. First, DOD  stated that it has not used its authority to transfer funds for security  cooperation assistance to other departments and agencies. However,  DOD indicated it would implement our recommendation to define roles  and responsibilities among agencies that participate in interagency  funding transfers, should such transfers become necessary. Second,  DOD stated that conducting impact evaluations was not a feasible in the  context of security assistance. Instead, DOD plans to conduct only  performance evaluations, but it would provide justifications for not  conducting impact evaluations, as required by the Guidelines. By  documenting these approaches in its M&E policies, DOD would help  ensure that those departments conducting M&E for DOD security  assistance initiatives implement them as required.

State agreed with the intent of the recommendation (see appendix V for  written comments). State explained that impact evaluations are often not  feasible in the context of assistance provided under PEPFAR and  described its alternative approach to evaluating new initiatives. State  indicated it would update appropriate PEPFAR policies to clarify when  agencies should conduct impact and/or performance evaluations. These  clarifications will reflect how State evaluates PEPFAR programs in  practice in accordance with OMB guidance and legislation, according to  State.

USAID provided written comments (see appendix VI). HHS and USDA  provided technical comments, which we incorporated as appropriate.  MCC did not provide comments.

We are sending copies of this report to the appropriate congressional  committees, the Director of the Office of Management and Budget;  Secretaries of Agriculture, Defense, Health and Human Services, and  State; Administrator of the U.S. Agency for International Development;  and the Executive Officer of the Millennium Challenge Corporation and  other interested parties. In addition, this report will be available at no  charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-3149 or GootnickD@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made major contributions to  this report are listed in appendix VI.

This report examines the extent to which (1) the Office of Management  and Budget’s (OMB) monitoring and evaluation (M&E) Guidelines  incorporate GAO leading practices and (2) agencies incorporate the OMB  Guidelines in their M&E policies and plans.

To address objective one, we examined the OMB Guidelines against  GAO’s 28 leading practices—14 for monitoring and 14 for evaluation— identified in GAO-16-861R. In 2016, GAO developed the 28 leading  practices. In 2019, we provide specific definitions for each of the practices  noted. We made slight modifications to the language to align with the  definitions provided. For monitoring, we developed this list of leading  practices based on our review of the GPRA Modernization Act of 2010;  OMB’s Uniform Administrative Requirements, Cost Principles, and Audit  Requirements for Federal Awards; GAO’s Standards for Internal Control  in the Federal Government (Greenbook); and others. The list of leading  practices for monitoring includes developing monitoring plans; collecting,  reviewing, and analyzing monitoring data; and establishing roles and  responsibilities of personnel responsible for monitoring.

For evaluation, we developed a list of leading practices based on the  American Evaluation Association’s (AEA) 2016 An Evaluation Roadmap  for a More Effective Government (AEA Roadmap) and Preface to  Evaluators’ Ethical Guiding Principles. The list of leading practices for  evaluation include developing evaluation plans; ensuring evaluator independence; developing staff skills regarding evaluation and use of  evidence; and establishing roles and responsibilities of personnel  responsible for evaluation. To perform these analyses, two analysts  assessed if the Guidelines incorporated specific GAO leading practices.  The analysts worked iteratively, comparing notes and reconciling  differences at each stage of the analysis. In addition, GAO staff  independent of the two analysts reviewed the final analysis, and made  modifications as appropriate. We also interviewed relevant OMB officials  in Washington D.C. involved in developing the memorandum and inquired  about specific requirements and plans to ensure the implementation of  these Guidelines.

To address our second objective, we examined U.S. agency M&E policies  against the requirements noted in the OMB Guidelines. We identified the  six major agencies administering the most foreign assistance funds. The  six agencies are the U.S. Agency for International Development (USAID),  the Department of State (State), the Millennium Challenge Corporation  (MCC), the Department of Health and Human Services (HHS), the U.S.  Department of Agriculture (USDA) and the Department of Defense  (DOD). We asked these agencies to identify or provide all relevant  policies and guidance relating to foreign assistance M&E, including,  where appropriate, standard operating procedures or other guidance.  For USDA, we reviewed the Foreign Agricultural Service’s food  assistance; for HHS, the President’s Emergency Program for AIDS Relief;  and for DOD, security cooperation programs. To perform these analyses,  two analysts assessed agency M&E policy documents against the  requirements in the OMB Guidelines. We identified requirements as  phrases that included the following language “required,” “must,”  “mandatory,” or “should.” The analysts worked iteratively, comparing  notes and reconciling differences at each stage of the analysis. In  addition, other GAO staff independent of the two analysts reviewed the  final analysis, and made modifications as appropriate. We also  interviewed relevant OMB staff and agency officials in Washington D.C.  involved in developing and implementing the M&E policies and inquired  about specific requirements, and plans to ensure their M&E policies are  implemented.

Appendix II: The Office of Management and Budget Monitoring and Evaluation Guidelines (OMB Memorandum M-18-04)

Appendix II: The Office of Management and  Budget Monitoring and Evaluation Guidelines  (OMB Memorandum M-18-04)

In January 2018, the Office of Management and Budget (OMB) released  (M-18-04) Monitoring and Evaluation Guidelines for Federal Departments  and Agencies that Administer United States Foreign Assistance (the  “Guidelines”) in response to the Foreign Aid Transparency and  Accountability Act of 2016 (FATAA). Table 4 shows the complete  description of the requirements noted in the Guidelines.

Appendix III: Assessment of the Foreign Aid Transparency and Accountability Act of 2016 and the Office of Management and Budget Monitoring and Evaluation Guidelines

The Foreign Aid Transparency and Accountability Act of 2016 (FATAA)  has required objectives on monitoring and evaluation for the Office of  Management and Budget (OMB) to include in the Guidelines. We  compared the 13 required objectives for the Guidelines set forth in the  FATAA legislation with those in the OMB Guidelines. We found that all of  the monitoring and evaluation requirements set forth in the legislation are  included in the OMB Guidelines. Table 5 shows the FATAA legislation  requirements, OMB Guidelines, and our assessment of the alignment  between the legislation and OMB’s Guidelines.

Appendix IV: Comments from the Department of Defense

Appendix V: Comments from the Department of State

Appendix VI: Comments from the United States Agency of International Development

Appendix VII: GAO Contact and Staff Acknowledgments

In addition to the individual named above, James B. Michels (Assistant  Director), Farahnaaz Khakoo-Mausel (Analyst-in-Charge), Paulina  Maqueda-Escamilla, Mark Dowling, Martin De Alteriis, Benjamin Licht,  John Hussey, Neil Doherty, Aldo Salerno, Carolina Morgan and Michael  Simon made key contributions to this report.

Government Auditing Standards 2018 Revision (Supersedes  GAO-12-331G. GAO-18-568G. Washington, D.C.: July 17, 2018.

Foreign Assistance: Agencies Can Improve the Quality and  Dissemination of Program Evaluations. GAO-17-316. Washington, D.C.:  March 3, 2017.

Foreign Assistance: Selected Agencies’ Monitoring and Evaluation  Policies Generally Address Leading Practices. GAO-16-861R.  Washington, D.C.: September 27, 2016.

Program Evaluation: Some Agencies Reported that Networking, Hiring,  and Involving Program Staff Help Build Capacity. GAO-15-25.  Washington, D.C.: November 13, 2014.

Government Efficiency and Effectiveness: Inconsistent Definitions and  Information Limit the Usefulness of Federal Program Inventories.  GAO-15-83. Washington, D.C.: October 31, 2014.

Standards for Internal Control in the Federal Government. GAO-14-704G.  Washington, D.C.: September 10, 2014.

State Department: Implementation of Grants Policies Needs Better  Oversight. GAO-14-635. Washington, D.C.: July 21, 2014.

Program Evaluation: Strategies to Facilitate Agencies’ Use of Evaluation  in Program Management and Policy Making. GAO-13-570. Washington,  D.C.: June 26, 2013.

President’s Emergency Plan for AIDS Relief: Agencies Can Enhance  Evaluation Quality, Planning, and Dissemination. GAO-12-673.  Washington, D.C.: May 31, 2012.

Grants Management: Action Needed to Improve the Timeliness of Grant  Closeouts by Federal Agencies. GAO-12-360. Washington, D.C.: April 16,  2012.

Designing Evaluations: 2012 Revision. GAO-12-208G. Washington, D.C.:  January 31, 2012.

International School Feeding: USDA’s Oversight of the McGovern-Dole  Food for Education Program Needs Improvement. GAO-11-544.  Washington, D.C.: May 19, 2011.

Program Evaluation: Experienced Agencies Follow a Similar Model for  Prioritizing Research. GAO-11-176. Washington, D.C.: January 14, 2011.

Managing for Results: Enhancing Agency Use of Performance  Information for Management Decision Making. GAO-05-927. Washington,  D.C.: September 9, 2005.

Program Evaluation: An Evaluation Culture and Collaborative  Partnerships Help Build Agency Capacity. GAO-03-454. Washington,  D.C.: May 2, 2003.

Managing For Results: Federal Managers’ Views Show Need for  Ensuring Top Leadership Skills. GAO-01-127. Washington, D.C.: October  20, 2000.

Performance Plans: Selected Approaches for Verification and Validation  of Agency Performance Information. GAO/GGD-99-139. Washington,  D.C.: July 30, 1999.

Agency Performance Plans: Examples of Practices That Can Improve  Usefulness to Decisionmakers. GAO/GGD/AIMD-99-69. Washington,  D.C.: February 26, 1999.

Executive Guide: Effectively Implementing the Government Performance  and Results Act. GAO/GGD-96-118. Washington, D.C.: June 1, 1996.

Are federal agencies doing a good job keeping track of the effectiveness of the foreign assistance programs they administer?

We looked at how agencies monitor and evaluate these programs. Specifically, we looked at the guidelines the Office of Management and Budget established for federal agencies with foreign assistance programs. We reviewed whether the guidelines incorporated leading practices for monitoring and evaluation, and whether agencies adopted the guidelines.

We found the OMB guidelines included 23 of 28 leading practices, and most agencies adopted most of them. We made 7 recommendations to increase the use of OMB's guidelines.

[Image updated to show examples of country flags.]


Private School Choice: Accountability in State Tax Credit Scholarship Programs

All TCS programs are state programs. States develop program policies and requirements, including establishing the roles and responsibilities of SGOs and participating private schools. The President's fiscal year 2020 budget request included a proposal for federal tax credits for donations to state-authorized SGOs. GAO was asked to review key characteristics related to accountability in state TCS programs that can fund K-12 educational expenses.

This report examines (1) key requirements state TCS programs have chosen to establish for SGOs, (2) key requirements for private schools participating in state TCS programs, and (3) how selected states implement TCS programs and assess whether SGOs and participating private schools are following key state requirements.

GAO identified key requirements states may choose to establish related to accountability for SGOs and schools based on relevant research and prior work. GAO also reviewed program documents from all 22 TCS programs to identify whether they had these key requirements as of school year 2018-2019 and then verified this information with state program officials. GAO did not conduct an independent review of state laws and regulations. GAO visited Arizona, Florida, and Pennsylvania, which have the largest TCS programs. In each of these states, GAO reviewed program documents and interviewed officials at state agencies and staff at selected SGOs and private schools (selected to provide variation in size and other characteristics).

State tax credit scholarship (TCS) programs—programs that offer state tax credits for donations that can fund scholarships for students to attend private elementary and secondary schools—have established various key requirements for the scholarship granting organizations (SGO) that collect donations and distribute awards. For example, all 22 TCS programs in operation as of January 2019 require SGOs to register with or be approved by the state and limit the percentage of donations they can use for non-scholarship expenses. In addition, almost all of these programs—which received over $1.1 billion in donations and awarded approximately 300,000 scholarships in 2017—also require SGOs to undergo annual financial audits or reviews (19 programs). Fewer programs have requirements about SGO fundraising practices (9 programs) or the qualifications of SGO leadership personnel (10 programs), such as restrictions on officials having previous bankruptcies.

States also have various key requirements that apply to private schools that enroll students with TCS scholarships. For example, private schools in most of the 22 programs must follow certain academic guidelines related to curriculum content (18 programs) and instructional time (19 programs), and have staff undergo background checks (18 programs). Schools in fewer programs are required to conduct academic testing (11 programs), ensure their teachers have specified qualifications (12 programs), or undergo an annual audit or financial review (4 programs).

The three states with the largest TCS programs—Arizona, Florida, and Pennsylvania—implement and oversee their programs in different ways. In all three states, state agencies administer the tax credits while SGOs are generally responsible for managing donations and awarding scholarships; the details of these processes varied based on the requirements of each program. For example, Arizona and Pennsylvania's programs allow donors to recommend that funds go to specific schools, which can affect how SGOs solicit donations and award scholarships. Florida does not permit recommendations. All three states require SGOs to report on operations and undergo annual financial audits or reviews, while the states differ in how participating private schools are overseen. Florida's TCS programs use multiple monitoring methods, while all Arizona programs and one of two Pennsylvania programs generally rely on SGOs to confirm that schools comply with program requirements.

State tax credit scholarship (TCS) programs offer state tax credits for  donations that can fund scholarships for students to attend private  elementary and secondary schools. Individuals or businesses donate to  scholarship granting organizations (SGOs) that then award scholarship  funds to eligible students. All TCS programs are state programs; there  are no federal TCS programs. States develop their TCS program policies  and requirements, including establishing the roles and responsibilities of  SGOs and participating private schools. As of January 2019, 17 states  operated 22 TCS programs, receiving over $1.1 billion in donations and  awarding approximately 300,000 scholarships during 2017, according to  state-reported data.

TCS programs are one type of private school choice program that states  have established to help families fund private school education. Other  types of private school choice include voucher programs and education  savings account programs, on which GAO reported in 2016 and 2017.

While TCS programs serve a relatively small number of students  nationwide compared to traditional public schools, promoting school  choice options—both private and public—through a variety of spending  and tax expenditure programs continues to be a topic of national debate.  Although no federal TCS program exists, bills to authorize federal tax  credits for contributions to SGOs have been introduced in recent years. In  addition, the President’s fiscal year 2020 budget request included a  proposal for federal tax credits for donations to state-authorized SGOs.  You asked GAO to review accountability in state-administered TCS  programs.

This report examines: (1) key requirements state TCS programs have  chosen to establish for SGOs, (2) key requirements for private schools  participating in state TCS programs, and (3) how selected states  implement TCS programs and how they assess whether SGOs and  participating private schools are following key state requirements.

To identify key requirements that states sometimes choose to establish  for SGOs and schools, we reviewed relevant research—including our  past work—on how states structure and oversee their private school  choice programs, as well as research on nonprofit management. By “key  requirements,” we mean requirements we identified as likely to be used  by multiple programs, but the list is not meant to be exhaustive. We  grouped the requirements into the broad categories of financial,  administrative, and scholarship award requirements for SGOs and  academic, administrative, and financial requirements for schools. To  determine which of these key requirements states chose to use for SGOs  and schools in each TCS program as of school year (SY) 2018-2019, we  reviewed program documents from all 22 TCS programs operating in  January 2019 and sent our findings to officials from each state program  for verification. GAO did not conduct an independent review of state laws  and regulations and is not evaluating the efficacy or appropriateness of  how states design or implement requirements.

To identify how selected states implement TCS programs and how they  assess whether SGOs and private schools follow program requirements,  we visited the three states with the largest TCS programs—Arizona,  Florida, and Pennsylvania—and conducted in-depth reviews of their TCS  programs. These three states were selected because they had the largest  TCS programs at the time of our study in terms of both donations and  scholarships awarded. Combined, these programs comprised  approximately 80 percent of donations received and TCS scholarships  awarded nationwide in 2017. For each of the three states, we reviewed  program documents and spoke with (1) officials at relevant state  agencies, (2) staff at selected SGOs, and (3) staff at selected private  schools to discuss TCS program operations and how the states monitor  SGOs and schools regarding program requirements. The information  gathered in the three selected states is not generalizable to all tax credit  scholarship programs.

We conducted this performance audit from October 2018 to September  2019 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.

The 22 TCS programs covered by our review varied widely in total  donations and number of scholarships awarded. For example, total  donations during 2017 ranged from $854,326 in New Hampshire’s  program to approximately $623 million in Florida’s largest program (see  fig. 1 and app. I). Of the 17 states operating these programs, Arizona,  Florida, and Pennsylvania operate the largest programs in terms of both  dollars donated and number of scholarships awarded. These are also the  only three states that operate multiple programs per state.

Decisions about whether to develop and operate a TCS program—as well  as how to structure requirements—are at the discretion of each state;  there is no federal role in establishing these programs. For example,  states sometimes choose to establish requirements that SGOs and  schools must follow as a minimum condition for participation. Since  scholarships are funded through donations rather than state  appropriations, the financial impact to states from TCS programs primarily  occurs through forgone revenue resulting from the associated tax  credits.

Various state agencies, SGOs, and participating private schools generally  all play a role in administering state TCS programs, with the specific  division of responsibilities varying by program.

State agencies that are responsible for tax administration, education,  or both, generally administer these programs. For example, in some  programs, state agencies disseminate information to donors,  scholarship students, or the public, and approve SGOs or private  schools to participate in the program.

SGOs are tax-exempt organizations that are generally responsible for  managing some aspects of the donation process—such as collecting  donations—as well as awarding scholarships to students.

Participating private schools educate students receiving tax credit  scholarships. They sometimes also facilitate donations or inform  parents about TCS scholarship awards. Participating schools can vary  in terms of characteristics such as their size, religious affiliation, and  whether they focus on specific student populations, such as students  with disabilities.

In September 2018, we issued a report that described state TCS program  requirements related to donations and student eligibility for programs  operating at the beginning of 2018. In that report, we found that these  programs offered tax credits to individuals, businesses, or both that  ranged from 50 percent to 100 percent of the donation amount. We also  found that most programs set limits on the amount of TCS program  donations that could be made per year. In terms of eligibility, we found  that TCS programs commonly determined student eligibility based on  their household income, with income limits varying widely across  programs.

TCS Programs Generally Require Financial Audits or Reviews for Participating SGOs and Prohibit Them from Awarding Scholarships to Only One School

States have established requirements that SGOs must follow as a  condition of participation in their TCS programs. See figure 2 for key  requirements that some states have chosen to put in place for SGOs.

All TCS programs have financial requirements that limit the percentage of  donations that SGOs are permitted to use for non-scholarship  expenses—such as staff salaries—and most also require SGOs to  undergo annual financial audits or reviews. (See fig. 3.)

The limit on the percentage of donated funds that SGOs are permitted to  use on non-scholarship expenses ranges from 2 percent to 10 percent for  20 of the 22 programs. The other 2 programs—both in Pennsylvania— have a limit of 20 percent. About half (12) of programs also require SGOs  to follow rules about how donated funds or related interest are managed and spent. For instance, South Dakota requires SGOs to spend all  revenue from interest or investments on TCS scholarships.

Most (19) programs require SGOs to undergo either an annual financial  audit or review. Some of these programs require more extensive audits  for SGOs that receive donations over a certain dollar threshold (e.g., over  $1 million). Three programs require at least some SGOs to submit proof  of fiscal soundness, such as a surety bond or letter of credit. For  example, to insure against potential financial loss, Florida requires SGOs  to submit a surety bond equal to the SGO’s unspent donations.

Some programs require SGOs to take steps to guard against conflicts of  interest. For example, four programs require SGOs to have a conflict of  interest policy or a policy designed to prevent individual financial gain  among SGO personnel. For instance, New Hampshire prohibits SGOs  from awarding scholarships to children of any SGO employee or to  children of any business owner whose business donates to the SGO.  About a third (8) of programs prohibit SGOs and participating schools  from sharing resources or personnel, while other programs do not include  such prohibitions. For example, in Pennsylvania, schools are permitted to  operate as SGOs and award scholarships directly to their students.

In addition, TCS programs require SGOs to be recognized as tax-exempt  organizations by the Internal Revenue Service (IRS), so they generally  are also subject to applicable federal requirements for tax-exempt  organizations, such as filing an annual information return or notice with  the IRS. In addition, SGOs may be subject to applicable state laws for  tax-exempt or non-profit organizations.

Figure 4 shows the number of state TCS programs that had selected  administrative requirements for SGOs.

All programs require SGOs to register or receive state approval by, for  example, submitting an application to the state or providing  documentation showing the SGO is a tax-exempt organization. Slightly  fewer than half (10) of programs require SGOs to follow rules about the  qualifications of SGO leadership personnel. For instance, Nevada  requires the SGO’s top board member to sign an affidavit stating that no  member of the board of directors or SGO employee has ever been  convicted of a felony, among other requirements. Nine of the programs  have requirements related to marketing and fundraising practices. For  instance, Louisiana requires SGOs to send the state any advertisements  so state officials can review the materials.

Almost all (21) programs require SGOs to report to the state the number  or total dollar amount of scholarships they awarded. Also, 19 programs  require SGOs to report information about the characteristics of  scholarship students, such as household income or geographic location.  These programs sometimes choose to require SGOs to report individual  student-level information or aggregated information for all of their  scholarship students combined. For example, Alabama requires SGOs to  report identifying information and scholarship amounts for each student to  the state. Alternatively, Georgia requires SGOs to report the number of  families of scholarship recipients by income group.

Figure 5 shows how many programs had key requirements for SGOs  related to scholarship awards. Almost all (20) programs prohibit SGOs  from awarding scholarships to only one school. Some of these 20  programs require SGOs to allow students to use their scholarships at any  qualified school whereas others allow SGOs to work with a subset of two  or more schools.

The majority of programs prohibit donors from recommending that  scholarships go to specific students while fewer programs prohibit donors  from directing funds to specific schools. More than half (16) of programs  prohibit SGOs from allowing donors to recommend that scholarships go  to specific individuals, such as students they know personally. About  one-third (8) of programs prohibit SGOs from allowing donors to  recommend that scholarship funds be used at a specific school.

Most States with TCS Programs Require Schools to Teach Core Subjects and Meet Minimum Attendance Standards; Few Require Financial Audits or Reviews of Schools

In addition to requirements for SGOs, states with TCS programs also  have requirements for private schools. See figure 6 for key requirements  that some states with TCS programs have chosen to put in place for  schools. Some requirements were specific to schools participating in the  TCS program and some requirements were for all private schools,  regardless of TCS program participation. We counted programs as having  a particular requirement as long as the requirement applied to at least  some of the participating private schools.

Programs generally have one or more academic requirements for  participating private schools. (See fig. 7.)

About half (9) of the programs require private schools to register or be  approved by the state TCS program before their students can receive  TCS scholarships. For example, in Nevada private schools must register  with the state by completing a form acknowledging that they will follow  program requirements. Other programs, such as Georgia’s, require SGOs  to determine if private schools meet program participation requirements.  Separate from any requirements to register with the TCS program itself,  over half (14) of the programs require participating private schools to be  accredited by the state or another organization, such as a regional  accreditation organization. For example, Pennsylvania requires  participating private schools to be 1) licensed by the state, 2) accredited  by an association approved by the state, or 3) operated by a religious  institution.

Private schools participating in most TCS programs are subject to  requirements regarding minimum instructional time or student attendance  and requirements related to curriculum content or core subjects, such as  reading, mathematics, social studies, and science. These requirements  may or may not be the same as requirements for all private schools in a  state. For example, Nevada requires all private schools, including those  participating in its TCS programs, to provide 180 days of academic  instruction per year. South Carolina’s TCS program generally requires  participating private schools to offer the courses required to receive a  high school diploma in the state.

Eleven programs require schools to give academic tests to TCS students.  Of these programs, three require participating schools to administer the  same test required of public school students and eight allow schools to  select among multiple tests. For example, Louisiana requires participating  private schools to give TCS students the same state tests in English and  math that are used in public schools, whereas Florida allows participating  private schools to choose from a state approved list of norm-referenced  tests. These 11 programs also require schools to report TCS students’  test results to the state, parents, or other entities. For instance, one of  Florida’s programs requires schools to report test results to a university  selected by the state to analyze TCS students’ test scores.

Participating schools are often required to ensure their teachers undergo  background checks or meet minimum qualifications, and less frequently  required to undergo site visits. (See fig. 8.)

Most (18) programs require participating private schools to conduct  background checks or fingerprinting of employees with direct,  unsupervised contact with children. Additionally, 12 programs require  schools to ensure teachers meet certain qualifications, such as holding a  state-issued certificate or a college degree. For example, Alabama  requires all private school teachers to hold teaching certificates issued by  the state.

Nine of the programs require private schools to permit state or other  officials to conduct site visits. In the majority of these programs, site visits  are a general requirement for private schools in the state or a component  of state accreditation that some or all schools are required to obtain  before participating in the TCS program. Across these programs, the  frequency of the site visit varies. For instance, Indiana state officials are  required to make random site visits to at least five percent of participating  private schools each year, while Iowa requires a site visit for all private  schools operating in the state at least once every five years.

Among the 22 programs, financial requirements for participating private  schools are generally less common than academic and administrative  requirements. (See fig. 9.)

Most (19) programs have requirements related to student withdrawals,  such as requiring schools to report or repay the scholarship when  students withdraw from the school for which the scholarship was originally  awarded. For example, Illinois requires schools and SGOs to prorate  scholarships for students who transfer to another private school during  the school year, while Louisiana requires schools to immediately notify  the SGO and state if a scholarship student is no longer enrolled.

Few (4) programs require schools to complete an annual financial audit or  review. Among programs with such requirements, Florida requires  schools that receive more than $250,000 in scholarship funds to submit  the results of a financial audit to the SGO that awarded the majority of  those funds. South Carolina requires all schools to include a copy of an  audit or other financial review when they initially apply to participate in the  program and annually afterwards.

About one-fourth (5) of programs require schools to provide surety bonds  or other evidence to demonstrate financial viability. For example,  Louisiana requires schools that have operated for fewer than five years  and will receive more than $50,000 in TCS funds to either provide a SGO  with a surety bond equal to the amount of TCS funds they expect to  receive during the school year or other information showing financial  viability. Florida requires schools operating for fewer than three years to  provide the state with a surety bond equal to the amount of scholarship  funds for any quarter.

States with the Largest TCS Programs Have SGOs Manage Key Program Features

SGOs Manage Donations and Award Scholarships while State Agencies Administer Tax Credits in the Three States

In the three states with the largest TCS programs—Arizona, Florida, and  Pennsylvania—SGOs are generally responsible for recruiting donors  while state agencies administer tax credits. (See fig. 10).

SGOs generally recruit potential donors and sometimes help them apply  for tax credits. SGO officials in all three states described ways they solicit  donations, such as meeting with representatives from corporations to  promote TCS programs or providing banners and pamphlets for private  schools to display. In Arizona—where SGOs are permitted to allow donor  recommendations for specific schools or students in certain programs— SGO and school officials described roles for schools or students and their  families in soliciting donations. For example, family members of  prospective scholarship students may encourage members of their  community to donate and recommend their child or child’s school,  according to officials from a SGO and a school we visited in Arizona. In  addition, SGO officials we spoke with in all three states noted that they  help donors navigate the process of obtaining tax credits. For example,  officials from a SGO in Pennsylvania described how they can fill out the  application for tax credit pre-approval as the donor’s delegate or review  completed applications for errors before donors submit them to the state.

State agencies administer state tax credits based on the rules of each  TCS program. Specifically, five of the eight programs in the three states  have a maximum total dollar limit on the amount of all scholarship tax  credits that can be awarded in a year. In those five programs, the state  requires donors to apply for pre-approval of the tax credits to ensure the  limit has not been reached and tax credits are still available. In the Florida  and Arizona programs that have such limits, state agencies consider all  donor applications for tax credit pre-approval on a first come, first served  basis. Pennsylvania considers returning donors for pre-approval before  considering new donors (while tax credits remain).

State officials in Pennsylvania and Florida described different methods for  reviewing donors’ tax compliance before and after they file their taxes and  claim the TCS tax credit. Specifically, the Pennsylvania Department of  Revenue checks for any outstanding tax liability before approving tax  credits for the TCS programs and reviews all tax returns that claim TCS  credits to ensure the amount of tax credits claimed does not exceed the  amount that was pre-approved. The Florida Department of Revenue  reviews tax returns that claim a TCS credit to ensure the amount of tax  credits used matches the approved amount that was allocated for the  donor. Officials from the Arizona Department of Revenue said it does not  have a separate tax compliance review process for its TCS credits.

All three states established scholarship requirements while SGOs  managed the scholarship awards process by determining which students  are eligible for TCS scholarships, which eligible applicants will receive  scholarships, and how much to allocate to scholarship recipients within  program limits. In contrast, state agencies have a limited role—or no  role at all—in determining the allocation of scholarships among eligible  students, according to state and SGO officials in all three states.  However, states may provide guidance documents to help SGOs  navigate the state’s general policies for the awards process.

The number of SGOs awarding scholarships varied across the three  states, as did program policies for how those SGOs determine which  eligible students receive scholarships. In fall 2017, Florida had two  SGOs—one of which awarded 99 percent of scholarships. Meanwhile,  Arizona’s four TCS programs had between 14 and 60 SGOs each and  Pennsylvania’s two programs had approximately 190 and 260 SGOs  each. In addition, as described in table 1, states varied in their  requirements for how SGOs prioritize eligible students, the degree to  which SGOs may work exclusively with a subset of schools, and whether  SGOs may consider donors’ recommendations when deciding which  students receive scholarships. Further, because each SGO is responsible  for establishing its own procedures for awarding scholarships within  program requirements, a TCS program with many SGOs could have  substantial variation in how scholarships are awarded.

Prioritizing eligible students: The three states provide varying levels of  discretion to SGOs in how they prioritize eligible students when awarding  scholarships. For example, Florida’s largest program requires SGOs to  award scholarships on a first-come, first-served basis with first priority to  students who previously received a scholarship, and then to students  from lower-income households or who are in foster care; its other  program requires SGOs to award scholarships to students on a first- come, first-served basis. Two programs in Arizona and one in  Pennsylvania outline requirements for how SGOs must prioritize among  eligible students, while the remaining programs do not. Further, in all four  Arizona programs and both Pennsylvania programs, SGOs are permitted  to set additional criteria for selecting scholarship recipients beyond  requirements set by the state, as long as those criteria align with program  rules and existing laws. For example, officials we spoke with at one SGO  in Arizona noted that their selection committee considers written  narratives from students and their parents about the student’s character  and academic achievement when prioritizing eligible students, among  other factors.

Working with a subset of schools: State policies about partnerships  between SGOs and schools can affect which students receive  scholarship awards and where students can use those awards. SGOs in  Florida award scholarships to students who can then use their  scholarship award at any private school that qualifies to participate in the  TCS program. In contrast, Arizona and Pennsylvania allow SGOs to  partner with subsets of participating schools and award scholarships  exclusively to students at those schools. For example, in Arizona and  Pennsylvania, some SGO officials noted that the scholarships awarded  through their SGO may only be used at partner schools that shared a  religious affiliation with the SGO.

Allowing donor recommendations: The TCS programs in the three  states also had different rules on whether donors may recommend that  scholarships be awarded to particular schools or students. Florida  prohibits donors from making scholarship recommendations or  designations for specific schools or students. In all Pennsylvania and  Arizona programs, donors can recommend or designate donations for  specific schools, but SGOs can take different approaches to distributing  any such recommended funds. For example, one SGO in Pennsylvania  sends recommended donation funds directly to the designated schools  and the school decides how to distribute the scholarship funds among  eligible students. In Arizona, one SGO tracks the amount of donations  donors recommend for each of its partner schools and awards those  funds to eligible students enrolled at those schools.

Further, Arizona’s two TCS programs designed for individual donors  (rather than businesses), allow donors to recommend that the funds they  donate go to specific students. An official at one SGO we visited in  Arizona said they provided its external scholarship award committee with  information about students’ applications, including any student-specific  recommendations, to inform the selection process. Pennsylvania  programs do not expressly prohibit donor recommendations for specific  students. In Pennsylvania, officials from one school that was also a SGO  told us that they did not accept student-specific recommendations, while  officials in a different SGO described how donors may not make student- specific recommendations, but may designate certain groups of students,  such as children of first responders.

In addition to establishing program policies regarding how eligible  students are selected, the three states also have requirements regarding  the amount of scholarship money that can be awarded per student, and  SGO officials described different methods for determining the amount and  frequency of scholarship awards for each student. Among the schools  we visited the proportion of students who received TCS scholarships  compared to students who did not receive TCS scholarships varied. For  example, in one school in Florida, less than one percent of students  received TCS scholarships and in a school in Arizona, school officials told  us that about 80 percent of students receive TCS program scholarships.  When awarding scholarships, officials at some SGOs we visited chose to  issue a limited number of awards at the maximum allowable scholarship  amount while others chose to issue scholarships to more students in  smaller amounts—sometimes for shorter periods. In addition, students in  Arizona and Pennsylvania may receive multiple concurrent scholarships  from different SGOs, different TCS programs in the state, or both. This  approach potentially increases the amount of funding students receive;  however, it can also present logistical difficulties for the schools and  families of scholarship recipients, according to SGO and school officials.  For instance, officials at one school described wanting students to receive  as much TCS funding as possible, but said it was also challenging to  track the different SGO award cycles, incoming funds, and the projected  impact on tuition balances for each student.

As part of the scholarship award process, some SGOs we visited told us  they collected information about tuition and fees at schools to ensure  scholarship award amounts do not exceed school tuition, per program  requirements. Costs for tuition ranged from approximately $6,000 per  year to approximately $37,000 per year among the schools we visited.

All Three States Require SGOs to Attest That They Will Follow Program Requirements, Submit Information on Their Operations, and Complete an Annual Financial Audit or Review

To participate in TCS programs, all three states require SGOs to provide  a description of some of their operating procedures and regularly report  donation and scholarship information. The type of information states  collect and how they determine whether SGOs are following applicable  TCS program requirements varies. (See table 2.)

All three states require SGOs to complete an application to participate in  TCS programs, which involves signing a form attesting that the SGO will  follow program requirements and providing other types of documentation.  This documentation includes evidence that the SGO is recognized as tax- exempt by the IRS, descriptions of the SGO’s procedures for awarding  scholarships, and other information, depending on the program. When  reviewing SGO applications, state officials in all three states described  how they check that all required information is included in accordance  with program rules, but generally do not evaluate the content. For  example, in Pennsylvania, a program official noted that the state agency  checks that SGOs submit all required documents and relies on attestation  statements signed by SGO officials as an essential step for certifying that  SGOs will follow program requirements. In addition, state agencies also  generally provide SGOs with some guidance on how to interpret program  requirements in all three states. State agencies in both Arizona and  Pennsylvania have developed guidance manuals for SGOs. Florida  collaborates closely with SGOs to interpret program rules and develop  guidance, according to SGO and state officials.

In addition to application materials, all three states require SGOs to  regularly submit information about donations received, scholarships  awarded, and the results of financial audits or reviews. The extent to  which these audits or reviews include an assessment of SGO compliance  with program requirements varies. For example, in addition to financial  audits, Florida also requires the state Auditor General to review SGO  operations for compliance annually. According to SGO officials, the  Florida Auditor General conducts file reviews and on-site visits during  these compliance reviews. In Arizona, the SGO manual includes optional  procedures financial auditors may use to determine if SGOs are following  certain program requirements as part of their review. Pennsylvania does  not require an assessment of SGOs’ compliance with program  requirements as part of its annual financial audit.

According to officials, states typically work with SGOs to resolve any  compliance issues and state agencies rarely permanently revoke SGOs’  approval to participate in TCS programs. According to officials in Florida,  no SGOs have been removed from the programs due to noncompliance.  The officials said that once a SGO has been approved through the states’  initial application process it is very likely that they will be renewed each  year unless a large compliance issue arises. Officials in Pennsylvania  noted that they contact SGOs to clarify discrepancies in documentation  and have temporarily revoked approval from a small proportion of SGOs  that failed to submit required information. Officials in Arizona described  one instance where a SGO was decertified due to noncompliance;  officials stated that the SGO would be recertified if it resolved the  compliance issues and reapplied to be a SGO.

State and SGO Responsibilities for Monitoring Private Schools Varied in the Three States

State officials in the three states described different approaches to  overseeing participating private schools’ compliance with program  requirements. Florida state officials described using a variety of  monitoring activities to oversee participating private schools, while  Arizona and Pennsylvania state agency officials said they do not conduct  ongoing monitoring activities due to the parameters of their statutory  authority. (See table 3.)

Florida state officials described conducting site visits to schools during the  initial application process and when they determine schools are at risk of  noncompliance. They also noted their monitoring practices have led them  to identify multiple issues of noncompliance at certain schools and, as a  result, they have temporarily or permanently suspended those schools  from receiving TCS program funds. They said the state also delegates  certain monitoring activities to SGOs, such as reviewing financial audit  results, following up with schools to resolve any issues resulting from  those audits, and reporting those issues to the state. State and SGO  officials said that SGOs in Florida may also be responsible for  implementing any penalties to schools, such as adjusting scholarship  payments to schools that do not meet certain reporting requirements.

For all Arizona programs and one of two Pennsylvania programs, officials  told us that they generally rely on SGOs to verify that schools receiving  scholarship funds meet program requirements. For example, in Arizona,  SGOs are responsible for determining if a school qualifies to participate in  the state’s TCS programs. SGO officials we spoke to in Arizona noted  they require schools to sign documents attesting that the school meets  the requirements to be a qualified school. In Pennsylvania’s Educational  Improvement Tax Credit Program, SGOs determine whether schools are  qualified to participate and each SGO may approach this differently,  according to state officials. For example, in addition to meeting the state’s  criteria for participation in the program, one SGO we spoke with in  Pennsylvania also required schools to be tax-exempt, have a board and  budget, and share its religious affiliation. Another SGO we spoke with in  Pennsylvania required that a school attest that it meets program  requirements. In Pennsylvania’s other TCS program (the Opportunity  Scholarship Tax Credit Program) state officials told us that they determine  whether schools are qualified to participate and do not conduct  subsequent monitoring activities.

We provided a draft of this report to the Department of Education for  review and comment. Education’s comments are reproduced in appendix  III. Education also provided technical comments, which we incorporated  as appropriate. We also provided relevant excerpts from the report to the  appropriate state agencies in each state we reviewed and incorporated  their technical changes as appropriate.

Education did not comment on the report’s findings. Instead, it provided  information about the administration’s tax credit proposal and made  observations about how the scope of TCS programs covered in our report  was both similar and different from educational programs that are  addressed in the administration’s proposal. In its comments, Education  further stated that given these scope differences, the GAO report may not  fully inform the debate around the administration’s proposal. As stated in  the draft report, promoting school choice options—both private and  public—through a variety of spending and tax expenditure programs  continues to be a topic of national debate. The purpose of this report was  to examine state TCS programs that are used to fund scholarships that  students can use to attend private elementary and secondary schools by  describing: (1) key requirements state TCS programs have chosen to  establish for SGOs, (2) key requirements for private schools participating  in state TCS programs, and (3) how selected states implement TCS  programs and how they assess whether SGOs and participating private  schools are following key state requirements.

Education stated that the draft report does not note that several TCS  programs allow scholarships to be used for educational expenses beyond  tuition. As stated in the draft report, some states use tax credit  scholarship programs to fund preschool, career and technical education,  or public school initiatives; the draft report further noted that these  programs are outside the scope of this review. This report is the second  of two GAO reports examining K-12 TCS programs. The prior report  (GAO-18-679) discussed various ways students can use state K-12 TCS  scholarships. Specifically, we reported that, as of SY 2017-2018, more  than half of the programs (13 of 22) allowed students to use their  scholarship money for costs like transportation and books in addition to  tuition, whereas the remaining programs (9 of 22) required scholarship  funds to be used for tuition only.

Education asserted positive fiscal effects associated with state TCS  programs and cited several studies to this end. As stated in the draft  report, tax credits are a form of forgone revenue. Assessing the fiscal  impact of these programs was not among the purposes of this report.   Thus, we did not assess the reliability of these studies or the validity of  their results. Education also commented on our decision to exclude  Montana’s TCS program from the scope of this report. As noted in our  report, we did not include Montana’s program because it was the subject  of pending litigation at the time of our review.

Finally, Education noted, as also stated in the draft report, that states and  school districts have obligations under the Elementary and Secondary  Education Act and the Individuals with Disabilities Education Act to make  equitable services available to eligible private school students, including  those who participate in TCS programs. We agree and have reported  extensively on equitable services in the context of private schools in  GAO-16-712 and GAO-18-94. In GAO-18-94, we recommended that  Education review information provided by states related to changes in  federal special education rights when a parent places a student with a  disability in private school and work with the states to correct inaccurate  or incorrect information. In that report, we identified some private school  choice programs that were providing information that Education confirmed  inaccurately described rights under the Individuals with Disabilities  Education Act when a student with a disability is moved from a public to a  private school. Education agreed with this recommendation – a  recommendation GAO considers to be among the highest priority of  recommendations we have made to the Department. However, Education  has not yet fully implemented this recommendation and, as of September  2019, we found that some information Education confirmed to be  inaccurate remains in several states’ private school choice program  documents.

As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies of this report to the  appropriate congressional committees, the Secretary of Education, and  other interested parties. In addition, the report is available at no charge on  the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (617) 788-0580 or nowickij@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who made key contributions to this report  are listed in Appendix IV.

Appendix I: State Tax Credit Scholarship Program Scholarship and Donation Information

of donors: businesses,  individuals, or both)

1,957b (both)  15 (business)  795 (business) $1,856  88,109b (individual)  $1,450  47,895b (individual)  ——- (both)

79 (both)  69d (business)  71 (business)

Average  donation  amount per  donor  ——- (both)

1,671 (business)  662 (business)  40 (business)  788d (both)  ——- (business)

Appendix II: Key Features of Largest Tax Credit Scholarship Programs in Arizona, Florida, and Pennsylvania

The profiles in this appendix describe key features of the largest tax credit  scholarship programs in Arizona, Florida, and Pennsylvania, as measured  in terms of the scholarships awarded in 2017. Information in these profiles  was obtained from interviews with state agency and selected scholarship  granting organization (SGO) officials and state program documents, and  was verified by state officials. GAO did not independently review state  laws or regulations. Unless otherwise specified, program requirements  are as of school year 2018-2019.

Financial • Limit non-scholarship spending to

Undergo annual audit or financial  review depending on amount of  donations received.   donations.

Ensure all promotional materials  include required language.  scholarship awards.

Follow guidelines for school  curriculum or core subjects.

Follow guidelines for minimum  instructional time or attendance.

Donors contributed over $68 million during calendar year 2017.

The program is open to individual donors only (business donors may  fingerprinting for staff. donate through other Arizona programs.)

Follow requirements if students  withdraw (e.g., report or repay).

Donations are eligible for 100 percent tax credits.

The program does not limit annual program-wide donations. Individual  donors (filing as single) were limited to $569 in tax credits during tax  year 2019.

34,632 scholarships were awarded between July 1, 2017 and June 30,  2018.

All K-12 students who live in Arizona and attend a private school are  eligible for the program. SGOs are expected to consider financial need  when prioritizing among eligible applicants.

Students may receive more than one scholarship from this program and  concurrently receive scholarships from other tax credit scholarship  programs in the state, up to the full cost of tuition.

SGOs may partner with a subset of qualified schools, such as schools  with shared religious views or teaching methods.

Donors may recommend that their donation fund scholarships at  specific schools or be awarded to specific students (within certain  parameters).

0 percent of donations in first 3  years and 3 percent afterwards.

Provide a surety bond or line of  credit.

Maintain separate accounts for  scholarship funds.

Not share resources or personnel  with private schools.

Not have an owner or operator  who recently filed for bankruptcy.

Donors contributed about $623 million during tax year 2017.

The program is open to businesses (not individuals).

Maximum allowable credit amounts per donor vary from 50 percent to

Administer an approved academic  test (not necessarily the test used  in public schools).

100 percent of tax liability, depending on the type of business tax to  which the tax credit is applied.

Program-wide limit on tax credits was $698 million in state fiscal year  2017-2018.

Hire teachers who meet minimum  qualifications.

Ensure background checks or  fingerprinting for staff.

108,098 scholarships were awarded during school year 2017-2018.

Students are eligible for full scholarships if their household income level  does not exceed 200 percent of the federal poverty level or if they are in  foster care.

Students may only receive one tax credit scholarship at a time, up to  $7,208 for tuition and fees, as of school year 2017-2018. given amount of scholarships.

SGOs must allow students to use scholarships at any private school

Provide a surety bond or line of  participating in the program. credit if in operation for less than 3  years.

Donors are prohibited from recommending that their donation fund  scholarships at specific schools or for specific students.

Follow requirements if students  withdraw (e.g., report or repay).

Undergo annual audit or financial  review.   donations.

Report information about  scholarship awards.

Be accredited, licensed by the  state, or a religious institution.

Follow guidelines for minimum  instructional time or attendance.

Follow guidelines for core  curriculum or academic subjects.

Donors contributed about $87 million during tax year 2017.

The program is open to businesses, including “special purpose entities”

Follow requirements if students  withdraw (e.g., report or repay). which allow individuals to donate to the program through the entity.

Donors are eligible to receive tax credits for 75 percent of their  donations if they donate for one year and tax credits for 90 percent of  their donations if they donate for two consecutive years.

The maximum allowable tax credit amount per donor is $750,000. The  program-wide limit on tax credits was $135 million in fiscal year 2017.  Donors must apply for pre-approval before claiming tax credits. Past  donors may apply for credits before potential new donors apply for any  remaining tax credits.

37,725 students received scholarships during school year 2017-2018.

Students are eligible for scholarships if their household income level  does not exceed $85,000, plus $15,608 per dependent in the household  (higher for students with disabilities).

Students may receive multiple tax credit scholarship at a time. Total  scholarship award amount may not exceed the cost of tuition and fees.

SGOs may partner with a subset of qualified schools, such as schools  with shared religious views or teaching methods.

Donors may recommend that their donation fund scholarships at  specific schools. Donors are not prohibited from recommending  scholarships to specific students.

Appendix III: Comments from the Department of Education

In addition to the individual named above, Nagla’a El-Hodiri (Assistant  Director), Barbara Steel (Analyst-in-Charge), Andrew Emmons, and  Jessica L. Yutzy made key contributions to this report. Also contributing to  this report were Isabella Anderson, Jeff Arkin, Deborah Bland, Lilia  Chaidez, Sarah Cornetto, Caitlin Cusati, Charles Ford, Monika Gomez,  Alison Grantham, Kirsten Lauber, Sheila R. McCoy, Mimi Nguyen,  Corinna Nicolaou, and Michelle Philpott.

Seventeen states offer state tax credits for donations to organizations that fund scholarships for students to attend private schools. The President’s 2020 budget request proposes new federal tax credits for such donations.

In 2017, scholarship organizations in these states received over $1.1 billion in donations.

We reviewed state requirements for finances, academics, and other topics and found:

All programs limit the percentage of donations that may be used to administer programs (2-20%)

States with the largest programs vary in how they monitor participating private schools