Juvenile Justice Grants: DOJ Should Take Additional Actions to Strengthen Performance and Fraud Risk Management

OJJDP administers grant programs to improve positive outcomes for juveniles in the justice system. In fiscal year 2018, OJJDP made 295 awards across 16 programs totaling over $290 million.

The Juvenile Justice Reform Act of 2018 included a provision for GAO to review OJJDP performance and internal controls intended to prevent fraud, waste, and abuse of grant funds.

This report examines the extent to which (1) OJJDP has goals and measures to assess the performance of its programs, and (2) DOJ has considered fraud risks for OJJDP grant programs. GAO reviewed DOJ documentation, such as OJJDP's Performance Measures Manual and OJP's risk management policy. GAO also reviewed performance data from selected OJJDP programs from October 2015 through December 2018, and interviewed DOJ officials.

The Department of Justice's (DOJ) Office of Juvenile Justice and Delinquency Prevention (OJJDP) has goal statements and performance measures for each of its programs, but has not established corresponding program-level targets (specific numeric goals). Rather, OJJDP has established several office-level targets to help assess progress across OJJDP grant programs collectively. For example, OJJDP has a target for the percent of youth who offend and reoffend across all applicable grant programs. Such office-level targets, while useful, might obscure the results of individual programs. Setting program-level targets would help OJJDP assess the progress of each program and reach its goal of increasing accountability for achieving results in individual programs.

DOJ's Office of Justice Programs (OJP) and DOJ's Justice Management Division (JMD) have taken steps to consider fraud risk affecting OJJDP programs. Specifically, OJP—the grant-making component in which OJJDP resides—has tools it uses to monitor grantee performance and compliance with award terms and conditions. According to OJP, these tools—such as checklists used during desk reviews and site visit audits—provide insight into grant fraud risks. Additionally, JMD—the component that manages fraud risk assessment across all components within DOJ—has taken steps to assess fraud risks affecting OJJDP grant programs. Specifically, JMD conducted department-wide fraud risk assessments in fiscal years 2017, 2018, and 2019. These assessments addressed all DOJ grants, including OJJDP's.

DOJ's 2017 assessment identified fraud risk scenarios and assessed their likelihood and impact—leading practices in GAO's Fraud Risk Framework. Building on the 2017 assessment, the 2018 assessment identified key fraud risk management activities, and the 2019 assessment resulted in a fraud risk profile. However, these assessments did not determine a fraud risk tolerance—i.e. managers' willingness to accept a specific level of risk—as it relates to OJJDP grant programs. JMD officials said they view this as the next step in the maturation of DOJ's fraud risk assessment processes, but did not have details or documentation of plans to do so. Determining a fraud risk tolerance—and assessing fraud risks against that tolerance to prioritize them—would help OJP calibrate resources to address grant fraud risk for OJJDP programs, helping ensure that resources are not under- or over-allocated.

GAO recommends that (1) OJJDP set performance targets for individual grant programs, and (2) DOJ determine the agency's fraud risk tolerance for all grants—which include OJJDP grant programs—and prioritize fraud risks based on an assessment against that tolerance. DOJ concurred with both recommendations.

The Department of Justice’s (DOJ) Office of Juvenile Justice and  Delinquency Prevention (OJJDP) provides federal funding to support  states, local communities, and tribal jurisdictions in their efforts to develop  and implement programs for juveniles. These programs are intended to  enhance public safety, ensure youth are held appropriately accountable  to both crime victims and communities, and empower youth to live  productive, law-abiding lives. In fiscal year 2018, OJJDP made 295  awards totaling over $290 million. Past allegations of improper program  administration have raised questions about the management and  performance of OJJDP programs; and previous investigations and audits  by the DOJ Office of the Inspector General (OIG) have identified  instances of potential fraud, waste, or abuse in OJJDP programs.

We have previously reported that a program performance assessment  system is an important component of effective program management and  contains key elements, including program goals and performance  measures. Goals communicate what the agency proposes to accomplish  and allow agencies to assess or demonstrate the degree to which those  desired results were achieved. Performance measures are concrete,  objective, observable conditions that permit the assessment of progress  made toward the agency’s goals.

The Juvenile Justice Reform Act of 2018 included a provision for us to  conduct a comprehensive analysis and evaluation regarding the  performance of OJJDP, and audit a sample of OJJDP’s grantees to  review internal controls intended to prevent fraud, waste, and abuse of  funds. This report examines the extent to which (1) OJJDP has goals  and measures to assess the performance of its programs, and (2) DOJ  has considered fraud risks for OJJDP grant programs. We will issue a  subsequent report to present the results of our audit of a selected sample  of grantees.

To address the extent to which OJJDP has goals and measures to  assess the performance of its programs, we reviewed documentation,  such as OJJDP’s Performance Measures Manual, program performance  reports, and grant solicitations. We interviewed officials from OJJDP,  which is an office within DOJ’s Office of Justice Programs (OJP). We  also interviewed officials from OJP’s Office of the Chief Financial Officer,  which has a role in OJJDP performance measurement. We assessed  OJJDP’s goals and measures against relevant leading practices for  performance management.

For the purpose of providing examples of performance results, we  selected three of OJJDP’s 16 programs funded in fiscal year 2018 (as  listed in appendix I). We then reviewed grantee-submitted performance  data covering the time period of October 2015 through December 2018.  We chose this timeframe because 2018 was the latest full calendar year  for which data were available, and looking back three years captured  variations in the programs’ funding levels. We assessed the reliability of  key data elements and methods used to calculate selected performance  measures. For instance, we tested grantee-submitted performance data  provided to us by OJJDP for missing data, outliers, and inconsistencies.  We determined that the performance-related data submitted to OJJDP by  grantees for programs we selected were not sufficiently reliable for this  purpose, and subsequently chose not to present these examples in this  report. Nevertheless, we present rounded numbers of “youth served” for  two of the programs we selected (2018 data only), as we found these  data to be reliable for the specific purpose of illustrating differences in the  relative sizes of the programs. See appendix II for details of our review of  grantee-submitted performance data.

To evaluate the extent to which DOJ considered fraud risks for OJJDP  grant programs, we reviewed documentation from DOJ’s Justice  Management Division (JMD), which is responsible for DOJ-level fraud risk  management, and from OJP, which is responsible for carrying out fraud  risk management activities for OJJDP grant programs. This  documentation includes DOJ- and OJP-level Enterprise Risk  Management (ERM) policies and risk assessment processes, and OJP- level grant monitoring and compliance auditing processes. We compared  DOJ’s fraud risk assessment efforts to relevant leading practices in A  Framework for Managing Fraud Risks in Federal Programs (Fraud Risk  Framework). We also interviewed relevant OJP and JMD officials about  fraud risk management as it relates to OJJDP grant programs.

We conducted this performance audit from February 2019 to December  2019 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.

In fiscal year 2018, OJJDP administered 16 programs, which collectively  included 53 grant solicitations and 295 associated awards. As shown in  table 1, the 16 programs included the Title II Formula Grant Program, two  discretionary grant programs for youth mentoring and missing and  exploited children that awarded the most funding, and 13 other  discretionary grant programs to support efforts such as helping opioid- affected youth, victims of child abuse, and girls in the juvenile justice  system. For a complete list of OJJDP programs and grants in fiscal year  2018, see appendix I.

On December 21, 2018, the Juvenile Justice Reform Act of 2018  (“Reform Act”) enacted various requirements to strengthen accountability  and oversight in OJJDP grant programs, including grants to states under  the Title II Formula Grant Program. The Reform Act expressed the  sense of Congress that OJJDP must restore meaningful enforcement,  and states must exercise vigilant oversight, to ensure compliance with  core requirements of the Title II Formula Grant Program. Among the  measures the Reform Act put into place to achieve this goal was a  requirement that states maintain an “effective” system of monitoring  compliance—a requirement that became effective for fiscal year 2020  awards. In contrast, the requirement prior to fiscal year 2020 was for  states to maintain an “adequate” system of monitoring compliance, which  was the requirement that applied to the grant performance period  reviewed for this report (October 2015 through September 2018).

This report is one of many called for by the Reform Act to improve  oversight of OJJDP grant programs, including the Title II Formula Grant  Program. Other oversight requirements in the Reform Act are addressed  to OJJDP; OJP’s Office of Audit, Assessment, and Management; and the  DOJ OIG. Appendix III provides information on selected Reform Act  requirements that relate to accountability and oversight in the Title II  Formula Grant Program, and the status of efforts to implement them.

OJP Grant Monitoring and Compliance Auditing Activities

All OJP grantees and awards are subject to “grant monitoring.” Grant  monitoring consists of (1) programmatic and (2) financial monitoring, and  according to OJP officials, helps ensure the programmatic and financial  integrity and accountability of grantees. OJP policy requires  programmatic desk reviews on all open awards each fiscal year and “in- depth” monitoring—consisting of enhanced programmatic desk reviews or  site visits—on at least 10 percent of the total number and dollar amount of  open and active awards annually. In addition, OJP’s Office of the Chief  Financial Officer plans to financially monitor at least 10 percent of the  award population annually.

States awarded Title II Formula Grants are subject to an additional form  of monitoring—“compliance auditing”—which is conducted by OJJDP  annually to fulfill statutory requirements unique to the program.  Specifically, “compliance auditing” refers to OJJDP’s process for (1)  auditing the compliance monitoring systems used by states, and (2)  evaluating states’ compliance with four core requirements specified in  law. See appendix IV for more information on compliance auditing in the  Title II Formula Grant Program associated with the performance period  we reviewed.

Both the grant monitoring and compliance auditing broadly consist of  three parts: (1) desk reviews that occur annually, (2) risk assessments  that assist officials in determining what additional monitoring or auditing  activities to perform or how to prioritize them, and (3) additional  monitoring or auditing activities, such as enhanced desk reviews or site  visits (see figure 1).

Fraud Risk Management and Related Guidance

Fraud and “fraud risk” are distinct concepts. Fraud—obtaining something  of value through willful misrepresentation—is challenging to detect  because of its deceptive nature. Fraud risk (which is a function of  likelihood and impact) exists when individuals have an opportunity to  engage in fraudulent activity, have an incentive or are under pressure to  commit fraud, or are able to rationalize committing fraud. Fraud risk  management is a process for ensuring program integrity by continuously  and strategically mitigating the likelihood and impact of fraud. When fraud  risks can be identified and mitigated, fraud may be less likely to occur.  Although the occurrence of fraud indicates there is a fraud risk, a fraud  risk can exist even if actual fraud has not yet been identified or  occurred.

According to federal standards and guidance, executive-branch agency  managers—including those at DOJ, OJP, and OJJDP—are responsible  for managing fraud risks and implementing practices for addressing those  risks. Federal internal control standards call for agency management  officials to assess the internal and external risks their entities face as they  seek to achieve their objectives. The standards state that as part of this  overall assessment, management should consider the potential for fraud  when identifying, analyzing, and responding to risks.

We issued our Fraud Risk Framework in July 2015. The Fraud Risk  Framework provides a comprehensive set of leading practices, arranged  in four components, which serve as a guide for agency managers  developing efforts to combat fraud in a strategic, risk-based manner.  The Fraud Risk Framework is also aligned with Principle 8 (“Assess  Fraud Risk”) of Standards for Internal Control in the Federal  Government. The Fraud Reduction and Data Analytics Act of 2015  requires agencies to establish financial and administrative controls that  are aligned with the Fraud Risk Framework’s leading practices. In  addition, guidance under Office of Management and Budget Circular A- 123 affirms managers should adhere to the leading practices identified in  the Fraud Risk Framework.

OJJDP Has Goals and Measures to Assess the Performance of Individual Programs, But Has Not Set Corresponding Performance Targets

OJJDP Has Established Goal Statements and Performance Measures for Individual Programs

OJJDP has established goal statements and performance measures for  individual programs. Goal statements broadly convey a program’s  overall intent, and performance measures assess program outputs or  outcomes. Figure 2 below shows current goal statements and  performance measures for the Youth Mentoring Program, for example.

OJJDP requires grantees to report on performance measures which are  listed in grant solicitations. OJJDP has designated 15 of its measures as  “core” measures that are generally applicable across most OJJDP  programs. OJJDP aggregates data from some of these core measures  across all applicable programs collectively to assess progress toward  office-level targets. Targets represent outputs or outcomes expressed as  a numeric goal. For example, OJJDP has office-level targets for the  percent of youth who offend and reoffend, as well as the percent of  grantees that have implemented an evidence-based program. OJJDP  obtains, reports on, and uses OJJDP program performance data, as  shown in figure 3 below.

OJJDP Is Taking Steps to Address Limitations in Grantee-Submitted Program Performance Data

We reviewed a selection of performance data that grantees provided to  OJJDP for three programs: Title II Formula Grant Program, Youth  Mentoring Program, and Gang Prevention Program. As a result of our  review—which covered data from October 2015 through December  2018—we determined that these data were not sufficiently reliable for the  purpose of providing examples of performance results in this report. For  example, we found that grantees reported inconsistent information, such  as different numbers of youth served for different performance measures  within the same data collection period. Further, some performance  measures double-count youth when presenting data by year. OJJDP  collects data from discretionary grantees during two 6-month reporting  periods and some measures are not designed to be aggregated across  time periods.

Nevertheless, we found that OJJDP was aware of most of the issues we  identified and is taking steps to improve the reliability of grantee- submitted data. For instance, beginning October 2017 (for formula grant  programs) and July 2018 (for discretionary grant programs), OJJDP  implemented a process to identify inconsistent or otherwise questionable  data and reach out to grantees for verification. Appendix II includes  information about the issues we found with the data we reviewed and  steps OJJDP is taking to improve the reliability of these data.

OJJDP Has Not Set Program-Level Targets, Which Limits Its Ability to Assess Progress toward Program Goals

While OJJDP has established goal statements and performance  measures, it has not set numeric targets by which it can assess progress  for each individual program. For example, one stated goal of the Title II  Formula Grant Program is to prevent youth already in the juvenile justice  system from reoffending. OJJDP’s annual Title II Formula Grant Program  performance reports state the percentage of youth who reoffended, the  performance measure for this goal; however, the reports do not provide a  target against which to evaluate whether the result reflects progress  toward the stated goal.

Further, while OJJDP has set several office-wide targets for all programs  collectively, these targets may not be appropriate for assessing the  progress of individual programs because programs vary in size. For  example, according to grantee-submitted performance data, the Title II  Formula Grant Program served approximately 100,000 youth in fiscal  year 2018, while the Gang Prevention Program served about 1,000 youth  in calendar year 2018. Thus, office-level targets, while useful, may be  more representative of the performance of OJJDP’s largest programs and  obscure the results of individual programs.

Since June 2019, OJJDP has been reviewing individual program goal  statements and performance measures as part of an OJP-wide review,  but this review does not include setting program-level targets. OJJDP  officials said that they have not set program-level targets for two reasons:  (1) there was uncertainty about whether OJJDP had the authority to do so  (versus OJP’s Office of the Chief Financial Officer), and (2) setting such  targets has not been a priority for OJJDP in the past, in part, due to a lack  of resources. In October 2019, OJP clarified that the OJJDP Administrator  has authority for OJJDP performance measurement, including setting  program-level and office-level targets. OJJDP officials also stated that  program oversight has recently become a higher priority and they plan to  bring on new staff; and they agreed with the need to set program-level  targets.

Tracking performance measures against established numeric targets is a  leading practice in performance management. Numeric targets establish  standards against which federal programs can measure progress towards  goals because comparisons can be easily made between projected  performance and actual results. Thus, updating program goal statements  and performance measures would be more effective with related numeric  targets. One goal of the OJP-wide review is to increase accountability for  achieving results. Setting program-level targets could help OJJDP meet  this goal by establishing a clear means by which progress toward goals  can be measured.

DOJ Has Taken Steps to Consider Fraud Risks Relevant to OJJDP Grant Programs, but Has Not Determined a Fraud Risk Tolerance

Tools Used in OJP’s Grant Monitoring and Compliance Auditing Efforts Provide Insight on Grant Risks Affecting OJJDP Grant Programs

According to OJP officials, some of the tools it uses in grant monitoring  and compliance auditing consider fraud risk affecting OJJDP grant  programs. As previously discussed, OJP’s grant monitoring and  compliance auditing broadly consist of three parts: (1) desk reviews that  occur annually, (2) risk assessments that assist officials in determining  what additional monitoring or auditing activities to perform or how to  prioritize them, and (3) additional monitoring or auditing activities, such as  enhanced desk reviews or site visits. To carry out these efforts, OJP  relies on various tools to assess the overall risk of grantees and awards.  Figure 4 provides additional information on the tools that provide insight  on fraud risks, according to OJP officials.

According to OJP officials, pre-award and ongoing risk assessment  processes that apply to all OJJDP grantees are the primary way in which  the office identifies fraud risks. The bullets below describe the nature of  the primary tools used during these risk assessment processes for all  OJJDP grantees and how they provide insights into potential fraud risks.

Financial Capability Questionnaire: This questionnaire includes 28  questions designed to provide insight on the financial systems and  internal controls a grantee has in place prior to receiving an award.  OJP developed the current version of this questionnaire in part as a  response to a 2013 DOJ OIG audit, according to OJP officials. All  applicants for OJJDP awards are required to fill out this questionnaire,  and new grantees’ pre-award risk scores and corresponding risk  levels rely, in part, on the applicants’ responses to it. These include  detailed questions related to the capabilities of the applicant’s  financial management system, such as whether it has the capability to  record expenditures by budget cost categories. According to OJP  officials, if an applicant’s accounting system cannot do so, the  opportunity for fraud increases because the commingling of funds  between budget categories would make it difficult to determine  whether federal funds were spent in accordance with the approved  budget. The questionnaire also includes items related to procurement,  travel policy, and subrecipient management and monitoring. These  questions may similarly indicate increased fraud risk depending on  how the applicant responds. For instance, according to OJP officials,  questions related to procurement are designed to determine whether  the applicant employs a fair, transparent, and competitive  procurement process. If an applicant’s procurement standards do not  meet these criteria, the likelihood of fraudulent activity may increase.

Grant Assessment Tool: The Grant Assessment Tool helps assess  open/active OJJDP awards and grantees against 38 risk criteria. OJP  officials identified 14 of these criteria as being indicators of potential  fraud risks, such as the results of recent audits, whether the award has  subawards or subcontracts, and whether grantees have completed  progress reports on time. Officials explained that progress report  delinquencies, for example, may be an indicator that a recipient does  not have adequate internal controls to handle federal awards, which  may provide a greater opportunity for fraud. The Grant Assessment  Tool generates a risk score and corresponding monitoring priority for  each open/active grantee and award quarterly. According to OJP  officials, Grant Assessment Tool criteria have evolved over time in  response to common audit or monitoring findings, as well as ongoing  coordination with the DOJ OIG, as discussed later.

For the Title II Formula Grant Program, grantees are also assigned risk  assessment scores and audited using the compliance auditing tools  described in the bottom half of figure 4. OJP officials stated that certain  responses to any of the questions in the tools used during compliance  auditing may indicate inadequate program management or weak internal  controls, which may increase the risk of fraudulent activity.

OJP officials use a variety of other tools during grant monitoring to  monitor each OJJDP grantee or award (see tools in the top half of figure  4). The tools used differ depending on the type and level of monitoring  being performed. According to OJP officials, all of the monitoring and  auditing tools include detailed questions that provide insight about the  strength of a grantee’s internal controls. In cases where a grantee is  unable to provide adequate documentation in response to these  questions, OJP officials stated that they may have weak internal controls  which may increase the risk of fraudulent activity.

OJP regularly coordinates with the DOJ OIG on issues related to fraud  risk affecting OJJDP grant programs through meetings, trainings, and  reviews of OIG audits. Specifically, according to officials, staff from OJP’s  Office of Audit, Assessment, and Management meet with the OIG two to  three times per year to discuss fraud allegations and ongoing fraud  investigations related to OJJDP grant programs. These discussions  assist OJP officials in determining their monitoring or auditing plans, but  also provide insights about the types of issues that are being referred to  the OIG for investigation, which can then inform needed changes to  monitoring and auditing tools, according to OJP officials. The OIG also  provides training to OJP staff every other year on how to identify and  report potential fraud. Additionally, Office of Audit, Assessment, and  Management officials regularly review OIG audit findings pertaining to  individual grantees and compare them to grant monitoring findings for the  same grantees. According to OJP officials, the purpose of these reviews  is to evaluate where OJP may be able to improve its monitoring  processes. Officials stated that one example of an outcome from such a  review is the previously discussed Financial Capability Questionnaire.

DOJ Has Not Determined the Department’s Fraud Risk Tolerance

According to JMD officials, the department is implementing fraud risk  management requirements through an iterative process that will be  completed over multiple years, which leverages the department’s overall  ERM processes. Leading practices for planning and conducting fraud  risk assessments acknowledge that assessing fraud risks is an iterative  process. According to JMD officials, as a result of this iterative  approach, the fraud risk assessments JMD conducted in fiscal years 2017  and 2018 did not fully align with selected leading practices in the Fraud  Risk Framework, which include documenting a fraud risk profile. Although  the Department continued to implement additional leading practices from  the Fraud Risk Framework by developing a fraud risk profile as part of the  fiscal year 2019 fraud risk assessment, the fraud risk profile did not  include any consideration of DOJ’s fraud risk tolerance—another leading  practice identified by the Fraud Risk Framework.

In fiscal year 2017, JMD conducted an assessment of fraud risks that  consisted of four facilitated discussion groups with relevant officials.  One discussion group was tailored to the specific concerns of grant  programs—which included grant programs managed by OJJDP—as  recommended by leading practices of the Fraud Risk Framework.  During the grant-focused facilitated discussion group, participants  identified five fraud scenarios relevant to DOJ grant programs, such as  misdirection of funds, which occurs when a recipient deliberately  misdirects funds in a manner inconsistent with the purpose outlined in the  award agreement. After participants reached consensus on the fraud  scenarios, they then ranked the inherent risk of each scenario, in  alignment with leading practices of the Fraud Risk Framework.

DOJ officials also told us that participants ranked residual risk using a  voting tool, which required the participants to understand the various  management controls in place to address a particular fraud risk.  However, the documented outcomes of these discussions did not identify  specific fraud risk controls or the extent to which those controls mitigate  specific fraud risks, and as a result, it is unclear how these residual risk  values were determined. Further, according to officials responsible for  managing the fiscal year 2017 effort, the facilitated discussion group did  not determine a specific and measurable fraud risk tolerance for DOJ  grant programs generally nor discuss specific fraud risk management  activities or controls for any specific grant programs. As discussed in  greater detail later in this report, leading practices for fraud risk  management state that managers should determine a fraud risk tolerance  and Standards for Internal Control in the Federal Government states that  managers should define risk tolerances in specific and measurable terms  so they are clearly stated and can be measured.

The fiscal year 2018 effort to assess fraud risk at DOJ consisted of a brief  survey about fraud-related issues that was distributed to all DOJ  components and was more limited in nature than the fiscal year 2017  effort. Specifically, the fiscal year 2018 questionnaire did not ask the  components to identify inherent or residual risks, as was done in the prior  year. Further, for the grants category, JMD included four of the five fraud  risk areas that were addressed in fiscal year 2017. However, the fiscal  year 2018 questionnaire did specifically ask components to identify key  fraud risk management activities designed to prevent, detect, or respond  to fraud, which had not been part of the fiscal year 2017 effort. Including  information about control activities indicates additional maturation of  JMD’s fraud risk management activities, but the questionnaire did not ask  components to consider the extent to which these control activities  mitigate the likelihood and impact of risk—as recommended by leading  practices of the Fraud Risk Framework. The questions asked in fiscal  year 2018 are shown in table 2 below.

According to JMD officials, the components’ responses to this  questionnaire were summarized for internal purposes, but no additional  analysis or work, such as defining a fraud risk tolerance and documenting  a fraud risk profile, was completed for any of the categories listed in table  2. Further, JMD officials stated that the department does not believe a  full-scale fraud risk assessment is warranted annually, and the fiscal year  2018 effort was designed to build on the prior year’s assessment.

According to officials, this is part of the department’s iterative approach to  implementing fraud risk management requirements, which, consistent  with leading practices, may not necessarily incorporate all relevant  leading practices in each iteration.

In December 2019, JMD officials provided the final summary of the fiscal  year 2019 fraud risk assessment, which included a fraud risk profile as  recommended by leading practices. According to JMD officials, to  conduct the fiscal year 2019 fraud risk assessment, JMD officials first  created a fraud risk profile template using information from the 2017 and  2018 assessments. Figure 5 shows an excerpt of JMD’s draft fraud risk  profile template, as of September 2019.

After senior leadership reviewed the pre-populated template, JMD held a  facilitated discussion with representatives from each component to  evaluate the risk information presented in the template for each topic  area. JMD identified several risks for each topic area, including the  same five risks for the grants area that were identified in fiscal year  2017.

Based on information provided, the fiscal year 2019 fraud risk  assessment and resulting fraud risk profile incorporate many leading  practices of the Fraud Risk Framework. These include consideration of  inherent fraud risk, current fraud risk controls and their suitability (the  extent to which control activities mitigate the likelihood and impact of risk),  and residual fraud risk. However, the 2019 fraud risk profile did not  determine a measurable fraud risk tolerance, or prioritize residual risk  against that tolerance for any of the assessed categories, including  grants. Managers’ defined risk tolerance may depend on the  circumstances of individual programs and other objectives beyond  mitigation of fraud risks. Leading practices for fraud risk management  state that managers should define a fraud risk tolerance, examine the  suitability of existing fraud controls, and then prioritize residual fraud  risks. In doing so, managers should consider the extent to which  existing control activities mitigate inherent risks and whether the  remaining risks exceed managers’ tolerance. Based on this analysis and  the defined risk tolerance, managers then rank residual risks in order of  priority, and determine their responses, if any, to mitigate those risks that  exceed their risk tolerance.

JMD officials stated that they did not yet define the department’s fraud  risk tolerance for any of the assessed categories because they view it as  the next step in the maturation of DOJ’s fraud risk assessment processes.  However, JMD did not provide details or documentation of its plans to  develop a specific and measurable fraud risk tolerance for the next  iteration of their fraud risk assessments. Although following an iterative  approach to fraud risk management is consistent with leading practices,  until DOJ defines a measurable fraud risk tolerance for the assessed  categories, the department may not effectively allocate limited resources  to address fraud risks—including those associated with OJJDP grant  programs. Specifically, by determining a measurable fraud risk tolerance  for the grants category and assessing identified residual fraud risks  against that tolerance to prioritize these risks, the department will help  ensure that OJJDP’s grant programs are not vulnerable to greater risks  than DOJ is willing to tolerate. Doing so will also provide assurance that  OJJDP does not unintentionally over-allocate limited funding to address  fraud risks the department is willing to tolerate.

Plans for Conducting Future Fraud Risk Assessments

According to JMD officials, they are in the process of awarding a contract  that will result in an implementation plan for addressing fraud risk  management requirements in the future. Specifically, in July 2019, JMD  released a Request for Quotes for a Blanket Purchase Agreement in  support of DOJ’s implementation of OMB Circular A-123. One of the  deliverables JMD expects to order under this agreement is an  implementation plan for addressing fraud risk management requirements,  which will include developing a plan for conducting regular fraud risk  assessments consistent with leading practices for fraud risk management.  According to officials, they expect to award the agreement by the end of  calendar year 2019, after which the contractor will perform task orders  issued by JMD that will include details related to the methodology,  timeframes, and staffing associated with each deliverable. Because  neither the award nor the task orders were in place at the time of our  review, we cannot determine whether DOJ’s planned efforts will fully align  with the leading practices of the Fraud Risk Framework, but we will  continue to monitor DOJ’s efforts during related ongoing work.

In fiscal year 2018, OJJDP made 295 awards totaling over $290 million to  support programs intended to ensure youth are held appropriately  accountable and empower youth to live productive lives. Performance  measurement helps ensure funding achieves such outcomes and fraud  risk management helps ensure funding is not improperly diverted from  this intended purpose. Both of these management principles facilitate  stewardship and accountability for federal funds.

Since June 2019, OJJDP has been reviewing and updating goal  statements and performance measures for individual programs. While  OJJDP has office-level targets, it does not have program-level targets.  Program-level targets (specific numeric goals) would help OJJDP assess  progress toward individual program goals and increase accountability for  achieving positive outcomes.

Over the past few years, DOJ has taken steps to consider fraud risk for all  DOJ grants, including OJJDP’s. Determining a fraud risk tolerance—and  assessing residual fraud risk against that tolerance to prioritize these  risks—would help OJP calibrate resources to address grant fraud risk for  OJJDP programs, helping ensure that resources are not under- or over- allocated.

We are making two recommendations, including one to OJJDP and one  to JMD. Specifically:  The OJJDP Administrator should set performance targets for individual  grant programs. (Recommendation 1)

The Assistant Attorney General for Administration should ensure that  future department-level fraud risk profiles (1) determine the department’s  fraud risk tolerance for DOJ grants—which include OJJDP grant  programs, and (2) prioritize residual fraud risks based on an assessment  against that tolerance, consistent with leading practices in GAO’s Fraud  Risk Framework. (Recommendation 2)

We provided a draft of this report to DOJ for review and comment. In an  email message, an official within JMD stated that the Department  concurred with our recommendations. In written comments provided by  OJP, which are reproduced in appendix V, the agency concurred with our  recommendation that it set performance targets for individual OJJDP  programs. Specifically, OJP stated that the OJJDP Administrator will set  performance targets, to the extent practical, for OJJDP's current and new  grant programs. Further, the OJJDP Administrator will ensure that the  performance targets are reviewed annually. We believe this action, if  implemented, would address our recommendation. DOJ also concurred  with our second recommendation to include a fraud risk tolerance for DOJ  grants in future department-level fraud risk profiles, but did not provide  details as to how they will address it.  DOJ also provided technical  comments, which we have incorporated as appropriate.

We are sending copies of this report to the appropriate congressional  committees, the Attorney General, and other interested parties. In  addition, the report is available at no charge on GAO’s website at  https://www.gao.gov.

If you or your staff have any questions about this report, please contact  Gretta Goodwin at (202) 512-8777 GoodwinG@gao.gov or Rebecca  Shea at (202) 512-6722 or SheaR@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made major contributions to  this report are listed in appendix VI.

Appendix I: Office of Juvenile Justice and Delinquency Prevention Programs and Grants in Fiscal Year 2018

Under the Department of Justice Appropriations Act, 2018, the Office of  Juvenile Justice and Delinquency Prevention (OJJDP) administered 16  grant programs in fiscal year 2018, corresponding with a total of 53 grant  solicitations and 295 awards. OJJDP issued certain grant solicitations to  carry out the purposes of more than one program. Table 3 shows  OJJDP’s fiscal year 2018 programs and associated information, including  the primary source of authority for each program, as identified by OJJDP.

Appendix II: GAO Review of Performance Data and Steps Being Taken to Improve Data Reliability

To provide examples of performance results from three of the Office of  Juvenile Justice and Delinquency Prevention’s (OJJDP) 16 programs  funded in fiscal year 2018, we assessed the reliability of key data  elements and methods used to calculate selected performance  measures. We determined that data grantees submitted to OJJDP for the  programs we selected were not sufficiently reliable for this purpose.  Nevertheless, we present rounded numbers of “youth served” for two of  the programs we selected (2018 data only), as we found these data to be  reliable for the specific purpose of illustrating differences in the relative  sizes of the programs. This appendix discusses our approach for  selecting programs to review, identifying the timeframe of review, and  selecting performance measures to review. It also discusses issues we  found with the data we reviewed and steps OJJDP is taking to improve  the reliability of data submitted by grantees.

We selected three of OJJDP’s 16 programs funded in fiscal year 2018  and listed in appendix I. Specifically, we selected the Title II Formula  Grant Program because it is OJJDP’s only formula grant program and  because it is a comprehensive nationwide program. We selected the  Youth Mentoring and Gang Prevention programs from among the 15  discretionary grant programs as a result of the following process: we  ranked programs by total amount awarded during fiscal years 2016  through 2018 and randomly selected one program from the top 50  percent and one from the bottom 50 percent, and we excluded programs  that we had recently reviewed in other reports—such as the Tribal Youth  and Victims of Child Abuse programs.

We obtained performance data for the selected programs from OJJDP  from October 2015 through December 2018. We chose this timeframe  because 2018 was the latest full calendar year for which data were  available, and looking back three years was sufficient to capture  variations in the programs’ funding levels.

We selected seven of OJJDP’s 15 core performance measures for  review. Specifically, we selected measures that focus on youth and that  objectively measure short-term outputs or outcomes, as follows: number of program youth served number and percent of youth with whom an evidence-based program number and percent of youth completing program requirements number and percent of program youth who offend (short-term) number and percent of program youth who re-offend (short-term) number and percent of program youth who are victimized (short-term); number and percent of program youth who are re-victimized (short- term)

Not all of these performance measures are applicable to all programs.  Performance measures that assess the number and percent of program  youth who offend, re-offend, and are victimized measure both short-term  and long-term outcomes, and we focused only on short-term because of  the challenge that grantees face in tracking youth after they exit  programs, as explained to us by OJJDP officials.

We reviewed relevant performance measures, their definitions, and the  methodology for calculating them. We reviewed data cleaning and  validation procedures that OJJDP uses to verify data provided by  grantees. We also tested grantee-submitted data provided to us by  OJJDP for missing data, outliers, and inconsistencies. For example, we  tested for illogical values, such as different numbers of youth served  within the same time period, or numerators that were higher than the  denominators (i.e. more youth successfully exiting the program than the  total number exiting). Finally, we interviewed knowledgeable OJJDP  officials and contractors on several occasions.

We determined that the data were not sufficiently reliable for the purpose  of providing examples of program performance results due to several  limitations (bulleted below). Although not every limitation applies to every  program we reviewed, the number and significance of the collective  issues identified led us not to use the data as examples of performance  results in this report.

Performance measure results for discretionary programs are  often double-counted when presenting data by year. OJJDP  collects data from discretionary grantees every 6-months; however,  some performance measures are not designed to be aggregated  across time periods. OJJDP acknowledges that aggregating data from  two, 6-month time periods often results in double-counting when  presenting annualized data.

Some grantees report inconsistent numbers of youth served. For  instance, in the Gang Prevention program, the total number of youth  served from January through June 2018 was reported as 387 and 267  for different performance measures in the same data set (within one  reporting period). We also identified instances where grantees  reported inconsistent numbers of youth across reporting periods.

Some grantees do not respond to requests to verify questionable  data. According to OJJDP officials grantees may lack resources or  staff capacity to collect data and track youth’s outcomes. As a result,  some grantees may submit incorrect data or submit data after the  reporting deadline. Beginning October 2017 (for formula grant  programs) and July 2018 (for discretionary grant programs), OJJDP  implemented a process whereby contractors flag inconsistent or  otherwise questionable data and reach out to grantees for  clarification. However, according to officials, grantees do not face any  consequences if they do not respond to requests from OJJDP for data  verification. For instance, for Youth Mentoring program data submitted  by grantees for July through December 2018, OJJDP contractors  flagged 122 of 630 grantees as having potentially inaccurate data, but  subsequently received responses from only 74 of the 122 grantees.

Inconsistent performance measure definitions. We found  inconsistencies in the definitions for the following performance  measures: (1) number and percent of program youth who offend  (short-term), and (2) number and percent of program youth who re- offend (short-term). For both measures, one documented definition  states that they apply only to youth who offend or re-offend during the  reporting period, and another documented definition states that it also  applies to youth who exited the program 0-6 months ago.

Reporting on a subset of youth not representative of all program  youth. OJJDP uses the number of youth “tracked” as the denominator  for several of its performance measures, including the number of  youth who offend and reoffend. According to OJJDP, the number of  youth tracked for the offend and reoffend measures should ideally be  the same number as total youth served by a program. However, the  number of youth tracked is usually lower than the number of youth  served. As a result, the measure often reflects a subset of youth that,  according to one official, may have characteristics that are not well  understood—such as youth or families who are more willing to be  tracked because they have not offended recently—and thus may  skew the results.

OJJDP Steps to Improve Reliability of Grantee- Submitted Data

OJJDP acknowledges there are concerns with the quality of the grantee- reported performance measures data. According to OJJDP officials, the  limitations are the result of several challenges which they are addressing:

Replacing outdated data collection tool. Officials said their current  data collection tool is outdated and can be unwieldy and confusing for  grantees. Along these lines, the tool only includes automated error  checks for a limited amount of data fields and does not include an  auto-populate feature, which would prevent grantees from entering  illogical or inconsistent data. However, according to Office of Justice  Program (OJP) officials, as of October 2019, OJP is designing a new  data collection tool for all OJP components—including OJJDP—that  will include automated error checks and an auto-populate feature, and  they plan to implement this tool beginning in October 2020.

Updating performance measures. Officials said that some  performance measures are also outdated, such as those that result in  duplication when reported annually. Officials also said that some  performance measures are confusing to grantees. Nevertheless, as  part of an ongoing OJP-wide review of performance measures,  OJJDP is in the process of reviewing and updating all OJJDP  performance measures and plans to provide updated definitions and  instructions to grantees.

Increasing grantee response rate for data verification. To increase  grantee response rates to data verification requests, OJJDP reported  that it is exploring possible consequences for grantees if they do not  respond, such as increased scrutiny by OJJDP staff who oversee  awards or temporary withholding of funds until verifications are  submitted. According to officials, whatever approach (or approaches)  they decide on, they will implement them by March 2020.

Appendix III: Selected Oversight Requirements Related to the Title II Formula Grant Program

The Office of Juvenile Justice and Delinquency Prevention (OJJDP)  within the Department of Justice is responsible for administering grant  programs under the Juvenile Justice and Delinquency Prevent Act of  1974. One of these programs, the Title II Formula Grant Program,  authorizes the award of formula grants to states to develop programs for  juveniles and improve their juvenile justice systems. On December 21,  2018, the Juvenile Justice Reform Act of 2018 (“Reform Act”) enacted  amendments to the Title II Formula Grant Program, including new  accountability and oversight requirements for grantees and OJJDP. The  amendments were not effective until the fiscal year 2020 grant award  cycle and did not apply to the period of performance we evaluated for this  report, which was through fiscal year 2018. Table 4 summarizes the  accountability and oversight requirements now in effect for the Title II  Formula Grant Program and the status of OJJDP’s efforts to implement  them.

The Reform Act also requires several evaluations and assessments to  help strengthen OJJDP’s internal controls and identify fraud, waste or  abuse in its programs. Table 5 summarizes selected oversight  requirements related to the Title II Formula Grant Program.

Appendix IV: Compliance Auditing Applicable to the Title II Formula Grant Program Prior to Fiscal Year 2020

The Title II Formula Grant Program Prior to Fiscal Year 2020

The Title II Formula Grant Program—so called because it was authorized  by Title II of the Juvenile Justice and Delinquency Prevention Act of 1974  (JJDPA)—is a state formula grant program, administered by the Office of  Juvenile Justice and Delinquency Prevention (OJJDP). The program has  been amended several times since 1974—most recently, by the Juvenile  Justice Reform Act of 2018 (“Reform Act”), which also called for this  evaluation of OJJDP’s performance. The performance data we reviewed  (which covers Title II Formula Grants from October 2015 through  September 2018) corresponds with statutory requirements in effect at that  time, not the current requirements, as amended by the Reform Act, which  apply to grant awards made in fiscal year 2020 and subsequent fiscal  years. To be consistent with the data we reviewed, this appendix  presents information on program requirements that applied prior to fiscal  year 2020. Because these requirements are no longer current, we will  differentiate them from those that are by citing the superseded edition of  the U.S. Code in which they appear—(2012 & Supp. V 2018)—in  comparison to the 2018 Main Edition (2018), which contains the  provisions now in force.

The term “compliance auditing” refers to OJJDP’s process for (1) auditing  the compliance monitoring systems used by states, and (2) evaluating  states’ compliance with four core requirements specified in law. During  the grant application process, the four core requirements are among  several (previously 28, now 33) that a state’s 3-year plan must satisfy for  the state to be eligible for award. However, unlike the other eligibility  requirements, the four core requirements can trigger a reduction to a  state’s grant allocation unless the state maintains compliance during  performance. States must provide adequate systems of monitoring their  compliance with three of the four core requirements—i.e. those related to  when and where juveniles may be detained in detention or correctional  facilities—and OJJDP must audit the adequacy of states’ compliance  monitoring systems. OJJDP must also determine whether states  maintained compliance with each of the four core requirements and, if  not, OJJDP must reduce the state’s allocation the following fiscal year by  at least 20 percent for each core requirement that the state failed to  meet.

During the period covered in our review (i.e., prior to fiscal year 2020), the  four core requirements subject to compliance auditing were:  1.  Deinstitutionalization of status offenders—which prohibits states  from using secure detention or correctional facilities to hold juveniles  charged with status offenses (except for a listed few). This  requirement also applies to juveniles not charged with an offense but  who enter the justice system as aliens or as dependent, neglected or  abused youths.  2.  Separation of juveniles from adult inmates—which prohibits a state  from detaining or confining juveniles protected by the  deinstitutionalization of status offenders requirement (see above), or  juveniles who are alleged or found to be delinquent, in any institution  where they have contact with adult inmates.  3.  Removal of juveniles from adult jails and lockups—which prohibits  a state from detaining or confining juveniles in adult jails or lockups,  except in limited circumstances and for specified periods of time, and  only if the juvenile has no contact with adult inmates.   4.  Addressing disproportionate minority contact—which requires a  state to address the disproportionate number of minority youth who  come into contact with the juvenile justice system.

OJJDP’s compliance auditing process during the majority of the period of  our review is set forth in a 2017 OJJDP policy document.  According to  this policy, OJJDP conducts a comprehensive assessment and makes a  determination whether the state is in compliance with each of the four  core requirements. The comprehensive assessment includes verification  of the data submitted, an analysis of the data submitted by the state to  evaluate compliance with each of the four core requirements, and a  review to assess the adequacy of internal controls over the state’s  compliance monitoring process for collecting and reporting compliance  monitoring data.

According to this policy, the OJJDP Administrator issues correspondence  annually regarding final compliance determinations. These determinations  include, as necessary, specific details regarding why a state was  determined to be out of compliance with any of the four core requirements  or the required compliance monitoring system. Per the policy, a state’s  formula grant funding will be reduced by 20 percent for each core  requirement with which OJJDP has determined a state to be out of  compliance. Additionally, if OJJDP determines that the state has an  inadequate system of monitoring, the state may have receipt of its  formula grant funding withheld or may be deemed ineligible for a formula  grant award.

Finally, according to this policy, OJJDP conducts field audits on a rotating  schedule. The purpose of the field audits is to confirm state compliance  monitoring activity and practices through direct onsite observation and file  review, and to identify needed areas for technical assistance. OJJDP  anticipates, with available funding and resources, that every state will  receive a field audit every three years.

Appendix V: Comments from the Office of Justice Programs

Appendix VI: GAO Contact and Staff Acknowledgements

In addition to the contacts named above, Tonnyé Conner-White  (Assistant Director), Jonathan Oldmixon (Assistant Director), Jeff Jensen  (Analyst-in-Charge), James Ashley, Dominick Dale, Christine Davis,  Caroline DeCelles, Elizabeth Dretsch, Eric Hauswirth, Elizabeth  Kowalewski, Ben Licht, Jan Montgomery, Heidi Nielson, and Abby Volk,  made key contributions to this report.

The Department of Justice’s Office of Juvenile Justice and Delinquency Prevention awarded over $290 million through 16 grant programs in FY 2018 to help juveniles in the justice system.

We found that this office only has numeric goals (targets) for what it hopes to achieve across its programs collectively. For instance, the office has a target for the percent of youth who offend and reoffend across all applicable grant programs. However, it doesn’t have similar targets for its individual grant programs.

We recommended (among other things) that this office set targets for individual grant programs to assess their performance.