Medicaid: Additional CMS Data and Oversight Needed to Help Ensure Children Receive Recommended Screenings

The EPSDT benefit is key to ensuring that Medicaid beneficiaries aged 20 and under receive periodic screening services, such as well-child screenings, and diagnostic and treatment services, such as physical therapy and eyeglasses, to correct or ameliorate conditions discovered during a screening.

GAO was asked to examine the extent to which Medicaid beneficiaries aged 20 and under receive health care services under the EPSDT benefit. Among other things, GAO examined (1) what is known about the provision of EPSDT services based on CMS-required annual state reporting, and (2) CMS oversight of the EPSDT benefit. To do this, GAO analyzed annual state reporting data from fiscal years 2010 through 2017, the most current year data were available; CMS documentation; and federal internal control standards. GAO also interviewed CMS officials and Medicaid officials from 16 states selected, in part, on the variation in number of beneficiaries and geographic diversity.

Approximately half of all Medicaid beneficiaries aged 20 and under received screenings and services recommended under the Early and Periodic Screening, Diagnostic, and Treatment (EPSDT) benefit in fiscal year 2017, but nearly as many did not. For example, GAO's analysis of state-reported data found that about 59 percent of all beneficiaries (20.2 million) who should have received at least one recommended well-child screening received one. About 48 percent of beneficiaries aged 1 to 20 (18.3 million) received a preventive dental service in fiscal year 2017. Older beneficiaries tended to have lower rates of screening.

Number of Medicaid Beneficiaries Receiving and Not Receiving Well-Child Screenings in Fiscal Year 2017, by Age Group

The Centers for Medicare & Medicaid Services (CMS), the agency that oversees Medicaid, including EPSDT, has taken steps to improve the quality of information that states report about the provision of EPSDT services. CMS has also set some EPSDT performance measure targets for states; yet, the agency has not taken other steps to oversee the EPSDT benefit, such as

collecting the data necessary to evaluate whether states are complying with CMS's policy for beneficiaries to receive a blood lead screening;

taking action, as needed, based on assessments of the appropriateness of some performance measures, such as well-child screening measures; and

using state-reported information to regularly evaluate states against CMS's EPSDT targets, or assisting states in planning improvements to meet the targets.

Absent these steps, CMS's oversight is limited and beneficiaries may not be receiving appropriate EPSDT services when they need them.

GAO is making six recommendations to CMS regarding its oversight of the EPSDT benefit, including collecting appropriate blood lead screening data; taking action, if needed, after assessing the appropriateness of performance measures and targets for EPSDT; and evaluating states' performance in meeting CMS's EPSDT targets. CMS agreed with three recommendations, but disagreed with three others regarding performance measures and targets. GAO maintains that these recommendations are valid, as discussed in this report.

The Early and Periodic Screening, Diagnostic, and Treatment (EPSDT)  benefit is key to ensuring that eligible beneficiaries aged 20 and under  receive appropriate services under Medicaid, a joint federal-state health  care program for low-income and medically needy individuals. The  EPSDT benefit is a comprehensive set of covered services for Medicaid’s  youngest beneficiaries that includes periodic screening services, such as  physical exams, and diagnostic and treatment services, such as physical  therapy and eyeglasses, among other services. In fiscal year 2017,  approximately 40 million Medicaid beneficiaries were entitled to receive  ESPDT services. However, we and the Department of Health and Human  Services (HHS) Office of Inspector General have previously found that  millions of beneficiaries had not received the services to which they were  entitled.

The Centers for Medicare & Medicaid Services (CMS), an agency within  HHS, is responsible for overseeing Medicaid at the federal level, including  the EPSDT benefit. To help inform its oversight of the EPSDT benefit,  CMS relies, in part, on several data sets submitted by states. States are  required to report annually on the provision of certain EPSDT services  through the Form CMS-416. States may also voluntarily report  information to CMS annually on health care services provided to EPSDT  beneficiaries, as well as individuals covered under the Children’s Health  Insurance Program (CHIP) through the Child Core Set. According to  CMS, the agency will increasingly rely on the Child Core Set to measure  health care outcomes for Medicaid and CHIP beneficiaries, rather than  the CMS-416. Finally, as part of a separate effort conducted jointly with  states, CMS has sought to improve the quality and usefulness of state- reported Medicaid data through its Transformed Medicaid Statistical  Information System (T-MSIS). CMS intends for T-MSIS to provide more  information to improve Medicaid oversight and to reduce the number of  reports CMS requires states to submit, including the CMS-416 and the  Child Core Set.

You asked us to review the extent to which Medicaid beneficiaries aged  20 and under receive health care services under the EPSDT benefit. In  this report, we examine  1.  what is known about the provision of EPSDT services in all states  according to what states report on the CMS-416;  2.  CMS oversight of the EPSDT benefit; and  3.  what is known about the capabilities of T-MSIS data to replace states’  CMS-416 and Child Core Set reporting.

Our report also describes state practices to promote and facilitate the  delivery of EPSDT services. (See app. I.)

To examine what is known about the provision of EPSDT services in all  states according to what states report on the CMS-416, we analyzed data  states reported to CMS through the CMS-416 from fiscal year 2010, the  year in which the current reporting template was implemented, through  2017, the most recent year of data available at the time of our reporting.  Specifically, we analyzed data on the three primary ESPDT services  reported on the CMS-416: (1) well-child screenings, (2) preventive dental  services, and (3) blood lead screenings. For our analysis, we calculated  the following, both nationally and for each state, using the fiscal years and  age groups that CMS uses for oversight:  the number and percent of beneficiaries recommended to receive at  least one well-child screening who received at least one screening  from fiscal years 2010 through 2017, both overall and for each of the  seven age groups reported on the CMS-416;  the number and percent of beneficiaries aged 1 to 20 who received  preventive dental services from fiscal years 2011 through 2017; and  the number of screenings conducted for blood lead levels for  beneficiaries aged 12 through 24 months in fiscal year 2017 using the  1 to 2 age group on the CMS-416.

We did not independently verify the accuracy of state-reported CMS-416  data; however, we checked those data for obvious errors and omissions,  compared analysis results with CMS’s publicly reported data about  EPSDT services, and communicated with CMS officials to resolve any  identified discrepancies. We also reviewed written guidance and  documents from CMS and interviewed CMS officials about the collection  and reliability of CMS-416 data. On this basis, we determined that these  data were sufficiently reliable for the purpose of our reporting objective.

To examine CMS oversight of the EPSDT benefit, we reviewed guidance  from CMS to states on the EPSDT benefit, CMS’s process for reviewing  EPSDT data on the CMS-416 and Child Core Set, and CMS summary  reports about EPSDT performance measures. We also interviewed  CMS officials about their oversight activities, including reviewing data,  setting and monitoring EPSDT-related performance measure targets, and  assisting states with meeting EPSDT targets. We compared these efforts  to best practices for results-oriented management as identified in  previous GAO work, and assessed them against federal standards for  internal control. We also selected a non-generalizable sample of 16  states and interviewed Medicaid officials in these states to obtain  information on data reporting and reliability, communications with CMS  about EPSDT oversight, and leading practices officials identified for  providing EPSDT services in their states. To obtain state variation, the  16 states were selected on the basis of (1) the high and low number of  Medicaid beneficiaries aged 20 and under, (2) the high and low number of  and percent of beneficiaries recommended to receive at least one well- child screening who receive at least one screening, (3) variation in the  way they deliver EPSDT services, (4) variation in whether they  participated in a program with CMS to report EPSDT information through  T-MSIS, and (5) their geographic diversity.

To examine what is known about the capabilities of T-MSIS data to  replace states’ CMS-416 and Child Core Set reporting, we interviewed  CMS officials about T-MSIS data accuracy and completeness, and CMS’s  plans for using T-MSIS data to replace state reporting of the CMS-416  and the Child Core Set. We also reviewed the results of two sets of pilot  studies CMS conducted to assess the extent to which T-MSIS could be  used to replicate certain parts of the CMS-416 and the Child Core Set.  In addition, we interviewed knowledgeable Medicaid officials in our  selected states regarding T-MSIS data quality and their interactions with  CMS about T-MSIS. We assessed CMS’s efforts to develop a timeline  with interim milestones for when T-MSIS will replace state reporting of the  CMS-416 or Child Core Set against federal standards for internal control.

We conducted this performance audit from March 2018 to August 2019 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

Federal law specifies that the EPSDT benefit covers screening, vision,  dental, and hearing services, as well as other Medicaid coverable  services that are necessary to correct or ameliorate any conditions  discovered through screening. The EPSDT benefit generally entitles  beneficiaries to these services regardless of whether such services are  covered in a state’s Medicaid state plan and regardless of any restrictions  that the state may impose on coverage for adult services. The EPSDT  screening component includes a wide range of preventive services, such  as comprehensive child health assessments known as well-child  screenings and age-appropriate blood lead screenings. Because EPSDT  covers any medically necessary service that could be covered for adults  in addition to the specified preventive screenings, the EPSDT benefit is  generally more comprehensive than the benefits provided for adult  beneficiaries.

The federal government and states jointly share responsibility for  implementing the EPSDT benefit. CMS, as part of its Medicaid oversight  responsibilities, approves state Medicaid plans, which describe how the  state administers its Medicaid program, including components related to  the provision of EPSDT services. CMS also develops and issues general  guidance to states about the EPSDT benefit, such as explanations of  covered services and strategies for providing those services.

Additionally, CMS has developed a goal for EPSDT, which is to assure  that beneficiaries get the health care they need when they need it: the  right care to the right child at the right time in the right setting. Further,  CMS established performance measures, some with associated targets,  to guide states’ implementation of EPSDT. For example, CMS set  performance measures and performance measure targets as part of its  Oral Health Initiative. CMS developed the performance measure targets  to carry out statutory requirements, quality improvement efforts, and  agency policy. (See table 1 for EPSDT performance measures that have  associated targets.)

States have flexibility, within federal parameters, to determine how  EPSDT services are provided. For example, states are required to ensure  that Medicaid-eligible beneficiaries and their families are aware of the  EPSDT benefit and have access to required services, but states can  choose whether to administer the benefit themselves or to oversee  managed care organizations that are contracted to provide the benefit.  States may also determine the frequency of screening services and  communicate them through periodicity schedules that meet federal  requirements.

CMS uses various sources of information to oversee the EPSDT benefit,  such as the CMS-416, the Child Core Set, and the Medicaid and CHIP  Scorecard.

States report information about the provision of select ESPDT services to  CMS annually through the CMS-416 and measures on the Child Core  Set. The CMS-416 provides CMS with basic information about EPSDT  services, such as the participant ratio and number of beneficiaries  receiving a preventive dental service. It includes the information  necessary for CMS to assess states’ performance on the participant ratio  and the screening ratio, among other things. The agency then can  compare performance on the two ratios with the agency’s ESPDT  performance measure targets. The Child Core Set provides CMS with  information about the quality of health care provided to Medicaid and  CHIP beneficiaries, and supports state efforts to improve health care quality and health outcomes. Child Core Set reporting becomes  mandatory on an annual basis beginning with the state reports on fiscal  year 2024. As of 2019, the Child Core Set included performance  measures related to the provision of EPSDT services, such as well-child  visits in the first 15 months of life. Because reporting is currently  voluntary, states vary in the number of performance measures they  choose to report. In fiscal year 2017, for example, 50 states and the  District of Columbia voluntarily reported on at least one of the 27 Child  Core Set performance measures, with states reporting a median of 18  Child Core Set performance measures. (See app. II for the information  reported in the CMS-416 and Child Core Set.) As shown in table 2, there  are both similarities and differences between the CMS-416 and Child  Core Set.

Since the Child Core Set performance measures include CHIP  beneficiaries who may not be entitled to the EPSDT benefit, data from the  Child Core Set are not directly comparable with reporting on the CMS- 416. In addition, CMS-416 data cover a longer period of time, as they  are available from 1995, while Child Core Set data are available from  2011. CMS officials said that having more years of CMS-416 data helps  identify trends in the provision of EPSDT services over a longer period of  time than possible with the Child Core Set. On the other hand, CMS  officials said it is difficult to compare states’ performance using the CMS- 416, because some performance measures are based on periodicity  schedules, which vary state-to-state and over time. In contrast, the Child  Core Set allows for more consistency in comparing data across states,  because each state is expected to calculate performance measures in the  same way.

In June 2018, CMS published the first Medicaid and CHIP Scorecard,  which includes performance measures about the provision of services to  Medicaid and CHIP beneficiaries. The scorecard includes 17  performance measures related to the provision of EPSDT services, six of  which are performance measures from the Child Core Set—and one of  these six measures is derived from the CMS-416. In January 2019,  CMS officials reported that the scorecard will be used to provide  increased transparency about state Medicaid program administration and  beneficiary health outcomes, and drive health care quality improvement  across states.

According to CMS officials, CMS envisions that the scorecard will be  strengthened as state reporting of data through T-MSIS becomes more  timely, accurate, and complete. CMS has been working since 2011 to  implement T-MSIS as a replacement for some current reporting to  improve and increase states’ reporting of Medicaid and CHIP data. CMS  intends for T-MSIS to provide a national data repository to support federal  and state Medicaid and CHIP program management, among other things.  T-MSIS includes data not previously reported by states and is intended to  improve Medicaid and CHIP program efficiency, in part, by allowing states  to compare their data with other states’ data. T-MSIS includes data that  can measure the provision of EPSDT services. According to CMS  officials, T-MSIS also includes aspects designed to improve the accuracy  of available state data. For example, states’ T-MSIS submissions undergo  approximately 2,800 automated quality checks, which provide states with  feedback on data format and consistency. As of January 2019, all 50  states and the District of Columbia were submitting data monthly,  according to CMS, but T-MSIS data were not being used to create the  CMS-416, Child Core Set, or the scorecard. Agency officials said  research-ready files are in development and T-MSIS data are improving  in quality over time with historical state resubmissions.

CMS Reports Indicate that Approximately Half of Beneficiaries Received Recommended Screenings and Services in 2017, but Nearly as Many Did Not

According to our analysis of CMS-416 data for fiscal year 2017, millions  of Medicaid beneficiaries received recommended EPSDT well-child  screenings and preventive dental services. However, nearly as many  eligible beneficiaries did not receive the various recommended  screenings and services, and few states met CMS’s performance  measure targets for EPSDT services. Additionally, while available data  show that millions of blood lead screenings were performed, the total  number of beneficiaries receiving blood lead screenings is unknown,  because the data are incomplete.

In fiscal year 2017, 20.2 million (59 percent) of the 34.2 million  beneficiaries who should have received at least one recommended well- child screening received that screening, known as the participant ratio,  according to our analysis of state-reported CMS-416 data. Additionally,  our analysis indicates that the national participant ratio has declined 5  percentage points since fiscal year 2010.

Three states met CMS’s participant ratio target of 80 percent in fiscal year  2017, as shown in figure 1. Our analysis also indicates that no more than  four states met CMS’s participant ratio target in any one fiscal year from  2010 through 2017. (See app. III, table 6, for participant ratios in each  state and nationally from fiscal years 2010 through 2017.)

Our analysis also indicates that as beneficiaries age, they tend to receive  fewer recommended well-child screenings, which results in lower  participant ratios. (See fig. 2 for participant ratios and numbers of  beneficiaries receiving and not receiving well-child screenings for each  CMS-416 age group in fiscal year 2017.) CMS has issued a guide on  serving older eligible beneficiaries, stating that regular preventive care  visits can lead to early identification of health issues. CMS officials said  the agency included measures focusing on these beneficiaries on the  Child Core Set and Medicaid and CHIP Scorecard to recognize the  importance of addressing these beneficiaries and to encourage states to  focus on this population. CMS officials noted that some states have  already taken steps to increase the number of well-child screenings that  older eligible beneficiaries receive, for example, by partnering with  schools.

In fiscal year 2017, 18.3 million (48 percent) of the 38.3 million Medicaid  beneficiaries aged 1 to 20 received a preventive dental service, according  to our analysis of CMS-416 data. This is an increase from the 42 percent  of beneficiaries receiving preventive dental services in 2011—the  baseline year for measuring state progress toward CMS’s Oral Health  Initiative targets—but less than CMS’s 52 percent national performance  measure target. Our analysis also shows that from fiscal years 2011  through 2017, nine states met CMS’s performance measure target of a 10  percentage point increase in each state’s percentage of beneficiaries  aged 1 to 20 receiving a preventive dental service. (See fig. 3 and table  8 in app. III for the percentage of beneficiaries aged 1 to 20 that received  preventive dental services in each state and nationally from fiscal years  2011 through 2017.)

Available data on blood lead screenings in the CMS-416 are incomplete  and, as a result, do not provide information necessary to determine how  many beneficiaries received the screenings. According to CMS’s  November 2016 guidance, CMS-416 data do not accurately represent the  number of beneficiaries receiving blood lead screenings.

The CMS-416 data capture screenings paid for by Medicaid, but not  those performed using funding from other sources, such as the  Centers for Disease Control and Prevention. This could under-count  the number of screenings performed.

In addition, the blood lead screening data reported on the CMS-416  show how many screenings were performed, but do not identify the  number of beneficiaries who received a blood lead screening.

Our analysis of available CMS-416 data shows that in fiscal year 2017  states reported 2.0 million blood lead screenings for beneficiaries aged 12  through 24 months, and there were 4.6 million beneficiaries aged 12  through 24 months.

CMS Has Improved EPSDT Data Quality; Additional Actions Are Needed to Improve Oversight of EPSDT Services, Particularly Blood Lead Screening Data

CMS Is Improving EPSDT Data, yet Does Not Regularly Take Action Based on Assessing the Appropriateness of the CMS-416 for Oversight

CMS has regularly taken actions to use both the CMS-416 and the Child  Core Set to improve the quality of information about the provision of  EPSDT services. These actions have made the data reported about  EPSDT services more complete and reliable. For example, CMS collects  data annually from states on performance measures for both the CMS- 416 and the Child Core Set. (See table 3.) Additionally, CMS annually  reviews the Child Core Set measures to determine whether measures  need to be added, deleted, or revised. CMS also regularly provides  technical assistance to states about data reliability, such as through its  monthly Quality Technical Advisory Group. For example, during one  group meeting, states shared challenges with reporting information about  developmental screenings on the Child Core Set and suggestions for how  to overcome these challenges. These actions are generally consistent  with federal internal control standards regarding information and  communication, which specify that management should use quality  information to achieve the entity’s objectives.

While CMS has taken actions to improve the quality of information about  EPSDT, and agency officials said they regularly assess whether the  information CMS collects on the CMS-416 is appropriate and useful for  EPSDT oversight, CMS has not taken action, as needed, based on such  assessments. For example, CMS has not added, removed, or amended  any performance measures on the CMS-416 since 2010, even though  officials acknowledge limitations in these measures.

The participant ratio, for example, is dependent, in part, on a state’s  chosen periodicity schedule, which means that the measure is not  consistently defined across states.

The screening ratio reflects the extent to which beneficiaries received  the recommended number of well-child screenings during the year,  but this information is aggregated and therefore cannot be used to  determine whether individual beneficiaries received the recommended  number of well-child screenings.

Although federal law requires collecting certain information about the  provision of EPSDT services, it provides the agency with flexibility to  determine the form and manner in which data are collected and to set  performance measures. For example, CMS could change the way  states are required to calculate the participant ratio or the screening ratio,  and could examine ways to do so to address the limitations that the  agency has identified and improve the quality of information about the  provision of EPSDT services.

Because CMS has not taken action, as needed, based on assessments  of the appropriateness of its CMS-416 performance measures, the  agency cannot be sure that it has the information it needs to oversee  state implementation of EPSDT. This is inconsistent with federal internal  control standards regarding information and communication, which  specify that management should identify information requirements in an  iterative and ongoing manner and ensure information remains relevant.  We have previously reported that results-oriented organizations set  performance goals to define desired program outcomes and develop  performance measures that are clearly linked to these performance goals  and outcomes.

CMS Has Set Some Performance Measure Targets, yet Does Not Consistently Evaluate States’ Performance against These Targets

CMS has taken steps to develop, assess, and use CMS-416 information  to improve states’ performance in providing EPSDT services. For  example, CMS has set performance measure targets for participant and  screening ratios reported on the CMS-416, and CMS publishes state-level  results of the participant and screening ratios. In addition, after identifying  issues with calculating the performance measure and target for the  permanent molar sealants, CMS removed them from the Oral Health  Initiative. CMS also convenes affinity groups and technical advisory  groups to provide assistance to states in improving performance, often  centered on specific services, such as dental services.

However, CMS and state Medicaid officials told us that CMS does not  consistently (1) communicate CMS-416 performance measure targets to  states, (2) evaluate state performance against performance measure  targets, or (3) provide states with assistance in reaching performance  measure targets. While it has not done so across all performance  measure targets, CMS did take these actions regarding targets for  preventive dental services as part of its Oral Health Initiative. For  example, CMS  communicated with states about the preventive dental service  performance measure target after it developed the Oral Health  Initiative; disseminated a national oral health strategy and published a review of  eight states identifying innovative approaches in providing preventive  dental services; and provided targeted outreach to states with the lowest performance on  the preventive dental service performance measure.

Improvements in the provision of dental services occurred in many states.  For example, in 2013, CMS met with state Medicaid officials in Florida  about improving the provision of preventive dental services. Five years  later, the percentage of beneficiaries receiving preventive dental services  had increased 18 percentage points.

CMS has not taken action in other areas. For example:

CMS does not communicate the participant and screening ratio  targets. Officials from CMS and from each of our 16 selected states  told us that CMS does not mention these targets in communications  with states, including discussions related to performance  improvement.

CMS has not evaluated state performance in meeting the participant  and screening ratio targets, nor has it provided focused assistance to  states to resolve gaps in states’ performance in reaching these targets  comparable to the assistance provided for the preventive dental  screening performance measure as part of the Oral Health Initiative.

CMS did not provide formal written notification to states when in  March 2016 the agency informed participants in two meetings that  CMS no longer planned to use the target for measuring states’  performance on the permanent molar sealants performance  measure. The notification was not provided through an official policy  document, such as an agency informational bulletin distributed to all  states. Despite removing the target, CMS issued a technical  assistance brief in March 2018 that referenced it, which could have  led to confusion among state officials.

With regard to the Child Core Set, CMS has not established any  performance measure targets and agency officials were not able to  provide information about plans for setting targets. CMS officials said that  the CMS-416 will remain a part of its EPSDT oversight. However,  because its information is not standardized across states, CMS plans to  increasingly rely on the standardized Child Core Set data to assess and  improve states’ performance on the provision of EPSDT services. CMS  officials noted that it publishes median, top quartile, and bottom quartile  information for each state for all the Child Core Set measures that are  publicly reported. Officials further reported in June 2019 that CMS and  states use these as performance benchmarks, with an aim of reaching  the national median on these measures if not the top quartile. Reporting  these data is an important step in ensuring better oversight of EPSDT.  However, CMS has not developed fixed targets that explicitly track states’  progress in increasing beneficiaries’ receipt of EPSDT screenings and  services. Using a median to assess states’ performance ensures that half  the states will not meet this target, regardless of their individual  performance.  Further, CMS officials have not provided plans or timelines  for when the Child Core Set would be used to help states achieve  performance measure targets.

CMS’s inaction regarding using the CMS-416 and Child Core Set to  improve performance on the provision of EPSDT services limits the  agency’s oversight and is inconsistent with federal internal control  standards for monitoring, and practices of leading organizations. Federal  internal control standards specify that management should (1) set  performance measure targets in measureable, numeric terms; (2)  communicate necessary information to achieve performance targets; (3)  evaluate progress toward desired targets; and (4) take action to resolve  identified issues. Without regularly using the CMS-416 and Child Core  Set to improve the provision of EPSDT services, CMS is unable to identify  whether state or federal efforts and policies are increasing the number of  beneficiaries receiving EPSDT services. As a result, CMS’s oversight is  limited and beneficiaries may not be receiving appropriate EPSDT  services when they need them—CMS’s stated goal for EPSDT. (See  table 4 for examples of actions CMS has and has not taken regarding  using the CMS-416 and Child Core Set for improving the provision of  EPSDT services.)

CMS Has Taken Limited Actions to Improve Data on the Number of Blood Lead Screenings, which Are Critical to Identifying Harmful Lead Exposure

CMS is unable to determine whether all eligible EPSDT beneficiaries are  receiving blood lead screenings in accordance with CMS policy. As  previously noted, CMS-416 data are incomplete, because they only  include blood lead screenings paid for by Medicaid, and the form reports  the number of screenings performed instead of the number of  beneficiaries receiving screenings.

State examples of collecting blood lead  screening data  Nebraska. Medicaid officials said that the  state has developed a database with the  Nebraska Health Information Initiative  containing laboratory testing data. Treating  providers and managed care organizations  can access the database to determine  whether a Medicaid beneficiary has received  a blood lead screening.  New Jersey. Medicaid officials said that it can  be difficult to track blood lead screenings that  are performed using funding from sources  other than Medicaid; for example, those  performed by the state health department.  Officials said that they have been building a  lead registry to capture data on lead  screenings performed, regardless of how they  are funded. New Jersey Medicaid officials  said they collect data every 6 months on  screenings not paid for by Medicaid and enter  the data into the state’s blood lead registry.

CMS has stated that screenings are important for identifying beneficiaries  with elevated blood lead levels at as young an age as possible, because  lead exposure can harmfully affect nearly every system of the body and  cause developmental delays. According to a presidential task force on  environmental health and safety risks to children, co-chaired by HHS,  early identification of developmental delays allows providers and  communities to intervene earlier to improve health outcomes. The  presidential task force issued goals in December 2018 to reduce lead  exposure and associated harms, including a goal to identify lead-exposed  individuals and improve their health outcomes.

Without complete information about blood lead screenings, CMS cannot  identify the number of beneficiaries who have not received blood lead  screenings. As a result, the agency may be unaware of beneficiaries with  unidentified lead exposures. CMS issued guidance in 2016 to states on  improving blood lead screening reporting, including correcting reporting  errors and partnering with providers to ensure beneficiaries receive blood  lead screenings. (See sidebar for examples of efforts states have taken to  improve available data about blood lead screenings.) However, as of  February 2019, the screening data remained incomplete, according to  agency officials. CMS officials also told us they are currently in  discussions with the Centers for Disease Control and Prevention about  how to capture more complete information about Medicaid beneficiaries  who are receiving blood lead screenings through programs funded by that  agency. However, as of February 2019, CMS officials had not identified  specific actions to gather this data. The lack of data is inconsistent with  federal internal control standards, which specify that management should  obtain relevant data from reliable sources based on identified information  requirements, and use such data for effective monitoring.

CMS Replicated Some CMS-416 and Child Core Set Information Using T- MSIS, but Lacks Time Frames and Interim Milestones for Using T-MSIS Data to Streamline State Reporting

According to CMS, the results of recent pilot studies indicate that T-MSIS  data can be used to replicate some information on the CMS-416 and  Child Core Set. CMS officials said that the results also suggest that  CMS may eventually be able to use T-MSIS data to produce the CMS- 416 and Child Core Set data, thus eliminating the need for states to report  this information themselves separately. As previously noted, CMS intends  for T-MSIS to both reduce the number of reports CMS requires states to  submit and to provide more information to improve Medicaid oversight.

CMS officials said that they were encouraged that the pilot studies to  replicate portions of the CMS-416 and Child Core Set generally yielded  positive results. For example, CMS was able to use T-MSIS to replicate  the total number of Medicaid beneficiaries aged 20 and under eligible for  EPSDT from the CMS-416 within 5 percent of state-reported values for  eight of nine pilot states—which CMS officials viewed as a positive  result. CMS officials noted some concerns with inaccurate state  Medicaid eligibility data; for example, multiple dates of birth reported  through T-MSIS for the same beneficiary. However, CMS officials believe  the accuracy and completeness of T-MSIS data has improved since the  pilot studies, which were conducted using data from 2015 and 2016.  Regarding the Child Core Set, CMS was able to use T-MSIS to replicate  some of the information, such as adolescent well-care visits, but not other  information, such as emergency department visits.

While CMS found generally positive results from the pilots, the agency  has not developed a plan with time frames and interim milestones for  when it will use state-reported T-MSIS data to produce the CMS-416 and  Child Core Set data sets instead of states separately producing both T- MSIS data and the two data sets. In April 2019, CMS officials said that  they were planning additional pilots beginning in fiscal year 2019 to  replicate portions of the CMS-416 and the Child Core Set. However, CMS  officials were unable to provide planned next steps, including time frames  and interim milestones, for using T-MSIS data to replace the CMS-416  and Child Core Set. This is inconsistent with federal internal control  standards related to using and communicating quality information to  achieve objectives. Without a specific plan with time frames with interim  milestones, CMS may miss opportunities to use T-MSIS data to  streamline state reporting and better oversee states’ provision of EPSDT  services. This limitation is similar to one we reported in December 2017  about the initial steps CMS had taken for using T-MSIS data. We found  CMS was limited in using T-MSIS for its broader oversight efforts of state  Medicaid programs, in part, due to the absence of an articulated plan and  time frames.

Under EPSDT, millions of Medicaid’s youngest beneficiaries received  well-child screenings and dental services in fiscal year 2017; however,  nearly as many of them did not. Further, existing data on blood lead  screenings are incomplete and inaccurate, leaving CMS unaware of  beneficiaries with unidentified lead exposures that can cause  developmental delays. The EPSDT data collected—whether via the CMS- 416, Child Core Set, or T-MSIS—have the potential to improve CMS  oversight of beneficiaries’ receipt of necessary services and screenings.  However, CMS has not taken sufficient steps to help ensure the  appropriateness of its state data collection, evaluations, and assistance;  and its plans for new reporting, including time frames and interim  milestones, are lacking.

We are making the following six recommendations to CMS:

The Administrator of CMS should work with states and relevant  federal agencies to collect accurate and complete data on blood lead  screening for Medicaid beneficiaries in order to ensure that CMS is  able to monitor state compliance with its blood lead screening policy,  and assist states with planning improvements to address states’  compliance as needed. (Recommendation 1)

The Administrator of CMS should regularly assess the  appropriateness of performance measures and targets for the EPSDT  benefit, and take any necessary actions to ensure their relevance and  use, including adding, changing, or removing measures, or targets,  and regularly communicating performance measures and targets to  states. (Recommendation 2)

The Administrator of CMS should conduct regular evaluations of state  performance by comparing states’ performance measurement data  with CMS’s EPSDT targets to identify gaps in states’ performance and  areas for improvement. (Recommendation 3)

The Administrator of CMS should assist states with planning needed  improvements, including providing focused assistance, to resolve  gaps in states’ performance in meeting CMS’s EPSDT targets.  (Recommendation 4)

The Administrator of CMS should develop a plan with time frames and  interim milestones for using T-MSIS data to generate the necessary  data from the CMS-416 to improve EPSDT oversight and streamline  state reporting. (Recommendation 5)

The Administrator of CMS should develop a plan with time frames and  interim milestones for using T-MSIS data to generate the necessary  data from the Child Core Set to improve EPSDT oversight and  streamline state reporting. (Recommendation 6)

We provided a draft of this report to HHS for comment, and its comments  are reprinted in appendix IV. HHS also provided us with technical  comments, which we incorporated in the report as appropriate. Overall,  HHS concurred with three recommendations and did not occur with three  recommendations.

HHS concurred with our first recommendation that CMS should work with  states and relevant federal agencies to collect accurate and complete  data on blood lead screening for Medicaid beneficiaries and assist states  with planning improvements to resolve gaps in states’ performance as  needed. However, HHS stated that it would not be possible to obtain  complete data on blood lead screenings, because some screenings are  not paid for by Medicaid. In our report, we noted some state and CMS  efforts to improve available data on blood lead screenings. We continue  to believe CMS needs to take additional actions to collect accurate and  complete data to oversee whether eligible EPSDT beneficiaries are  receiving blood lead screenings in accordance with CMS policy.

HHS did not concur with our second recommendation, which stated that  CMS should regularly assess the appropriateness of performance  measures and targets for the EPSDT benefit, and take any necessary  actions to ensure their relevance and use.

HHS noted that it assesses the appropriateness of Child Core Set  measures annually and may update existing measures based on that  assessment, including measures on the CMS-416. We acknowledge  CMS’s actions to assess the appropriateness of Child Core Set  measures annually and update those measures as appropriate, and  we found these actions generally consistent with federal internal  control standards regarding information and communication.  However, CMS has not taken action, as needed, related to any  assessments of the CMS-416 performance measures, even though  officials acknowledge limitations in these measures, such as the  participant and screening ratios.

HHS also stated that it may set targets in key areas as appropriate,  and has done so as part of the Oral Health Initiative, but that HHS  does not believe it would be productive at this time to set targets for  every measure. We are encouraged that HHS agreed that it may set  targets in key areas as appropriate. This is consistent with our  recommendation for CMS to regularly assess the appropriateness of  its targets. Our recommendation does not assume that targets should  be set for every measure—rather, that CMS needs to regularly assess  the appropriateness of performance measures and targets for the  EPSDT benefit and communicate them to states.

HHS did not concur with our third recommendation, which stated that  CMS should conduct regular evaluations of state performance by  comparing states’ performance measurement data with CMS’s EPSDT  targets.

HHS stated that it offers a wide range of technical assistance on  quality improvement to help states address performance goals. HHS  commented that it believes this is the most effective method of helping  states identify and address areas for potential improvement. We  acknowledge that CMS has provided states with technical assistance  and individual state snapshots of selected Child Core Set measures  over time. However, regular evaluations of states’ performance  against appropriate EPSDT targets are necessary to help identify  gaps in states’ performance and areas for improvement.

HHS noted that states recently received snapshots about their  performance on publicly reported Child Core Set measures for the  past 5 years, through fiscal year 2017. According to HHS, the  snapshots include information about a state’s performance on each  measure relative to other states’ performance and highlights  significant changes in a state’s performance for each measure.  However, these snapshots include descriptions of all states’  performance—using medians, and top and bottom quartiles—which  are subject to change over time. Moreover, because the median is the  midpoint of all states’ performance, this target ensures that half of  states will not meet it, regardless of their individual performance. A  fixed target—or targeted improvement goal, such as the one  developed as part of the Oral Health Initiative—would provide states  with the opportunity to measure performance over prior years’ results,  which is a more meaningful measure that all states can strive to  achieve.

HHS did not concur with our fourth recommendation, which stated that  that CMS should assist states with planning needed improvements to  resolve gaps in states’ performance in meeting EPSDT targets.

HHS stated that it has developed national and state-specific  improvement goals for children enrolled in Medicaid with respect to  receipt of at least one preventive dental service and provided targeted  technical assistance to the lowest performing states. In this report, we  noted states’ progress in meeting targets once CMS developed a  performance measurement target for preventive dental services,  including actions to improve state performance. Developing additional  targets on performance measures critical to beneficiaries’ health and  well-being could help improve oversight of EPSDT.

HHS also described other examples of targeted technical assistance  to remedy gaps in states’ performance, which included working with  states on improving their performance on certain Child Core Set  measures and improving access to EPSDT services by better  leveraging schools as settings for care. Such technical assistance  could be valuable for CMS to provide to states after identifying gaps in  states’ performance relative to EPSDT targets. Doing so would allow  CMS to share additional strategies to help states plan and implement  needed improvements.

HHS concurred with our fifth and sixth recommendations that CMS should  develop a plan with time frames and interim milestones for using T-MSIS  data to generate the necessary data from the CMS-416 and Child Core  Set to improve EPSDT oversight and streamline state reporting.

As agreed with your office, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to the Secretary of Health  and Human Services, the Administrator of the Centers for Medicare &  Medicaid Services, and other interested parties. In addition, the report is  available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact  Carolyn L. Yocom at (202) 512-7114 or yocomc@gao.gov. Contact points  for our Offices of Congressional Relations and Public Affairs may be  found on the last page of this report. Major contributors to this report are  listed in appendix V.

Appendix I: Selected States’ Practices for Delivering Early and Periodic Screening, Diagnostic, and Treatment Services

Selected states used several types of practices to promote and facilitate  the delivery of Early and Periodic Screening, Diagnostic, and Treatment  (EPSDT) services, according to Medicaid officials in the 16 selected  states we interviewed and profiles of these states created by the  American Academy of Pediatrics. The practices selected states used  included outreach and education, financial incentives, collaboration in  EPSDT administration, and EPSDT service delivery initiatives, as shown  in figure 4.

Appendix II: Information Reported on Form CMS-416 and Child Core Set

States annually report information about Early and Periodic Screening,  Diagnostic, and Treatment (EPSDT) services to the Centers for Medicare  & Medicaid Services (CMS), through the Form CMS-416 and the Child  Core Set. The CMS-416 provides basic information about EPSDT for  Medicaid beneficiaries aged 20 and under, such as the participant ratio  and number of beneficiaries receiving a preventive dental service. The  Child Core Set provides CMS with information about the quality of health  care provided to Medicaid beneficiaries and individuals aged 18 and  under who are covered under the Children’s Health Insurance Program.  In fiscal year 2024, annual reporting of the Child Core Set will become  mandatory. As of 2019, the Child Core Set included performance  measures related to the provision of EPSDT services, such as well-child  visits in the first 15 months of life. Because Child Core Set reporting is  currently voluntary, states vary in the number of performance measures  they choose to report. In fiscal year 2017, for example, 50 states and the  District of Columbia voluntarily reported on at least one of the 27 Child  Core Set performance measures, with states reporting a median of 18  Child Core Set performance measures. Some information is only  reported on the CMS-416 or Child Core Set, while other information— well-child visits, preventive dental services, and dental sealants—is  reported on both CMS-416 and Child Core Set. (See table 5 for  information reported on the CMS-416, the Child Core Set, or both.)

Appendix III: Summary of Selected Early and Periodic Screening, Diagnostic, and Treatment Data

Tables 6 through 8 present annual state-reported data from the Centers  for Medicare & Medicaid Services’ (CMS) Form CMS-416 on the  provision of selected Early and Periodic Screening, Diagnostic, and  Treatment (EPSDT) services by state and nationally. Well-child  screenings are presented from fiscal year 2010, the year in which the  current reporting template was implemented, through fiscal year 2017, the  most recent year for which data were available at the time of our review.  Preventive dental services data are presented from fiscal year 2011, the  baseline year for measuring states’ progress toward CMS’s Oral Health  Initiative targets, through fiscal year 2017, the most recent year for which  data are available.

Appendix IV: Comments from the Department of Health and Human Services

Appendix V: GAO Contact and Staff Acknowledgments

Carolyn L. Yocom at (202) 512-7114 or yocomc@gao.gov.

In addition to the contact named above, Karen Doran (Assistant Director),  Peter Mangano (Analyst-in-Charge), Matthew Green, Erika Huber, Drew  Long, Jennifer Rudisill, and Kelly Turner made key contributions to this  report. Also contributing were Muriel Brown, Giselle Hicks, Erika Lessien,  and Madeline Ross.

Medicaid, which provides health care for low-income and medically needy people, has an early screening and treatment benefit to help young people get the health care services they need. Goals of the benefit include getting blood lead screenings for young children and well-child exams for at least 80% of eligible young people.

59% of those eligible got at least one recommended exam in 2017

Data on blood lead screenings is incomplete, so nobody knows how many young children were screened. Our recommendations include improving lead screening data and evaluating state performance in meeting targets.


VA Vocational Rehabilitation and Employment: Additional Assessment Could Enhance Consistency among Counselors

VA's VR&E program helps veterans with service-connected disabilities obtain and maintain suitable employment. VR&E participants work with vocational counselors to develop career goals and employment plans. However, some veteran service organizations have questioned the consistency with which participants are treated by counselors in developing these plans. GAO was asked to review how VR&E vocational counselors work with participants to select employment plans, and VA's efforts to ensure high quality and consistency.

This report examines (1) the factors that vocational counselors considered when developing VR&E participants' plans and how consistently they applied those factors, and (2) the extent to which VA trains and monitors vocational counselors to ensure a consistent, high-quality approach to helping veterans develop plans. GAO analyzed VR&E quality review data from fiscal years 2016 through 2018; reviewed a random, non-generalizable sample of 34 VR&E case files from 2019; reviewed relevant federal laws, regulations, and VA policy; and interviewed VR&E counselors and other program officials.

The Department of Veterans Affairs' (VA) Vocational Rehabilitation and Employment (VR&E) counselors in GAO's review generally considered a set of common factors when developing plans to help veterans with disabilities obtain employment, but counselors explained that inconsistent application of those factors likely occurs. These factors included the veteran's disability, his or her interests, and local labor market conditions. The 34 VR&E plans GAO reviewed showed that counselors' generally considered and documented these factors (see table). Counselors in each of the three regional offices GAO visited said that plans are individualized to suit the veteran's needs and as a result will differ because each veteran's case is unique. Nonetheless, these counselors acknowledged that some veterans with similar circumstances likely receive different types of plans given differences in counselor judgment and experience.

VA trains and monitors counselors to develop complete VR&E plans but does not assess the consistency of plans across counselors for veterans with similar circumstances. VA's training for VR&E counselors emphasizes that plans should accommodate each veteran's individual needs, abilities, aptitudes, and interests. In designing training for counselors, VA followed principles identified by GAO for strategically developing training. VA monitors the completeness of VR&E plans through national and regional quality reviews that check, among other elements, whether plans have an employment focus and include needed services. However, these quality reviews do not assess the consistency of plans developed by different counselors. VR&E officials explained that the agency has not yet conducted such an analysis because of other priorities, but agreed that it could do so. One of the objectives of VR&E's central office is to provide training and guidance to help ensure consistency among field staff. Assessing consistency across counselors would better position VA to mitigate any unfair differences in plans for similarly-situated veterans.

GAO recommends that VA assess the consistency of VR&E plans among counselors by, for example, comparing counselors' responses to identical hypothetical cases, and take mitigating steps if warranted. VA concurred with the recommendation and planned to develop a consistency study.

About 4.7 million veterans—including more than 1.7 million from the Iraq  and Afghanistan era—have a disability connected to their military  service, and many of these veterans face special challenges obtaining  employment because of these physical or mental health conditions. The  Department of Veterans Affairs (VA) operates the Vocational  Rehabilitation and Employment (VR&E) program, which is intended to  help veterans with service-connected disabilities obtain and maintain  suitable employment, among other goals. The VR&E program provides  services including vocational assessment, education and training, and job  placement. In fiscal year 2018, VR&E received funding of about $1.6  billion to cover program expenses, including tuition and subsistence  payments for participants. In the same year, over 900 VA vocational  counselors worked directly with the program’s approximately 125,000  participants to help develop individualized employment plans, which entail  identifying appropriate career goals and managing progress toward  achieving them.

Testimony from several veteran service organizations at congressional  hearings has raised questions about the consistency with which  counselors develop plans for veterans with similar circumstances. You  asked us to examine how counselors work with VR&E participants to  develop plans and VA’s efforts to ensure quality and consistency. This  report examines (1) the factors vocational counselors considered when  developing VR&E participants’ plans and how consistently counselors  applied these factors, and (2) VA’s training and monitoring of counselors  to ensure a consistent, high-quality approach to helping veterans develop  plans.

To identify factors that vocational counselors considered when developing  VR&E plans, we reviewed program guidance and interviewed VR&E  officials about the plan development process. To gain further insight, we  reviewed and analyzed a random, non-generalizable sample of veterans’  VR&E case files. Specifically, we reviewed all 34 case files randomly  selected by VA for its February 2019 quality review of VR&E plan  development. We assessed VA’s selection methodology by reviewing  documentation and interviewing knowledgeable officials and found it to be  reliable for our purposes. VR&E’s quality reviews cover all phases of  counselors’ casework, whereas our review focused on plan development.  VR&E’s quality reviews assess whether counselors conducted a complete  evaluation to determine the veteran’s needs and circumstances. In  contrast, our review focused on identifying specific factors that counselors  considered to generate veterans’ plans, identifying how consistently they  applied them, and describing examples related to plan development and  any changes that occurred in the development.

To identify how VA monitors and trains counselors, we visited VR&E’s  centralized quality review office, reviewed training materials and program  guidance for counselors, and interviewed program officials. We evaluated  VA’s practices against its program objectives and a guide we developed  for assessing training and development efforts. To inform both  objectives, we: analyzed VR&E quality review data from fiscal years 2016  through 2018; reviewed relevant federal laws, regulations, and VA policy;  interviewed representatives from two veteran service organizations  selected for specializing in education benefits; and interviewed  counselors and the VR&E official responsible for each of three regional  offices selected for variation in geography and VR&E plan development  accuracy rates. Across these three regional offices, we interviewed a  total of 27 counselors. Their views are not generalizable but provided  insights. We limited our review to veterans currently seeking  employment.

We conducted this performance audit from August 2018 to December  2019 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.

To be entitled to VR&E services and related benefits, veterans generally  must (1) have at least a 20 percent service-connected disability rating  from VA and (2) be in need of rehabilitation because of an employment  handicap. Entitled veterans may generally receive up to 48 months of  vocational rehabilitation services and up to an additional 18 months of  employment services, which include counseling, and placement and  postplacement services.

If a veteran is entitled to receive VR&E services and found to be  employable, a counselor is to work with the veteran to identify a suitable  employment goal, and to incorporate that goal and the needed services  and benefits to achieve it into a vocational rehabilitation and employment  plan (hereafter “employment plan”). To develop an employment plan, the  counselor and veteran review labor market information for jobs within the  veteran’s identified abilities, aptitudes, and interests that will not  aggravate his or her service-connected disability or disabilities. After  assessing obstacles to employment, they agree on a written employment  plan that describes the employment goal and the services needed to  achieve it. Common services provided by VR&E are funding for higher  education, career counseling, and short-term employment services like  job search assistance. Counselors have the authority to approve a wide  variety of educational programs and may approve employment plans that  have an annual cost of up to $25,000.

Within VA’s Veterans Benefits Administration (VBA), the VR&E central  office is responsible for overseeing the VR&E program, including training  staff and monitoring their work to ensure high performance and  consistency. Among other elements, VR&E’s quality assurance efforts  entail reviewing a subset of case files on a monthly basis to ensure that  the entitlement decisions, development of plans, and delivery of services  are performed and documented in accordance with VA regulations,  VR&E’s operations manual, and other directives.

VR&E services are provided by field staff at 56 regional offices and about  300 satellite locations. The satellite locations include college campuses to  help veterans successfully complete their training and find employment,  as well as military sites to help servicemembers with disabilities as they  begin their transition to veteran status and the civilian workplace. VR&E  field positions include (1) VR&E officers who manage the program and its  staff in each region; (2) vocational rehabilitation counselors who work  directly with veterans to assess their entitlement, develop their  employment plans, and manage their progress; and (3) staff to support  the administration of the program.

As of June 2019, 1,394 field staff members were administering the VR&E  program, of which nearly 75 percent (1,026) were counselors. From  September 2013 to June 2019, VR&E’s total caseload peaked in fiscal  year 2016 with almost 135,000 participants (see fig. 1).

Over the same period, the number of counselors changed little until 2019.  In 2019, the number increased after VA hired an additional 88 counselors  in response to a provision in an appropriations law suggesting that the  agency aim to serve 125 veterans or fewer per full-time equivalent  counselor. The increase in staffing helped reduce the average caseload  of 130-141 cases per counselor during fiscal years 2013 through 2016 to  113 cases in June 2019 (see fig. 2).

Counselors Generally Considered Common Factors When Developing Veterans’ Plans but Noted Inconsistent Application of Those Factors Likely Occurs

Counselors in Our Review Generally Considered a Set of Common Factors When Developing Plans

VR&E counselors consider a set of common factors, including the  veteran’s disability or disabilities, interests, and local labor market  conditions, when developing and approving veterans’ employment plans.  Program regulations require an assessment of some of these factors  when the veteran is initially evaluated. VR&E quality review data from  fiscal years 2016 through 2018 suggest that counselors generally  documented certain plan considerations during the evaluation. For  example, in 98 percent of the 1,080 cases VA reviewed for accuracy in  fiscal year 2018, counselors documented the veteran’s service needs  based on their functional limitations. In 95 percent of cases, counselors  documented that they assessed the veteran’s abilities, aptitudes, and  interests. Lastly, in nearly 99 percent of cases, counselors documented  that the veteran was involved in vocational exploration activities such as  career searches and labor market research.

During our more focused review of how counselors developed plans for a  non-generalizable sample of 34 VR&E case files, we found that  counselors generally documented a set of common factors. Consistent  with program guidance stipulating that counselors are to consider a  veteran’s service needs, abilities, aptitudes, and interests, we identified  common consideration factors including one’s functional limitation from  disability, prior education, aptitude results, and career interests. Table 1  presents these factors and the number of files in which the factors were  documented.

Our case file review found that 30 of the 34 counselors also documented  the estimated cost of VR&E employment plans. According to testimony  from a veteran service organization, many VR&E participants are  dissuaded by their counselor from pursuing education at a top tier  university because of cost. VA’s VR&E operations manual states that if  more than one local training or educational facility will meet a veteran’s  needs, counselors must justify their decision to select a school that is  more expensive than the least costly one. Counselors are not required to  document all of the educational facilities that would serve a veteran’s  needs; therefore, we could not determine the extent to which counselors chose the lowest cost facility. Counselors we interviewed in each of the  three regional offices we visited said that while mindful of cost, they strive  to develop employment plans that best meet the needs of the veteran.  For example, counselors at one regional office described a situation in  which a higher priced school was chosen because the school offered  smaller class sizes that better suited the veteran’s particular mental  health conditions. Of the 34 files we reviewed, the annual plan cost  exceeded $25,000 in 3 cases.

Counselors we interviewed said that they considered the veteran’s career  interests but weighed these interests against other factors, such as the  veteran’s functional limitations and information about the local labor  market. All 34 plans we reviewed aligned with the veteran’s stated career  goals, though in some cases the veteran’s goals evolved after talking with  the counselor about alternative occupations. In a few instances among  these cases, the final plan’s career goal was notably different from the  initial goal that the veteran had stated on the program intake form. Table  2 presents examples of how a plan can evolve as a result of career  exploration activities and conversations between the veteran and their  counselor.

Counselors in Our Review Stated They Strive to Develop Individualized Plans but Acknowledged That Some Unintended Differences in Plans for Similarly-Situated Veterans Likely Occurs

Counselors we interviewed described how veterans’ employment plans  are individually designed to suit a veteran’s needs and, as a result, may  differ from one another even when veterans have similar goals,  characteristics, and circumstances. In some instances two veterans may  appear to be similar, but may actually differ in some critical respect that  results in appropriate variation across plans. A common difference among  veterans is the geographical location where they are seeking  employment. Counselors described how a veteran may be encouraged to  explore an occupation with many job opportunities within a specific  region, while a veteran with similar characteristics and interests living in a  different area may be dissuaded from pursuing the same occupation for a  lack of job opportunities in the area for that occupation. Local labor  markets may also drive the need for a certain type of educational  credential. For example, counselors said that some veterans will be  competitive in certain labor markets with a bachelor’s degree, while  others living in a different region with a more educated population may  need a master’s degree. Likewise, they said that certain occupations,  such as certified public accountants or school teachers, may require  different forms of credentialing in different states.

Other characteristics of individual veterans may also cause counselors to  develop different plans for veterans who appear to have similar  circumstances. One counselor we interviewed described a scenario in  which one veteran who received a high score on an aptitude test for  reading comprehension skills might obtain a certain employment plan  while another veteran who received a much lower score would be steered  toward a different plan. If the veterans were to compare their final plans,  but were unaware of the differences in their aptitude test scores, they  could perceive inconsistent treatment. Counselors also described how  conversations they have with veterans as they work to develop  employment plans can reveal other character traits, such as interpersonal  skills, which can lead them to suggest different plans to two otherwise  similar veterans. The counselors said that such conversations play an  important role towards the development of successful plans.

However, counselors we interviewed in each of the three regional offices  we visited acknowledged that unintended variation likely occurs across  plans developed for similarly-situated veterans. They explained that the  reasons for such potential inconsistency can include (1) the prominent  role professional judgment plays in the program and the potential for  unintended bias, (2) counselors’ different VR&E experience levels, and  (3) variations in regional offices’ policies.

Judgment and bias. The counseling role is inherently subjective and  requires counselors to use their professional judgment in each case.  The VR&E operations manual describes counselors’ responsibilities in  broad terms, stating that counselors are to guide and assist the  veteran in making an informed decision on an appropriate plan based  on the veteran’s abilities, aptitudes, and interests. According to  counselors we interviewed, professional judgment enables them to  develop a plan that is best suited for the veteran’s unique needs,  although it also introduces the potential for personal bias and  inconsistent plans for veterans with similar circumstances. For  example, a counselor we interviewed cited a case in which he saw the  need to develop a plan that allowed for a school that was closer to a  veteran’s home over other, less costly options because of his  sensitivity to the veteran’s childcare responsibilities. Another  counselor in the same office may not have seen the need for that  accommodation. Further, counselors we interviewed said that some of  their colleagues may be more comfortable suggesting that a veteran  reconsider his or her career goal given circumstances such as the  veteran’s disabling conditions or the local labor market. They  explained that while some counselors would be hesitant to make the  veteran unhappy, and possibly angry, other counselors would be  more inclined to work through the conflict. Counselors said that they  try to mitigate inconsistency by asking their fellow counselors to weigh  in on these sorts of judgments, either informally, or at periodic  information-sharing meetings.

Counselor experience. Although all counselors have at least a  master’s degree in rehabilitation counseling or a related field,  differences in counselors’ levels of VR&E experience may affect their  approach to plan development. Counselors at two regional offices  noted that the focus of the VR&E program has oscillated between  education and employment, with employment being the current  primary focus. They said, as a result, a counselor’s general approach  to plan development could be influenced by the prevailing focus that  existed at the time he or she was hired. Counselors also said that, in  general, counselors with more experience will tend to approach plan  development differently than a less seasoned counselor because they  will apply lessons learned from serving many other veterans. For  example, one counselor said that, based on years of prior experience  and observation, he has developed a better understanding of which  local educational programs offer veterans the best chance for success  and which do not. He said while he is able to apply his institutional  knowledge and experience to do what is best for veterans, a less  experienced counselor may not have the same level of knowledge,  which could lead to inconsistent plans for veterans with similar  circumstances. According to counselors we interviewed, because of  the recent hiring of new counselors to meet caseload targets,  differences in VR&E experience among counselors may be more  pronounced at this time.

Regional office variation. Differences in administrative policies  specific to individual regional offices may also contribute to  inconsistent plan development. For example, according to program  officials, to ensure the soundness of employment plans in the local  labor market, some regional offices require management to approve  plans involving a master’s degree, while others do not. Counselors in  one region told us that requiring management approval might  dissuade a counselor from developing a plan focused on a master’s  degree because of the time the extra step would require. They  acknowledged that this sort of approval policy could cause  inconsistency across counselors’ plans and also cause a discrepancy  in the number of master’s degree programs being approved at one  regional office versus another.

In general, the large number of variables involved in the development of  employment plans may complicate the ability to determine the extent to  which differences among counselors lead to inconsistent plans among  veterans. Counselors we interviewed said that given the subjective nature  of the program, such inconsistency is likely. However, counselors  cautioned against making the plan development process overly structured  and formulaic. In their view, a more restrictive approach would eliminate  the flexibility that they need to generate plans that suit each veteran’s  unique needs.

VA Trains Counselors and Monitors Their Performance but Does Not Monitor the Consistency of Employment Plans

VA Trains Counselors to Develop Sound Employment Plans for Veterans

VA trains counselors on developing sound and complete employment  plans for veterans. New counselors receive a series of training courses  that are developed and deployed through VR&E’s central office, and then  receive additional courses and mentorship that are delivered through the  regional offices. Course topics for new counselors include understanding  vocational impairments, developing a rehabilitation plan, and  documenting a narrative of the plan. The formal training emphasizes that  plans should be individualized to accommodate the veteran’s  rehabilitation needs, abilities, aptitudes, and interests. Collectively, these  trainings take up to 80 hours. As of 2019, experienced counselors—those  on the job for at least a year—take up to 20 hours of refresher training  each year determined according to how they score on an annual  assessment. The assessment evaluates counselors’ technical  competencies such as knowledge of relevant regulations, vocational  assessment and evaluation, and case management. If a counselor scores  low on a particular topic, related courses are identified for the counselor  to complete.

In designing training for counselors, VA followed principles for  strategically developing training that are consistent with a related guide  for federal managers. For instance, VA obtained and considered input  from multiple sources—including field advisory committees, quality  assurance reviewers, and internal site visit auditors—to identify needs for  counselor training. For example, questions and input from the field about  a policy clarification led to a training about veterans’ entitlement to VR&E  services. In addition, VA built flexibility into its training curricula for  counselors so they could receive training on emerging topics such as  implementing new policies throughout the year as needed. VA also has  evaluated its training efforts in multiple ways. For example, it has  evaluated training courses by surveying counselors to get immediate  feedback and by checking with attendees and their supervisors to gauge  improvements in skills and knowledge.

VA Checks If Plans Are Complete but Does Not Monitor Consistency among Counselors

VA monitors employment plans to ensure that they are complete, but  does not check for consistency among counselors for veterans with  similar circumstances. Quality reviews occur nationally as well as locally  at each regional office. The purpose of the national reviews is to monitor  the quality of regional offices’ work such as plan development, whereas  the purpose of the local reviews is to help evaluate the performance of  individual counselors. Nationally, a centralized quality assurance team  monitors the completeness of regional offices’ VR&E entitlement  decisions, employment plans, and service delivery by reviewing a  randomly-selected subset of case files from each regional office on a  monthly basis. Among other criteria, reviewers check whether a veteran’s  plan identified goals and objectives, included an employment focus, and  incorporated the veteran’s need for various services. Locally, VR&E  officers or their designees are to review plans using the same criteria.  Officers are supposed to review at least three cases per counselor per  quality category (e.g., accuracy of evaluation, planning and rehabilitation  services) per quarter. Reviewers do not check for consistency among  counselors for similarly-situated veterans, at either the national or local  levels.

VR&E officials we interviewed identified challenges to completing and  monitoring local reviews, but VA is addressing these challenges.  According to the VR&E officer in each of the three regional offices we  visited, it is difficult to complete local reviews given system limitations and  their other job responsibilities such as implementing case management  initiatives. They said that it is likely that some officers are not completing  the required reviews while others are conducting them with varying  degrees of thoroughness. Historically, VA has not identified the specific  cases VR&E officers are supposed to review locally. Consequently, VA  could not determine if VR&E officers conducted the requisite number of  reviews or whether officers were selecting cases for quality review  uniformly and fairly. In June 2019, during the course of our review, VA  began a pilot in five regional offices to centrally and systematically identify  the cases officers are to review to gauge individual counselors’  performance. The new process and system are intended to help officers  conduct and track local reviews as well as to help VA monitor the  completion of local reviews. VA plans to expand this process to all  regional offices in fiscal year 2020.

Although VA trains counselors to develop complete employment plans  and reviews the completeness of some plans, it does not monitor the  consistency of plans among different counselors. The code of  professional ethics for rehabilitation counselors calls for counselors to be  fair in the treatment of all clients and to provide appropriate services to  all. In addition, one of the objectives of VR&E’s central office is to  provide training and guidance to ensure high performance and  consistency among field staff. Several veteran service organizations have  testified at congressional hearings that VR&E is marked by inconsistent  treatment of similarly-situated veterans. For example, one testimony cited  veterans who allegedly received different plan approvals, such as access  to graduate level education, merely on the basis of their counselor.  Unlike for VA staff members who work on disability claim decisions, VA  does not compare the output of VR&E counselors by, for example,  analyzing responses to identical hypothetical cases for training or  monitoring purposes. As a result, in addition to missing a training  opportunity for counselors about employment plan development, VA does  not know the degree to which inconsistency among counselors occurs.  For example, the agency does not know the extent to which counselors  would agree to a particular veteran’s pursuit of a master’s degree through  VR&E. Moreover, VA cannot respond in an informed way—and take  mitigating steps if warranted—to criticisms of subjectivity in the program.  VR&E officials explained that the agency has not yet conducted such a  comparative analysis because of other priorities, but agreed that it could  do so particularly through its training efforts.

VA uses several training and monitoring practices to help ensure that  VR&E counselors develop employment plans that help veterans with  disabilities obtain and sustain employment. In approving these plans,  VR&E counselors use their judgment and discretion fostered in part by  their formal education and professional experience in vocational  rehabilitation. While our review of a non-generalizable sample of 34 cases  found that counselors generally considered common factors in developing  employment plans, counselors we interviewed nevertheless  acknowledged that counselors may apply the factors differently because  of their varying backgrounds and experience levels. The variability of  counselors’ experiences and veterans’ circumstances may make it  difficult to determine the full extent of any inconsistency. However, taking  steps to examine the prevalence and type of any inconsistency among  counselors who, for example, consider the same hypothetical case, would  better position VA to mitigate any unfair differences in plans for veterans  with similar circumstances. An understanding of how effectively and  consistently counselors assist veterans will be even more important in the  coming years as VA fully integrates the new counselors hired to decrease  the average caseload.

The Secretary of VA should ensure that the Director of VR&E assesses  the consistency of VR&E plans among counselors and takes mitigating  steps if results warrant. For example, as part of its training efforts, VA  could have counselors respond to identical hypothetical veteran cases  and, if unfair inconsistencies in plans result, the agency could enhance  training on plan development. (Recommendation 1)

We provided a draft of this report to VA for comment, and its written  comments are reproduced as appendix I in this report. VA concurred with  our recommendation and said that VBA will develop a consistency study  of VR&E plan development. It emphasized that no two veterans are the  same. It also provided technical comments which we incorporated as  appropriate.

We are sending copies of this report to the appropriate congressional  committees, the Secretary of Veterans Affairs, and other interested  parties. In addition, the report will be available at no charge on the GAO  website at https://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-7215 or curdae@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who made key contributions to this report  are listed in appendix II.

Appendix I: Comments from the Department of Veterans Affairs

Appendix II: GAO Contact and Staff Acknowledgments

In addition to the individual named above, Mark Glickman (Assistant  Director), Joel Green (Analyst in Charge), and David Perkins made  significant contribution to the report. In addition, Jennifer Cook, Holly Dye,  Alex Galuten, Monica Savoy, Mimi Nguyen, Almeta Spencer, Jeff Tessin,  Rosemary Torres Lerma, and Sonya Vartivarian made key contributions.

Nearly 5 million veterans have a disability connected to their service, and many face challenges finding jobs as a result. VA’s Vocational Rehabilitation and Employment program helps veterans prepare for, find, and keep a job, usually with help from a counselor who prepares an employment plan. It may include recommendations for a particular college degree or training.

We found plans may differ among similarly situated veterans because of counselors’ experience and judgment. VA checks the quality of the plans but does not look at consistency from counselor to counselor.

We recommended VA assess plan consistency to reduce any unfair differences.