K-12 Education: Education Needs to Address Significant Quality Issues with its Restraint and Seclusion Data

Every 2 years, Education requires nearly all school districts to report incidents of restraint and seclusion. Generally, restraint is restricting a student's ability to move, and seclusion is confining them alone in a space they cannot leave.

The House Committee on Appropriations' explanatory statement accompanying the Consolidated Appropriations Act of 2018 included a provision for GAO to evaluate the CRDC's restraint and seclusion data. This report examines (1) the effectiveness of CRDC data quality control procedures, (2) selected districts' interpretation of CRDC's restraint and seclusion definitions, and (3) selected districts' use of data. GAO analyzed CRDC's quality control processes for school year 2015-16, and interviewed officials from seven stakeholder groups and over 50 school and district officials in three states. GAO selected states, districts, and schools to obtain a range of perspectives on using restraint and seclusion data and interpreting CRDC definitions of restraint and seclusion. Selection criteria included changes in reported incidents year to year and laws requiring districts to report incidents to states.

The Department of Education's (Education) quality control processes for data it collects from public school districts on incidents of restraint and seclusion are largely ineffective or do not exist, according to GAO's analysis of school year 2015-16 federal restraint and seclusion data—the most recent available. Specifically, Education's data quality control processes were insufficient to detect problematic data in its Civil Rights Data Collection (CRDC)—data Education uses in its efforts to enforce federal civil rights laws (see figure). For example, one rule Education used to check the quality of data submitted only applied to very large school districts, although GAO and Education's own analyses found erroneous reporting in districts of all sizes. Education also had no rules that flagged outliers that might warrant further exploration, such as districts reporting relatively low or high rates of restraint or seclusion. GAO tested for these outliers and found patterns in some school districts of relatively low and high rates of restraint or seclusion. Absent more effective rules to improve data quality, determining the frequency and prevalence of restraint and seclusion will remain difficult. Further, Education will continue to lack information that could help it enforce various federal civil rights laws prohibiting discrimination.

Officials in the nine school districts GAO visited lacked a common understanding of the CRDC's restraint and seclusion definitions. Similarly, officials GAO interviewed in all three state educational agencies (Kentucky, Washington, and Wisconsin) and all seven stakeholder groups expressed similar concerns about the clarity of these definitions. For example, officials inconsistently interpreted the word alone in the definition of seclusion and, therefore, on whether to count an incident if a teacher was in the room. Absent clearer definitions, Education will continue to lack quality information on restraint and seclusion in public schools.

Officials in school districts GAO visited identified several benefits to collecting these data, including identifying patterns in student behavior and developing interventions that can reduce the need for restraint and seclusion. Officials also said that analyzing their data helped them identify needs for additional staff training and student support services.

GAO made six recommendations, including that Education expand its CRDC business rules to cover all districts, develop additional quality controls to address misreporting, address factors underlying misreporting, and refine and clarify its definitions. Education agreed with these recommendations.

Although Department of Education (Education) data show that the  number of incidents of restraint and seclusion of students in K-12 public  schools is small, data analyzed for our June 2019 report showed that the  true nationwide extent of these practices cannot be determined because  the data do not reflect all incidents of restraint and seclusion. This is  particularly concerning because, as we reported earlier, some of the most  vulnerable public school students—students with disabilities—are  disproportionately affected. Our June 2019 report found that many of the  nation’s largest school districts erroneously reported to Education that  they had zero incidents of restraint or seclusion during the 2015-16  school year. In all, 70 percent of school districts nationwide reported  having zero incidents during that school year. As a stop-gap measure to  improve the quality of the 2017-18 school year data before it was  published, we made several recommendations to improve the accuracy  and completeness of that data. Education has taken some steps to  address them. At the same time, school districts around the country have  been in the spotlight for misreporting and underreporting incidents of  restraint and seclusion, further undermining confidence in the accuracy of  the data.

In broad terms, Education defines restraint as restricting a student’s  ability to freely move their torso, arms, legs, or head, and defines  seclusion as involuntarily confining a student alone in a room or area from  which the student is physically prevented from leaving. Education’s most  recent general resource document on the use of restraint and seclusion  states that restraint or seclusion should never be used except when a  child’s behavior poses imminent danger of serious physical harm to self  or others.

As part of its Civil Rights Data Collection (CRDC), Education collects and  publicly reports a range of data from nearly all public school districts,  including data on the use of restraint and seclusion. Education’s Office  for Civil Rights (OCR) uses CRDC data in its enforcement of various  federal civil rights laws prohibiting discrimination on the basis of race,  color, national origin, sex, and disability, making its accuracy of  paramount importance.

The explanatory statement from the House Committee on Appropriations  accompanying the Consolidated Appropriations Act of 2018 includes a  provision for us to study the CRDC’s restraint and seclusion data and  provide examples of practices schools are adopting to reduce the  incidence of restraint and seclusion. This report examines (1) the  effectiveness of CRDC data quality control procedures for its restraint and  seclusion data, (2) how selected districts interpret the CRDC definitions of  restraint and seclusion, and (3) how selected districts use data on  restraint and seclusion and encourage staff to report incidents.

To gather information on the extent to which Education ensures the  quality of federally reported data, we analyzed Education’s CRDC for the  2015-16 and 2013-14 school years. (The CRDC is administered every  two years; these were the most recent available during our review.) To  assess the effectiveness of Education’s procedures in identifying  inaccuracies, we reviewed the data quality control procedures Education  had in place for both the 2015-16 and 2017-18 school years, including  automated system checks, which Education refers to as “business rules.”  We reviewed the CRDC contractor’s post-collection data quality report for  the 2015-16 school year. Education did not identify any quality issues  related to restraint and seclusion data during its 2015-16 review. We also  reviewed documentation related to Education’s efforts to follow up with  school districts that had reported potentially inaccurate data, such as  correspondence between Education and school districts concerning data  quality issues.

To gather information on how selected districts use the restraint and  seclusion data and interpret federal restraint and seclusion definitions, we  visited nine school districts and 11 schools in Kentucky, Washington, and  Wisconsin. We interviewed over 50 school and district officials and  teachers. We selected states, districts, and schools to obtain a range of  perspectives on federal reporting of restraint and seclusion data and to  account for other criteria. Examples of selection criteria included districts  that had significant changes—an increase or decrease—in the number of  incidents of restraint and seclusion they reported to the CRDC across  reporting periods and states that had laws that require school districts to  report incidents of restraint and seclusion to the state. Information we  collected from our nine selected districts and 11 selected schools, while  not generalizable to all districts and schools, offers insights into how  some districts use their data and interpret the CRDC definitions of  restraint and seclusion. We also interviewed representatives from several  nonfederal advocacy organizations, including some that represent  parents and families of individuals with disabilities, and others that  represent relevant professional associations.

Finally, to inform all our objectives, we interviewed federal agency officials  and examined relevant federal statutes, regulations, guidance, and  selected state statutes. For more information on our methodology, see  appendix I.

We conducted this performance audit from November 2018 to April 2020  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

Schools generally report incidents of restraint and seclusion to their  district, and districts are responsible for reporting incidents of restraint  and seclusion to Education’s CRDC. Districts are expected to report the  number of incidents and the number of students affected for all schools in  their district and to use Education’s definitions of restraint and seclusion  to determine whether an incident occurred. Education defines two types  of restraint: physical and mechanical (see table 1). Education’s definition  of a physical restraint makes a distinction between a restraint and a  physical escort. Similarly, Education’s definition of seclusion makes a  distinction between seclusion and a timeout (see figures 1 and 2).

Every two years, OCR administers the CRDC to nearly every public  school district in the country. In turn, districts self-report information on a  wide variety of topics, including course offerings, discipline, and restraint  and seclusion. Education collects these data through an online  submission tool. CRDC activities, such as data collection and quality, are  managed by a company under contract with Education. The data  submission period for the 2017-18 school year ended June 21, 2019.  School districts have one year from the end of the submission period to  make a request to amend submitted data. As of March 2020, Education  had not announced when it will publicly release these data.

Education’s primary data quality checks for the CRDC data, including the  restraint and seclusion data, are built into the CRDC submission tool. The  online CRDC submission tool automatically performs checks that flag  data errors or potential errors. These “business rules” occur in real time  as districts enter data or after they upload files. The 2015-16 submission  tool used three business rules related to restraint and seclusion; for the  purpose of this report, we focused on the two rules most relevant to our  work.

The first business rule pertains to the reporting of zeros for very large districts only—that is, those districts with 100,000 or more enrolled students (see fig. 3). If a very large school district enters zero incidents of restraint and seclusion, it receives a message prompting it to review its enrollment counts and reported incidents, or provide an explanation using a reason code and comment. Importantly, if districts have not collected the data required for the CRDC—or if the data are unavailable for some other reason—districts are to leave relevant data cells blank. A zero in a data cell should represent an actual count— that is, the district restrained or secluded zero students.

The second rule applies to schools that both report more than 100 incidents of restraint or seclusion and enter a greater number of students affected by restraint or seclusion than the number of incidents reported (see fig.4). Districts are asked to resolve this error by adjusting their counts so the number of students subjected to restraint or seclusion is less than the number of incidents of restraint or seclusion, or to provide an explanation using a reason code and comment.

The last step in the data submission process is the district certification. To  complete this step, the district superintendent or an authorized designee  must indicate agreement with a statement that acknowledges that they  are responsible for verifying the data, the information provided is “true  and correct,” and a willfully false statement is punishable by law. The  CRDC submission tool is designed so that it should not allow a district to  certify its submission unless all required data pass the system validation  checks, or all errors are explained.

Education also reviews CRDC data quality during other phases of data  collection to identify potential data quality issues to improve future  collections. For school year 2015-16, Education’s contractor reviewed  data quality during and after the collection phase. Education’s contractor  contacted school districts about potential errors that Education  determined were easily adjustable and asked them to review and correct  data or provide an explanation if no corrections were determined to be  necessary.

Further, some states and school districts have laws, regulations, and/or  policies regarding restraint and seclusion. These laws vary from state to  state, and sometimes require schools or districts to annually report  incidents of restraint and seclusion to either the state or local education  agency.

In January 2019, Education announced an initiative to address  inappropriate use of restraint and seclusion on children with disabilities.  As part of this initiative, OCR announced plans to conduct 50 data quality  reviews of the 2015-16 restraint and seclusion CRDC data submitted  across each of OCR’s 12 enforcement regions. This review had not been  completed at the time of our review.

In August 2019, Education announced plans for OCR to work  collaboratively with Education’s National Center for Education Statistics  (NCES). Education stated that the collaboration would help ensure that  CRDC data are reliable and authenticated in a manner that provides a  more accurate picture of key civil rights issues in education. According to  the announcement, under a new agreement, NCES and OCR will work  together to improve the quality of the CRDC data by providing school  districts with technical assistance, and by reviewing and revising data  quality procedures. NCES is the primary federal entity responsible for  collecting and analyzing statistical data related to education.

Key Quality Control Processes for Restraint and Seclusion Are Ineffective or Do Not Exist

We identified four key issues for which Education’s quality control  processes for its CRDC restraint and seclusion data are largely ineffective  or do not exist (see table 2).

CRDC Business Rule Targeting Zero Reporting Is Largely Ineffective

Education’s business rule to detect potentially erroneous reporting of zero  incidents applied to only 30 of the more than 17,000 school districts  nationwide, rendering it largely ineffective for checking the 70 percent of  districts that reported zero incidents of restraint or seclusion. This is  because the rule only applied to districts with over 100,000 enrolled  students. However, in its January 2019 data quality review of 50 districts’  restraint and seclusion data, OCR found erroneous reporting of zeros in  districts of all sizes. Of the 50 districts OCR contacted, OCR determined  that 40 districts should amend their original 2015-16 submissions.

When we analyzed the 2015-16 CRDC restraint and seclusion data, we  found that almost three-quarters of small districts reported zeros, while  about one-third of large and one-fourth of very large districts reported  zero incidents (see table 3). The findings from Education’s data quality  review, along with those from our analysis, suggest that misreporting is a  problem among districts of all sizes.

For the 2017-18 CRDC data collection, Education lowered the threshold  for the rule to detect potentially erroneous reporting of zeros to include  districts with 50,000 enrolled students, rather than only districts with more  than 100,000. However, the new rule counts students with disabilities and  students without disabilities separately. To trigger the rule, a district would  have to have at least 50,000 students with disabilities and report zero  incidents for them, or have at least 50,000 students without disabilities  and report zero incidents for them. Of the nation’s more than 17,000  school districts, only 3 (Chicago, Los Angeles, and New York) have at  least 50,000 students with disabilities; only 95 have at least 50,000  students without disabilities.

Education’s CRDC data show that restraint and seclusion  disproportionately affect students with disabilities and its data quality  review showed that substantial portions of districts of all sizes  inaccurately reported zeros. However, Education could not provide a  data-driven basis for the 100,000 or 50,000 student enrollment thresholds  or for creating separate thresholds for students with and without  disabilities. Rather, Education stated that the thresholds were a  management policy decision inherited from previous administrations.

Because Education’s business rule targeting districts that report zeros  was inadequate, in June 2019, GAO recommended, among other things,  that for the 2017-18 data collection Education contact districts that  reported all zeros for restraint and seclusion to ask them to ensure that  the zeros actually represented zero incidents, and Education did so after  the data collection closed. Absent a business rule targeting all districts  reporting zeros during data submission, inaccuracies in future data  collections will likely be missed, and Education risks expending more time  and resources with repeated manual follow up after the fact. Federal  standards for internal control state that management should evaluate both  internal and external sources of data for reliability. Absent reliable and  accurate data, the public’s confidence in the CRDC restraint and  seclusion data may be further undermined, and the utility of a dataset  intended to assist with federal civil rights monitoring, enforcement, and  oversight will remain limited.

While it is clear that some school districts have reported inaccurate  restraint and seclusion data, Education officials do not fully understand  why this is occurring. In technical comments on a draft of this report,  Education stated that it is committed to learning more about why this is  occurring. While we do not know all of the reasons districts fail to report  accurate data, our interviews with over 50 school and district officials  provide some insight. School officials in the nine districts we visited cited  a variety of reasons districts might not report, including that they were not  collecting the data because their state did not require reporting, and that  their school district only required them to collect data for students with  Individualized Education Programs (IEPs). More fully understanding  why districts report inaccurate data is key to correcting the issue. Federal  standards for internal control also state that managers should use quality  information to achieve the entity’s objectives, assess the risks facing the  entity as it seeks to achieve its objectives, and use this assessment to  develop appropriate risk responses. By not identifying school districts’  reasons for reporting zero incidents of restraint and seclusion, Education  will not know how to best support districts in improving the accuracy of  their reported data. Future CRDC data will remain inaccurate, significantly  limiting the utility of a key tool on which OCR relies to help it enforce  federal civil rights laws.

CRDC Lacks Business Rules to Detect Very Low or Very High Rates of Restraint or Seclusion

Education has no business rules that flag school districts reporting very  low or very high rates of restraint or seclusion, nor has it completed initial  efforts to determine a range of rates that might warrant further  exploration. Given widespread concerns about misreporting, we devised  two possible ways to test for these types of outliers.

First, we looked beneath the district level to examine school-level  reporting patterns within districts. When we tested the nation’s 30 largest  school districts (those with more than 100,000 students), we found  patterns that may suggest underreporting in at least 13 of them, in  addition to the 10 that reported zeros for the 2015-16 school year. In  these 13 districts, we found that all of the incidents of restraint reported  occurred in no more than 15 percent of a district’s schools; the rest of the  schools in those districts reported zero incidents. (See fig. 5.) For  example, the Chicago Public School District—the third largest school  district in the country, with nearly 400,000 students enrolled—reported a  total of 47 incidents of restraint for school year 2015-16. All of these  incidents were reported by just two of its 579 schools. The district’s six  incidents of seclusion were clustered in one school. In the Los Angeles  Unified school district, the second largest school district in the country, 82  of its 785 schools reported a total of 108 incidents of restraint, with 65  schools reporting exactly one incident each. The district reported no  incidents of seclusion.

Education has a business rule that targets large districts (for 2017-18,  those with over 50,000 students), but only when all schools in a district  report zeros. Thus, as long as a large district reports at least one incident  of either restraint or seclusion, the business rule would not be triggered.  Education’s post-collection data quality reviews for school year 2015-16  did not test below the district level to look for potential underreporting  within a district.

Second, we tested for outliers by comparing per capita rates of restraint  or seclusion in the 30 very largest districts (over 100,000 students  enrolled) to average rates in all school districts. In the 30 districts, we  found that in addition to the 10 districts that reported zeros, nine districts  reported fewer than three incidents of physical restraint per 10,000  students, which is lower than 95 percent of all districts reporting incidents.  (See fig. 6.) For example, DeKalb County school district in Georgia  reported 0.3 incidents per 10,000 students, and Charlotte Mecklenburg  school district in North Carolina reported 0.5 incidents per 10,000  students.

We also tested for districts with very high rates of physical restraint. For  the 2015-16 school year, we found 52 districts that were outliers, most of  which served comparatively large populations of students with disabilities.  Forty-nine of these 52 districts had rates of physical restraint per enrolled  student higher than 99 percent of all districts that reported incidents of  physical restraint. Almost half reported an average of 10 or more  incidents per student affected, and almost two-thirds of the districts  reported restraining from 25 to100 percent of their students. The Learning  Tree preschools in Alabama, which enrolled a combined 135 preschool  students ages 3 to 5, reported that it restrained nearly two-thirds of its  students in school year 2015-16. Further, Learning Tree reported 5,963  incidents of physical restraint affecting 84 students, or an average of 71  incidents of physical restraint per preschooler. The Morris-Union Jointure  Commission School District in New Jersey, where almost all of its 281  students were identified as having a disability, reported restraining over  one-third of its students. These students were restrained an average of  20 times in school year 2015-16. (See table 4.)

We found a similar pattern in the 2015-16 seclusion data, identifying 36  outlier school districts. For 22 of the 36 districts, the rates of seclusion  were higher than 99 percent of districts reporting. (See appendix IV for  more information.) For example, CRDC data for the Sangamon Area  Special Education District in Illinois, which enrolled 74 students in grades  1 through 12, showed the district secluded one-third of its students an  average of 27 times each in school year 2015-16. Similarly, data for the  Bi-County Special Education Cooperative, also in Illinois, showed the  district secluded over two-thirds of its 48 enrolled students an average of  13 times each in school year 2015-16.

Federal standards for internal control state that management is to  determine if controls individually and in combination with other controls  are capable of achieving an objective and addressing related risks. An  internal control design may be deficient when a control necessary to meet  an objective is missing. Without business rules or similar analytical  processes to flag these outliers, they may continue to go undetected by  Education and other stakeholders. Education may be missing an  opportunity to identify districts with disproportionately low or high  incidents of restraint and seclusion to determine where technical  assistance or other intervention may be warranted. Such information is  particularly critical given widespread concerns about underreporting and  misreporting, and its stated interest in protecting students’ civil rights.

CRDC Business Rule Targeting Illogical Data Is Largely Ineffective

Education has a business rule that identifies illogical data; that is, when  schools report more students affected than incidents of either restraint or  seclusion. However, the rule is largely ineffective because it was not  designed in a way that would detect logical inconsistencies in the majority  of cases, as the rule would have only applied to schools with at least 100  incidents. When we tested Education’s rule on the 462 schools that  reported at least 100 incidents in 2015-16, we found no logical  inconsistencies in the data. However, when we tested the rule on all  schools, we found logical inconsistencies in the data reported by 592  schools with fewer than 100 incidents. For example, a school in Indiana  reported that it restrained 156 students, but only reported 80 incidents.  (See table 5.) Education could not provide any data-driven basis for its  threshold of 100 incidents for this business rule. Education officials said  that the threshold was inherited from previous administrations’ business  rules.

Collecting accurate data is key to the Office for Civil Rights’ (OCR)  mission to ensure equal access to education and to promote educational  excellence throughout the nation. In addition, federal standards for  internal control state that when evaluating the design of internal controls,  such as business rules, management should determine if controls are  capable of achieving an objective and addressing related risks. An  internal control design is deficient if, even though it operates as designed,  it does not meet the control objectives. Our analysis shows that the  business rule is not effective in its current form, because 592 schools  were able to report illogical, and therefore incorrect, data.

School and School District Officials Lacked a Common Understanding of Federal Restraint and Seclusion Definitions

We talked to more than 50 officials in nine school districts in Kentucky,  Washington, and Wisconsin about their interpretations of the CRDC’s  definitions of restraint and seclusion. These school districts all reported  incidents of restraint and seclusion in 2015-16, but officials we  interviewed differed in their interpretations of terms used in the CRDC  definitions, such as alone and escort. As a result, districts varied in how  they counted incidents of restraint and seclusion. Further, officials we  spoke with in the three state educational agencies and all seven  stakeholder groups with expertise on the use of restraint and seclusion in  public schools also said there was ambiguity regarding terms used in the  definitions. For example, an official from one stakeholder group that  represents some of the nation’s largest school districts said that its  constituents provided feedback that restraint and seclusion terms were  ambiguous, open to interpretation, and do not provide enough clarity.

Civil Rights Data Collection Definition of  Seclusion:   Seclusion is the involuntary confinement of a  student alone in a room or area from which  the student is physically prevented from  leaving. It does not include a timeout, which is  a behavior management technique that is part  of an approved program, involves the  monitored separation of the student in a non- locked setting, and is implemented for the  purpose of calming.

With respect to the definition of seclusion, district and school officials  varied in their interpretations of the word alone, and consequently,  whether the incident should be counted as seclusion. Officials in three  districts said that an incident was not seclusion as long as a teacher was  in the room with the student, while officials in several other districts said  that even if a teacher was present, it could still be seclusion if the student  was prevented from leaving. (See sidebar.)

Officials in the nine districts we visited also varied in their interpretation of  the word area. Because the CRDC’s definition of seclusion states that  seclusion can occur in an area, officials from one stakeholder group  representing thousands of school administrators wondered whether it  should be considered seclusion if a child is in a classroom with 20 other  students and is required to stay alone in the corner of the room. Officials  from a district in Wisconsin said that if a student is taken away from peers  and placed in one area of the same room, but cannot leave that area, it  still might be seclusion, even if the student and peers are in the same  room. Officials in another district in Wisconsin said that sending a student  to a corner does not count as seclusion. However, they said that the use  of mobile partitions to close off an area of a room could constitute  seclusion.

The phrase physically prevented from leaving also elicited differing  interpretations. Officials from the stakeholder group representing  administrators said the definition is not clear about what counts as  “prevented from leaving”: a barrier, such as a door; the presence of  another adult watching the child; or both. School officials we spoke with  had differing interpretations of this phrase, which affected how they  counted and reported incidents of seclusion. School officials in a district in  Kentucky said that the phrase means closing the door and keeping it  closed. However, officials in another school in the same district did not  specify the use of a door, stating instead that “physically prevented from  leaving” means the student cannot walk out of the room. A school official  in Washington said that it would count as seclusion if staff put a student in  a motorized wheelchair in a room and deactivated the wheelchair’s  power.

Civil Rights Data Collection Definition of  Physical Restraint:   Physical restraint is a personal restriction that  immobilizes or reduces the ability of a student  to freely move their torso, arms, legs, or head.  The term does not include a physical escort.  Physical escort means a temporary touching  or holding of the hand, wrist, arm, shoulder or  back for the purpose of inducing a student  who is acting out to walk to a safe location.

With respect to the definition of physical restraint, school district officials  we interviewed varied in their interpretations of the term escort, which the  CRDC definition specifies is not a physical restraint. (See sidebar.) While  officials in three districts said that an escort meant providing a physical  prompt to a student who was not resisting relocation, officials in four  districts said that moving a student who was resisting staff still counted as  an escort. For example, officials in a school in Wisconsin said that if the  student who is resisting is “carried away” from a location, that action  would not meet the definition of escort and would count as restraint. Yet a  district in Kentucky counted moving students against their will—including  by carrying them—as escorts and did not report them as restraints.  Further demonstrating differing interpretations of these terms, officials in  four districts said they reported all escorts as restraints in the CRDC.

Education does not provide schools or school districts with any  information that could help clarify its definitions or provide examples on  how schools and school districts should apply the definitions of restraint  and seclusion to common classroom situations. For example, while  Education’s “Restraint and Seclusion: Resource Document” includes the  CRDC definitions of restraint and seclusion, it does not include clarifying  information or examples about how to apply the definitions.

Officials in the schools and districts we visited inconsistently interpreted  the definitions for restraint and seclusion; moreover, officials from the  seven stakeholder groups we interviewed said the definitions were  unclear. These findings raise concern about whether restraint and  seclusion data reported by school districts to the CRDC are being  reported in a way that is consistent with the CRDC definitions. Federal  guidance on data reliability states that data should be well defined  enough to yield similar results in similar analyses. In addition, federal  standards for internal control state that agency management should use  quality information to achieve the entity’s objectives, noting that such data  should be reasonably free from error and bias and faithfully represent  what they purport to represent. Absent data on restraint and seclusion  that is what it purports to be, Education will continue to lack quality  information key to fulfilling its mission of ensuring equal access to  education nationwide.

All Nine School Districts We Visited Used Data to Reduce Incidence of Restraint and Seclusion and Developed Strategies for Improved Reporting

All Nine Districts Used their Data on Restraint and Seclusion to Reduce Use of Restraint and Seclusion

Officials in all nine school districts we visited said they used their data on  restraint and seclusion to help reduce its use. In addition to collecting  data for CRDC reporting purposes, these districts also collected and used  more current and more detailed data to help reduce the use of restraint  and seclusion. Officials in seven of the nine districts said they began  collecting the data when their state passed a law requiring reporting.  District officials identified several benefits to collecting data and using it to  develop strategies to reduce use of restraint and seclusion. Specifically,  officials said that the data helped them identify the following:  Behavior patterns. Officials in several districts told us that collecting and  reviewing data on restraint and seclusion helps them identify patterns in  staff and student behavior that may contribute to use of these practices.  Specifically, by identifying the circumstances under which a student’s  behavior tends to escalate, staff can strategize how to more effectively  respond so as to prevent the need to use restraint or seclusion. For  example, one official in an elementary school in Wisconsin said that if  staff notice more incidents occur on particular weekdays, they can  examine those days to understand what may be affecting students’  behavior. Similarly, a teacher of students with autism in a middle school in  Washington said that reviewing data helps staff, such as teachers,  paraprofessionals, and administrators, determine what triggered a  student’s behavior and then determine what to do differently to avoid  triggering the student. In all three states, we visited districts that required  staff to participate in a debriefing after each incident in an effort to  understand what might have triggered the event and to discuss strategies  to deescalate future incidents. For example, officials in Washington said  that the building administrator and all staff involved discuss every  incident. District officials consider this an important step for reducing use  of restraint and seclusion, and said holding the discussions was a “game- changer.”

Need for training. Officials in several districts said they examine data on  restraint and seclusion at the classroom and school level to determine if  staff need additional training, including on how to manage student  behavior, or appropriately use restraint or seclusion. For example, a  behavior coach for a Kentucky school district said that the data on  restraint and seclusion helps her determine if certain teachers could  benefit from more training on de-escalation techniques. A director of  student services in Washington said that he was concerned about the  rates of restraint and seclusion in the district, and after implementing  more training for teachers, the rates declined. Officials in another  Washington district said that after the district began collecting data in  response to state law, they discovered that staff were using restraint and  seclusion as punishment. As a result, district officials said that they  coached teachers on how to manage behavior differently and  emphasized that restraint and seclusion should not be an everyday  occurrence. Officials at an elementary school in Wisconsin said that de- escalation training helps staff understand that students are trying to  communicate with their behaviors. They said that when staff adopt the  perspective that students are trying to communicate, staff also see the  value of collecting data to improve how they respond to the students’  behaviors.

Need for student supports. Officials in five of the nine districts we  visited spoke about using restraint and seclusion data to assess when a  student required additional support services to be successful in the  classroom. For example, officials in one district in Kentucky said their  data provides evidence for obtaining additional staff or social emotional  learning resources for students. Similarly, officials in one Wisconsin  district said the data can be used to allocate funding for school-based  services to help address underlying causes of behavior. Officials in  another Wisconsin district said that a jump in restraints or seclusions of a  particular student could indicate that the student’s individualized  education program needs to be adjusted.

All Selected Districts Developed Strategies to Encourage Reporting of Incidents

Officials in all the districts we visited also shared strategies on how they  improved their CRDC data reporting, including communicating with staff  about how data are used, training on how to report, and developing  processes that encourage reporting. Specifically:  Communication and culture. Officials in the majority of school districts  said they routinely reviewed their data with school staff and emphasized  the value of collecting data on restraint and seclusion. Officials in a district  in Wisconsin said that they monitor data on restraint and seclusion on a  monthly basis for students with and without disabilities, which increases  interest among school staff about what causes the incidents. Officials in  three districts we visited said that they explain to staff that documenting  incidents of restraint or seclusion ensures that students obtain the support  services they need. In a Wisconsin district, officials said they emphasize  that reporting helps the students and keeps the school safe by making the  district aware that more supports are needed. Officials at a school in  Wisconsin said that some staff might worry that the data reflect poorly on  them or might fear repercussions, but district officials have worked to shift  the culture of reporting to focus on continuous improvement and problem  solving. Similarly, officials in another Wisconsin district said that schools  might be concerned about the data being used against them; therefore,  district officials try to create a culture of curiosity around the data, rather  than a culture of punishment.

Accountability. To encourage staff to report incidents, officials in some  districts developed processes that increased accountability for reporting.  For example, school officials at an elementary school in Wisconsin said  an administrative assistant in the main office immediately logs calls from  classroom teachers requesting help managing a student’s behavior.  Officials said this process provides accountability. Two districts said that  they used a team approach for restraint or seclusion, which included  someone to observe and someone to record details of the intervention,  such as the time it began or the events that preceded it. Having multiple  people involved increased the likelihood that relevant facts were  recorded. Officials in a district in Washington said that keeping teachers  and staff honest about reporting requires reiterating the process and  procedures, reviewing the forms with staff, and following up with schools  that fail to submit reports. Officials in a district in Wisconsin said they  have advised staff to write the incident down on paper until staff are able  to enter it in the district’s electronic reporting system.

Training. Officials from all nine districts said they encourage reporting by  provided training on how to report incidents. Generally, this information  was incorporated into trainings on when to use restraint and seclusion  and how to deescalate a student’s behavior. Officials from five  stakeholder groups we interviewed, all of whom have expertise related to  the use of restraint and seclusion in public schools, stated that training  was necessary to both raise awareness of the requirement to report  incidents and to ensure that incidents were reported accurately. For  example, in de-escalation training for teachers in a Washington district,  the trainer provides examples of restraint and seclusion; presents a  variety of scenarios, including ambiguous ones, for discussion; and  reviews the appropriate staff response.

The Civil Rights Data Collection (CRDC) is a longstanding and critical  aspect of Education’s Office for Civil Rights’ overall enforcement and  monitoring strategy. Collecting accurate data through the CRDC can help  Education in its mission to ensure equal access to education, promote  educational excellence for all, and enforce various federal civil rights laws  prohibiting discrimination on the basis of race, color, national origin, sex,  and disability. However, the significant data quality problems that both  Education and we identified with the CRDC data on restraint and  seclusion, combined with the significant weaknesses we found in  Education’s data quality control processes, cast serious doubt on the  accuracy of these data. As a result, it is impossible to accurately  determine the frequency and prevalence of restraint and seclusion among  K-12 public school students. The four recommendations in our June 2019  report urged Education to take immediate steps to address the  widespread potential misreporting of zeros for its 2017-18 CRDC.  Education took some steps to address the issues we raised, but has not  yet fully addressed them. Moreover, those recommendations were  intended as stop-gap measures to improve the quality of the 2017-18 data being collected in real time precisely because the CRDC’s business  rules related to restraint and seclusion are inadequate. Therefore,  addressing our recommendations would not solve the issues that are the  subject of this report. Our work makes it clear that an overhaul of the  quality control processes is needed to correct fundamental problems with  federal restraint and seclusion data collected through the CRDC.

Two of the CRDC’s key business rules meant to check data quality and  flag potential errors in restraint and seclusion data are poorly designed  and the thresholds that trigger these rules have no data-driven basis.  Further, Education does not have business rules designed to flag outlier  schools and school districts that report relatively low or high rates of  restraint and seclusion, nor has it determined a range of rates that might  warrant further exploration. Until Education more fully understands why  so many school districts are underreporting and misreporting federal  restraint and seclusion data, it will likely not be able to help districts  improve their reporting, thereby improving the accuracy and utility of the  data.

There were widely varied interpretations of federal restraint and seclusion  definitions among the 50 school and district officials with whom we spoke  and officials from the seven key stakeholder groups we interviewed  echoed these concerns. As a result, we have concerns that school  districts may be inconsistently counting and reporting instances of  restraint and seclusion for federal reporting purposes. Clarifying the  definitions, for example by explaining to districts how they can be applied  to common classroom scenarios, could help produce more consistency in  reporting.

Ultimately, the issues we found with Education’s restraint and seclusion  data have consequences for the students who are restrained or secluded  in school and whose restraint or seclusion goes un-reported. When  federal data are misreported to the public, it undermines confidence in  that data and fails to provide decision makers with reliable information on  which to make informed policy decisions to protect students. In addition,  Education lacks information that could help it determine whether schools’  use of these practices may be excessive, discriminatory, or both.

GAO is making six recommendations on restraint and seclusion to the  Department of Education’s Office for Civil Rights.

The Assistant Secretary for the Office for Civil Rights should revise its  CRDC business rule to require that every school district reporting zeros,  regardless of district size or numbers of students with disabilities, affirm  the zeros are correct during the CRDC data submission process.  (Recommendation 1)

The Assistant Secretary for the Office for Civil Rights should develop and  implement a CRDC business rule that targets schools and school districts  that report very low numbers of incidents and set data-driven thresholds  to detect such incidents. (Recommendation 2)

The Assistant Secretary for the Office for Civil Rights should develop and  implement a CRDC business rule that targets schools and schools  districts that report very high number of incidents and set data-driven  thresholds to detect such incidents. (Recommendation 3)

The Assistant Secretary for the Office for Civil Rights should apply the  CRDC business rule targeting illogical data at the school level to all  schools, regardless of the number of incidents reported.  (Recommendation 4)

The Assistant Secretary for the Office for Civil Rights should identify the  factors that cause underreporting and misreporting of restraint and  seclusion and take steps to help school districts overcome these issues.  (Recommendation 5)

The Assistant Secretary for the Office for Civil Rights should further refine  and clarify federal restraint and seclusion definitions and take steps to  ensure that this information is conveyed to school districts. This could  include providing common classroom scenarios that highlight the  differences between a restraint and an escort, and a time out and a  seclusion. (Recommendation 6)

We provided a draft of this report to the Department of Education for  review and comment. In its formal comments, which are reproduced in  appendix II, Education agreed with all six recommendations. Education  also provided technical comments, which we incorporated, as  appropriate.

In agreeing with GAO’s six recommendations, Education stated that it  would determine the best means to implement them. Education also  stated that it is fully committed to working with public schools, state  educational agencies, and school districts to help ensure accurate  reporting of  federal restraint and seclusion data, and to improve the  quality of the information for all users of CRDC data. We appreciate  Education’s willingness to address the serious data quality issues  affecting the CRDC restraint and seclusion data.

In its response, Education stated that the agency has already made  significant improvements to the CRDC in general and has made specific  improvements with respect to restraint and seclusion data, especially in  response to the four recommendations we made in our June 2019  correspondence. Education asked that we acknowledge the progress it  feels it has made in this regard, and we have done so. Importantly,  however, our June 2019 recommendations were intended as stop-gap  measures to improve the quality of the 2017-18 data that was already  being collected in real time precisely because the CRDC’s business rules  related to restraint and seclusion were inadequate. Therefore, steps  Education has taken toward addressing them do not address the  underlying data quality issues that are the subject of this report. In other  words, the recommendations in this report urge Education to address  data quality problems at the front-end by applying adequate business  rules at the time districts submit their data. This could reduce the need for  follow-up with districts to correct potentially inaccurate data. More  information about our assessment of the steps Education has taken to  address the four recommendations from the June 2019 report are  available on our website.

Education also stated that because our draft report did not mention the  methodological improvements OCR made to address the quality of  restraint and seclusion data for the 2017-18 CRDC data collection, our  draft report overstates the relevance of the data issues from the 2015-16  collection. Education also stated that it provided us with information about  the methodological improvements in December 2019, and, in its formal  response, requested that we reflect the information in this report. We  disagree with this perspective. After we completed our audit work for this  engagement, Education provided us an excerpt from its post-collection  data quality report for school year 2017-18. At that time and again in its  technical comments on this report, Education stated that “information  shared with GAO about the results of the 2017-18 data quality review  process and what might be addressed is still confidential.” As of March  23, 2020 Education described the 2017-18 data quality control process as  “incomplete.” Lastly, the 2017-18 CRDC data, which are the topic of the  excerpt Education provided to us, are not yet available. Under our  auditing standards, we cannot opine on the quality of data we could not  independently assess or on the efficacy of process improvements  associated with those data.

Education also raised concerns about how we weighted our interviews  with school officials, and it questioned the relevance of our discussions  about selected school districts’ use of restraint and seclusion data not  reported for CRDC purposes. We disagree.

Education was concerned about the weight GAO placed on  information obtained from 50 officials in 11 school districts across 3  states whom we interviewed during the course of our audit work. As  stated in the report, this information cannot be generalized to all  districts. However, we believe it provides useful insights into how  some districts use their restraint and seclusion data to reduce the  incidence of these practices and improve the accuracy of their data.  The widespread disagreement among the 50 school officials with  whom we spoke also highlights confusion about how to accurately  and consistently apply CRDC definitions of restraint and seclusion.  This finding is supported by the views of seven nonfederal advocacy  organizations that represent parents and families; individuals with  disabilities; and other stakeholders, such as representatives of  relevant school and special education professional associations.

Education questioned the relevance of discussing the benefits that  selected school districts said they derive from using restraint and  seclusion data not reported for CRDC purposes. Education stated that  “attempting to generalize comments about how these nine school  districts use restraint and seclusion data” seems inconsistent with  Governmental Accounting Standards Board (GASB) statistical  principles. We believe that describing selected school districts’ use of  their restraint and seclusion data is within the scope of our stated  audit objectives. In addition, the explanatory statement from the  House Committee on Appropriations accompanying the Consolidated  Appropriations Act of 2018 includes a provision for us to provide  examples of how schools are adopting effective alternatives to these  practices and reducing the incidence of seclusion and restraint,  among other things. Further, Education mistakenly asserts that none  of the data and analyses that the school districts collected, performed,  or used are part of the CRDC and none could be feasibly collected by  the CRDC. We have further clarified in the final report that portions of  the data these school districts collect are used for CRDC reporting  purposes. For example, some of the data elements are the same  ones that districts use to calculate aggregate incident counts, which  are required by the CRDC. We do not recommend that Education  collect such detailed data or perform such analyses. Regarding  Education’s concern about “GASB statistical principles” and case  selection, all GAO performance audits are subject to Generally  Accepted Government Auditing Standards (GAGAS); in contrast,  GASB’s Generally Accepted Accounting Principles apply to financial  audits of public entities. The applicable methodological guidance we  followed -- Selecting a Sample of Nongeneralizable Cases for Review  in GAO Engagements -- is designed to ensure that GAO policies on  evidence and GAGAS are met, and conforms to the generally  accepted principles and practices of the appropriate disciplines. When  providing illustrative examples, it is neither necessary nor appropriate  to use statistical methods to analyze and interpret evidence.

Finally, in its comments, Education stated that it is critical that we  emphasize that the CRDC is an aggregate of self-collected and self- reported data from school districts and that the district superintendent or  an authorized designee certifies that the data they submit are “true and  correct.”  We agree, and acknowledged this in several places in both the  draft and final reports. At the same time, we believe that self-certified data  does not absolve Education of its responsibility to ensure the quality of  the data it collects and publicly reports – especially given the CRDC’s  longstanding role in Education’s overall enforcement of various federal  civil rights laws prohibiting discrimination on the basis of race, color,  national origin, sex, and disability. Self-reported data by nature are  subject to error, making the need for effective quality control measures  before, during, and after collection a necessity.

We are sending copies to the appropriate congressional committees, the  Secretary of Education, and other interested parties. In addition, the  report will be available at no charge on the GAO website at  http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (617) 788-0580 or nowickij@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who made key contributions to this report  are listed in appendix V.

Appendix I: Objectives, Scope, and Methodology

This report examines (1) the effectiveness of CRDC data quality control  procedures for its restraint and seclusion data, (2) how selected districts  interpret the CRDC definitions of restraint and seclusion and (3) how  selected districts use data on restraint and seclusion and encourage staff  to report incidents. Below are the details of our analysis to determine the  extent to which Education ensures the quality of restraint and seclusion  data reported by school districts, and of our interviews with officials in  selected districts about how they apply Civil Rights Data Collection  (CRDC) definitions of restraint and seclusion and use restraint and  seclusion data.

To inform all of our objectives, we interviewed federal agency officials,  representatives from several nonfederal advocacy organizations that  represent parents and families, individuals with disabilities, and other  stakeholders, such as representatives of professional associations. We  also reviewed agency documentation, relevant federal laws, regulations  and policies, and selected state laws.

Analysis of National Restraint and Seclusion Data  To determine the extent to which Education ensures the quality of  restraint and seclusion data reported by school districts, we analyzed  Education’s Civil Rights Data Collection (CRDC) for school year 2015-16.  Specifically, we analyzed the CRDC to determine the extent to which  districts reported zero incidents of restraint and seclusion, to identify  outliers (districts that reported a high or low incidence of restraint and  seclusion), and to identify illogical data. CRDC is a biennial survey that is  mandatory for nearly every public school and school district in the United  States and is conducted by Education’s Office for Civil Rights. The  CRDC collects data on the nation’s public schools (pre-K through 12th  grade) that includes the use of restraint and seclusion, student  demographics and enrollment numbers, educational and course offerings,  and disciplinary actions. In school years 2013-14 and 2015-16, the CRDC  collected data from nearly every public school in the nation  (approximately 96,000 schools in 17,000 school districts in school year  2015-16).

CRDC data are self-reported by districts and schools, and consequently  there is potential for misreporting of information. After reviewing their  CRDC data, school districts can submit revised data to Education. The  public-use data file of the CRDC for school year 2015-16 was the primary  source of data for our analyses and the most recent data available at the  time. We also used restraint and seclusion data from school year 2013- 14 primarily to analyze how use of restraint and seclusion may have  changed between the two time periods.

The CRDC collected data on (1) mechanical restraint, (2) physical  restraint, and (3) seclusion. Using these data, we performed the following  analyses to determine potential inaccuracies or underreporting in the  CRDC.

Analysis of Extent of Districts Reporting Zeros  To examine the extent to which school districts reported zeros, we  calculated the percentage of districts and schools reporting zeros for  restraint (both mechanical and physical) and for seclusion. We performed  this calculation for both districts and schools nationally and by state,  district size, and school type (e.g., charter, traditional, and special  education schools). Although Education has a business rule that targets  very large districts that report zero incidents of restraint or seclusion, we  calculated the number of all districts and schools that reported zeros to  understand the prevalence of zeros in the reported data.

Analysis of Relatively Low Rates of Restraint and Seclusion  Incidents  To test for potential underreporting, we first limited our analysis to the  restraint and seclusion data reported by the 30 largest school districts in  the nation (districts with over 100,000 students enrolled). Because of  these districts’ size, we reasoned that they would be more likely to have  incidents of restraint and seclusion to report. Our analysis found that 20 of  the 30 largest school districts reported incidents, and thus we focused our  analysis of underreporting on the 20 largest districts that reported  incidents. For each of the 20 districts, we calculated the percentage of  schools that reported incidents. To compare the 20 largest districts that  reported incidents with all 5,252 districts that reported incidents, we  calculated the rates of restraint and seclusion per enrolled student and  calculated percentile ranges. (See table 6.) We determined that nine of  the 20 districts had incidents of physical restraint per enrolled student that  were below the 5th percentile of all districts reporting incidents of physical  restraint.

Analysis of Relatively High Rates of Restraint and Seclusion  Incidents  To identify school districts with relatively high rates of restraint and  seclusion, we examined districts that reported having more incidents than  students enrolled. This analysis potentially indicates that some students  may have been restrained or secluded multiple times. To illustrate, if a  school district reported that it had 24 students enrolled, and also reported  that it had 100 incidents of restraint, these reported data would indicate  that the reporting was erroneous or that some students were restrained  multiple times. Based on this analysis, we then calculated the average  number of incidents (of restraint and seclusion) per student affected.

Analysis of Extent of Illogical Data  To test for illogical data, we analyzed the restricted-use restraint and  seclusion data file for schools that reported more students affected than  incidents. To illustrate, if a district reported that a school had restrained  80 students, and also reported that the school had 40 incidents of  restraint, these reported data are illogical. Education has a business rule  to detect illogical data at the school level, but the rule applies only to  schools with more than 100 incidents. For our analysis, we looked for all  schools with illogical data to determine the prevalence.

School District Interviews on Interpreting CRDC Definitions of  Restraint and Seclusion  To determine how selected school districts interpret the CRDC definitions  of restraint and seclusion, we selected 11 schools and nine school  districts in three states to serve as illustrative examples. In total, we  interviewed about 50 school officials. Information we collected from our  11 selected schools and nine districts cannot be generalized to all  districts and schools nationwide.

We selected states, districts, and schools to obtain a range of  perspectives on federal reporting of restraint and seclusion data. Our  selection also accounted for other criteria, such as selecting states that  had laws requiring reporting; high or low rates of reporting zeros among  districts; relatively high or low rates of restraint or seclusion per capita;  grade levels served (e.g., K-6 or 9-12); school type (e.g., traditional or  charter); and significant changes—increase or decrease—in incidents  across reporting periods. We also selected districts that had reported  incidents. As a result, we selected nine school districts to visit: two in  Kentucky, three in Washington, three in Wisconsin, and a charter district  in Wisconsin (see table 7).

To determine how district and school officials, such as assistant  superintendents, program managers, department directors, principals,  and teachers, were interpreting the CRDC definitions of restraint and  seclusion, we made the following statements and asked the following  questions in our interviews.  1.  We are going to talk to you about the definitions of restraint and  seclusion that appear in the CRDC. We have heard that these  definitions are not always clear to educators, so we want to get your  feedback.

Mechanical Restraint: the use of any device or equipment to  restrict a student’s freedom of movement.

Do you think this definition is clear or does it leave room for  ambiguity?

Physical Restraint: a personal restriction that immobilizes or  reduces the ability of a student to move his or her torso, arms,  legs, or head freely. The term does not include a physical escort.  Physical escort means a temporary touching or holding of the  hand, wrist, arm, shoulder or back for the purpose of inducing a  student who is acting out to walk to a safe location

How do you differentiate between physical escort and physical  restraint?

Does breaking up a fight constitute a restraint?

Seclusion: the involuntary confinement of a student alone in a  room or area from which the student is physically prevented from  leaving. It does not include a timeout, which is a behavior  management technique that is part of an approved program,  involves the monitored separation of the student in a non-locked  setting, and is implemented for the purpose of calming.

How do you differentiate between timeout and seclusion?

What does physically prevented (from leaving) mean?

In what types of physical spaces can seclusion occur?

Does your district have dedicated spaces for seclusion rooms?

Can you describe where they are generally located, e.g.,  which types of schools or classrooms?  2.  How do staff determine when an incident needs to be recorded as a  restraint?  3.  How do staff determine when an incident needs to be recorded as a  seclusion?

Appendix II: Comments from the Department of Education

Appendix III: Restraint and Seclusion Data for 20 Largest School Districts Reporting Incidents

Appendix IV: School Districts with Relatively High Rates of Reported Seclusion

Appendix V: GAO Contact and Staff Acknowledgments

In addition to the contact named above, Sherri Doughty (Assistant  Director), Lara Laufer (Analyst-in-Charge), Morgan Jones, and Kristin  Petroff, made key contributions to this report. Also contributing were  James Bennett, Deborah Bland, Tonnye Conner-White, Holly Dye, Gretta  Goodwin, Sheila R. McCoy, Jean McSween, John Mingus, James Rebbe,  and Manuel Valverde.

The Department of Education requires public school districts to biennially report incidents of restraint (restricting a student’s movement) and seclusion (confining a student to a space alone).

Education’s data quality checks may not catch misreporting or statistical outliers:

70% of districts reported 0 incidents of restraint and seclusion, but Education’s quality check only applies to fewer than 100 large districts

Education doesn’t have a quality check for districts reporting relatively high incident rates like one that reported an average of 71 restraint incidents per student per year

Our recommendations are to improve data quality