Higher Education: Approaches and Strategies Used in College Campus Surveys on Sexual Violence

Sexual violence–which can include crimes such as rape and other forms of sexual coercion–is widely acknowledged as a problem on college campuses. Although Education collects some data on sexual violence at colleges that receive federal funding, measuring the prevalence of campus sexual violence has proven difficult, due in part to underreporting of these incidents to law enforcement. While some researchers have used surveys to gather additional information regarding sexual violence on college campuses, estimates from these surveys can vary widely due to factors such as differing methodologies and response rates.

This report examines (1) key stakeholders' views on the strengths and limitations of campus climate surveys on sexual violence, (2) approaches selected colleges have taken to survey their students, and (3) the role federal agencies play in helping colleges develop and implement these surveys. GAO reviewed documentation for three widely administered survey instruments, and relevant federal laws, regulations, and guidance. GAO interviewed 25 stakeholders, including researchers; Education and Justice officials; officials in four states that required or recommended campus climate surveys as of January 1, 2017, a date selected to allow time for implementation; and 10 colleges—including seven that conducted campus climate surveys—selected based on program length (2- or 4-year), geographic diversity, and other factors.

Campus climate surveys that examine sexual violence occurring on individual college campuses have several strengths and limitations, according to stakeholders GAO interviewed.

Strengths. Nearly all stakeholders said colleges can use these surveys to gather more comprehensive information about incidents of campus sexual violence, such as those not previously reported to the colleges or law enforcement. Surveys can also provide information on students' knowledge of the colleges' procedures for reporting incidents, among other topics, which can help colleges identify areas for improvement.

Limitations. Most stakeholders said getting students to respond can be challenging. In addition, about half of stakeholders said some colleges may not have the resources to effectively administer these surveys, and results across colleges that use different surveys may not be comparable.

The seven selected colleges that conducted surveys reported using various approaches to survey their students about the incidence of campus sexual violence. Each college used one of three widely used surveys, but six modified them to some extent. Six colleges sent the survey to all undergraduates, and one surveyed a representative sample of students. Colleges also reported using multiple outreach strategies to increase participation, including offering incentives, such as gift cards, to students who completed the survey; using social media; and involving student leaders (see figure). Colleges' reported response rates ranged from less than 10 percent to more than 60 percent.

The Departments of Justice (Justice), Education (Education), and Health and Human Services (HHS) have created and disseminated informational resources for colleges interested in conducting campus climate surveys. For example, from 2014 to 2017, Justice made funding available for the development of a campus climate survey instrument for public use, and developed technical assistance materials covering various topics, including how to choose survey respondents and protect their confidentiality. In addition, in 2015, Education issued guidance encouraging colleges to develop ways to survey students about the campus climate. Justice and HHS have also funded grant programs that allowed grantees to use some funding to conduct campus climate surveys.

Sexual violence, which can include crimes such as rape and other forms  of sexual coercion, is widely acknowledged as a problem on college  campuses. College students who are victims of sexual violence may  suffer from a variety of physical and mental health conditions, which may  also impact their academic performance. Measuring the prevalence of  campus sexual violence, however, has proven difficult. Data collected by  the Department of Education (Education) from colleges under the Jeanne  Clery Disclosure of Campus Security Policy and Campus Crime Statistics  Act (Clery Act) indicate that an estimated six sex offenses per 10,000  college students were reported to law enforcement or campus security  authorities in 2016, but research shows that traditional crime statistics  like these tend to understate rates of sexual violence. Victims may not  report incidents of sexual violence to law enforcement or other designated  officials for various reasons, including feelings of guilt, shame,  embarrassment, or fear.

Due to concerns about underreporting of sexual violence to law  enforcement, some researchers have administered surveys to gather  additional information regarding sexual violence, such as estimating its  prevalence on college campuses. For example, a widely cited 2007 study  conducted at two colleges estimated that nearly one in five female  undergraduate students experienced attempted or completed sexual  assault since entering college. However, estimates from these types of  studies can vary widely due to factors such as differing survey  methodologies, response rates, and metrics.

In 2014, a White House Task Force recommended that colleges  voluntarily begin using campus climate surveys to help examine campus  sexual violence. These surveys may include questions about students’  experiences with sexual violence, knowledge of the college’s policies and  procedures for reporting incidents, and perceptions about how well  campus authorities and local law enforcement handle reports of sexual  violence. In recent years, colleges across the country have conducted  campus climate surveys focused on the issue of sexual violence.  However, as researchers and others have noted, the findings from  individual campus climate surveys may not reflect the experiences of  students nationwide and the different survey instruments share some of  the same methodological limitations, such as a lack of standard  definitions and metrics.

You asked us to review issues related to campus climate surveys on  sexual violence and efforts by federal agencies to help colleges develop  and implement these surveys. This report examines (1) what key  stakeholders view as the strengths and limitations of using campus  climate surveys to examine the incidence and characteristics of sexual  violence on college campuses; (2) what approaches selected colleges  have used to survey their students about the incidence of sexual violence  on campus; and (3) what role federal agencies play in helping colleges  develop and implement campus climate surveys on sexual violence.

To address these objectives, we reviewed survey instruments and  methodological documentation for three widely administered and publicly  available campus climate surveys to understand the strengths and  limitations of using such surveys. Specifically, we reviewed surveys  developed by the Association of American Universities (AAU), the  Administrator Research Campus Climate Collaborative (ARC3), and the  Department of Justice (Justice).

We also interviewed a total of 25 stakeholders, including campus climate  survey developers and other researchers who have studied campus  sexual violence, officials from selected states and colleges, and federal  officials from Education and Justice. We identified researchers to interview by reviewing research on sexual violence. We also spoke with  survey developers and officials at organizations responsible for  developing the three climate survey instruments we reviewed. The  findings from our interviews summarize selected stakeholders’ views  regarding campus climate surveys on sexual violence and do not  represent the views of all researchers, states, and colleges on these  topics.

We also selected 10 colleges for interviews with school officials. We took  several steps to identify these colleges since there is no central repository  of information on whether colleges have conducted a campus climate  survey. We analyzed data from Education’s Integrated Postsecondary  Education Data System on the characteristics of colleges to develop an  initial list of colleges that participate in federal student aid programs. We  then conducted targeted web searches to identify colleges that had  publicly reported using one of the three survey instruments included in  our review. We judgmentally selected 10 colleges, including seven that  have conducted campus climate surveys to examine the incidence of  sexual violence on their campuses and three that have not. We selected  these colleges to obtain variation in size, sector (i.e., public, private not- for-profit, and private for-profit), program length (i.e., 2-year and 4-year),  geographic location, survey instrument used, and whether the college  was located in a state that as of January 1, 2017 had a statutory  requirement in effect for at least some colleges in their state to conduct a  climate survey.

We also conducted semi-structured interviews with officials from four  states regarding the use of campus climate surveys in their state. We  selected three states (Louisiana, New York, and Washington) that had, as  of January 1, 2017, a statutory requirement in effect for at least some  colleges in their state to conduct a campus climate survey, and one  state (Ohio) that recommended colleges conduct such surveys. We  identified these states by consulting with researchers and reviewing  relevant reports, and we confirmed applicable state requirements with  state officials. The selected states varied in the nature of the survey  requirement or recommendation, such as the types of colleges covered  (e.g., public or public and private) and how frequently the survey was  required or recommended to be administered.

To inform all of our objectives, we reviewed relevant federal laws and  regulations, and also reviewed federal guidance pertaining to campus  sexual violence. We reviewed informational resources from Education  and Justice to help colleges conduct campus climate surveys, and  information about grants provided by Justice and the Department of  Health and Human Services (HHS). See appendix I for additional details  on our scope and methodology.

We conducted this performance audit from July 2018 to April 2020 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

Campus Climate Surveys on Sexual Violence

Campus climate surveys on sexual violence are designed to collect  information on the incidence and characteristics of sexual violence on  college campuses as well as related student attitudes and behaviors.  The topics covered by campus climate surveys can vary, depending on  the questions included on the survey instrument. For example, these  surveys may include questions about incidents of sexual violence, such  as the number of incidents of sexual assault, intimate partner violence, or  stalking, among other topics. There are two different methods that  colleges can use to administer these surveys:  In a census survey, all members of a group, such as the student body  of a college, are surveyed. This type of survey can be used when the  group that is the focus of the survey is small, when substantial  resources and time are available to obtain enough responses to the  survey, or when there is reason to provide all members of the group  the opportunity to participate.

In a sample survey, a portion of the group is selected using statistical  methods to provide accurate information about the larger group.  Administering a survey to a sample of students can reduce the time  and resources needed to obtain enough survey responses to produce  accurate data. Sample-based surveys are appropriate when it is not  practical or desirable to survey every member of a group.

With both sample and census-based surveys, collecting data that  accurately represents the experiences of respondents requires taking a  variety of steps when designing, administering, and analyzing the survey,  such as weighting or analyzing the completed responses to ensure that  they represent the larger group.

A 2014 White House Task Force report recommended conducting these  surveys as an initial step in a college’s plan to address campus sexual  assault. The report also suggested follow-up actions for colleges to  consider, such as providing training for college officials and creating  partnerships with community sexual assault support services. In addition,  there have been efforts to compare campus climate survey results across  colleges.

Some states have also enacted laws requiring colleges in their state to  administer campus climate surveys. These state laws may vary in the  nature of the survey requirements, such as the types of colleges covered  by these requirements and whether a particular survey instrument must  be used. For example, Louisiana requires public colleges in the state to  administer these surveys, while New York requires all colleges located  within the state to do so. Additionally, Louisiana requires that schools  use a standard survey instrument developed by the state, while New York  allows colleges to select their own survey instrument. However, there is  currently no federal requirement for colleges to conduct campus climate  surveys on sexual violence.

Federal Efforts Related to Addressing Campus Sexual Violence

Education, Justice, and HHS currently engage in a variety of efforts to  address sexual violence on college campuses, including overseeing  relevant federal laws and funding prevention and response activities.  Education and Justice oversee colleges’ compliance with Title IX of the  Education Amendments of 1972 (Title IX), which prohibits discrimination  on the basis of sex in any education program or activity that receives  federal financial assistance. Title IX prohibits sex discrimination— including sexual harassment and sexual violence—that effectively denies  victims equal access to recipients’ educational opportunities or benefits.  Under Education’s regulations, colleges receiving federal financial  assistance from Education, such as those participating in federal student  aid programs, must establish procedures for resolving Title IX complaints,  and take steps to ensure that members of the college community are  aware of their rights under Title IX. In addition, these colleges must  designate at least one employee to coordinate their efforts to comply with  and carry out their responsibilities under Title IX. According to Education  guidance, the Title IX coordinator is responsible for coordinating the  college’s response to all complaints involving possible sex discrimination,  including monitoring outcomes, identifying and addressing any patterns,  and assessing effects on the campus climate.

Education also oversees the Clery Act, which requires colleges that  participate in student financial assistance programs under Title IV of the  Higher Education Act, as amended, to collect statistics on certain crimes  that occur on or near their campuses, including specified sex offenses,  publish those statistics in an annual security report, and annually report  them to Education. Colleges must also include a policy statement in  their annual security reports describing their sexual violence prevention  and awareness programs for students and employees. In addition, Justice  and HHS have funded grants for campus sexual violence prevention and  response efforts. HHS has also developed a technical assistance  document for planning and implementing sexual violence prevention  strategies on college campuses.

Federal Data Sources Related to Campus Sexual Violence

Education and Justice also manage key efforts to collect data related to  campus sexual violence (see table 1). For example, Education oversees  the Campus Safety and Security Survey, which collects information from  colleges that participate in student financial aid programs on reported  criminal incidents, including specified sex offenses, which occur on or  near campuses that the colleges own or control, as required by the Clery  Act. Colleges are required to include data on specified crimes that are  reported to local police or campus security authorities and that occurred  (1) on campus (including the subset of crimes that occurred in on-campus  student housing facilities), (2) on public property within or immediately  adjacent to campus, and (3) in or on non-campus buildings or property  the college owns or controls. The survey collects data on the following  offenses related to sexual violence: rape, fondling, incest, statutory rape,  domestic violence, dating violence, and stalking. Education publishes the  data on a public website.

Justice collects data on crimes, including sex crimes, through the Bureau  of Justice Statistics’ National Crime Victimization Survey (NCVS). The  NCVS captures data on a range of offenses related to sexual violence:  completed rape, attempted rape, threatened rape, sexual assault other  than rape or attempted rape, unwanted sexual contact with or without  force (e.g., grabbing, fondling), verbal threat of sexual assault other than  rape, and stalking. The NCVS collects data through in-person interviews  and phone calls with a nationally representative sample of households on  the frequency, characteristics, and consequences of criminal victimization  in the United States. In particular, the NCVS collects information about  crimes reported and not reported to the police.

Although the NCVS includes certain group residences, such as college  residence halls, in its sample of households, the resulting data may not  fully represent the sexual victimization experiences of college students  residing on campus because the sample is primarily comprised of  households. Research has found that individuals living in group  residences may be at higher risk of sexual violence. Research has also  noted concerns with how the NCVS is administered. Specifically,  interviews are conducted in person at respondents’ homes or over the  phone. As a result, victims may be less likely to honestly answer sensitive  questions, such as those related to sexual violence, as their responses  might be overheard by other members of their household or the  offender.

Stakeholders Said Campus Climate Surveys Provide Insights into Campus Sexual Violence, but Colleges Face Challenges Administering Them

Stakeholders we interviewed, including survey developers, other  researchers, and federal, state, and college officials, considered campus  climate surveys a useful tool for learning more about the incidence of  campus sexual violence and identifying areas for improvement to address  it. However, stakeholders also noted that colleges face a variety of  challenges with developing and conducting surveys, such as limited  access to needed survey expertise and low response rates, which can  affect the reliability of campus climate survey results.

Campus Climate Surveys Can Provide More Information about Students’ Experiences with Sexual Violence and Their Awareness of Related Policies, Procedures, and Services

Surveys Can Provide More Comprehensive Information on Incidents of Sexual Violence

Nearly all stakeholders said that campus climate surveys provide an  opportunity to learn more about the incidents of sexual violence occurring  on individual campuses, such as those that students may not have  previously reported to campus authorities or law enforcement.  According to Justice officials and one researcher, campus climate  surveys, which collect data directly from victims, can help overcome  limitations in law enforcement data that rely on victims reporting to  authorities (see sidebar). For example, the three campus climate surveys  we reviewed are designed to capture information on incidents of sexual  violence that students have experienced regardless of whether the  incidents were previously reported to campus authorities or law  enforcement.

Underreporting of Traditional Crime  Statistics  According to a National Research Council  panel, rape and sexual assault are generally  underreported to law enforcement, which can  affect traditional crime statistics for these  incidents. For example, according to a 2014  Department of Justice report, National Crime  Victimization Survey (NCVS) data showed the  rate of rape and sexual assault for female  college students was 6.1 per 1,000 (the 95  percent confidence interval ranges from 5.0 to  7.2 percent) for the period 1995–2013, and an  estimated 80 percent of rape and sexual  assault incidents went unreported to police  (the 95 percent confidence interval ranges  from 75 to 85 percent). College students  responding to the NCVS who indicated they  did not report incidents of rape and sexual  assault to police cited a variety of reasons,  such as considering the assault to be a  personal matter, fear of reprisal, or not  considering the victimization important  enough to report. Similarly, Clery Act data are  based on reports made to campus security  authorities and law enforcement. According to  one federally funded pilot study, data from  campus climate surveys at nine colleges  suggested that the majority of rapes are not  represented in a college’s Clery numbers.

In contrast, Clery Act data collected through Education’s Campus Safety  and Security Survey provides information only on incidents that are  reported to campus security authorities or law enforcement and that  occurred on or near campuses that the colleges own or control. This can  result in campus climate surveys identifying a larger number of campus  sexual violence incidents than federal Clery Act data. For instance, a pilot  study of campus climate surveys at nine colleges found that  undergraduate students attending these colleges experienced an  estimated 2,380 incidents of rape during the 2014-2015 academic year, of  which an estimated 770 occurred on campus. In contrast, Clery Act data  documented 40 reported rape incidents for these colleges during the  2014 calendar year.

Several stakeholders we spoke with also said that campus climate  surveys can provide information on a broader range of sexual violence  incidents than federal crime statistics data, such as the National Crime  Victimization Survey and Clery Act data, which collect information  specifically on criminal offenses. For example, one researcher we spoke  with noted that campus climate surveys can collect information about  sexual harassment, which is not included in federal Clery Act crime  statistics. The three surveys we reviewed collect information on a range  of sexual violence incidents, including sexual assault, coerced sexual  contact, stalking, intimate partner violence, and sexual harassment.

Behavioral Questions  Campus climate surveys on sexual violence  ask students about a variety of topics that are  often sensitive. One challenge of conducting  these surveys is that students’ understanding  of what behaviors are considered “sexual  assault” or “rape” may differ and the words  used to describe sexual violence will  determine what is measured by the survey,  such as incidents of rape. Research has  found, for example, that surveys asking  directly whether students experienced specific  types of sexual violence can produce  inaccurate data. To improve the quality of  information collected by campus climate  surveys, researchers ask students about  specific behaviors and events that describe  the incident rather than referring to it using a  label such as “sexual assault” or “rape.” campus climate survey instruments we reviewed included questions that  asked for additional context on incidents of sexual violence reported by  students, such as the victim’s relationship to the perpetrator.

Researchers we interviewed also noted that campus climate surveys can  include behaviorally specific questions to identify conduct that survey  respondents might not categorize as sexual violence (see sidebar). Each  of the three surveys we reviewed used behaviorally specific questions to  describe behaviors that may constitute sexual violence for survey  respondents, without using specific terms, such as rape. For example:

One survey we reviewed asks, “Since the beginning of the current  academic year, has an intimate partner threatened to hurt you and  you thought you might really get hurt?” instead of asking whether the  respondent has experienced “intimate partner violence.”

Another survey we reviewed asks, “How many times have one or  more people left you unwanted messages (including text or voice  messages)?” instead of asking if the respondent has experienced  “stalking.”

Seven of the nine researchers we spoke with considered behavioral  questions to be a best practice for collecting data on sexual violence,  including one that noted the general public may not be aware of the  definitions of rape or other types of unwanted sexual contact or  behaviors. However, one researcher we spoke with expressed concern  that the wording of behavioral questions can be imprecise.

Surveys May Help Colleges Identify Specific Areas for Improvement

Each of the three campus climate surveys we reviewed included  questions regarding student knowledge of the administering college’s  policies and resources related to preventing and responding to sexual  violence on campus. According to nearly all of the stakeholders we  interviewed, these data can help colleges identify areas for improvement.  In particular, about half of these stakeholders noted that campus climate  survey results can help colleges address barriers to reporting. For  example, officials from one college we spoke with reported increasing  their efforts to educate students about where to go if they experienced  sexual assault based on gaps in awareness identified through survey  results. Further, information from campus climate surveys also helped  one state identify how it could better assist colleges, such as by providing  training on intimate partner violence, according to a state official.

Several stakeholders reported that campus climate surveys may also help  colleges assess their performance on reducing sexual violence. For  example, two researchers said that these surveys can help colleges see  where improvements were made and where additional action might be  needed. Another researcher we spoke with noted that colleges are very  interested in using campus climate surveys to establish baseline data and  are beginning to understand the usefulness of having data on sexual  violence prevalence.

Colleges Face Challenges Administering Surveys and Analyzing Results

While campus climate surveys can provide additional information on  campus sexual violence, stakeholders reported that colleges face a  variety of challenges with developing and conducting surveys, as well as  analyzing the results.

Surveys Can Be Costly and Require Technical Expertise to Administer

Although some survey instruments are free, about half of the  stakeholders we interviewed offered that some institutions, particularly  smaller colleges, may not have the resources to effectively administer  surveys on their own. Stakeholders cited costs associated with hiring  contractors or relying on faculty and staff to administer and analyze the  results of a survey. For example, one researcher we spoke with said that  administering a survey can require having people available to respond to  student questions about the survey. Officials from one college we spoke  with said they relied on faculty volunteers to analyze survey results over a  school break, due to a limited survey budget. About half of the  stakeholders also noted that providing incentives to students can help  increase survey response rates, yet incentives can also be the most  expensive part of a college’s survey budget. For example, one college  reported that the $10 incentives offered to students who completed the  survey constituted the college’s largest survey expense. Given the  potentially high costs of these surveys, officials in one state we spoke  with reported that the state provided funding to help its colleges  administer surveys, analyze results, prepare reports, and translate the  survey results into action.

About half of the stakeholders also reported that some colleges may not  have technical expertise readily available to conduct a campus climate  survey on sexual violence. As previously noted, colleges can administer  campus climate surveys to a sample of students (sample approach) or all  students at a college (census approach). According to federal guidance, a  sample approach can reduce the amount of follow-up needed to  encourage survey completion; however, expertise is needed to create a  sampling frame that includes all, or nearly all, of a target population, and  then to accurately select a sample of that population to survey that still  represents the target population. Several stakeholders said colleges  may face challenges in creating a representative sample of their students,  in particular. For example:

One researcher noted that colleges may not collect sufficient  demographic data or have adequate funding to create a  representative sample of their students.

Another researcher observed that for both sample and census  surveys, colleges may also lack the expertise to ensure, through  statistical methods such as non-response bias analysis or weighting  responses, that respondents are representative of the student body.

Justice officials noted that properly administering campus climate surveys  requires personnel with adequate statistical expertise, as well as support  from college administration. Research shows that statistical methods like  testing for non-response bias and weighting responses are an important  consideration when developing estimates on prevalence since non- response bias can potentially limit the extent to which the results can be  generalized to the entire student population. One college that had not  conducted a campus climate survey also noted that doing so would be a  challenge due to limited expertise with conducting surveys on sensitive  topics, such as sexual violence. However, two of the selected colleges  that conducted campus climate surveys reported working with a third  party to ensure more reliable results.

Surveys May Not Yield Reliable Results Due to Low Response Rates

Response rates are a key factor in producing reliable survey results, and  most stakeholders reported that obtaining a sufficient number of  responses from students can be a challenge. Achieving a sufficient  response rate can help ensure that the survey results are representative  of the target population, so that the results can be used with confidence to  inform decisions. However, our prior work on federal sexual violence  data found surveys are subject to variable response rates over time, and  different surveys may have different response rates, which may affect the  resulting estimates and the validity of the data. The seven selected  colleges that conducted surveys reported response rates ranging from  less than 10 percent to more than 60 percent. Additionally, officials we  interviewed in two of the selected states reported that their survey  response rates were not high enough to generalize or draw meaningful  conclusions regarding campus sexual violence, as originally intended.  Officials in these states said they primarily included the data in required  state reports, with limitations noted as needed.

Most stakeholders noted that survey design or administration factors can  affect response rates. About half of the stakeholders noted that keeping  the survey short is critical to ensuring more students complete it, but  some topics of interest to the colleges may not be covered as a result.  For example, at one college, officials included survey questions about  sexual assault and sexual harassment, but did not pursue questions  about stalking due to concerns about survey length. One researcher also  told us that long and complicated surveys may not work well on smart  phones, which is how many students take these surveys. As for survey  administration, most stakeholders noted that choices on how to  administer the survey can also affect the response rates for surveys. For  example, two researchers we spoke with said that it is better to leave  surveys open to respondents for a longer time period to increase the  response rate. In addition, a researcher and one state official noted that  technical issues can interfere with obtaining a high response rate, such as  sending survey invitations to university email accounts that students may  not check regularly.

Survey Results May Not Be Comparable Across Colleges

About half of the stakeholders stated that differences in survey  instruments and methodology may make it difficult for colleges to  compare their results with the results of other colleges.

Variation in questions and definitions. The surveys we reviewed  varied in the wording used to ask respondents about their knowledge  of institutional policies for reporting sexual violence. According to one  researcher, differences in the wording of questions and structure of  questionnaires can affect comparability across surveys. Officials in  one state also reported that colleges used different definitions of key  terms on their campus climate surveys, which made it challenging to  reach general conclusions across colleges. Similarly, another  researcher stated that differences in the definitions of terms used in  colleges’ campus climate surveys make accurate comparisons  difficult.

Variation in time periods. The surveys we reviewed ask  respondents about incidents of sexual violence occurring over  different time periods, which may also limit the comparability of survey  results across colleges. According to one researcher we spoke with,  colleges using different survey instruments should not compare  prevalence estimates with results from other surveys that ask about  incidents of sexual violence for different time periods. For example,  one survey instrument we reviewed asks students about their  experiences with sexual violence during the current academic year. In  contrast, the two other survey instruments we reviewed ask students  about their experiences with sexual violence since first enrolling at  college, which covers a longer time period for seniors than first-year  students. Time periods may also affect the accuracy of the data  collected. One researcher we spoke with, for example, stated that  survey questions that cover longer time periods can introduce bias,  such as the telescoping effect, whereby respondents recall certain  events as being more recent than they actually are. Additionally,  longer time periods may yield larger numbers of incidents, since more  individuals may experience the behavior over time.

To address these comparability challenges, some colleges have used the  same survey instruments as other colleges. For example, two colleges  included in our review participated in a survey effort among multiple  colleges that was designed to allow for comparisons across participating  schools. To make these comparisons, a third party administered the  survey at participating colleges using a survey instrument with  standardized questions and a standardized methodology to enable the  measurement of prevalence, and then analyzed the results.

In summary, while all stakeholders noted the value of conducting campus  climate surveys, about half of them generally cautioned against requiring  colleges to administer them in light of the associated challenges  previously discussed. Officials at one college that voluntarily conducted a  campus climate survey using a one-time grant stated they would have to  use funds from faculty and staff salaries if they were required to conduct a  survey in the future. Additionally, an official from one college that had not  conducted a campus climate survey noted that high turnover in the Title  IX coordinator position would make it difficult for the college to sustain a  survey effort over time. Further, another college that has not conducted a  campus climate survey to examine the incidence of sexual violence noted  it would be difficult to design a standard survey instrument that would  apply across all colleges, such as those that primarily serve students who  take courses online.

Selected Colleges Have Used Various Survey Design, Administration, and Outreach Strategies to Increase Understanding of Campus Sexual Violence

The seven selected colleges that conducted campus climate surveys  used various survey design, administration, and outreach strategies to  learn about the incidence of campus sexual violence. Most of these  colleges also chose to publicly report some survey results.

Choosing a survey instrument. These seven colleges considered  several factors when choosing a survey instrument:

Rigor. Officials from each of the seven colleges that conducted a  campus climate survey said it was important to use a rigorous survey  instrument, such as one that survey developers have validated or  colleges have widely adopted. One college official explained that  using a validated instrument provided assurances that helped secure  a timely approval from the college’s institutional review board.

Flexibility. Officials from five colleges said they valued the flexibility  of using a survey instrument that could be modified based on the  specific characteristics and needs of their colleges. For example,  officials from one college said that the chosen instrument enabled  administrators to use gender-inclusive language and ask questions  about incidents of sexual violence from the perspective of the  perpetrator in addition to the victim.

Comparability. Officials from four colleges noted that comparability  was a consideration when selecting a survey instrument, including the  potential to compare survey results across colleges that share similar  characteristics or at their own colleges over time. However, as  previously discussed, stakeholders noted that differences across  survey instruments can limit the comparability of survey results.

Cost. Officials from four colleges said the cost of conducting campus  climate surveys informed their selection of a survey instrument. For  example, officials at one college said they used a free, publicly  available survey instrument because the college lacked the resources  to pay for an instrument.

Length. Officials from four colleges identified survey length as  another factor they considered. Officials from three of these colleges  specifically noted that longer surveys may result in lower response  rates. In addition, officials from one of these colleges stated that  because longer surveys collect more data, the college would need  more time to analyze the results. An official from another college  expressed concern that longer surveys with multiple follow-up  questions about incidents of sexual violence risk re-traumatizing  victims.

Modifying the survey instrument. Six of the seven colleges modified  their survey instruments to some extent. Officials at five of the six  colleges reported adding questions to their survey instruments. For  example, two colleges reported adding questions to comply with a state  survey requirement, while another college reported adding follow-up  questions to collect information on events prior to an incident of sexual  violence. Officials at three of the five colleges reported limiting the  number of questions they added to keep the survey short. Officials at two  of these colleges noted that lengthening the survey could result in fewer  students completing it. Officials at one of these colleges cited additional  fees that the vendor charged for such modifications as another factor in  their decision to limit the number of questions they added. Officials from  two colleges also said they modified the language in the survey  instruments to reflect the names of specific offices and programs on their  campuses. In contrast, officials at one college reported making no  changes to their survey instrument because they planned to use the  original survey as a baseline against which to compare future survey  results.

Identifying the survey population. Six of the seven selected colleges  that conducted a campus climate survey distributed their surveys via  email to all students in the target population (i.e., a census approach),  and one worked with a third party to select a representative sample of  students to receive the survey. As previously discussed, surveying a  sample of students can reduce the amount of follow-up work needed to  obtain sufficient responses to provide information about the student body  as a whole. However, officials at four of the seven colleges cited other  considerations for choosing a census approach. Specifically, officials from  three of these colleges said that a census approach provided every  student the opportunity to share their experiences and perspectives  through the climate survey. Officials from two of these colleges further  explained that administering the survey to a sample of students could  give the appearance they were excluding students, some of whom might  be victims of sexual violence, from participating in the survey. Another  college reported using a census approach because it lacked the  resources needed to develop a representative survey sample.

Determining survey timing and frequency. All seven of the selected  colleges that conducted a climate survey administered at least one survey  during the spring semester. Officials from three of these colleges said that  administering climate surveys in the spring ensures that first-year  students have spent time on campus prior to taking the survey. However,  officials at four colleges said that competing demands for students’ time,  such as other surveys and final exams, are a tradeoff to administering  these surveys in the spring. As a result, students may experience “survey  fatigue”—that is, they may be less likely to respond to or complete the  survey. The seven selected colleges administered surveys with varying  frequencies. For example, one college reported administering its survey  biennially in accordance with a state requirement, while two others  administered their surveys less frequently (e.g., every 4 years) to avoid  survey fatigue and low response rates.

Protecting confidentiality. Six of the colleges reported taking steps to  preserve the confidentiality of survey respondents. For example, officials  from five colleges explained that in order to maintain respondents’  confidentiality they had to redirect students who completed the survey to  a separate webpage to claim their incentive or enter a drawing. Officials  from four colleges reported using a third-party vendor to help protect  students’ confidentiality or, at a minimum, signal that the college had no  direct role in collecting or storing student responses. For example, to  protect students’ confidentiality, officials from three of these colleges said  their vendors provided summary data, rather than student-level data, and  did not report results with a low number of respondents. Officials from  three of the six colleges reported consulting their institutional review  boards to help ensure that the colleges protected respondents’  confidentiality. Officials from another college reported limiting how often  they administered campus climate surveys to head off potential student  concerns that they were being “tracked” during their time on campus.

Offering survey incentives. As part of their outreach efforts, six of the  seven colleges offered incentives to students who completed surveys,  which some research suggests can increase web-based survey  participation rates (see fig. 1). For example, one college offered a $20  gift card to survey respondents, which college administrators considered  critical to achieving a higher response rate. This comports with a study  funded by Justice that found incentives between $20 and $30 appear to  help maximize survey participation, whereas a $40 incentive does not  clearly offer any additional advantage. To manage the cost of  incentives, two colleges offered a limited number of incentives to students  via lottery drawings. Officials from one of these colleges said it funded its  lottery for five $200 gift cards with proceeds from an on-campus student  event. Another college offered a coupon for a free drink at a campus  coffee shop to the first 300 survey respondents. An official from the  college that did not offer incentives in its most recent climate survey said  that incentives would help improve response rates for future surveys.

While incentives can help increase survey participation, two colleges  noted that offering incentives may require additional precautions to  prevent abuse. For example, one college had to put its survey on hold to  fix a technical error that enabled a student to collect additional incentives  by completing the survey multiple times. Another college with experience  offering survey incentives reported that it received calls and emails from  other colleges requesting assistance with preventing such abuses.

Marketing the survey. Each of the seven selected colleges that  conducted a climate survey used email to invite students to respond to  the survey and various marketing efforts to encourage survey  participation (see fig. 1). Officials from five of the colleges reported  following up with email reminders. For example, one college reported  adding the incentive dollar amount to the subject lines and another  reported varying the gender of those who sent follow-up emails and the  timing of them to increase student responses. In addition, five colleges  reported using social media to advertise their climate surveys.  Recognizing the importance of gaining institutional buy-in, officials at all  seven colleges said they engaged college administrators or faculty in their  marketing efforts. For example, officials at one college said that deans of  its various schools were asked to send emails encouraging students to  take the survey. The officials credited this particular email strategy for  doubling the college’s survey response rate. Officials from five colleges  also reported involving student leaders and influencers in their marketing  efforts, such as creating t-shirts for students to wear that included  information about the survey; having students publish an op-ed in the  campus newspaper promoting the survey; and asking student leaders to  share information about the survey with student organizations.

Six of the seven selected colleges publicly reported at least some of the  results of their surveys. Five colleges, for example, published survey  results on their respective websites, and another created a campus  poster with an infographic illustrating key survey results. Two of these  colleges also presented the results during meetings with different student  populations, such as fraternities and sororities and lesbian, gay, bisexual,  transgender, and queer/questioning students. Officials from four colleges  expressed that they felt a responsibility to be transparent. However,  according to an official at one college, a potential drawback to making  survey results publicly available is that the results could create or  reinforce negative perceptions of a college’s climate regarding campus  sexual violence. Finally, officials from the one college that had not publicly  disclosed any survey results explained that, due to a lack of resources  and in-house expertise, they did not feel sufficiently confident in their  analysis of the survey results to publish them.

Federal Agencies Have Provided Information to Colleges about Developing and Implementing Campus Climate Surveys, Among Other Efforts

Since the issuance of the White House Task Force to Protect Students  from Sexual Assault report in 2014, federal agencies have created and  disseminated informational resources for colleges interested in  conducting campus climate surveys. For example, Justice’s Bureau of  Justice Statistics and Office on Violence Against Women funded the  development of a publicly available survey instrument and a validation  study from 2014 to 2016, to provide colleges and researchers with access  to a free and reliable survey instrument to collect school-level data on  campus climate and sexual victimization. In 2017, Justice also  collaborated with HHS’s Centers for Disease Control and Prevention to  provide funding and project planning assistance for a pilot study to  develop and test a campus climate survey for use at two Historically  Black Colleges and Universities. According to Justice and HHS officials,  this survey instrument was based on the validated Justice survey  instrument, with some modifications made to the campus climate  questions. In October 2019, Justice officials told us the agency had  decided not to proceed with funding for the study due to concerns that  modifications to the original validated survey instrument would result in  data that are not comparable to data from the validation study.

In addition, Justice has developed technical assistance materials for  colleges interested in conducting a campus climate survey. For example,  from 2016 to 2017 Justice’s Office on Violence Against Women issued  documents outlining lessons learned from the Justice survey validation  study, talking points to help college administrators and students  communicate about climate surveys, and a frequently asked questions  sheet on campus climate surveys. These documents covered a range of  topics, including the goals of a campus climate survey, best practices for  developing survey content, and tips for choosing survey participants and  protecting their confidentiality, among others. Justice’s campus climate  survey, validation study, and technical assistance documents are publicly  available on Justice’s website.

Justice’s campus climate survey and validation study are also available  through the Center for Changing Our Campus Culture, an online  clearinghouse developed and maintained by a nonprofit organization with  funding from Justice’s Office on Violence Against Women. The  clearinghouse provides resources for colleges on addressing sexual  assault, domestic violence, dating violence, and stalking. For example,  the clearinghouse includes documents outlining (1) selected research  initiatives and resources on campus climate surveys, (2) suggested  campus sexual assault policies and procedures, and (3) steps college  institutional review boards and administrators can take to oversee  research on sexual violence while maintaining participant confidentiality.  Most stakeholders we interviewed were aware of federal information and  resources available to assist colleges in conducting campus climate  surveys. For example, officials at two of the colleges reported using  Justice’s survey instrument for their campus climate surveys, with officials  from one college noting they selected the instrument because it had been  validated as a reliable instrument. An official from another college  reported using Justice’s validation study during the survey instrument  selection process, to better understand the strengths and weakness of  survey instruments and potential sources of bias in the data collected.

In addition to the resources provided by Justice, Education has offered  information to colleges regarding the prevention of campus sexual  violence. For example, Education’s 2015 Title IX Resource Guide  encouraged Title IX coordinators to help colleges develop a method,  appropriate to their college, for surveying students about the campus  climate. Additionally, to address Title IX concerns or complaints,  Education may enter into voluntary resolution agreements with colleges.  These agreements describe the changes colleges agree to make to  ensure their procedures for preventing and responding to sex  discrimination comply with the law. According to agency officials,  Education may include campus climate surveys as part of these voluntary  agreements, on a case-by-case basis.

Justice and HHS have also funded campus sexual assault prevention and  response grants. For example, Justice’s Office on Violence Against  Women provides grant funding to colleges to help improve responses to  sexual assault and other types of domestic and sexual violence through  its Grants to Reduce Sexual Assault, Domestic Violence, Dating Violence,  and Stalking on Campus Program. According to a Justice official, colleges  receiving these grants are allowed, with prior approval, to use a small  percentage of the grant funds to conduct campus climate surveys for  program improvement purposes, but it is not a requirement of the  program. Additionally, HHS’s Office on Women’s Health provided  funding for the College Sexual Assault Policy and Prevention Initiative  from 2016 to 2019 to organizations that partnered with colleges to provide  technical assistance and support in developing sexual assault policies  and prevention strategies. According to HHS officials, grantees were  encouraged to conduct campus climate surveys to establish baseline data  for their partner campuses.

HHS officials also reported providing grantees with information on  different campus climate survey instrument options, including a free,  publicly available survey instrument. One college we spoke with reported  partnering with one of these HHS grantees to conduct baseline and  follow-up campus climate surveys and to develop comprehensive campus  prevention strategies. For example, officials from the college and the  grantee told us they used funds from the grant to help the college  establish memoranda of understanding with community-based  organizations, such as the local women’s crisis center, to support  students living off-campus who may have experienced sexual violence.

We provided a draft of this report to the Departments of Education and  Justice for review and comment. The Departments of Education and  Justice provided technical comments, which we incorporated as  appropriate. We also provided relevant report sections to the Department  of Health and Human Services, and to third parties, including survey  developers and states included in our review, for technical comments.  The Department of Health and Human Services, survey developers, and  state officials provided technical comments, which we incorporated as  appropriate.

As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to the appropriate  congressional committees, the Secretary of Education, the Attorney  General, and other interested parties. In addition, the report will be  available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-7215 or larink@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. GAO staff who made key contributions to this report are  listed in appendix II.

Appendix I: Objectives, Scope, and Methodology

This report examines (1) what key stakeholders view as the strengths and  limitations of using campus climate surveys to examine the incidence and  characteristics of sexual violence on college campuses, (2) what  approaches selected colleges have used to survey their students about  the incidence of sexual violence on campus, and (3) what role federal  agencies play in helping colleges develop and implement climate surveys.

To inform our examination of stakeholders’ views on the strengths and  limitations of campus climate surveys, we reviewed three commonly used  survey instruments that included questions regarding the incidence of  sexual violence, including sexual assaults, coerced sexual contact,  stalking, and intimate partner violence. Each of these survey instruments  is also available, online or by request, for any college to use for free. For  each of the surveys, we reviewed survey questions and methodological  reports, and conducted interviews with representatives from the  organizations involved in developing them.

The Association of American Universities (AAU), an association of 65  research universities, developed its survey instrument in conjunction  with the research firm Westat. AAU administered its survey to  participating colleges in spring 2015 and spring 2019.

The Administrator Research Campus Climate Collaborative (ARC3),  an organization of sexual assault researchers, university  administrators, and student and legal affairs professionals, developed  and tested its campus climate survey from 2014 to 2015. The final  survey instrument was made available to colleges in 2015. According  to the survey developers, there is no comprehensive list of schools  that have conducted the ARC3 survey.

The Department of Justice (Justice) survey instrument was initially  developed by the White House Task Force to Protect Students from  Sexual Assault in 2014, and later refined and tested by Justice in  collaboration with RTI International, a research organization. The  survey instrument, also known as the Campus Climate Survey  Validation Study, is publicly available online. According to Justice  officials, there is no comprehensive list of schools that have  conducted the Justice survey.

Additionally, we reviewed two key federal data sources on campus sexual  violence: Clery Act data from Education’s Campus Safety and Security  Survey and the National Crime Victimization Survey (NCVS) from  Justice. We identified these data sources based on a review of prior  GAO work and interviews with Education and Justice officials. We  examined documentation for these data sources and interviewed the  responsible agency officials to determine the type of data they collect on  campus sexual violence, the methods for collecting this information, and  their limitations. We determined these data sources were sufficiently  reliable for our purposes.

To inform all three objectives, we also interviewed a total of 25  stakeholders with relevant expertise, including representatives of four  organizations involved in developing the three surveys we reviewed and  five additional researchers who have studied campus sexual violence;  officials from 10 colleges; officials from four states; and federal officials  from Education and Justice. We refer to the representatives of these  organizations and entities collectively as “stakeholders” in our report.  When discussing stakeholder views, we group them into the following  categories: “several” (between four and nine), “about half” (between 10  and 14), “most” (between 15 and 19), and “nearly all” (20 or greater). In  instances where we report on the views of specific groups, such as  colleges, researchers, or state officials, we refer to the individual group  and enumerate the number of group members. During these interviews,  we gathered information on issues related to designing and conducting  campus climate surveys and analyzing and communicating survey  results. We also discussed federal information and resources available to  help colleges develop and implement campus climate surveys. Findings  from our interviews summarize selected stakeholders’ views regarding  campus climate surveys on sexual violence. These findings do not  represent the views of all researchers on these topics and do not  represent the experiences of all colleges developing or implementing  these surveys.

To identify researchers with a variety of perspectives, we reviewed  research on sexual violence and conducted targeted web searches. We  then selected individuals or organizations with experience conducting  research on campus sexual violence or developing and administering a  campus climate survey on sexual violence. We also spoke with  representatives of the organizations responsible for developing the three  climate survey instruments we reviewed.

We used multiple approaches to identify the 10 selected colleges  included in our review since there is no central repository of information  on whether colleges have conducted a campus climate survey on sexual  violence.

Colleges that conducted a campus climate survey. Based on  targeted web searches, we identified colleges that had conducted a  campus climate survey and then grouped them according to which of  the three survey instruments they used. We analyzed data from the  Department of Education’s Integrated Postsecondary Education Data  System to identify the characteristics of these colleges, including  sector (i.e., public, private not-for-profit, and private for-profit),  program length (i.e., 2-year and 4-year), size, and geographic  location.

Colleges that had not conducted a campus climate survey. We  also used data from the Integrated Postsecondary Education Data  System to help identify colleges that had not conducted campus  climate surveys on sexual violence. Specifically, we grouped colleges  into categories by sector and program length and randomized the lists  within each category. To select specific colleges, we started with the  college in each category at the top of the randomized list and  conducted targeted web searches in an effort to ensure the college  had not publicly reported conducting a campus climate survey.

We conducted outreach to the Title IX coordinators at each of the  selected colleges via email or telephone to confirm whether or not the  college had conducted a campus climate survey on sexual violence. In  total, we selected 10 colleges, including seven that have conducted  campus climate surveys that examine the incidence of sexual violence on  their campuses and three that have not. We selected these colleges to  ensure variation in size, sector (i.e., public, private not-for-profit, and  private for-profit), program length (i.e., 2-year and 4-year), geographic  location, survey instrument used, and whether the college was located in  a state that as of January 1, 2017 had a statutory requirement in effect for  at least some colleges in their state to conduct a campus climate survey  (see table 2 for selected colleges by program length, sector, and use of  campus climate survey). We interviewed Title IX coordinators and other  knowledgeable officials regarding the selected colleges’ experiences with  conducting campus climate surveys and their perspectives on the  strengths and limitations of these surveys.

As part of our efforts to obtain a variety of perspectives, we also  conducted semi-structured interviews with officials from four states  regarding the use of campus climate surveys in their states. We selected  three states (Louisiana, New York, and Washington) that as of January 1,  2017, had a statutory requirement in effect for at least some colleges in  their state to conduct a campus climate survey, and one state (Ohio) that  recommended colleges conduct such surveys. To identify states that  required or recommended that colleges conduct campus climate surveys,  we used several approaches to develop a preliminary list, including  consulting with researchers, reviewing annual reports from 2014 to 2019  prepared by the National Conference of State Legislatures on state higher  education legislation, and conducting targeted web searches. Based on  these reviews, we judgmentally selected four states to ensure a diversity  of state experiences with requiring or recommending campus climate  surveys. We also confirmed applicable state requirements with state  officials. The selected states differed in the nature of the survey  requirement or recommendation, such as the types of colleges covered  (e.g., public or public and private) and how frequently the survey was  required or recommended to be administered.

To supplement information gathered from our interviews, we also  identified and reviewed studies and reports that examined the design and  use of campus climate surveys. We conducted a targeted search of  various databases to identify studies on leading survey practices. We  selected studies for additional review based on their relevance to our  objectives, and, using a standard review instrument, assessed the quality  and rigor of each study’s findings and methods. Our report includes  information about leading survey design and implementation practices  from those studies we found were appropriate through this review  process.

To examine the role that federal agencies play in helping colleges  develop and implement climate surveys, we reviewed Justice and  Education resources available to help colleges conduct campus climate  surveys. Additionally, we reviewed information on campus sexual  violence prevention grants provided by the Department of Health and  Human Services. We also reviewed relevant federal laws and regulations,  as well as federal guidance and other documentation pertaining to  campus sexual violence and campus climate surveys.

In addition to the contact named above, Debra Prescott (Assistant  Director), Maria Gadel (Analyst-in-Charge), Jonathan Adams, Will Colvin,  Caitlin Cusati, Kirsten Lauber, and Erica Vilay made key contributions to  this report. Additional assistance was provided by MacKenzie Cooper,  Sarah Cornetto, Holly Dye, Monika Gomez, Dana Hopings, Connor  Kincaid, Sheila R. McCoy, Mimi Nguyen, and Almeta Spencer.

The prevalence of sexual violence on college campuses, including rape and other forms of sexual coercion, can be difficult to measure. Some colleges have surveyed students to better understand the issue of sexual violence on their campuses.

Researchers and others told us surveys can help colleges gather more comprehensive information about campus sexual violence, but cautioned that such surveys can be difficult and expensive to administer.

Federal agencies have supported the development of a survey colleges can use for free; encouraged colleges to develop ways to survey students; and provided grants that could be used for surveys.