Border Security: DHS Should Improve the Quality of Unlawful Border Entry Information and Other Metric Reporting

According to DHS, the United States has approximately 6,000 miles of land borders, 95,000 miles of coastline, and more than 300 ports of entry where travelers and cargo are inspected and processed for entry. Securing U.S. border areas is a key part of DHS's mission, and the department's ability to measure its border security efforts is essential for it to manage its responsibilities effectively and efficiently.

The NDAA for Fiscal Year 2017 requires DHS to report annually on 43 border security metrics. DHS issued its first report in May 2018. The Act also includes a provision for GAO, within 270 days of receipt and biennially for the following 10 years, to review and report on the data and methodology contained in DHS's report. This report assesses the extent to which DHS: (1) reported metrics as outlined in the NDAA using quality information; and (2) validated assumptions and conveyed statistical uncertainty for unlawful entry metrics, among other objectives. GAO assessed the methodology and data in DHS's report, analyzed DHS's use of statistical models, and interviewed officials from DHS offices and components involved in developing the metrics.

The Department of Homeland Security (DHS) reported on 35 of 43 metrics called for by the National Defense Authorization Act (NDAA) for Fiscal Year 2017 (see figure); it generally used quality information, but did not identify some data limitations. GAO found that about half of the 35 metrics generally included elements as called for by the NDAA, while 17 metrics differed, such as in scope or calculation. For example, DHS only provided information on the southwest border for some metrics, such as the estimate of undetected unlawful border crossers for which a methodology for estimating unlawful crossings for the northern border had not yet been completed. DHS components responsible for collecting the metric data generally have processes in place to ensure the reliability of the data and the quality of the information provided. DHS also identified and disclosed limitations for some, but not all, of the data elements and metrics used. For example, GAO found that DHS did not disclose limitations on data related to apprehensions of individuals that were assisted by unmanned aerial systems. By developing and implementing a process to systematically review the reliability of the data and comprehensively identify and communicate limitations, DHS would improve the quality of the information provided.

DHS used a statistical model to estimate three metrics on unlawful border entries but did not validate some assumptions the model employs through sensitivity analyses and provide measures of statistical uncertainty in accordance with standards for federal agencies. For example, DHS's model assumes that 100 percent of families unlawfully crossing the border will be apprehended, but DHS did not provide information on the extent to which the assumption affected its metrics. DHS also did not provide information on the level of statistical uncertainty for the metrics, such as margins of error. Providing such information would allow Congress and the public to better understand the potential limitations and accuracy of these metrics of unlawful entry. Additionally, DHS's statistical model, which is based on Mexican adults not seeking asylum, represents a small and declining share of those apprehended at the border and DHS is developing a new model to account for current border conditions.

GAO is making four recommendations, including that DHS develop and implement a process to systematically review the reliability of metric data, identify and communicate limitations of the metrics, and include the results of sensitivity analyses and measures of statistical uncertainty for metrics derived from statistical models. DHS concurred with the recommendations.

Securing the nation’s borders against illegal entries, smuggling of drugs  and contraband, and terrorist activities is a key part of the Department of  Homeland Security’s (DHS) mission. According to DHS, the United States  has approximately 6,000 miles of land borders, 95,000 miles of coastline,  and 328 ports of entry (POE). DHS’s ability to measure border security  activities, outputs, and outcomes is essential for the department to make  evidence-based decisions about resource allocation and investments and  manage its border security responsibilities effectively and efficiently. In  our prior work, we have reported on the need for DHS to improve its  measures for assessing its border security efforts.

The National Defense Authorization Act for Fiscal Year 2017 (NDAA)  requires DHS to provide an annual report to appropriate congressional  committees, the Comptroller General, and certain other entities,  containing 43 specific metrics to measure the effectiveness of border  security in four domains—between POEs, at POEs, the maritime border,  and with respect to aviation assets and other air and marine operations in  the land domain. The majority of the 43 metrics are counts and rates of  border security activities, such as the number of detected unlawful entries  between POEs and a rate that measures traffic volume at land POEs  against the physical and staffing capacity at each land POE. The  remaining metrics are estimates, such as the number of undetected  unlawful entries, or were not specifically described. DHS issued its first  report to respond to the NDAA requirement in May 2018, titled Border  Security Metrics Report. The NDAA also includes a provision for us,  within 270 days of receipt and biennially for the following 10 years, to  review and report to Congress on DHS’s report. Specifically, the provision  directs us to analyze the suitability and statistical validity of the data and  methodology contained in the report, and, as appropriate, include  recommendations on improvements needed to the metrics and the  feasibility of other suitable metrics. This report addresses the following  questions:  1.  To what extent has DHS reported metrics as outlined in the NDAA  using quality information?  2.  To what extent has DHS validated the assumptions and conveyed  statistical uncertainty for its unlawful entry metrics?  3.  What, if any, other metrics have been identified that may be used to  measure the effectiveness of border security?

To determine the extent to which DHS reported metrics outlined in the  NDAA using quality information, we first determined which of the 43  metrics DHS included in its first annual report and which it did not. For  metrics DHS included, we identified the specific data sources and sets  DHS used to develop them, such as administrative data collected by DHS  components (e.g., data on apprehensions, POE wait times, drug seizures,  and flight hours). We also interviewed officials from DHS offices and  components involved in developing the metrics, including the Office of  Immigration Statistics (OIS), U.S. Customs and Border Protection’s (CBP)  Office of Field Operations, U.S. Border Patrol (Border Patrol), and Air and  Marine Operations (AMO); and the U.S. Coast Guard (Coast Guard). In  these interviews we obtained information about the methodologies DHS  components used to develop the metrics, including any limitations they  identified and their plans to update or revise existing metrics in the future.  To determine the extent to which DHS reported metrics as outlined in the  NDAA, we assessed how, if at all, the metrics DHS presented and the  methods DHS used to calculate the metrics were similar to, or different  from, the metrics listed in the NDAA. Where we identified clear  differences between the metrics DHS reported and those described in the  NDAA, we reviewed documentation and obtained additional perspectives  from DHS officials, as necessary, to determine the reasons for the  differences.

To determine the quality of the information used for the metrics, we  assessed the extent to which DHS has processes to ensure data  reliability and quality. Specifically, we reviewed any of our ongoing or  completed work relevant to the metrics, relevant DHS Office of Inspector  General (OIG) reports, and the metrics included in DHS’s Annual  Performance Reports to determine which data we had previously  assessed or which had been assessed by the OIG or DHS, and the  results of those assessments. For data that had not previously been  assessed, we collected information from DHS to determine what  processes are in place to ensure the overall reliability and quality of the  data. We reviewed this information to determine the extent to which  DHS’s processes are consistent with Standards for Internal Control in the  Federal Government, good practices for verifying and validating  performance information we have identified in our prior work, and DHS’s  Management Directive on Information Quality.

To determine the extent to which DHS validated the assumptions and  conveyed statistical uncertainty for its unlawful entry metrics, we first  identified the metrics for which DHS utilized a statistical model (i.e., the  use of a capture-recapture model to estimate the number of undetected  unlawful entries). We interviewed officials from DHS Office of Immigration  Statistics (OIS) and the Institute for Defense Analyses, DHS’s contractor,  to obtain information on the statistical model used to estimate unlawful  border entry metrics, including assumptions made and how, if at all, they  were validated. We further analyzed DHS’s use of the statistical model  and compared it against practices for the use of statistical models  outlined in the Office of Management and Budget’s (OMB) Standards and  Guidelines for Statistical Surveys. Using OMB’s standards, we identified  principles and practices to determine the extent to which DHS’s modeling  was consistent with them and what, if any, improvements could be made.  We also analyzed DHS’s modelling assumptions on the composition of  the unlawful migrant population to determine the extent to which  assumptions DHS made about the unlawful migrant population reflect  data on individuals apprehended between POEs. We interviewed Border  Patrol officials and reviewed documentation to obtain information on  possible alternative approaches DHS is considering for modelling  unobserved events.

To identify other metrics that may be used to measure the effectiveness  of border security, we reviewed our prior work and DHS OIG reports  related to border security to identify open recommendations focused on  establishing border security measures in the four domains listed in the  NDAA. We focused our search for prior work on reports that we and the  DHS OIG issued from 2010 through 2018.

We conducted this performance audit from May 2018 to March 2019 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

The NDAA requires DHS to develop and implement 43 border security  metrics in four domains—between POEs, at POEs, the maritime border,  and air and marine security in the land environment. Within DHS, CBP  and the Coast Guard have primary responsibility for border security within  these four domains. CBP and its subcomponents are to secure U.S.  borders at and between POEs by preventing inadmissible people and  illicit goods from entering the United States, among other responsibilities.  Within CBP, the primary offices and components involved in border  security are the Office of Field Operations at POEs, Border Patrol  between POEs, and Air and Marine Operations for air and marine security  in the land and maritime domains. The Coast Guard and CBP’s Air and  Marine Operations share responsibility for security of the nation’s  maritime borders. Table 1 shows examples of border security metrics by  domain and responsible DHS component.

According to DHS officials, within DHS, two subcomponents within the  Office of Strategy, Policy, and Plans were responsible for coordinating the  department’s effort to develop the fiscal year 2017 Border Security  Metrics Report. A senior DHS official explained that the report was initially  tasked to the Unity of Effort Integration Office, which was part of the Unity  of Effort initiative started in 2014 to better understand border security  efforts along the southwest border including exploring the development of  border security metrics. OIS assumed responsibility for the report in  2017. According to OIS officials, to prepare the report, they obtained  data and information related to each NDAA metric from the administrative  records of the DHS components with primary responsibilities for border  security in the four domains. For example, OIS requested data and  information on “turn backs” and “got aways” from Border Patrol—the lead  component for the between POE domain—which records sector  estimates of turn backs and got aways based on direct and indirect  observations.

Of the 43 metrics the NDAA listed for inclusion in the Border Security  Metrics Report, the majority were counts and rates of border security  activities. The remaining metrics were estimates or were not specifically  described. For example, the number of apprehensions in each Border  Patrol sector is a count metric. In contrast, a rate metric compares one  value or number against another. For example, the wait time ratio  compares the average wait times to total commercial and private  vehicular traffic volumes being processed at a land POE. An estimate is  used for metrics of flows or activities that are largely undetected and  therefore cannot be measured directly and must be estimated, such as  the number of undetected unlawful entries. A few metrics are a  combination of counts or rates with an estimate. For example, the metric  for total inadmissible travelers at POEs counts known inadmissible  travelers that are intercepted at POEs, and also requires an estimate of  how many inadmissible travelers may have successfully entered at a  POE without being detected, which cannot be directly measured. The  NDAA did not specifically describe some metrics. For example, while the  NDAA asked for an examination of each of the eight consequences under  the Consequence Delivery System, it did not specify how this examination  was to be carried out or what it was to include.

While many of the metrics required by the NDAA can be addressed with  data from DHS’s administrative records, certain metrics that rely on  estimates necessitate the use of alternative methodologies and in some  cases, specialized technical expertise. For example, DHS contracted with  the Institute for Defense Analyses to assist with the development of a  statistical model for estimating undetected unlawful entries. In its fiscal  year 2017 Border Security Metrics Report, DHS provided information on  its methodological approaches, such as how it estimated undetected  unlawful entries.

DHS Reported Information on Most Required Metrics and Generally Used Quality Information but Did Not Identify Some Limitations

DHS Reported Information on 35 of the 43 Required Metrics and Generally Included Elements Listed in the NDAA

In its first Border Security Metrics Report, DHS reported information on 35  of the 43 metrics called for by the NDAA. The metrics DHS provided  spanned the four domains outlined in the NDAA and included a mix of  counts, rates, estimates, or a combination thereof as shown in figure 1.

For 18 of the 35 border security metrics DHS included in its report, we  found DHS generally included elements listed in the NDAA. For example,  the NDAA asked for the number of detected unlawful entries between  POEs, and in its report DHS provided information on the number of  detected unlawful entries over a 10-year period. As another example,  the NDAA asked for the number of cargo containers at sea ports that  were identified to be potentially high-risk. In response, DHS provided  information on the number of potentially high-risk containers from fiscal  years 2013 through 2016 and also provided contextual information about  trends in the volume of such containers over time. See table 2 for more  information on these examples as well as other examples of the types of  information included in DHS’s fiscal year 2017 Border Security Metrics  Report.

For some metrics, DHS also provided information in addition to the  elements listed in the NDAA. For example, the NDAA described the  “AMO apprehensions assisted” metric as a count of the number of  apprehensions that were assisted by CBP’s AMO through the use of  unmanned aerial systems and manned aircraft. In addition to the counts  for such assists, DHS also provided data on the flight hours expended to  assist with these apprehensions.

For 17 of the 35 reported metrics, we identified differences between the  metric as described by the NDAA and as reported by DHS. The  differences we identified generally fell into two categories:

Metric differed in scope or calculation. Some of the metrics DHS  reported on differed in scope or in their calculation from what the  NDAA described for reasons such as data availability, among other  factors. For example, DHS’s fiscal year 2017 Border Security Metrics  Report scoped three metrics on unlawful border crossings between  POEs (the “attempted unlawful border crosser apprehension rate,” the  “estimated undetected unlawful entries,” and the “probability of  detection rate”) to only include data for the southwest border. In  these instances, the report noted that a methodology for estimating  data on unlawful crossings for the northern border had not yet been  completed but that research was underway to do so. As an example  of a difference in calculation, DHS presented the interdiction  effectiveness rate for each southwest border sector as an alternative  to the metric “unlawful border crossing effectiveness rate in each  Border Patrol sector.” According to DHS’s report, the department  used the interdiction effectiveness rate because it had not yet  produced and validated sector-level estimates of unlawful entries  required to calculate the unlawful border crossing effectiveness rate.  In its report, DHS stated it expects these estimates to be available for  the 2019 report.

Alternative metric provided. For the situational awareness in the  maritime environment metric, DHS stated that it is in a multi-year  process to develop a metric that meets the intent of the NDAA. As  an alternative, DHS instead provided data on the number of aircraft  and vessel operational hours that contributed to maritime domain  situational awareness.

See appendix I for additional information about any differences we  identified for each metric.

The eight metrics on which DHS did not provide information spanned all  four domains. In its report, DHS explained that the eight omitted metrics  were either still in development, under review within the department, or  officials were in the process of collecting data for them. Table 3 lists the  eight metrics on which DHS did not provide information and the date DHS  estimated it will report on each metric.

DHS Components Generally Have Processes to Help Ensure Reliable Data and Quality Information, but DHS Does Not Have a Systematic Process for Reviewing the Reliability of Data to Identify Limitations

In general, DHS components responsible for collecting the data used in  the metrics DHS reported have processes to help ensure the reliability of  the data and the quality of the information provided. DHS also identified  and disclosed limitations with some of the data elements or  methodologies used for the metrics in its report. However, DHS does not  have a systematic process for reviewing the reliability of data to identify  limitations related to the metrics, and we identified at least one additional  limitation for 21 of the 35 metrics on which DHS reported where DHS did  not disclose such limitations or could have been more transparent about  the limitations or assumptions in its report.

Data are considered reliable when they are reasonably free from error  and bias. Quality information is derived from relevant and reliable data  and is considered to be, among other things, complete, accurate, and  timely. The specific processes DHS components use to ensure data  reliability vary from metric to metric. Examples of processes DHS or its  components have implemented to help ensure the reliability of the data  and the quality of information provided include:  Issuing guidance and monitoring implementation. In September  2012, Border Patrol headquarters officials issued guidance to help  provide a more consistent, standardized approach for the collection  and reporting of turn back and got away data by Border Patrol  sectors. Each sector is individually responsible for monitoring  adherence to the guidance. According to DHS’s report, command staff  at Border Patrol stations ensure agents are aware of and utilize  proper definitions for apprehensions, got aways, and turn backs at  their respective stations and also ensure that the necessary  communication takes place between and among sectors and stations  to minimize double-counting when subjects cross over multiple areas  of responsibility.

Supervisory reviews of data entries. With regard to data on AMO  vessel and aircraft missions, AMO guidance mandates that  supervisors perform a review of all pre- and post-mission data entries  to help ensure accurate entry of mission information. AMO officials  confirmed that supervisors review the data being entered into the  database. Additionally, officials said AMO data teams run monthly  validation checks of data entered to check for completeness and  accuracy, such as out-of-range values.

Using built-in electronic safeguards. CBP’s databases for entering  and maintaining data elements—including travelers or passengers  seeking admission, known inadmissible aliens at POEs, referrals for  secondary examinations, major infractions, and private vehicles  processed at a POE—have built-in processes to detect and prevent  potential data entry errors. More specifically, as an officer enters a  record, the systems check for valid entries into relevant fields and  provide an error message to the officer for entries that appear to be  invalid (e.g., if an officer leaves a mandatory field blank or enters  contradictory information such as charging an individual with a crime  while also entering a request for expedited removal). In some cases,  the systems will prevent a record from being saved if any required  fields are blank.

Comparing data against other sources. As part of the Coast  Guard’s data reliability processes for data on maritime migrant  interdictions used in the “known maritime migrant flow rate” metric,  Coast Guard officials said that analysts cross-check the data entered  into their database with other Coast Guard reporting documents, such  as internal spreadsheets, to ensure accuracy.

Independent assessment of performance measure data. Some  border security metrics are similar to, or use the same data elements  as, performance measures DHS reports annually in response to the  Government Performance and Results Act Modernization Act  (GPRAMA) of 2010. For those performance measures, DHS  annually assesses a subset of measures and their data for  completeness and reliability using independent review teams. For  example, in May 2017 an independent review team assessed the  “migrant interdiction effectiveness in the maritime environment”  performance measure, which uses the same data as the border  security metric, “known maritime migrant flow rate.” The review team  found the measure to be complete and reliable and the data to be of  good quality overall, but also recommended that the Coast Guard and  DHS continue work on an improved database to enhance the  consistency of data collection, among other things.

In addition to the components having processes to help ensure the  reliability of the data and the quality of the information used in the report,  DHS took steps to be transparent in its presentation of the metrics by  identifying and disclosing known limitations with some of the data  elements or methodologies used for the metrics in its report.  Communicating the extent to which such limitations exist and their  potential impact is important to help facilitate the appropriate use and  understanding of the data and the metrics. DHS identified and disclosed  limitations related to the potential for misclassification of observations, the  potential for cases not being entered or recorded correctly, and  methodological limitations, among other things. For example, one of the  key limitations DHS’s report identified for the data on turn backs and got  aways is that they are based on potentially subjective observations of  agents who have to make a determination on how to classify them based  on what they observed or the available evidence (e.g., tracks, sensor  activations, interviews with apprehended subjects, camera views, etc.).  Further, DHS’s report explained that agents may face challenges in  making that determination because some unlawful border crossers may  enter the United States to drop off drug loads or to act as decoys to lure  agents away from a certain area and then return to Mexico, and therefore  may be misidentified as turn backs, for example. As another example,  DHS identified limitations due to cases not being entered or recorded  correctly. For the “known maritime migrant flow rate” metric, DHS used  data on the total number of maritime migrants interdicted. In its report,  DHS explained that a potential limitation of this data element is that the  Coast Guard relies on international and domestic partners to report their  interdictions for compilation in its database. Consequently, the accuracy  and completeness of the data depend on whether those reports are made  by those partners and the accuracy of their reports. See appendix I for  additional information about the limitations identified for each metric.

Even as DHS identified and disclosed limitations related to some of its  metrics, we identified at least one additional limitation for 21 of the 35  metrics on which DHS reported where DHS did not disclose such  limitations or could have been more transparent about the limitations in its  report. Examples of such instances include:

Potential for cases not being entered or recorded correctly. In our  previous work we found that mission data for unmanned aerial  systems were inconsistently collected across operation locations.  Specifically, in February 2017 we reported that there were instances  where no assist information was recorded in AMO’s data system even  though such assets participated in investigations and operations.  Because AMO’s data may not reflect all asset assists, we  recommended that AMO update and maintain guidance for recording  mission information in its data collection system and provide training  to users of the system. For its fiscal year 2017 Border Security  Metrics Report, DHS used asset assists data in metrics such as the  “AMO individuals detected,” “AMO apprehensions assisted,” and “illicit  drug seizures assisted by AMO,” but did not disclose this limitation in  its report.

Potential for data to be changed over time. Border Patrol officials  told us that data on the apprehension of unaccompanied alien  children may change over time because original apprehension  records from a shared database have, in some instances, been  updated by staff from U.S. Immigration and Customs Enforcement  (ICE) Enforcement and Removal Operations (ERO). Officials said  that in January 2015 they noticed that ERO staff were inadvertently  overwriting Border Patrol’s original data entries about the status of  apprehended children when they made updates to those children’s  records. For example, if a child was unaccompanied at the time of his  or her apprehension and was recorded as such by Border Patrol in  the initial record entry, ERO may have changed the “unaccompanied”  status in the system after they matched the child with a family  member or sponsor. As a result, data may not be reconcilable with  initial apprehension counts over time.

DHS did not fully disclose limitations for some metrics. We  identified instances where DHS could improve transparency about the  assumptions or limitations of the data presented in its report. For  example, in 2014 Border Patrol implemented a standard, southwest  border-wide methodology to improve reporting of turn backs and got  aways. While DHS made mention of this change in the text of the  report, the data for these metrics are presented in tables without any  table notes or disclosures within the table about this change. Further,  DHS’s report does not discuss how the change may affect  comparability of the data. Consequently, a reader may not be aware  that data for before 2014 in a table are not necessarily comparable to  the data for 2014 and after in the same table.

Without a comprehensive identification of the limitations of the metrics  and their associated data, and without an adequate disclosure of those  limitations, the value of DHS’s report as a source of information to  Congress, policymakers, and the public may be diminished. The metrics  in the report were specifically identified and requested by Congress in the  NDAA, and provide Congress with important information about the  outputs and outcomes of DHS’s border security policies and investments  that could be used to inform decision-making. However, those reading the  report may not be aware of important contextual information because  DHS did not identify and disclose some limitations, thereby creating the  potential for the data to be misinterpreted.

According to DHS officials who prepared the report, while they took steps  to identify methodological limitations of the metrics, no process currently  exists to systematically review the reliability of operational data used for  public reporting purposes, such as in the metrics report. Specifically, DHS  officials within OIS told us that while they were responsible for leading  and managing the preparation of the report, they largely relied on the  DHS components from which they collected the data to assess the data’s  reliability and communicate identified limitations. OIS officials explained  that many of the data elements used, such as those from AMO or the  Coast Guard, were ones with which they were not familiar or had not  worked with previously in their area of immigration statistics. OIS officials  also noted that in some cases, the data had previously been used in  performance measures or had been collected and tracked for several  years, so they trusted the components’ processes for ensuring their  reliability and identifying limitations, but reviewed the data provided where  possible and consulted with the components as needed. However, OIS  officials said that while they included as much information in the report as  was known about identified limitations with the existing operational data,  no additional effort was made to systematically review the underlying  reliability of the data to comprehensively identify limitations that should be  acknowledged when publicly reported because no department-wide  process exists to do so.

Standards for Internal Control in the Federal Government state that  management officials should evaluate data sources for reliability and  communicate quality information, including relevant data from reliable  sources, to achieve an agency’s objectives. The quality information can  then be used by agency management and external stakeholders such as  policymakers, to make informed decisions and evaluate performance,  among other things. Further, DHS’s Management Directive on Information  Quality states that data and information disseminated by the department  should, among other things, have full, accurate, transparent  documentation, and error sources affecting data quality should be  identified and disclosed to users. Additionally, our previous work on  approaches for verifying and validating performance information found  that communicating significant data limitations and their implications  allows stakeholders to judge the data’s credibility for their intended use  and to use the data in appropriate ways. By developing and  implementing a process to systematically review the reliability of the data  or consider the results of assessments components have completed,  comprehensively identify any limitations, and communicate the data or  methodological limitations with the metrics, DHS would improve the  quality of the information available to Congress, DHS leadership, and the  public. Doing so would also facilitate a better understanding and  appropriate interpretation and use of the data in the context of the Border  Security Metrics Report, thereby enhancing the report’s value as a source  of information for future decision-making.

DHS’s Model to Estimate Unlawful Border Entries Uses Assumptions that Have Not Been Validated and Does Not Convey Uncertainty of Estimates

DHS Used a Statistical Model to Estimate Unlawful Border Entries for Three Metrics

Based upon statistical modelling, DHS developed a Model-based  Apprehension Rate to calculate the total number of unlawful border  entries between land POEs, including entries both detected by Border  Patrol and “estimated undetected unlawful entries.” DHS reported that in  fiscal year 2016 there were about 624,000 detected entries (which include  apprehensions, turn-backs, and got aways) and estimated that there were  about 62,000 undetected unlawful entries. DHS also used the Model- based Apprehension Rate to develop two other metrics in the fiscal year  2017 Border Security Metrics Report:  (1)  A “probability of detection rate,” which is the estimated proportion of  the number of detected unlawful border entries to the total number of  unlawful entries between land POEs. DHS estimated that in fiscal  year 2016, 91 percent of unlawful border crossers were detected and  9 percent were not detected.  (2)  The “attempted unlawful border crosser apprehension rate,” which is  the estimated proportion of unlawful border entrants apprehended by  Border Patrol to the total number of unlawful entrants between land  POEs. DHS estimated that in fiscal year 2016, 65 percent of  individuals were apprehended by Border Patrol and 35 percent of  individuals attempting an unlawful border entry either got away or  entered the United States undetected.

DHS based its statistical model upon research conducted by the Institute  for Defense Analyses that leveraged long-standing research using  capture-recapture models. Originally developed and utilized in biological  and ecological sciences, capture-recapture models have been applied to  other disciplines, including social science. According to the Institute for  Defense Analyses, capture-recapture models have been the core  approach for academic efforts to model the process of unlawful entry into  the United States across land borders for several decades.

To develop its statistical model, DHS used a capture-recapture  methodology to calculate a probability of apprehension by counting the  number of unlawful border crossers that were apprehended multiple  times. At a high-level, capture-recapture involves taking an initial sample  of the population of interest, in this case individuals attempting to cross  the border unlawfully. Then, separately, a second, independent sample of  the same population is taken. The samples are then compared to  determine the number of individuals who appear in both samples. When  the number of individuals who appear in both samples (e.g., individuals  who have been apprehended twice) is low, it can be inferred that the  overall population of interest (e.g., total unlawful border crossers) is much  larger than the total number of individuals in the two samples. On the  other hand, if the recapture rate is high, then it can be inferred that the  overall population of interest is not much larger than the total number of  individuals in the two samples.

In the context of unlawful border crossing, when an individual’s first  attempt at unlawfully crossing the border is successful, the individual  enters the United States and no apprehension is made. However, if an  individual is apprehended, Border Patrol records an apprehension of this  individual in a DHS data system and the individual is potentially subject to  consequences for entering unlawfully, such as administrative  enforcement and removal, criminal prosecution, or being barred from  legally entering the United States in the future. The individual is then  returned to his or her home country, where the individual can then choose  whether or not to make another attempt to unlawfully cross the border.  During a second attempt to unlawfully cross the border, the individual  faces the same possible outcomes (enter the United States unlawfully or apprehension by Border Patrol). Figure 2 provides the framework for  DHS’s Model-based Apprehension Rate.

DHS modified the traditional capture-recapture methodology by  calculating a deterrence rate of 60 percent in fiscal year 2016 to account  for individuals who choose not to make another unlawful border crossing  attempt. The deterrence rate accounts for an individual being deterred  from attempting to unlawfully cross the border again; that is, DHS  assumed that some percentage of apprehended individuals, once  returned to their country, will remain in their home country. DHS  calculated the deterrence rate based upon a survey of Mexican  individuals who were apprehended and returned to the border region of  Mexico by U.S. immigration authorities. DHS assumed the remaining 40  percent of individuals who were apprehended and removed to their home  country in fiscal year 2016 remain undeterred and will attempt to  unlawfully cross the border again.

Historically, DHS (and its predecessor the Immigration and Naturalization  Service) did not use statistical models to calculate an apprehension rate  but relied on apprehensions as a proxy measure for all unlawful entries  (both observed and unobserved) between POEs. DHS also included in its  report information on the apprehension rate using this method.  Specifically, DHS also calculated an Observational Apprehension Rate  based on direct observations (unlawful border crossers observed by  Border Patrol) and indirect observations (residual evidence of a border  crosser, i.e., footprints) of attempted unlawful border crossers. Using the  observational apprehension rate, DHS calculated that in fiscal year 2016,  it apprehended 79 percent of unlawful border crossers.

DHS’s Statistical Model Uses Assumptions about Border Crossers that Have Not Been Validated and May Affect Results

DHS made assumptions about border crossers to develop its statistical  model and described these assumptions in its report; however, DHS did  not validate some of these assumptions or determine how they potentially  could affect the accuracy of the Model-based Apprehension Rate through  the use of sensitivity analyses.

More specifically, DHS’s model incorporates several assumptions related  to border crossers. Among others, these assumptions include:  the rate at which individuals will be deterred from crossing again  remains the same, regardless of the number of attempts an individual  has made;   individuals who indicate an intent to stay near the U.S.-Mexico border  will attempt re-entry; a single apprehension rate applies to diverse groups of border  crossers, regardless of their nationality or the number of attempts an  individual has made; and  certain individuals will not evade Border Patrol.

However, the validity of some of these assumptions—which affect the  Model-based Apprehension Rate—is uncertain. For example, DHS’s  model estimates the rate at which a diverse group of border crossers  attempting to evade detection will be apprehended by Border Patrol. This  group includes both Mexicans and non-Mexicans and individuals who  attempt to cross again after varying amounts of time. Despite this  diversity, the model assumes that all crossers have the same chance of  apprehension on each attempt to cross the border. This assumption  allows DHS to apply the estimated apprehension rate developed based  on a sample of Mexicans re-apprehended within 90 days—the group for  whom relevant data exist—to a broader population of individuals  regardless of the number of attempts the border crossers have made or  their nationality. However, DHS did not make efforts to determine the  extent to which an apprehension rate based on Mexican citizens re- attempting entry within 90 days would reflect apprehension rates for non- Mexicans and individuals crossing again after longer periods.

Additionally, DHS assumes that the apprehension rate never varies  between an individual’s attempts at crossing the border. For example,  DHS assumes that an individual making a first attempt at crossing the  border faces the same odds of apprehension as an individual making a  fourth or fifth attempt at crossing the border. However, DHS has not  explored the possibility that, for example, individuals may gain experience  and knowledge from border-crossing attempts that could help them better  evade Border Patrol on subsequent attempts.

Further, DHS’s model assumes that certain individuals unlawfully  crossing the border, such as those seeking asylum, will not evade  apprehension and will turn themselves in to Border Patrol. Specifically, in  addition to individuals who ultimately do seek asylum, DHS also includes  within this group and applies this assumption to individuals apprehended  as a family unit and unaccompanied minors. Under this assumption, 100  percent of such individuals are apprehended. According to DHS’s fiscal  year 2017 Border Security Metrics Report, these individuals have  historically been released into the United States with a Notice to Appear  in immigration court for legal proceedings on a future date, rather than  being subject to immediate DHS enforcement consequences such as  voluntary return. Therefore, DHS assumes that 100 percent of these  individuals will self-present to Border Patrol because, in doing so, they  are able to claim asylum or other protection and potentially remain in the  United States.

However, representatives from the Institute for Defense Analysis stated  that while anecdotally self-presenting rates of these individuals are high,  more rigorous analysis is needed to accurately estimate a self- presentation rate. For example, it is possible that not all families crossing  the border unlawfully may seek to self-present to Border Patrol; some  may attempt to evade capture and enter the United States undetected. In  this case, DHS may be underestimating the number of individuals who  unlawfully cross the border and enter the United States by assuming 100  percent of these individuals will self-present to Border Patrol agents.  Additionally, DHS noted in its fiscal year 2017 Border Security Metrics  Report that this assumption does not reflect the actual behavior of all  border crossers in this group. OIS officials stated that they based this  assumption on interviews with Border Patrol agents but had not done  formal or quantitative analysis to support this assumption. Further, OIS  officials stated that they did not have a strong alternative assumption to  use instead and therefore assumed that 100 percent of individuals within  this group are apprehended.

DHS described these assumptions in its report but did not provide  quantitative information on the extent to which these assumptions  affected the Model-based Apprehension Rate through the use of  sensitivity analyses. Sensitivity analyses help to convey the extent to  which changing the values of variables, assumptions, data, or other input  affects statistical estimates. For example, sensitivity analyses could  provide information on how different assumptions about unlawful border  crossers’ behavior and other inputs to the statistical model could have  affected the Model-based Apprehension Rate. OIS officials stated that  while they had started to run sensitivity analyses by modifying certain  assumptions, they had not completed the analysis and did not include  results of the sensitivity analyses in the report.

The Office of Management and Budget’s (OMB) statistical standards for  federal agencies include providing the results of sensitivity analyses for  key methodological assumptions to ensure that these assumptions do not  unduly affect the results of the model. By including the results of  sensitivity analyses in its Border Security Metrics Report, DHS would  allow Congress and the public to better understand the potential  limitations associated with its model and make independent assessments  on its accuracy.

DHS Did Not Convey the Statistical Uncertainty of Its Estimated Apprehension Rate

DHS used a statistical model to develop the Model-based Apprehension  Rate but did not provide information on the level of uncertainty related to  this estimate. Rather, the fiscal year 2017 DHS Border Security Metrics  Report provided a single rate that does not fully convey the difficulty and  uncertainty of estimating partially unobserved metrics, such as unlawful  entries and the probability of detection. Specifically, using the Model- based Apprehension rate, DHS estimated that 65 percent of unlawful  border crossers were apprehended in fiscal year 2016, and the remaining  35 percent entered the United States. However, like all statistical models,  DHS’s estimate is based upon a limited sample of data and may be  affected by random variation, meaning that DHS does not have complete  certainty that its rate is accurate. DHS included a discussion of limitations  in the report but did not quantify its degree of uncertainty.

According to the OMB statistical standards for federal agencies, possible  variation in estimates should be noted, such as by reporting the range of  each estimate. Measures of statistical uncertainty, such as margins of  error or confidence intervals, help to convey the amount by which  estimates might vary due to randomness in the data and allows  consumers of the estimates to evaluate their accuracy.

OIS officials stated that they agree that providing measures of statistical  uncertainty would help Congress and the public better understand the  Model-based Apprehension Rate to evaluate border security. Officials told  us that the office had begun to develop measures of statistical uncertainty  but did not complete this effort because the staff member who was  working on the analyses recently left the office. Further, OIS officials  stated that they were unsure when they would be able to provide  measures of statistical uncertainty in future reports. Including measures of  statistical uncertainty in future reports would allow Congress, policy  makers, and the public to more fully evaluate the extent to which the  metrics that use the Model-based Apprehension Rate are valid. Further,  while DHS may ultimately adopt a new, simulation-based model in the  future, described later in this report, it plans to use the current Model- based Apprehension Rate for estimates in its Border Security Metrics  Report for the foreseeable future. Therefore, providing this additional  information about the estimates would allow DHS to more accurately  convey how limitations in available data and methods could affect the  results and provide more useful information about migration and border  enforcement. Additionally, to the extent DHS adopts a new estimating  metric, that estimate may have some level of uncertainty associated with  it.

DHS Is Developing Another Model to Better Reflect Unlawful Border Entries at the Border

DHS is developing another model because its current statistical model  may not sufficiently reflect conditions at the southwest border.  Specifically, DHS’s current statistical model does not fully account for the  changing population of unlawful border crossers. The capture-recapture  methodology, which underlies the Model-based Apprehension Rate, was  developed to sample homogenous populations that behave in set,  uniform ways. However, those crossing the border have become  increasingly diverse in recent years. Our analysis of DHS data used to  develop the Model-based Apprehension Rate shows that the number of  unlawful border crossers whose characteristics and behavior are best  reflected in the statistical model has declined. For example, our analysis  illustrated that the population that conforms best to the model’s  assumptions—adult Mexicans travelling without dependents who do not  plan to claim asylum and who are returned to Mexico in a short amount of  time—has fallen from over 60 percent of apprehensions in fiscal year  2000 to less than 25 percent of apprehensions in 2016, as shown in  figure 3.

Conversely, the number of individuals who are excluded from the  statistical model such as non-Mexicans, and individuals whose behavior  may not reflect the model’s assumptions, such as asylum-seekers or  those who have not departed the United States (e.g., because they are  awaiting immigration court proceedings) have increased over time, as  shown in figure 4. For example, the percentage of individuals  apprehended at the border who are excluded from the model because  they await immigration court proceedings increased from 26 percent in  fiscal year 2000 to almost 70 percent in fiscal year 2016.

DHS acknowledged these trends in its fiscal year 2017 Border Security  Metrics Report and noted them as a limitation to the effectiveness of its  model. OIS officials further noted that some of these limitations are  difficult to address within the bounds of the statistical model. For example,  to properly account for non-Mexicans, OIS officials stated that they would  need information on the rate at which non-Mexicans are deterred from  crossing the border. However, it would be difficult and costly to obtain this  information through the use of a survey and real-world data does not  already exist, according to OIS officials.

To help address limitations of its current statistical model, DHS has  invested in another research project to estimate the number of unlawful  border crossers between land POEs, including unknown border entries.  Border Patrol contracted with Johns Hopkins Applied Physics Laboratory  to undertake a project that aims to use a combination of statistical  modeling and data from sensors along the border to estimate the total  number of unlawful border entries between land POEs, including entries  both detected by Border Patrol and those not detected by Border Patrol.  According to project documentation we reviewed, the project plans to  leverage the CBP Tactical Simulation, an agent-based simulation of  tactical border operations. CBP Tactical Simulation incorporates  information on terrain at the border based on geographic information  systems and sensors along with probability models that reflect how  Border Patrol agents and unlawful border crossers behave in given  circumstances.

Border Patrol and OIS officials told us that this project would be more  adaptable to changing border conditions and could help the agency  address limitations associated with the Model-based Apprehension rate.  Specifically, according to OIS officials, a simulation-based estimate would  rely upon fewer assumptions about the types of individuals who unlawfully  cross the border as compared to the current Model-based Apprehension  rate. However, Border Patrol officials noted that estimates of  unobservable phenomena, such as unobserved border entries, always  face some limitations in their accuracy and that the new model may still  rely upon samples of data that would have associated uncertainty as well  as assumptions that would need to be validated. Ultimately, though,  Border Patrol officials stated that the simulation-based model may be an  improvement upon the current Model-based Apprehension rate.

Border Patrol officials stated that the first iteration of the model would be  presented to Border Patrol leadership for their review at the end of fiscal  year 2019 and if at that time Border Patrol leadership approves the  model, the earliest the simulation-based estimate could potentially be  incorporated into the DHS Border Security Metrics Report would be for  fiscal year 2020. Exploring alternative models is a positive step for DHS,  however given that the project is in the early stages, it is too early to tell if  it will be able to address the limitations we identified associated with the  current model.

Our Prior Work Has Identified Other Metrics DHS Could Use to Help Measure the Effectiveness of Border Security

In addition to the NDAA metrics, we have identified other metrics that  DHS could use to help measure the effectiveness of border security. In  particular, based on the findings from our previous reviews of border  security programs and efforts, we have recommended that DHS use  metrics that are relevant to each of the four domains listed in the NDAA— between POEs, at POEs, the maritime border, and for air and marine  security in the land domain. For example,

Between POEs domain. In February 2017, we reported on the use of  border fencing along the southwest border and found that CBP  collects data that could help provide insight into how border fencing  contributes to border security operations, including the location of  illegal entries. For example, we found that CBP collects data it could  potentially use to determine the extent to which border fencing diverts  illegal entrants into more rural and remote environments, and border  fencing’s impact, if any, on apprehension rates over time. However,  CBP had not developed metrics that systematically use these data to  assess the contributions of border fencing to its mission. To better  position CBP to make resource allocation decisions with the best  information available to inform competing mission priorities and  investments, we recommended that the Chief of the Border Patrol  develop metrics to assess the contributions of pedestrian and vehicle  fencing to border security along the southwest border using CBP data.  DHS agreed with the recommendation and stated that it planned to  develop metrics for use in its operational control framework for  southwest border security operations. As of October 2018, DHS  stated that the department planned to test the metrics and implement  them in the framework by September 2019.

At POEs domain. In July 2017, we reported on the Importer Security  Filing (ISF) program and found that while ISF rule data have  improved the program’s ability to identify high-risk cargo shipments,  CBP could collect additional performance information to better  evaluate program effectiveness. While evaluating the direct impact  of using ISF rule data to assess shipment risk is difficult, we identified  examples of how CBP could better assess the ISF program’s  effectiveness. For example, CBP could track the number of containers  not listed on a manifest—which could pose a security risk—it identifies  through reviewing vessel stow plans. Collecting this type of additional  information would help CBP better assess whether the ISF program is  improving its ability to identify high-risk shipments. Therefore, we  recommended that CBP identify and collect additional performance  information on the impact of the ISF rule data, such as the  identification of shipments containing contraband, to better evaluate  the effectiveness of the ISF program. DHS agreed with the  recommendation and reported that it is working to assess additional  performance metrics to evaluate the effectiveness of the ISF program  and anticipates completing the assessment by end of December  2019.

Maritime border domain. In October 2017, we reported on the Coast  Guard’s performance goals and found that although the Coast  Guard’s performance goals are generally aligned with its statutory  missions, the Coast Guard does not explain why certain aspects of  mission performance are measured while others are not. For  example, we found that while the Coast Guard’s mission is to interdict  all illegal drugs, the agency’s two performance goals related to that  mission were for cocaine interdiction only, excluding many other  substances. We recommended that the Coast Guard either develop  new performance goals to address mission activity gaps, or explain in  the Coast Guard’s Annual Performance Report why certain aspects of  mission performance are measured while others are not. Developing  new goals to address missions, or describing how existing goals  sufficiently assess mission performance, could better convey the  Coast Guard’s progress in achieving its missions.

DHS agreed with the recommendation and in February 2018, the  Coast Guard provided us with its updated fiscal year 2017 Annual  Performance Report. We found that while the updated report  explained why performance goals related to its drug interdiction  mission focus solely on cocaine interdiction, for the four other  performance goals we previously identified as not fully addressing all  related mission activities, the updated report did not include additional  goals or explain why certain aspects of mission performance are not  measured. We continue to believe that in instances in which  performance goals do not fully address all of the respective mission  activities, the Coast Guard’s Annual Performance Report should  include an explanation.

Air and marine security in the land domain. In May 2017, we  reviewed DHS’s efforts to address subterranean, aerial, and maritime  smuggling of drugs and humans. We found that while DHS  established high-level performance measures and collected data on  smuggling by ultralight aircraft, it had not assessed its efforts specific  to addressing this smuggling method. Additionally, we found that DHS  had similarly not assessed smuggling methods such as tunnels,  panga boats (a fishing vessel), and recreational vessels. We  recommended that DHS direct CBP, ICE, and Coast Guard to  establish and monitor performance measures and targets related to  ultralight aircraft, cross-border tunnels, panga boats, and recreational  vessel smuggling to help provide reasonable assurance that efforts to  address these smuggling methods are effective. By establishing  measures and monitoring performance against targets, managers  could obtain valuable information on successful approaches and  areas that could be improved to help ensure that technology  investments and operational responses to address these smuggling  methods are effective.

DHS agreed with the recommendations for measures related to  ultralight aircraft and cross-border tunnels. DHS reported that AMO  and Border Patrol have drafted a performance measure for ultralight  aircraft, however, reviews and approval of the measure will not be  completed until November 2019. As of June 2018, DHS reported that  ICE was leading the development of measures related to cross-border  tunnels. DHS did not agree with the recommendation to establish  measures and monitor performance against targets for smuggling by  panga boats and recreational vessels because the department  believed measures and targets would not provide the most useful  strategic assessment of operations to prevent all illicit trafficking,  regardless of area of operations or mode of transportation. We  continue to believe that the recommendation is valid and recognize  the value of high-level strategic performance measures. However,  such high-level measures may not provide sufficiently detailed  performance information to allow DHS to identify successful  approaches to addressing smuggling by panga boats and recreational  vessels and areas for improvement. Further, establishing performance  measures and targets related to smuggling by panga boats and  recreational vessels could, in turn, better position DHS to understand  the overall smuggling threat.

Appendix II provides additional information on these and other metrics we  have previously recommended that DHS could use to help measure the  effectiveness of border security in the four domains.

Securing U.S. borders is a complex undertaking that spans multiple  domains and locations. It is also a key part of DHS’s mission for which  DHS has made significant investments over the years. Given the  complexity and breadth of border security efforts, having data and  information available on the state of border security is important for DHS  as well as policymakers and the public to understand the effectiveness of  those investments. DHS’s fiscal year 2017 Border Security Metrics Report  makes an important contribution in providing such data and information.

DHS components generally have processes to help ensure the reliability  of the data used in the metrics report and DHS identified and disclosed  some data and methodological limitations with the metrics. However,  DHS did not systematically review the reliability of data used in all metrics  to identify and disclose limitations and their potential implications for the  metric. Without complete information about the limitations of the data or  the metric methodologies used in the report, Congress, policymakers, and  the public may not be aware of important context or information needed to  fully and appropriately understand the data being presented. By  developing and implementing a process to systematically review the  reliability of the data, as well as comprehensively identify limitations and  communicate limitations of the metrics, DHS would improve the quality of  the data and information provided in the report which would facilitate a  better understanding and appropriate interpretation of the data and  information provided.

To develop three metrics in the report, DHS used a statistical model that  incorporated untested assumptions about the behavior of unlawful border  crossers that may not reflect real-world conditions. DHS was transparent  about the limitations of its model, but providing the results of sensitivity  analyses and measures of statistical uncertainty related to the model  would allow Congress, policymakers, and the public to better understand  its potential limitations and more fully evaluate the validity of DHS’s  metrics that use estimates.

We are making the following four recommendations to DHS:  The Secretary of Homeland Security should develop and implement a  process to systematically review the reliability of the data used in its  Border Security Metrics Report and comprehensively identify any  limitations with the data and methodologies that underlie its metrics.  (Recommendation 1)

The Secretary of Homeland Security should ensure the communication of  the limitations of the metrics identified through the systematic review in  the department’s annual Border Security Metrics Report.  (Recommendation 2)

The Under Secretary for the Office of Strategy, Policy, and Plans should  include the results of sensitivity analyses to key assumptions in its  statistical models of unlawful entry estimates in its annual Border Security  Metrics Report. (Recommendation 3)

The Under Secretary for the Office of Strategy, Policy, and Plans should  include measures of statistical uncertainty for all metrics based on  estimates derived from statistical models in its annual Border Security  Metrics Report. (Recommendation 4)

We provided a draft of this report to DHS and the Office of National Drug  Control Policy for review and comment. DHS provided written comments,  which are reproduced in appendix III and discussed below. DHS also  provided technical comments, which we incorporated as appropriate. The  Office of National Drug Control Policy indicated via e-mail that it did not  have any comments on the draft report.

In its comments, DHS concurred with our recommendations and stated  that it planned to implement 3 of the 4 by October 2020. With respect to  our second recommendation, DHS requested that we consider it closed  as implemented because the department already detailed some of the  limitations in its fiscal year 2017 report, and plans to continue to identify  known limitations and the progress made to mitigate previously identified  limitations in future reports. As discussed in this report, we agree that  DHS identified and disclosed limitations for some metrics in its fiscal year  2017 Border Security Metrics Report; however, we identified at least one  additional limitation for 21 of the 35 metrics on which DHS reported that  DHS did not disclose or about which it could have been more transparent.  To address the intent of this recommendation, once DHS has  implemented a process to systematically review the reliability of the data  used in its report and comprehensively identified related limitations, it  should disclose those limitations in its annual Border Security Metrics  Report.

We are sending copies of this report to the appropriate congressional  committees and the Secretary of the Department of Homeland Security.  In addition, the report is available at no charge on the GAO website at  http://www.gao.gov. Contacts points for our Offices of Congressional  Relations and Public Affairs may be found on the last page of this report.

If you or your staff have any questions about this report, please contact  me at (202) 512-8777 or gamblerr@gao.gov. GAO staff who made key  contributions to this report are listed in appendix IV.

Appendix I: Additional Information on Metrics Contained in the Department of Homeland Security’s Fiscal Year 2017 Border Security Metrics Report

This appendix provides additional information on our analysis of the  suitability and validity of the metrics the Department of Homeland Security  (DHS) reported in its fiscal year 2017 Border Security Metrics Report for  each of the four domains listed in the National Defense Authorization Act  for Fiscal Year 2017 (NDAA)—between ports of entry, at ports of entry,  the maritime border, and air and marine metrics in the land domain.  Specifically, this appendix provides information on the metrics including  their status, descriptions, differences between what DHS reported for the   metrics and how they were described or defined by the NDAA, limitations,  and any additional information or planned actions by DHS, where  applicable.

Description   This metric is a rate comparing apprehensions to the total number of attempted unlawful border crossers. As  such, this metric requires an estimate of the number of unlawful entry attempts that are not detected, which is  added to the number of detected unlawful border crossers to create the denominator. The Department of  Homeland Security (DHS) provided two methods for calculating this rate in its report. The first method, called  the Model-based Apprehension Rate, uses a statistical model based on the capture-recapture methodology to  estimate the rate.The second method, called the Observational Apprehension Rate, calculates the ratio of  apprehensions to the sum of apprehensions and got aways. unlawful entries is limited to the southwest border. According to the report, research is underway on methods  to produce estimates for the northern border.

Limitations   The Observational Apprehension Rate incorporates data on apprehensions, and got aways, while the Model- based Apprehension Rate is based on an estimate for undetected unlawful entries. Consequently, the  limitations for those metrics also apply here. For more information on the limitations for those metrics, see the  respective sections below.

DHS identified: The observational apprehension rate excludes unobserved got aways.

Additional information and planned actions by the Department of Homeland  Security   In its report, DHS noted that it has taken steps to improve situational awareness along the border and mitigate  limitations. These steps include investing in technology, refining observational estimates, and developing a  methodology to estimate statistical reliability. According to U.S. Border Patrol officials, investments in new  technology have enabled U.S. Border Patrol to better detect cross-border activities. For additional information  on the data elements used for this metric and DHS’s planned actions, see the respective sections below on  apprehensions, got aways, and the estimate for undetected unlawful entries.

Description   This metric is a count of the total  number of attempted unlawful  border crossers between land  ports of entry who were directly or  indirectly observed or detected by  U.S. Border Patrol. The  Department of Homeland Security  (DHS) calculated this metric by  adding turn backs, got aways, and  apprehensions of unlawful border  crossers.

Patrol officials, the northern border has different immigration dynamics than the southern border and accounts  for a significantly smaller number of turn backs and got aways overall, so northern border data were not  included.

Limitations   Because this metric incorporates data on apprehensions, got aways, and turn backs, the limitations for those  metrics also apply here. For more information on the limitations for those metrics, see the respective sections  below.

Additional information and planned actions by the Department of Homeland  Security   For additional information on the data elements used for this metric and DHS’s planned actions, see the  respective sections below.

Description   This metric is an estimate of the number of attempted unlawful border crossers that are not directly or indirectly  observed or detected by U.S. Border Patrol (Border Patrol). The Department of Homeland Security (DHS) used  a statistical model, based on capture-recapture methodology, to estimate total successful unlawful entries, and  subtracted detected got aways to calculate the total number of undetected unlawful entries. unlawful entries is limited to the southwest border. According to DHS’s report, research is under way to  produce this estimate for the northern border.

DHS does not currently have reliable data on the estimated share of migrants who, following an  unsuccessful unlawful entry attempt, are deterred from making a subsequent reentry attempt. For its  model, DHS used data from a survey of recently removed Mexicans, which asked them about their  intentions to re-enter the United States. According to DHS’s report, a shortcoming of the survey is that it  does not take account of shifting border enforcement efforts, potential changes in behavior by individuals  who have been exposed to consequence programs, or other deterrent factors along the border.  Consequently, any resulting undercount in the estimate of the deterred population results in a downward  bias.

The population that conforms best to the model’s assumptions represents a diminishing share of southwest  border apprehensions. Specifically, in its report DHS said that Mexican adults removed to the nearest  border accounted for about 95 percent of apprehensions in the 1990s. However, because of recent  changes at the border, including changes in the composition of border flows (i.e., rising numbers of Central  Americans and asylum seekers) and in Border Patrol’s enforcement strategy, the population best reflected  in the model has declined to as few as 20 percent of apprehensions in recent years. Further, DHS noted  that some alien populations, such as those seeking asylum and who do not evade detection by Border  Patrol agents, are also excluded from the model. However, these populations make up an increasing share  of apprehensions in recent years.

The model uses restrictive assumptions about which re-apprehensions to include. For example, the model  excludes apprehensions occurring at check points and other remote locations and those occurring more  than 4 days after an illegal entry. According to DHS, these assumptions result in a downward bias.

DHS described assumptions it made in its report but did not provide quantitative information on the extent  to which they affected its estimated undetected unlawful entries through the use of sensitivity analyses.

Sensitivity analyses help to convey the extent to which changing the values of variables, assumptions,  data, or other input affects statistical estimates. By including the results of sensitivity analyses in its Border  Security Metrics Report, DHS would allow Congress and the public to better understand the potential  limitations associated with its model and make independent assessments on its accuracy. DHS did not  provide information on the statistical level of uncertainty related to this rate, such as margins of error or  confidence intervals. This information would help convey how the estimates might vary due to randomness  in the data. Instead, DHS provided a single rate that does not fully convey the difficulty and uncertainty of  the estimate.

This metric incorporated data on apprehensions and got aways. For more information on the limitations  associated with those metrics, see the respective sections below.

Additional information and planned actions by the Department of Homeland  Security   According to DHS, officials are continuing to improve the accuracy of the existing statistical model for  estimating unlawful border crossers but are also considering alternative methodologies. U.S. Customs and  Border Protection has contracted with Johns Hopkins Applied Physics Laboratory to develop a new model for  estimating the flow of unlawful border crossers. This model uses a combination of statistical modelling, data  from sensors along the border, and probability models that reflect how Border Patrol agents and unlawful  border crossers behave in given circumstances. Border Patrol officials estimated that the earliest the  simulation-based estimate could potentially be incorporated into the DHS Border Security Metrics Report would  be for fiscal year 2020.

Description   This metric is a count of the number of unlawful border crossers who, after making an unlawful entry into the  United States, responded to law enforcement efforts by returning promptly to the country from which they  entered. These data came from U.S. Border Patrol (Border Patrol) records.

Border Patrol officials, the northern border has different immigration dynamics than the southern border and  accounts for a significantly smaller number of turn backs overall, so northern border data were not included.  Officials stated that while the current emphasis of reporting is on the southwest border, efforts are underway to  identify and find ways to capture data that are important and reflective of the effectiveness in addressing  threats specific to the northern border.

The estimate aggregates potentially subjective observations from thousands of individual agents.

Some unlawful border crossers may enter the United States to drop off drug loads or to act as decoys to  lure agents away from a certain area and then return to Mexico, and therefore may be misidentified as turn  backs.

In our previous work we identified differences in the procedures for reporting and classifying turn backs  across sectors, and noted how factors such as terrain and weather may impact agents’ abilities to  accurately detect turn backs.

According to DHS, since 2014, Border Patrol has implemented a standard, southwest border-wide  methodology to improve reporting and mitigate the potential subjectivity of observations by agents.  Therefore, data before 2014 are not necessarily comparable to data from 2014 and later. DHS presented  the data in a table without explaining that the methodology used to categorize and count turns backs  changed in 2014.

Additional information and planned actions by the Department of Homeland  Security   According to DHS’s report, Border Patrol has taken steps to implement a standard, southwest border-wide  methodology to improve reporting of potential turn backs. In addition, DHS’s report said that command staff  ensure all agents are aware of and utilize proper definitions for apprehensions, got aways, and turn backs at  their respective stations. They also ensure necessary communication takes place between and among sectors  and stations to minimize double-counting when subjects cross through more than one station. DHS’s report  noted that Border Patrol headquarters components validate data integrity.

Description   This metric is a count of the number of unlawful border crossers who are directly or indirectly observed  entering unlawfully, are not apprehended, and are not turn backs. These data came from U.S. Border Patrol  (Border Patrol) records.

Border Patrol officials, the northern border has different immigration dynamics than the southern border, so  northern border data were not included. Officials stated that while the current emphasis of reporting is on the  southwest border, efforts are under way to identify and find ways to capture data that are important and  reflective of the effectiveness in addressing threats specific to the northern border.

Limitations   DHS identified: The count aggregates potentially subjective observations from thousands of individual agents.

In previous work we identified differences in procedures for reporting and classifying got aways across  sectors, and noted how factors such as terrain and weather may impact agents’ abilities to accurately  detect got aways.

For information on limitations with the model-based estimate for undetected unlawful entries, see the  section for estimated undetected unlawful entries above.

Additional information and planned actions by the Department of Homeland  Security   According to DHS’s report, Border Patrol has taken steps to implement a standard, southwest border-wide  methodology to improve reporting of potential got aways. In addition, DHS’s report said that command staff  ensure all agents are aware of and utilize proper definitions for apprehensions, got aways, and turn backs at  their respective stations. They also ensure necessary communication takes place between and among sectors  and stations to minimize double-counting when subjects cross through more than one station.  DHS’s report  noted that Border Patrol headquarters components validate data integrity.

As a comparison against the counts of documented got aways, DHS also provided an estimate of total  successful unlawful entries along the southwest border using a statistical model based on capture-recapture  methodology. For more information on the methodology for this estimate, see the section titled “Estimated  Undetected Unlawful Entries” in this appendix.

Description   This metric is a rate comparing the number of apprehensions and turn backs to the number of apprehensions,  estimated undetected unlawful entries, turn backs, and got aways in each U.S. Border Patrol sector. rate is not available because sector-level estimates of unlawful entries and attempts have not yet been  produced and validated. As an alternative, DHS presented data using the interdiction effectiveness rate. With  this rate, the estimated undetected unlawful entries measure is replaced with known got aways. However, DHS  does not have an interdiction effectiveness rate for the northern border so it solely provided data for the  southwest border. According to DHS’s report, the department has not yet developed a northern border  interdiction effectiveness rate because there are only a small number of attempted and successful entries  along the northern border.

Additional information and planned actions by the Department of Homeland  Security   DHS reported that sector-level estimates of unlawful entries and attempts are projected to be available in its  2019 annual Border Security Metrics Report to Congress.

Authorization Act for Fiscal Year  2017 (NDAA) defined this metric  as a rate comparing the estimated  total undetected unlawful border  crossing attempts to the unlawful  border crossing effectiveness rate.  The Department of Homeland  Security (DHS) calculated this  metric by dividing the detected  unlawful entries by the estimated  total unlawful entries. The number  of detected unlawful entries is  calculated by adding turn backs,  got aways, and apprehensions.  Estimated total unlawful entries is  calculated by adding turn backs,  apprehensions and estimated total  successful unlawful entries derived  from DHS’s statistical model. unlawful entries is limited to the southwest border. Additionally, DHS used detected unlawful entries as the  numerator, instead of the estimated total unlawful border crossing attempts not detected as called for in the  NDAA. For the denominator DHS used the estimated total unlawful entries instead of the unlawful border  crossing effectiveness rate, as called for in the NDAA.

Limitations   Because this metric incorporates data on apprehensions, got aways, and turn backs, as well as the estimate  for undetected unlawful entries, the limitations for those metrics also apply to this metric. For more information  on the limitations for those metrics, see the respective sections for those metrics.

Additional information and planned actions by the Department of Homeland  Security   For additional information on apprehensions, got aways, turn backs, and the estimate for undetected unlawful  entries, and any planned actions by DHS for those metrics, see the respective sections for those metrics.

Description   This metric is a count of the  number of apprehensions in each  U.S. Border Patrol (Border Patrol)  sector. Data come from  Border  Patrol records, and each  apprehension of the same  unlawful crosser in a fiscal year is  counted separately, meaning  these data do not represent a  count of unique crossers  apprehended.

Limitations   DHS identified: In its report, DHS said that apprehensions are not a useful indicator of successful unlawful  border crossings over the long-term and across multiple locations because the relationship between  apprehensions and successful unlawful entries depends on the apprehension rate, which changes over time  and may differ by location.

Description   This metric is a count of the number of apprehensions of unaccompanied alien children (UAC), and the  nationality of such children, in each U.S. Border Patrol (Border Patrol) sector. A UAC is a child under 18 years  old with no lawful immigration status, and no parent present and available in the United States to provide care  and physical custody.

Differences between the National Defense Authorization Act for Fiscal Year 2017  and the Department of Homeland Security’s report   The Department of Homeland Security (DHS) only included data for the southwest border.

Limitations   DHS identified: Agents may not be able to reliably distinguish among older children and young adults or  confirm whether children are traveling alone or in family groups.

We previously reported that it can be challenging to obtain accurate information about a child’s country of  origin because of absence of documentation, language barriers, and coached responses by smugglers,  among other reasons.

Border Patrol officials said that the data on UAC may have reliability issues because original data from a  shared database had been changed. Specifically, officials said that in January 2015 they noticed that  Enforcement and Removal Operations staff were inadvertently overwriting Border Patrol’s original data  entries about the status of migrant children apprehended once those children were placed with relatives or  a foster family.

Additional information and planned actions by the Department of Homeland  Security   According to Border Patrol officials, agents rely on statements provided by the child to determine the nationality  of UACs when verifiable documentation is not available. Verifiable documentation could include biometric  checks, birth certificates, state-issued identification cards, and passports. However, officials noted that this list  is not all-inclusive and the processing agent determines the validity of any presented documents. Border Patrol  officials said that a data integrity team regularly examines data on apprehensions and they conduct biweekly  data reliability checks. Additionally, they are working with Enforcement and Removal Operations to modify the  data entry process so that updates can be made without overwriting the original apprehension data entered by  Border Patrol.

Description   This metric is a count of the  number of apprehensions of family  units, and the nationality of such  family units, in each U.S. Border  Patrol (Border Patrol) sector. A  family unit is the number of  individuals apprehended with a  family member. For example, a  mother and child apprehended  together are counted as two family  units.

Limitations   DHS identified: DHS noted that the count of apprehensions for family units is considered reliable, but that  agents may not be able to reliably identify family units.

We identified: According to Border Patrol officials, their data entry system did not have a dedicated field for  agents to record apprehensions of persons within a family unit for all of the years presented in the report. In  December 2014, Border Patrol added specific data entry fields to its data entry processes for agents to input  information about family units. These fields incorporated built in safeguards and edit checks to help ensure that  agents make an appropriate family unit classification. Previously, Border Patrol officials said they used proxy  data to identify family units. Given the additional safeguards and checks included with the new family unit data  entry fields, Border Patrol officials stated that the data after December 2014 may be more reliable overall  compared to previous years. Border Patrol officials stated that they have high confidence in the proxy count for  data pre-2014, but acknowledged that those data may contain misclassifications of family units.

Additional information   According to Border Patrol officials, agents are trained in interviewing techniques and the processing agent will  consider all available evidence to determine the validity of claims to familial relationships. Border Patrol officials  also noted that in order to be categorized as a family unit, at least one member of the family unit must be at  least 18 years of age. Consequently, related individuals younger than 18 years of age that are apprehended  together would not be categorized as a family unit.

Description   This metric is a rate comparing the amount and type of illicit drugs seized between ports of entry in any fiscal  year to the average of the amount and type of illicit drugs seized between ports of entry in the immediately  preceding 5 fiscal years.

Differences between the National Defense Authorization Act for Fiscal Year 2017  and the Department of Homeland Security’s report   None identified.

Description   This metric was not specifically defined in the National Defense Authorization Act for Fiscal Year 2017 (NDAA);  the NDAA called for an estimate of the impact of the Consequence Delivery System (CDS) on the recidivism  rate of unlawful border crossers over multiple fiscal years. The Office of Immigration Statistics (OIS)  calculated this metric by providing the average annual recidivism rate for the 3 years prior to fiscal year 2012— when the CDS was implemented—and the average annual recidivism rate for the subsequent 3 years. The  annual recidivist rate is calculated by dividing the number of unique crossers apprehended multiple times in a  fiscal year by the total number of unique crossers in the fiscal year. to DHS’s report, recidivism data for the northern border were not available due to the small number of  attempted illegal entries along the northern border.

Noting the findings from our January 2017 review, DHS stated that its current recidivism measure could be  strengthened by using the date an unlawful border crosser is removed or returned instead of the date they  are apprehended, as well as by counting re-apprehensions within a fixed period of time defined by the  crosser’s repatriation date instead of by the fiscal year. In January 2017, we reported that using a  crosser’s apprehension history beyond 1 fiscal year, and excluding crossers that have not been previously  removed, among other things, produces a significantly different rate compared to how DHS currently  calculates it. Consequently, we recommended that DHS calculate recidivism for a period of time longer  than 1 fiscal year and that DHS exclude from the recidivism calculation aliens for whom there is no record  of removal and who may remain in the United States. As of December 2018, this recommendation  remained open.

DHS stated that changes in the recidivism rate after 2012 cannot be attributed solely to CDS because  enforcement is a complex, dynamic system.

We identified: Given that DHS’s methodology is to provide the 3-year average of the recidivism rate before and  after CDS was implemented in fiscal year 2012, the data presented will remain static for subsequent annual  reports because the periods of comparison for analyzing recidivism are fixed around a specific point in time.  According to OIS officials, to help address this issue, in the next report they plan to provide individual rates for  each year instead of the 3-year average.

Additional information and planned actions by the Department of Homeland  Security   In its report, DHS noted that future reports will include estimates of the impact of CDS on both the annual  recidivism rate and a longer-term recidivism rate. For example, OIS officials said they plan to update the way  they calculate recidivism for future issues of the report and are developing a multivariate impact analysis that  would take into consideration factors such as crossers’ demographics and immigration history.

Description   This metric was not specifically defined in the National Defense Authorization Act for Fiscal Year 2017 (NDAA);  the NDAA called for an examination of each consequence under the Consequence Delivery System (CDS),  including (1) voluntary return, (2) warrant of arrest or notice to appear, (3) expedited removal, (4) reinstatement  of removal, (5) alien transfer exit program, (6) criminal consequence program, (7) standard prosecution, and  (8) Operation Against Smugglers Initiative on Safety and Security. The Department of Homeland Security  (DHS) presented data on the recidivism rates for each consequence between fiscal years 2012 through 2016.

Differences between the National Defense Authorization Act for Fiscal Year 2017  and the Department of Homeland Security’s report   As noted above for the metric “estimates of the impact of the Consequences Delivery System on recidivism,”

DHS only included data for the southwest border because recidivism data for the northern border were not  available due to the small number of attempted illegal entries along the northern border.

Differences in recidivism rates among the consequences may also reflect differences in the propensity of  the targeted populations to attempt to re-enter.

As with the metric for estimating the impact of the CDS on recidivism discussed above, DHS noted the  limitation that current recidivism data are based on apprehensions within a given fiscal year, and not the  date when an individual was repatriated to their country of origin.

In January 2017, we reported that some unlawful border crossers were incorrectly classified based on CDS  guidance. U.S. Border Patrol (Border Patrol) agents implement CDS by classifying apprehended aliens  into one of seven noncriminal or criminal categories and then applying one or more of eight different  consequences; therefore, determining the correct classification of the unlawful border crosser is important  for identifying and applying the appropriate consequence. Our analysis of Border Patrol apprehension data  from fiscal year 2013 through 2015 showed that Border Patrol did not classify 11 percent of apprehensions  in accordance with the agency’s guidance. We recommended that Border Patrol provide consistent  guidance for classification and take steps to ensure the integrity of classification data. Border Patrol  implemented this recommendation as of December 2017, but the issue could potentially have implications  for the data DHS used in this metric, which was for fiscal years 2012 through 2016.

Additional information and planned actions by the Department of Homeland  Security   According to its report, DHS is refining its analysis and will seek to specifically address the limitations  discussed above in the fiscal year 2018 version of the Border Security Metrics Report.

Ports of entry are U.S. government facilities that provide for the controlled entry into or  departure from the United States. There are 328 ports of entry in the United States. Specifically,  a port of entry is any officially designated location (seaport, airport, or land border location)  where U.S. Customs and Border Protection (CBP) officers or employees are assigned to clear  passengers, merchandise and other items, collect duties, and enforce customs laws; and where  CBP officers inspect persons seeking to enter or depart, or apply for admission into, the United  States pursuant to U.S. immigration law and travel controls. CBP’s Office of Field Operations  (OFO) is the lead DHS component responsible for carrying out activities at POEs. The 15  metrics in this domain measure the number of travelers attempting to enter the United States at  ports of entry, illicit drugs seized at ports of entry, and cargo entering the United States, among  other things. DHS included 11 of the 15 metrics called for in the NDAA for this domain in its  fiscal year 2017 Border Security Metrics Report, as shown in table 5.

DHS reported that the four metrics for which it did not provide information did not yet have a  reliable methodology or were under review, and that DHS was in the process of developing  methodologies to capture the data needed for the requested metrics. DHS officials said these  four metrics would not be ready for inclusion in the next annual report.

Description   This metric is a count of total inadmissible travelers, and requires an estimate of the number of inadmissible  travelers who successfully enter at a port of entry without being detected. The metric is the sum of the  number of inadmissible travelers interdicted and the estimated number of inadmissible travelers who  successfully enter at a port of entry without being detected. inadmissible travelers who successfully enter at a port of entry without being detected. Therefore, DHS only  presented data on known inadmissible travelers.

Additional information and planned actions by the Department of Homeland  Security   DHS projected that the department may be able to include estimates on the number of inadmissible travelers  who successfully enter at a port of entry in its fiscal year 2019 Border Security Metrics Report to Congress.  According to U.S. Customs and Border Protection (CBP) officials, they are in the process of determining  whether CBP’s Compliance Measurement Examination (COMPEX) program could be used as a means to  reliably measure undetected inadmissible travelers.

Description   These metrics are rates that  require data on travelers seeking  admission at a port of entry,  interdictions of inadmissible  travelers, and an estimate of the  number of inadmissible travelers  who successfully enter at a port of  entry without being detected. The  refusal rate is calculated by  dividing the number of  inadmissible travelers interdicted  by all people seeking admission at  a port of entry. The interdiction  rate is calculated by dividing the  number of inadmissible travelers  interdicted by the total number of  inadmissible travelers who attempt  to enter at a port of entry. inadmissible travelers who successfully enter at a port of entry without being detected. Therefore, DHS only  presented data on the refusal rate.

Additional information and planned actions by the Department of Homeland  Security   DHS projected that the department may be able to include estimates on the number of inadmissible travelers  who successfully enter at a port of entry in its next Border Security Metrics Report to Congress. According to  U.S. Customs and Border Protection (CBP) officials, they are in the process of reviewing data and program  policies for CBP’s Compliance Measurement Examination program to determine if the program could be used  as a means to reliably measure undetected inadmissible travelers, which would then be used in calculating the  interdiction rate.

Description   This metric is a count of the amount in kilograms of illicit drugs seized by U.S. Customs and Border Protection  officers at ports of entry. In an appendix to the report, the Department of Homeland Security listed out 34  different types of illicit drugs and the amounts seized for each for fiscal years 2007 through 2016.

Differences between the National Defense Authorization Act for Fiscal Year 2017  and the Department of Homeland Security’s report   None identified.

Description   This metric is a rate that compares  the amount of illicit drugs seized  (in kilograms) by Office of Field  Operations officials at ports of  entry in 1 fiscal year to the  average amount seized in the  immediately preceding 5 fiscal  years. the metric. The Department of Homeland Security provided rates for cocaine, methamphetamine, marijuana,  and heroin for fiscal years 2012 through 2016.

Description   This metric is a count of the number of infractions related to travelers and cargo committed by major violators,  and an estimate of the number of major infractions not interdicted. The Department of Homeland Security  (DHS) calculated an infraction rate by dividing the number of major infractions by the total number of  passengers at ports of entry for fiscal years 2007 through 2016.

National Defense Authorization Act for Fiscal Year 2017 (NDAA). As an alternative, for the purpose of its  report, DHS defined a major infraction as an arrest, including arrests related to terrorism, drugs, criminal  aliens, and currency, among other things.

DHS reported that it does not have a methodology in place to estimate the number of undetected major  infractions. Therefore, only data on known infractions are included.

DHS only included data for passenger infractions and not cargo-related infractions.

Although not requested by the NDAA, DHS provided an infraction rate by dividing the number of known  infractions by the total number of travelers at ports of entry.

Limitations   We identified: Given that DHS’s alternative approach to this metric involves using arrests as a proxy for major  infractions, it is unclear whether there is a one-to-one correspondence between the arrest of a major violator  and the number of infractions committed.

Additional information    According to U.S. Customs and Border Protection (CBP) officials, they plan to use data from CBP’s  Compliance Measurement Examination program as a means to report estimated undetected major infractions  starting with DHS’s fiscal year 2019 report.

Description   This metric is a rate that compares the amount of cocaine seized at land ports of entry to the total estimated  flow of cocaine. the total flow of cocaine through land ports of entry. The Office of National Drug Control Policy produces  annual estimates for total cocaine flow into the United States, but does not have a methodology to estimate the  flow of cocaine through land ports of entry alone. Therefore, the estimates the Department of Homeland  Security used included cocaine flow through all domains. According to the U.S. Drug Enforcement  Administration’s National Drug Threat Assessment, the southwest border remains the key entry point for the  majority of the cocaine entering the United States.

Description   This metric is a rate that compares  the average wait time for vehicles  to pass through a land port of entry  to the total number of commercial  and private vehicles at each land  port of entry. data were not available for every port of entry, such as small ones with negligible wait times.

Limitations   We identified: We reported in July 2013 that commercial vehicle wait time data were unreliable due to  inconsistent data collection processes at ports, and made two recommendations to DHS to improve the  reliability of the data. While DHS implemented these recommendations in 2018, older data, including the data  for the years presented in the report (fiscal years 2012 through 2016), remain unreliable.

Additional information and planned actions by the Department of Homeland  Security    U.S. Customs and Border Protection (CBP) officials clarified that the wait times shown in the report reflect the  average of all hourly recordings for wait times at ports of entry rather than the average passenger or vehicle  experience because CBP did not report a volume-weighted measure of wait times. According to the report,  CBP’s wait time policy is currently under review and new guidance will be issued in the future to account for  improvements in automation and recording.

Description   This metric is a rate that measures traffic volume at land ports of entry against the physical and staffing  capacity at each land port of entry. The Department of Homeland Security (DHS) calculated the average  number of vehicles processed per booth, per hour at each land port of entry.

Differences between the National Defense Authorization Act for Fiscal Year 2017  and the Department of Homeland Security’s report   None.

Additional information   In addition to reporting utilization at each port of entry, DHS provided the average utilization rate for all northern  border land ports of entry and all southern border land ports of entry.

This metric is a rate that measures the frequency of secondary examinations at each land port of entry. The  Department of Homeland Security (DHS) calculated the rate by dividing the recorded number of passengers  sent for secondary inspection by the total number of recorded passengers at each land port of entry.

Differences between the National Defense Authorization Act for Fiscal Year 2017  and the Department of Homeland Security’s report   DHS did not include data on secondary examinations of cargo or shipments.

Description   This metric is a count of the  number of cargo containers at sea  ports that DHS identified as  potentially high-risk using National  Targeting Center (NTC) security  criteria. According to the  Department of Homeland Security  (DHS), all international cargo  shipments coming to the United  States are screened to identify  potentially high-risk containers,  which may then be reviewed,  scanned, or physically inspected  prior to lading at a port of entry.

Differences between the National Defense Authorization Act for Fiscal Year 2017  and the Department of Homeland Security’s report   None identified.

Limitations   We identified: U.S. Customs and Border Protection (CBP) officials said that the process of defining and  identifying “high-risk” shipments can exclude some shipments, such as those in free trade zones.

Additional information   DHS’s report said that the NTC periodically refines, improves, and revises the security criteria for high-risk  shipments, which can affect the number of cargo shipments identified as high-risk.

Description   This metric is a rate comparing the number of potentially high-risk cargo containers scanned at each sea port  of entry during a fiscal year to the total number of high-risk cargo containers that entered the United States at  each sea port of entry during the previous fiscal year. separate from cargo containers that were reviewed or assessed; instead, DHS tracks these inspection  methods collectively. Therefore, DHS also included data on potentially high-risk cargo containers that were  reviewed or assessed as well as those that were scanned in its report.

Limitations   DHS identified: In its report, DHS noted that ratio data are not available for fiscal year 2014 because U.S.

Customs and Border Protection did not collect comparable container-level data (as opposed to shipment-level  data) in fiscal year 2013. DHS also noted that the totals across the ports or field offices may include duplicate  container counts.

NTC officials said that the definition of “high-risk” shipments excludes some shipments, such as those in  free trade zones.

NTC officials noted that assessing, reviewing, and scanning containers are different activities and reflect  different levels of inspection or review. For example, NTC officials said that while all containers are  “assessed” in order to determine their risk level, only higher risk containers may be scanned using radiation  detection and nonintrusive inspection equipment. Consequently, when DHS included data on containers  that were assessed or reviewed but not scanned, the resulting count was higher.  In an appendix to its report, DHS presented a column of data called the “percentage of potentially high-risk  containers scanned (same fiscal year)” for each fiscal year. Given DHS’s inability to separate data on the  different inspection methods, the data in this column included containers that were reviewed by all  inspection methods, not just scanning.   In its appendix, DHS did not present data on the number of containers that “entered the United States,”  even though it used those data to calculate the ratio and they are specified in the National Defense  Authorization Act for Fiscal Year 2017. As a result, it is not possible to verify the accuracy of DHS’s ratio  calculations.

The U.S. maritime border domain encompasses ports, internal or inland waters, and coastal  waters, as well as the territorial sea (waters 12 nautical miles seaward of the U.S. coast),  contiguous zone (waters adjacent to and seaward of territorial sea and extending 24 nautical  miles from shore), and exclusive economic zone (waters seaward of and adjacent to territorial  sea and extending out to 200 nautical miles from shore). U.S. Coast Guard (Coast Guard), Air  and Marine Operations, and U.S. Border Patrol share responsibility for patrolling the U.S.  maritime borders, and territorial sea. The Coast Guard is a component of DHS and the lead  federal maritime law enforcement agency on the high seas (waters beyond 12 nautical miles  seaward of the U.S. coast) and all other waters under U.S. jurisdiction. The Coast Guard  responds to a variety of maritime border security issues, including trafficking of narcotics,  people, illicit goods, unlawful migration, illegal exploitation of natural resources, potential  terrorist activities, and the disruption of maritime commerce. The metrics in this domain  measure the number of migrants and illicit drugs removed, among other things. DHS included 4  of 6 metrics called for in the NDAA for this domain in its fiscal year 2017 Border Security Metrics  Report, as shown in table 6.

Description   This metric was not specifically defined in the National Defense Authorization Act for Fiscal Year 2017 (NDAA).

The NDAA described situational awareness as the knowledge and understanding of current unlawful cross- border activity, including (1) threats and trends concerning illicit trafficking and unlawful crossings, (2) the  ability to forecast future shifts in such threats and trends, (3) the ability to evaluate such threats and trends at a  level sufficient to create actionable plans, and (4) the operational capability to conduct persistent and  integrated surveillance of the international borders of the United States. developing a measure for situational awareness in the maritime domain that meets the intent of the NDAA.  While this effort is in process, DHS presented data on U.S. Coast Guard and U.S. Customs and Border  Protection (CBP) asset (aircraft and cutter or boat) hours contributing to situational awareness or interdiction  support and the number of vessel manifests screened.

Additional information   According to CBP Air and Marine Operations officials, they did not have confidence that the data for years prior  to fiscal year 2016 were consistent enough for making comparisons across years. Consequently, only data for  fiscal year 2016 were included in DHS’s report for the metrics related to CBP.

Description   This metric is a count of the total  number of undocumented migrants  interdicted, identified directly or  indirectly but not interdicted, or  otherwise believed to have  unlawfully entered the United  States through the maritime  border.

Limitations   Department of Homeland Security (DHS) identified: The accuracy of migrant flow counts depends on partners  to report interdictions and the ability to detect migrants. According to the DHS report, the U.S. Coast Guard  relies on partners to report interdictions for compilation in the U.S. Coast Guard’s database. Interdictions may  be double-counted by the U.S. Coast Guard and its partners because they cooperate during operations and  some interdictions by partners may not get reported. Further, some migrants may not be apprehended and  leave no evidence, and are therefore excluded from the known flow figures.

We identified: According to U.S. Coast Guard officials, there is no centralized database for tracking migrant  interdictions, and the decentralized nature of the data collection could lead to errors.

Additional information   According to the U.S. Coast Guard, about 90 percent of the data on migrant interdictions and flow originate  from U.S. Coast Guard records. U.S. Coast Guard officials said that as part of a department-wide initiative to  standardize illegal immigration statistics, they are in the preliminary stages of building a centralized database  to enter and maintain information on migrant interdictions. Additionally, officials said they take steps to  ensure the reliability of externally reported data such as communicating with partners and working together to  reconcile any errors. Within the U.S. Coast Guard, meetings are held regularly to discuss and vet the accuracy  of migrant flow data.

Description   This metric is a rate comparing the amount and type of illicit drugs removed by the Department of Homeland  Security (DHS) maritime security components in any fiscal year, including drugs abandoned at sea, to the  average amount removed or abandoned in the immediately preceding 5 fiscal years. by all DHS maritime security components, but DHS only provided data on removals by the U.S. Coast Guard.

DHS did not explain in its report why it only included data from the U.S. Coast Guard. DHS officials said  that the U.S. Coast Guard is the primary DHS component involved in this activity and was the only  component that provided data for this metric, but this was not noted in the report.

According to U.S. Coast Guard officials, some of the data for fiscal 2013 was misreported. Specifically, the  quantity removed for methamphetamine should be 0 (report shows 17.4) while the value should be 7.9  kilograms for heroin (report shows 0).

Description   This metric is a rate comparing the  amount of cocaine removed by the  Department of Homeland Security  (DHS) maritime security  components inside and outside the  maritime transit zone to the total  documented cocaine flow rate.  DHS used estimates of  noncommercial maritime cocaine  flow from the Consolidated  Counter Drug Database, which are  derived from intelligence reporting  and case data.

Differences between the National Defense Authorization Act for Fiscal Year 2017  and the Department of Homeland Security’s report   None identified.

Limitations   DHS identified: There is less robust intelligence on noncommercial maritime cocaine flow outside the transit  zone than inside the transit zone, so data for outside the transit zone are not considered reliable.

Precise cocaine flow estimates through a particular mode or domain can be difficult to obtain. In our prior  work, officials with the Office of National Drug Control Policy and other departments and agencies involved  in U.S. counternarcotics efforts told us that it is difficult to obtain precise estimates of cocaine flow because  of the difficulty in obtaining specific information about the production of cocaine and how it gets to the  United States. We have also previously reported that when confronted with threats to their activities,  drug-trafficking organizations use a variety of techniques to quickly change their modes of operation, thus  avoiding capture of their personnel and seizure of their illegal drugs. For example, when air interdiction  efforts have proven successful, traffickers have increased their use of maritime and overland transportation  routes.

Additional information   According to U.S. Coast Guard officials, DHS officials hold quarterly inter-agency meetings to review the  reliability of performance data related to cocaine interdiction performance.

Air and Marine Operations (AMO) is a federal law enforcement agency within CBP that  interdicts unlawful people and cargo approaching U.S. borders, investigates criminal networks,  and provides domain awareness in the air and maritime environments, among other things.

The metrics in this domain measure AMO’s flight hours, individuals detected, and  apprehensions, among other things. DHS included 7 of 8 metrics within this domain called for in  the NDAA in its fiscal year 2017 Border Security Metrics Report, as shown in table 7.

DHS reported that the “AMO actionable intelligence” metric was under review and estimated  that the department would provide information on this metric in its 2019 annual report to  Congress.

Description   This metric is a rate comparing the number of flight hour requirements to the number of flight hours flown by Air  and Marine Operations (AMO) in the land domain.

Limitations   Department of Homeland Security (DHS) identified: Data prior to fiscal year 2016 were unavailable. According  to AMO officials, this is because AMO did not collect these data prior to fiscal year 2016, or because older data  were not comparable.

We identified: DHS used the terms “funded flight hours,” “unfunded flight hours,” and “unconstrained flight  hours” in the report without clearly defining them. AMO officials stated that a definition of these terms will be  included in the next report.

Additional information   AMO officials said they have taken steps to improve how they track flight hour data, such as by adding new  data fields to AMO’s system and providing training to staff.

Description   This metric is a rate comparing the number of funded flight hours appropriated to Air and Marine Operations  (AMO) to the number of actual flight hours flown.

Description   This metric is a rate comparing the  number of aviation missions flown  by Air and Marine Operations  (AMO) to the number of aviation  missions cancelled by AMO due to  maintenance, operations, or other  causes. the number of missions cancelled due to causes within AMO control, such as maintenance, personnel, and  asset availability. However, the Department of Homeland Security (DHS) used the total number of mission  requests, which also includes the number of missions flown in addition to the number of missions cancelled for  reasons within AMO control.

Limitations   DHS identified: Data prior to fiscal year 2016 were unavailable. According to AMO officials, this is because  AMO did not collect these data prior to fiscal year 2016, or because older data were not comparable.

Description   This metric is a rate comparing the number of missions cancelled by Air and Marine Operations (AMO) due to  weather compared to the total planned missions.

Description   This metric is a count of the number of individuals detected by Air and Marine Operations (AMO) through the  use of unmanned aerial systems and manned aircraft.

Limitations   Department of Homeland Security (DHS) identified:

Data prior to fiscal year 2016 were unavailable. According to AMO officials, this is because AMO did not  collect these data prior to fiscal year 2016, or because older data were not comparable.

DHS data on detections from manned aircraft were limited to those that led to apprehensions and arrests,  and data from unmanned aircraft were limited to the number of Vehicle and Dismount Exploitation Radar  (VADER) detections. AMO did not track data from all sensors on unmanned and manned aircraft, and  considers this metric to be a work in progress.

We identified: In February 2017 we reported that some mission data (such as asset assists) for unmanned  aerial systems were collected inconsistently across operation locations, which could affect the accuracy of the  counts provided. We recommended that U.S. Customs and Border Protection—of which AMO is a  component—update and maintain guidance for recording mission information in its data collection system, and  provide training to users of the system. DHS completed implementation of these recommendations in July  2018. Although the recommendations have been implemented, this limitation is relevant because the data  presented (for fiscal year 2016) were collected prior to their implementation.

Additional information and planned actions by the Department of Homeland  Security   DHS expects to provide more comprehensive data for this metric in the next annual report. AMO officials said  they have taken steps to improve how they track flight hour data, such as by adding new data fields to AMO’s  system and providing training to staff.

Description   This metric is a count of the  number of apprehensions assisted  by Air and Marine Operations  (AMO) through the use of  unmanned aerial systems and  manned aircraft.

Differences between the National Defense Authorization Act for Fiscal Year 2017  and the Department of Homeland Security’s report   None.

We identified: In February 2017 we reported that some mission data (such as asset assists) for unmanned  aerial systems were collected inconsistently across operation locations, which could affect the accuracy of the  counts provided. We recommended that U.S. Customs and Border Protection—of which AMO is a  component—update and maintain guidance for recording mission information in its data collection system, and  provide training to users of the system. DHS completed implementation of these recommendations in July  2018.

Although the recommendations have been implemented, this limitation is relevant because the data presented  (for fiscal year 2016) were collected prior to their implementation.

Additional information   In addition to the number of apprehensions assisted, DHS also provided the number of enforcement flight  hours used for the assists. AMO officials said they have taken steps to improve how they track flight hour data,  such as by adding new data fields to AMO’s system and providing training to staff.

Description   This metric is a count of the number and quantity of illicit drug seizures assisted by Air and Marine Operations  (AMO) through the use of unmanned aerial systems and manned aircraft.

Limitations   Department of Homeland Security (DHS) identified: Data prior to fiscal year 2016 were unavailable. According  to AMO officials, this is because AMO did not collect these data prior to fiscal year 2016, or because older data  were not comparable.

We identified: In February 2017 we reported that some mission data (such as asset assists) for unmanned  aerial systems were collected inconsistently across operation locations, which could affect the accuracy of the  counts provided. We recommended that U.S. Customs and Border Protection—of which AMO is a  component—update and maintain guidance for recording mission information in its data collection system, and  providing training to users of the system. DHS completed implementation of these recommendations in July  2018. Although the recommendations have been implemented, this limitation is relevant because the data  presented (for fiscal year 2016) were collected prior to their implementation.

Additional information   In addition to the drug seizures assisted (in pounds), DHS also provided the number of enforcement flight  hours used for the assists. AMO officials said they have taken steps to improve how they track flight hour data,  such as by adding new data fields to AMO’s system and providing training to staff.

Based on findings from previous reviews of border security programs and  efforts, we have recommended other metrics that the Department of  Homeland Security (DHS) could use to help measure the effectiveness of  border security. The tables that follow provide information about these  recommended metrics in each of the four domains listed in the National  Defense Authorization Act for Fiscal Year 2017—between ports of entry,  at ports of entry, in the maritime border domain, and the air and marine  security in the land domain. The recommendations listed in the tables  below remain open; however, implementing them would provide DHS with  additional indicators and metrics that could provide important insights into  the state of border security.

Appendix III: Comments from the Department of Homeland Security

Appendix IV: GAO Contact and Staff Acknowledgments

In addition to the contact named above, Taylor Matheson (Assistant  Director), David Alexander, Kelsey Burdick, Lilia Chaidez, Kathleen  Donovan, Michele Fejfar, Sally Gilley, Christopher Hatscher, Eric  Hauswirth, Mikaela Meyer, Sasan J. “Jon” Najmi, Kevin Reeves, and Jeff  Tessin made key contributions to this report.

The Department of Homeland Security is required by law to report annually on 43 specific measures of border security effectiveness.

DHS reported on 35 of those metrics in its 2017 report and generally used data and methods that should produce reliable results.

However the report disclosed some, but not all, of the data's limitations. For example, the report didn't indicate that different locations may have recorded some border-related data differently. Complete information about the data would provide context for Congress and the public to fully and appropriately understand the report.

We recommended ways to help DHS improve report quality.