2020 Census: Bureau Generally Followed Its Plan for In-Field Address Canvassing

The decennial census is a costly and complex undertaking and its success depends largely on the Bureau's ability to locate every person residing in the United States. To accomplish this monumental task, the Bureau must maintain accurate address and map information for every person's residence. If this information is inaccurate, people can be missed, counted more than once, or included in the wrong location. To help control costs and to improve accuracy, the Bureau used new procedures to build its address list for 2020.

GAO was asked to review how the in-field address canvassing operation performed. This report (1) determines the extent to which the Bureau followed its plans and schedule for in-field address canvassing, and (2) identifies the successes and challenges that occurred during 2020 Census In-Field Address Canvassing that have potential implications for future operations.

To address these objectives, GAO reviewed key documents including the 2020 Census operational plan that discussed the goals and objectives for the operation. GAO observed in-field address canvassing across the country at 18 area census offices, including a mix of rural and urban locations. GAO also interviewed field supervisors, listers, and office management to discuss the operation's successes and challenges.

GAO provided a draft of this report to the Bureau. The Bureau provided technical comments, which were incorporated as appropriate.

The Census Bureau (Bureau) completed in-field address canvassing as scheduled on October 11, 2019, despite nationwide hiring shortfalls. The Bureau credits this success to better-than-expected productivity—the actual hourly productivity rate for the operation was 19.8 addresses versus the anticipated rate of 15.8 addresses. The total workload included more than 50 million addresses.

GAO observations of in-field address canvassing found that a majority of field staff (listers) generally followed procedures, but there were a number of exceptions. For example, 14 of 59 listers we observed did not consistently knock on every door as required to confirm the address and ask about “hidden” housing units. Not knocking on doors or asking about hidden housing units represents missed opportunities to potentially add missing addresses to the Bureau's address file. GAO communicated to Bureau officials that listers were not following procedures and they sent out a nationwide reminder for listers to do so.

The Bureau credits efficiency gains to new systems for assigning work and a new reporting mechanism for collecting timecards, but experienced delays in hiring for address canvassing. Though address canvassing productivity was higher than expected, in some parts of the country the operation was at risk of falling behind because of a shortage of listers. The Bureau told GAO that it filled the gap with listers who lived well outside of the area in which they were supposed to work—in some cases from a different state. The Bureau is taking actions to address hiring problems for later operations, including nonresponse follow-up, when the Bureau intends to hire between 320,000 to 500,000 enumerators to follow up with households that did not initially respond to the census. Those actions include increasing wage rates in 73 percent of the counties nationwide.

The federal government’s constitutionally-mandated efforts to undertake a  decennial census have begun. On August 4, 2019, the U.S. Census  Bureau (Bureau) launched its in-field address canvassing operation to  update and verify its master address file. The decennial census is a costly  and complex undertaking and its success depends largely on the  Bureau’s ability to locate every person residing in the United States. To  accomplish this monumental task, the Bureau must maintain accurate  address and map information for every person’s residence. If this  information is inaccurate, people can be missed, counted more than  once, or included in the wrong location.

To help control costs and to improve accuracy, the Bureau used new  procedures to build its address list for 2020. Over the past few years, the  Bureau tested the new procedures several times to help minimize risks  and ensure the new approach would function as planned. In February  2017, we added the 2020 Census to our high-risk list because operational  and other issues are threatening the Bureau’s ability to deliver a cost- effective enumeration.

You asked us to review how the 2020 Census Address Canvassing  operation performed. This report (1) determines the extent to which the  Bureau followed its operational plans and schedule for 2020 In-Field  Address Canvassing Operation, and (2) identifies the successes and  challenges that occurred during 2020 Census In-Field Address  Canvassing that have potential implications for future operations.

To accomplish these objectives, we reviewed key documents including  the 2020 Census Operational Plan that discussed the goals and  objectives for the operation, the address canvassing study plan, quality  control plans, as well as training manuals and e-training modules and  other related documents for address canvassing. To obtain a first-hand  perspective of the conduct of in-field address canvassing we visited 18 of  the 39 Area Census Offices (see appendix II). These sites were selected  based on several factors, including rural-urban mix, participation in the  2018 End-to-End Test, and whether a site was scheduled to start the  operation earlier than others.

At each location, we interviewed Bureau staff, including census field  supervisors, address listers, and office management, to discuss what  went well and what challenges they faced during in-field address  canvassing. At each location, we interviewed and observed staff working  on the days of our visits. We observed production listers conduct address  canvassing and at several locations we observed quality control listers  perform quality control of addresses that had been canvassed. In  addition, we used the training manuals to determine whether listers  collected address information as prescribed by the Bureau. In total we  conducted 86 interviews with Bureau staff.

We conducted 64 in-field observations (59 listers, and five quality control  listers) using a data collection instrument to document our observations.  We also interviewed 22 field supervisors about what went well and what  challenges they faced during address canvassing. These observations  are not generalizable. We also interviewed Bureau headquarters officials  to discuss the use of management reports for monitoring and overseeing  the operation.

We reviewed workload estimates, address lister productivity rates, and  hiring information for the operation, including how many addresses the  Bureau expected to canvass per hour and how many people the Bureau  needed to hire. To assess the reliability of these data, we reviewed  available documentation and interviewed Bureau officials. We found the  data to be sufficiently reliable for the purposes of our reporting objectives.  We also met periodically with Bureau headquarters staff to discuss  progress of the operation.

We conducted this performance audit from July 2019 to February 2020 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

The Bureau’s address canvassing operation updates its address list and  maps, which are the foundation of the decennial census. An accurate  address list both identifies all living quarters that are to receive a notice by  mail to respond to the census, and serves as the control mechanism for  following up with households that fail to respond to the initial request.  Precise maps are critical for counting the population in the proper  locations—the basis of congressional apportionment and redistricting.

Our prior work has shown that developing an accurate address list is  challenging—in part because people can reside in unconventional  dwellings, such as converted garages, basements, and other forms of  “hidden” housing. For example, as shown in figure 1, what appears to be  a single-family house could contain an apartment, as suggested by its two  doorbells.

During address canvassing, the Bureau verifies that its master address  file and maps are accurate to ensure the tabulation for all housing units  and group quarters is correct. For the 2010 Census, the address  canvassing operation mobilized almost 150,000 field workers to canvass  almost every street in the United States and Puerto Rico to update the  Bureau’s address list and map data—and in 2012 reported the cost at  nearly $450 million. The cost of going door to door in 2010, along with the  emerging availability of imagery data, led the Bureau to explore an  approach for 2020 address canvassing that would allow for fewer boots  on the ground.

To reduce costs for the 2020 Census, the Bureau took a new approach  and some address canvassing work was completed in-office. The  Bureau compared current satellite imagery to the contents of its master  address file to determine if areas had housing changes, such as new  residential developments or repurposed structures. If the satellite imagery  and the master address file matched, then the Bureau considered those  areas to be resolved or stable and did not canvass them in-field.

These areas that were unresolved by the in-office review were sent to in- field address canvassing. Field staff called listers used laptop computers  to compare what they saw on the ground to the address list and maps.  Listers confirmed, added, and deleted addresses or moved addresses to  their correct map positions. The listers were trained to speak with a  knowledgeable resident at each housing unit to confirm or update  address data, ask about additional units, confirm the housing unit location  on the map (known as the map spot), and collect a map spot either using  global positioning systems (GPS) or manually. If no one was available,  listers were to use house numbers and street signs to verify the address  data. The data were then transmitted electronically to the Bureau.

The Bureau Completed In-Field Address Canvassing on Schedule and under Budget, but Listers Did Not Always Follow Procedures

The Bureau completed in-field address canvassing on time despite  nationwide hiring shortfalls. The Bureau credits this success to better- than-expected productivity. The Bureau conducted “in-field” address  canvassing for approximately 35 percent of the housing units  (approximately 50 million housing units) across the country (see fig. 2).  The Bureau had already determined “in-office” that the other 65 percent  of addresses (approximately 93 million housing units) were part of stable  blocks.

The Bureau began the in-field address canvassing operation at seven of  its 39 Area Census Offices on August 4, 2019, and then rolled out the  operation to the remaining 32 offices on August 18, 2019. It conducted  this phased approach to ensure all operations and systems worked  together before commencing the operation nationwide. The total in-field  address listing workload was more than 50 million addresses from the  Bureau’s address file. Bureau officials reported that listers were generally  more productive than expected, thus allowing the Bureau to complete the  operation as scheduled on October 11, 2019 (see fig. 3).

The actual hourly productivity rate for the operation was 19.8 addresses  versus the anticipated rate of 15.8 addresses. According to Bureau  officials, listers were more productive due to efficiency gains from the  Bureau’s new approach, including an automated time and attendance  system, the use of computer laptops to collect census data, and a new  operational control system that was used to electronically optimize  assignments and transmit work to listers. Bureau officials stated that the  high productivity also helped the operation come in under budget. The  operation’s cost was $118.6 million—while the anticipated cost was $185  million—a reduction of 36 percent.

The Bureau Missed Potential Opportunities to Improve the Address List When Listers Did Not Follow Procedures

For in-field address canvassing, listers received online training, which  detailed the procedures they were to follow, such as:  comparing the housing units they see on the ground to the housing  units on the address list,  knocking on all doors so they could speak with a resident to confirm  the address (even if the address is visible on the mailbox or house)  and to confirm that there are no other living quarters such as a  basement apartment,  looking for hidden housing units, and  confirming the location of the housing unit on a map with GPS  coordinates collected on the doorstep.

In our observations of in-field address canvassing, the majority of listers  generally followed these procedures. However, some listers we observed  did not always follow procedures. For example,

Ten out of 59 listers did not work ground to book (i.e., compare what  they saw on the ground to what was on their list).

Nine out of 59 listers did not walk up to the doorstep to collect the  GPS coordinate. Specifically, we observed listers use mailboxes to  confirm address information and collect the GPS coordinates from the  mailbox. Following proper procedures is important because getting a  GPS reading from the doorstep of every address contributes to the  accuracy of the address file.

Fourteen of 59 listers did not consistently knock on every door as  required to confirm the address and ask about “hidden” housing units.

Seventeen of 59 listers did not always look for or ask about “hidden”  housing units. Not knocking on doors or asking about hidden housing  units represents missed opportunities to potentially add missing  addresses to the Bureau’s master address file.

Further, not all listers we observed provided the required confidentiality  notices to occupants. Seven listers we observed did not provide  confidentiality notices. Occupants may be more willing to provide their  information if they know their responses will not be shared. We  communicated the information regarding our observations to the Bureau,  and on August 26, 2019, the Bureau instructed its field offices to remind  listers of the appropriate procedures.

According to Bureau officials, some amount of temporary staff deviates  from following procedures with every decennial census. As such, to  control for this, the Bureau implemented a Quality Control (QC)  component for in-field address canvassing that is designed to detect and  correct deficient production listers’ work. QC started on August 11, 2019,  and included a total workload of around 3.4 million addresses. For this  operation, an automated system selected the sample of addresses to  review; these addresses were assigned to QC listers. QC listers received  instructions to begin canvassing at a specified location, usually an  intersection, and to continue canvassing addresses until the system  identified the work unit as “complete” for QC purposes.

An address worked by a production lister was considered to have “failed”  QC if the QC lister recorded changes, or if the lister missed the address  and the QC lister found it. Depending on the size of the block, after a  predetermined number of addresses fail within a block, the system fails  the entire block. Once a block fails, the QC lister must recanvas all the  addresses in that block. Based on preliminary results, Bureau officials  estimate that 4.3 percent, or about 2.2 million addresses, failed.  According to Bureau officials, while they did not have a predetermined  target for what was an acceptable range for the total number of  addresses that failed QC, they nevertheless are reasonably confident that  this was in an acceptable range for QC errors encountered during the  operation. They further stated that they could not compare 2020 QC  results to 2010 because the 2010 Address Canvassing Operation  canvassed 100 percent of the addresses in-field, while the 2020 In-field  Address Canvassing Operation only covered approximately 35 percent of  the addresses across the country.

Lister productivity for QC was also higher than expected. The Bureau  anticipated the QC productivity at 8.03 addresses per hour compared to  the actual rate of 14.05 addresses per hour. Higher-than-expected  productivity rates contributed to a reduction in costs and the actual cost of  QC production was $10.3 million versus the anticipated cost of $25.6  million, a savings of $15.3 million. Additionally, Bureau officials stated that  QC came in so far under budget because the use of laptops increased  efficiency and the actual QC workload was lower than the budget  estimate.

Planned Evaluations Will Ultimately Determine the Quality of the Operation

While the Bureau conducted real-time quality control follow-up of selected  blocks during address canvassing, it also has two studies underway that  will evaluate the re-engineered address canvassing approach, as well as  the in-field address canvassing operation. Similar studies conducted by  the Bureau in 2010 found that 95.7 percent of addresses were correctly  deleted and 83.6 percent of addresses were correctly added. Both studies  underway have a set of research questions designed to evaluate the  accuracy and effectiveness of address canvassing. For example the  Bureau seeks to answer questions such as:

What percentage of the housing units added during in-field address  canvassing were correctly added (and added-in-error)?

What percentage of the housing units identified as deleted or  duplicated by the listers during in-field address canvassing were  correctly deleted or duplicated (and deleted-in-error)?

Answering these and other questions contained in both studies will be  critical to determining the quality of the operation, as not all listers  followed procedures, which may have led to errors in the address file. It is  anticipated that the final report for the 2020 Census In-Field Address  Canvassing Operational Assessment study will be available September  2020, and the 2020 Census Evaluation: Reengineered Address  Canvassing study will be available March 2023.

The Bureau Had Successes and Challenges during In- Field Canvassing, Which Have Potential Implications for Future Operations

The Bureau Cited Successes with the Operation

In addition to completing in-field address canvassing on schedule and  under budget Bureau officials highlighted other successes from the  operation including:

Automated solutions for training staff. Bureau-developed training  materials that used a blended training approach including instructor- led, computer-based, and hands-on training. This is a change from  the 2010 paper-based and classroom-only training approach.

Efficiency gains from conducting reengineered field operations using:

New operational control systems, which were used to  electronically assign and transmit work to the listers.

New automated time and expense reporting (timecards) for  employees. In 2010, timecards were paper-based and the listers  had to meet with their supervisors to submit them.

Enhanced software application for validating and updating  addresses.

Implementation of rapid response to Hurricane Dorian, which  affected areas of the Southeastern United States, resulted in  minimal disruptions to the operation.

Additionally, the Bureau was able to resolve some unforeseen challenges  at the seven Area Census Offices that opened early. For example, the  Bureau identified issues with training login and new hires not being on the  training roster and rectified those issues before the operation expanded to  the rest of the country.

The Bureau Is Taking Steps to Address Challenges with Hiring and Onboarding Staff

The Bureau experienced delays in hiring for its early operations, raising  concerns about hiring for peak operations. The Bureau’s target was to  hire 40,300 listers by September 7, 2019, but as of September 9, 2019,  the Bureau had hired 31,151 listers. Though address canvassing  productivity was higher than expected, in some parts of the country the  operation was at risk of falling behind because of a shortage of listers.  The Bureau told us it filled the gap with listers who lived well outside of  the area in which they were supposed to work—in some cases from a  different state. This strategy allowed the Bureau to complete the  operation on schedule; however, though the operation as a whole was  under budget, the Bureau incurred unplanned costs for travel (airfare,  personal mileage rates, rental cars, hotel stays, and per diem).

As we previously reported, these hiring problems are an early warning for  what may occur later in the census during nonresponse follow-up, when  the Bureau intends to hire between 320,000 to 500,000 enumerators to  follow up with households that did not initially respond to the census. The  Bureau said the hiring issues were caused by delays in processing  background checks and greater-than-expected attrition. According to the  Bureau, these delays arose, in part, due to early shortages of staff to  review background checks and because a significant number of  applicants did not completely or accurately fill out related forms. In  February 2019, the Bureau began to bring on about 130 temporary staff  to review forms for accuracy and completeness prior to submission for  investigation and to help investigators conduct the pre-employment  background checks.

Those delays in turn contributed to subsequent challenges in onboarding  listers for address canvassing. For example, according to Bureau officials,  the delays in early hiring for Area Census Office staff meant some offices  did not have enough clerks in place to process paperwork for listers or  make reminder phone calls to hire and onboard listers.

Regarding attrition, more listers quit than expected at two points in the  hiring process:

Fingerprinting: The Bureau expected about 15 percent of applicants  would leave the hiring process after being selected and before  submitting fingerprints. However, the attrition rate was closer to 25  percent. Bureau officials told us they attributed this to selected  applicants, in some cases, having to travel long distances to be  fingerprinted.

Training: The Bureau found that fewer selected and cleared  applicants attended training than anticipated. Bureau officials  attributed this to fewer clerks being available to call trainees with  reminders to attend training due to delays in clerks receiving their own  background checks.

Bureau officials also attributed some of this attrition to the 60-day period  between the selection of applicants and their training. This new time  frame was put in place for the 2020 Census to provide adequate time for  adjudication of background checks.

The Bureau has begun to address these challenges by adapting its hiring  and onboarding processes for peak operations, such as nonresponse  follow-up, which is to begin May 2020. For example, the Bureau:  Increased the number of fingerprinting locations and machines.  According to Bureau officials, it added 133 additional sites and 300  additional machines, bringing the total number of vendor sites for  fingerprinting to 829.

Staffed Area Census Offices to help newly-selected applicants for  positions complete their forms and initiate the background check  process.

Hired additional staff to help clear background checks. The Bureau  hired 200 staff at the National Processing Center and an additional  150 at the Regional Census Centers.

Changed the recruiting goals due to the attrition experienced during  address canvassing. The recruiting goal has increased from 2.3  million to 2.7 million to ensure it has a large enough applicant pool.  This increases the ratio of recruited applicants to positions from 5:1 to  6:1.

Completed a wage rate study and increased wages in 73 percent of  counties by an average of $1.50 per hour for enumerators.

Developed an email campaign to maintain contact with individuals in  the recruiting pool.

Decreased the types, and therefore the number, of positions that  required a full background check.

Included additional training for replacement hires in the training  schedules. A make-up session was added to the nonresponse follow- up training schedule, May 14-19, 2020.

If effectively implemented, these steps hold promise for helping to  address the hiring issues.

The Bureau Experienced Challenges with Management’s Use of Information

To effectively manage address canvassing, the Bureau provides data- driven tools for the census field supervisors to manage listers, including  system alerts that identify issues that require the supervisor to follow up  with a lister. Operational issues such as listers not working assigned  hours or falling behind schedule need to be resolved quickly because of  the tight time frames of the address canvassing and subsequent  operations. For the address canvassing operation, the system generated  codes that covered a variety of operational issues such as unusually high  or low productivity (which may be a sign of fraud or failure to follow  procedures) and administrative issues such as compliance with overtime  and completion of expense reports and time cards.

During the operation, more than 621,000 alerts were sent to census field  supervisors. Each alert requires the supervisor to take action and then  record how the alert was resolved. To assist supervisors, these alerts  need to be reliable and properly used. However, nine out of 22 census  field supervisors we spoke to indicated the alerts were not always useful.  For example, almost 40 percent of those alerts were related to no  progress being made on a block. This was due in part to listers opening  all of the blocks they were assigned on their laptops in order to manage  their workload, triggering the system that work had begun on all assigned  blocks when in fact the lister was only working one block. We first heard  about this issue from field supervisors in late August. Census field  supervisors we spoke to indicated that these alerts took an inordinate  amount of time to resolve, in part because almost every lister would open  every block to plan his or her day.

We alerted Bureau officials in headquarters, and they notified area  census offices to remind supervisors to instruct listers not to open all of  their blocks at once. After the notification was sent out, Bureau officials  reported that the number of alerts due to blocks not being worked  declined. Bureau officials further stated that this issue would not impact  nonresponse follow-up because enumerators do not receive multiple  assignments, but instead receive, work, and transmit only one  assignment of housing units for follow-up a day.

Another challenge faced by census field supervisors was providing  feedback to listers on why addresses failed quality control. Four of 22  census field supervisors we spoke with were not aware that they had  access to the reasons why addresses on a block failed quality control.  Knowing where to find this information would have allowed census field  supervisors to communicate this information to listers, thus improving  lister performance as well as the accuracy of the data collected. We  shared this information on some census field supervisor’s lack of  awareness with the Bureau and on August 26, 2019, the Bureau notified  its field offices to remind supervisors that detailed information on why  addresses failed quality control was available on their laptops.

For nonresponse follow-up, Bureau officials told us QC information about  any enumerator with a specified number of failed cases will be sent  directly to the Regional Census Center rather than the census field  supervisor. The Regional Census Center will decide whether the  enumerator should continue working and, if so, what corrective action to  take, such as retraining. However, if it is determined that an enumerator  falsified data, then the enumerator would not be given new assignments  and all of his or her work would then be reinterviewed.

We provided a draft of this report to the Secretary of Commerce. In its  written comments, reproduced in appendix I, the Bureau noted that our  report made no formal recommendations and that we highlighted several  successes of the in-field address canvassing operation. The Bureau also  described several claims of cost savings and efficiency gains which it  attributed to various address list-building activities. While we have  previously reported on the Bureau’s 2020 address list-building efforts, we  have not audited claims made in the Bureau’s response or elsewhere  regarding potential cost savings from innovations for the 2020 Census.  The Bureau also provided us with technical comments, which we  incorporated as appropriate.

We are sending copies of this report to the Secretary of Commerce, the  Under Secretary of Economic Affairs, the Director of the U.S. Census  Bureau, and interested congressional committees. The report also will be  available at no charge on GAO’s website at http://www.gao.gov.

If you or your staff have any questions about this report please contact  me at (202) 512-3236 or mihmj@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff that made major contributions to this report  are listed in appendix III.

Appendix I: Comments from the Department of Commerce

Appendix II: Area Census Offices Responsible for Locations Visited in This Review

Appendix III: GAO Contact and Staff Acknowledgments

J. Christopher Mihm, (202) 512-3236 or mihmj@gao.gov  In addition to the contact named above, Lisa Pearson, Assistant Director;  Timothy Wexler, Analyst-in-Charge; Margaret Fisher; Robert Gebhart;  Richard Hung; Cynthia Saunders; Anna Sorrentino; Kate Sharkey; Dylan  Stagner; Jon Ticehurst; Peter Verchinski; and Alicia White made key  contributions to this report.

The Census Bureau began its decennial count of the U.S. population in January 2020. Early operations were generally on schedule—with field staff successfully completing address canvassing for more than 50 million addresses. Ensuring accurate addresses is vital or people may be missed, counted more than once, or included in the wrong location.

But, a shortage of staff put the operation at risk of falling behind. The Bureau is taking steps to recruit enough workers for later operations, including hiring up to 500,000 people to follow up with households that do not initially respond to the census. The 2020 Census is on our High Risk list.


Ready Aircrew Program: Air Force Actions to Address Congressionally Mandated Study on Combat Aircrew Proficiency

In September 2016, GAO reported that annual combat aircrew training requirements delineated in the Air Force's Ready Aircrew Program might not address pilot training needs, and that the Air Force did not systematically evaluate the effectiveness of its training. As a result, Congress included a provision in Section 351 of the NDAA for Fiscal Year 2017 for the Air Force to commission an independent review of its Ready Aircrew Program, report on actions it planned to take in response to any recommendations, and provide an estimate of any resources required.

Section 351 also included a provision for GAO to assess the Air Force report. This report examines whether (1) the independent review conducted by the RAND Corporation addressed statutory requirements to review and assess the Ready Aircrew Program, and (2) the Air Force has reported on completed or planned actions to implement the RAND report recommendations.

To address these objectives, GAO reviewed the RAND and Air Force reports on the Ready Aircrew Program, assessed the study against generally accepted research standards, and interviewed officials at RAND, Air Force Headquarters, and the Air Combat Command.

A July 2018 RAND report—commissioned by the Air Force—addressed the statutory requirements of the National Defense Authorization Act (NDAA) for Fiscal Year 2017 to review and assess the Air Force's Ready Aircrew Program and make recommendations for ways to improve it. The Ready Aircrew Program establishes minimum annual training requirements for combat aircrew. RAND's report, entitled Independent Review and Assessment of the Air Force Ready Aircrew Program , made nine recommendations to improve its management:

1. Leverage internal expertise to implement measures for proficiency.

2. Invest resources to design data collection and storage solutions that facilitate analysis and readiness reporting.

3. Document the Ready Aircrew Program Tasking Memorandum development process in Air Force instruction supplements and ensure that the process incorporates squadron-level input and feedback.

4. Establish a more explicit and formal link between proficiency and Ready Aircrew Program requirements.

5. Document training quality to support requests for training resources.

6. Identify the conditions under which Ready Aircrew Program requirements, including mission types, can be accomplished.

7. Consider changing how Ready Aircrew Program requirements affect the Flying Hour Program.

8. Invest in data systems to correct data collection and assess deficiencies.

9. Leverage the Air Force Research Laboratory's performance data work and invest in added analysis to produce enterprise-wide proficiency metrics.

The nine RAND recommendations aligned with two GAO recommendations made in 2016 to comprehensively assess the assumptions underlying the annual aircrew training requirements and develop a process to collect data to assess the effectiveness of the training.

The Air Force's August 2018 one-page report to Congress included three broad actions in response to RAND's recommendations. The Air Force planned to

build training matrices to help commanders assess their units' effectiveness,

establish common data architecture through the Air Force's Chief Data Officer–led effort, and

evaluate aspects of the Ready Aircrew Program to increase lethality and improve readiness as the Air Force shifts to executing the mandates of the 2018 National Defense Strategy.

The Air Force, however, did not explain how these three efforts would specifically address the nine recommendations. Air Force officials said that, though they generally agreed with RAND's recommendations, the Air Force lacked the resources to fully implement them beyond actions that were underway prior to the RAND report, and considers all recommendations as “closed.” In part due to its not fully implementing RAND's recommendations, the Air Force has not fully addressed GAO's two recommendations. Fully implementing GAO's recommendations would better position the Air Force to ensure its aircrews receive effective training to achieve a range of missions.

GAO is not making any new recommendations in this report but restates the need to fully implement GAO's two 2016 recommendations. In comments on a draft of this report, DOD did not concur with the two 2016 recommendations. GAO maintains that the Air Force should implement both recommendations. recommendation. GAO will consider these in its annual recommendations follow-up.

The Air Force, through its Ready Aircrew Program, requires its combat  pilots to complete a defined number of live training events, or “sorties,”  and virtual simulator training events, or “simulator missions,” each year to  maintain mission readiness. As part of this training, the Ready Aircrew  Program establishes the minimum numbers and types of sorties and  simulator missions that aircrews of a particular combat aircraft must  complete during the annual training cycle to sustain different levels of  mission readiness. For example, in fiscal year 2016, experienced  aircrews for the F-15E were required to complete a minimum of 96 annual  live sorties (eight per month) to be considered combat mission ready, of  which 49 were to be flown to fulfill primary mission requirements.  Experienced aircrews for the F-15E were also required to complete 36  annual simulator missions (three simulator missions per month) to be  considered combat mission ready.

In addition to quantifying the minimum number of annual flying hours  required to train combat aircraft squadrons, the Ready Aircrew Program is  an important input for establishing the resources necessary to make the  required training available to pilots and units. For example, the Air Force  uses the Ready Aircrew Program sortie requirements for its combat  aircraft, along with other inputs, to calculate the Air Force’s annual flying  hour program and includes this cost in the Air Force’s annual budget  submission to Congress. In fiscal year 2018, for example, the Air Force  executed 826,606 flying hours at a cost of about $5.3 billion.

Section 351 of the National Defense Authorization Act (NDAA) for Fiscal  Year 2017 included a provision for the Air Force to commission an  independent study to (1) review and assess the assumptions underlying  its annual continuation training requirements, (2) review and assess the  effectiveness of the Ready Aircrew Program in managing aircrew training  requirements, and (3) make recommendations for the improved  management of these training requirements. Section 351 also required  the Air Force to report on actions it plans to take in response to the  independent study’s recommendations and to estimate the resources  required to implement the recommendations. In accordance with Section  351, the Air Force contracted with the RAND Corporation in 2017 to  independently study the Air Force’s Ready Aircrew Program at a cost of  slightly over $1 million. The RAND Corporation provided the results of its  study in a July 2018 report to the Air Force entitled Independent Review  and Assessment of the Air Force Ready Aircrew Program. Hereafter, we  refer to this report as the RAND report. In August 2018, the Air Force  released its report, which consisted of a one-page transmittal letter with  the RAND report attached as an enclosure. Hereafter, we refer to this  report as the Air Force report.

Section 351 also included a provision that we review the Air Force report.  Based on our review, this report (1) assesses whether the RAND report  addressed the three statutory requirements identified in Section 351 and  whether RAND followed generally accepted research standards in  performing its review and (2) describes the Air Force’s response and  planned actions to the RAND recommendations, as required by Section  351.

For objective one, we reviewed the RAND report to determine whether it  addressed the elements specified in Section 351 of the NDAA for Fiscal  Year 2017. As part of this review, we compared the RAND report findings,  recommendations, and the work described in the report to the NDAA- specified elements. Additionally, we discussed RAND’s work with RAND  officials, Air Force Headquarters Training and Readiness officials, and Air  Combat Command training officials. To understand the extent to which  the RAND report complied with generally accepted research standards,  we reviewed research literature and Department of Defense (DOD)  guidance and identified frequently occurring, generally accepted research  standards that are relevant for defense studies. We selected standards  related to the design, execution, and presentation of the study from  among the standards we identified as generally accepted for research.  We assessed the RAND study against these three major elements for  generally accepted research standards, as based on prior GAO work. To  make our assessments, we considered the individual components of  those elements, as appropriate for the RAND study. We discussed these  standards with RAND officials, who agreed that they were generally  consistent with their own quality standards for research and were  applicable to this study. See appendix I for a more detailed description of  the generally accepted research standards we focused on in this review.

For objective two, we reviewed and compared the Air Force report and  briefing to us with the findings and recommendations in the RAND report  to determine whether the Air Force concurred with the RAND findings and  recommendations. We then discussed the Air Force report with officials  from the Air Force Headquarters Training and Readiness Directorate and  the Office of the Under Secretary of Defense for Personnel and  Readiness to determine completeness of the report vis-à-vis Section 351  requirements. On the basis of these discussions of RAND findings and  recommendations with Air Force officials, we reviewed any efforts that the  Air Force officials reported being considered during the course of our  review in response to the RAND recommendations. We discussed the  planned actions identified in the Air Force report in interviews with officials  from Air Force Headquarters, Air Combat Command, and the Office of the  Under Secretary of Defense for Personnel and Readiness. For both  objectives, we met with officials from the Office of the Under Secretary of  Defense for Personnel and Readiness, Air Force Headquarters, Air  Combat Command, and RAND to discuss the methodologies and results of the RAND report. We did not assess actions that the Air Force briefed  to us after the completion of our field work to determine whether these  actions were sufficient to fully address RAND findings and  recommendations. However, Air Force officials told us that any actions  the Air Force had taken had begun before RAND made its  recommendations.

We conducted this performance audit from September 2018 through  February 2020 in accordance with generally accepted government  auditing standards. Those standards require that we plan and perform the  audit to obtain sufficient, appropriate evidence to provide a reasonable  basis for our findings and conclusions based on our audit objectives. We  believe that the evidence obtained provides a reasonable basis for our  findings and conclusions based on our audit objectives.

Air Force pilots are required to complete various phases of training before  they are considered to be mission ready. To assess pilot candidates’  flying aptitude for both traditional (i.e., manned) and remotely-piloted  aircraft, the Air Force first requires these candidates to attend initial flight  training. Successful candidates for traditional aircraft then attend one of  two schools:

Euro-NATO Joint Jet Pilot Training, the graduates of which become

Specialized Undergraduate Pilot Training, the graduates of which  become fighter, bomber, airlift, or tanker aircraft pilots, depending on  pilot strengths and Air Force needs.

During or following this training, a pilot is assigned to a specific aircraft,  and the flight training program proceeds through three stages:  Initial Qualification Training. This stage of training qualifies a pilot for  basic flying duties associated with the type of aircraft (e.g., an F-16).  The pilot accomplishes this stage at a formal training unit before  moving to an assigned squadron. Graduates of initial qualification  training courses have basic aircraft qualification status.

Mission Qualification Training. This stage of training occurs once the  basic aircraft qualified pilot is at the assigned unit. The pilot  undergoes mission qualification training in the type of aircraft  assigned to qualify for the specific missions the unit is required to  perform.

Continuation Training. This stage of pilot training has two  components. Pilots participate in continuation training to (1) in some  instances upgrade their qualifications to fill certain positions, such as  flight lead, instructor, or forward air controller through specialized  continuation training; and (2) in all instances maintain proficiency and  improve their capabilities to perform their units’ assigned missions.

The Ready Aircrew Program establishes the minimum number of live  training events, or “sorties,” and virtual simulator training events, or  “simulator missions,” that aircrews of a particular combat aircraft must  complete during the annual training cycle to maintain mission readiness.  These sorties and simulator missions are aligned with the units’ primary  missions, for which the units must maintain “proficiency,” and secondary  missions, for which they must maintain “familiarity.” The Air Combat  Command, as lead command for the Ready Aircrew Program, with the  assistance of other major commands (including the Air Force Global  Strike Command, Pacific Air Forces, and U.S. Air Forces in Europe) and  associated subordinate organizations (i.e., air wings and squadron  commanders), develops tasking memorandums for the Ready Aircrew  Program that delineate and specify the annual continuation training  requirements for personnel assigned to each of the subordinate combat  units.

The RAND Report Addressed Statutory Requirements, Followed Generally Accepted Research Standards, and Identified Deficiencies

The RAND Report on the Ready Aircrew Program Addressed Section 351 Requirements by Assessing Training Requirements and Making Recommendations

On the basis of our analysis, we found that the RAND report addressed  each of the three statutory elements required by Congress in Section 351.  First, the RAND report addressed two statutory elements by reviewing  and assessing  the assumptions underlying annual continuation training requirements  for the Ready Aircrew Program and  the overall effectiveness of the Ready Aircrew Program in managing  aircrew training requirements.

These two statutory elements focus on issues raised in our prior report  recommendations, which we discuss in more detail later in this report.  Table 1 provides detailed information about these statutory elements, our  assessment of RAND’s findings, and RAND’s findings associated with  each element.

The RAND report addressed the third statutory element by making  recommendations for the improved management of training requirements.  Specifically, the RAND report made nine recommendations, listed in table  2.

The RAND Report on the Ready Aircrew Program Was Consistent with Generally Accepted Research Standards

The RAND report and its underlying analysis is consistent with generally  accepted research standards for design, execution, and presentation.  Table 3 summarizes our assessment of the extent to which the RAND  report conformed with these standards.

The RAND Report on the Ready Aircrew Program Identified Similar Deficiencies as Reported by Us in 2016

The RAND study and our previous audit work identified similar  deficiencies in the management and operation of the Ready Aircrew  Program. Specifically, in September 2016, we reported that the Air Force  had used the same underlying assumptions to establish its annual  training requirements in the Ready Aircrew Program from 2012 through  2016, which may not reflect current and emerging training needs. We  concluded that without fully reassessing the assumptions underlying its  training requirements, the Air Force could not be certain that its annual  training plans are aligned with its stated goals to ensure a full-spectrum- capable force that can successfully achieve missions across a broad  range of current and emerging threats. We recommended that the Air  Force comprehensively reassess the assumptions underlying its annual  training requirements—and make any appropriate adjustments in future  aircrew training plans to ensure that its forces can accomplish a full range  of missions.

Additionally, in our September 2016 report, we also reported that the Air  Force did not systematically evaluate the effectiveness of training  performed as part of the Ready Aircrew Program. We recommended that  the Air Force establish desired learning objectives and training support  elements needed to accomplish the training expectations identified by the  Ready Aircrew Program and develop a process to collect data to assess  the effectiveness of annual training against these features. We discuss  these recommendations and their status in more detail later in this report.

The Air Force Reported Some General Actions to Link Readiness to Training but Plans No Further Actions in Response to RAND Recommendations

Section 351 of the NDAA for Fiscal Year 2017 required the Air Force to  report on any actions it plans to take in response to RAND’s  recommendations and to estimate the resources required to implement  the recommendations. On August 30, 2018, the Air Force provided its  report—a one-page transmittal letter from the Secretary of the Air Force  with the RAND report incorporated as an enclosure—to congressional  committees in fulfillment of the Section 351 requirements. In its report, the  Air Force agreed with RAND that more investment is needed in data  collection because its current system does not lend itself to analysis that  could be used to gain efficiencies. The Air Force also stated that it is  addressing the RAND recommendations by working to link readiness to  Ready Aircrew Program training requirements. Specifically, the Air Force  stated that it was taking the following three actions: building training matrices to help commanders assess their units’ establishing common data architecture through the Air Force’s Chief  Data Officer–led effort, and evaluating aspects of the Ready Aircrew Program to increase lethality  and improve readiness as the Air Force shifts to executing the  mandates of the 2018 National Defense Strategy.

However, in its August 2018 report, the Air Force did not provide any  additional details to further describe or link these broad actions to the  RAND findings and recommendations. Therefore, the extent to which  these three actions are responsive to the RAND recommendations is  unclear.

In October 2019, upon completion of our audit work, Air Force officials  provided us with a briefing they described as a corrective action plan that  further elaborated on the Air Force’s position with respect to each of  RAND’s nine recommendations. Air Force officials conceded that the  actions described in that briefing and plan were already underway at the  time of the study and were not initiated in response to the study’s  recommendations. They stated that, though they generally agreed with  the recommendations, the Air Force lacked the manpower, resources,  and means to implement them. As such, the Air Force considers each of  the recommendations closed and plans no further actions. Further, the Air  Force has no plans for future follow-up on implementation of the RAND  recommendations. Accordingly, we did not further assess the actions  described by the Air Force in relation to the RAND study. Table 4  summarizes the Air Force’s position as provided in its corrective action  plan briefing.

As described in table 4, the Air Force concurred with three  recommendations, partially concurred with five, and did not concur with  one. The following summarizes the Air Force position by concurrence  category:  Concur: In concurring with RAND Recommendations 2 and 8—to  invest in data systems for the collection, access, and storage of data  to correct deficiencies in current systems and improve analysis and  readiness reporting—the Air Force stated that, before the  recommendations were made, it had made available $5.15 million in  fiscal year 2020 funding to develop an Aircrew Readiness Training  Management Module as an upgrade to the Air Force’s Aviation  Resource Management System. The Air Force expects that this  module will centralize management of the Air Force Ready Aircrew  Program Tasking Memorandum at the command level, transfer  aircrew training data whenever a member moves to a new station,  and improve capability to track the types of flight simulators used for  training. Further, because the Air Force does not plan to take  additional actions, it estimated no resources are required, beyond the  $5.15 million already funded, prior to RAND making these  recommendations.

In concurring with RAND Recommendation 5—to document training  quality to support requests for training resources—the Air Force  explained that it is documenting the quality of training prescribed in its  annual Air Force Ready Aircrew Program Tasking Memorandum at  the squadron level. However, this is not a change based on the RAND  recommendation and was being done prior to RAND making this  recommendation. The Air Force factors unit training,  accomplishments, and readiness inputs into the Air Force Ready  Aircrew Program Tasking Memorandum and overall flying hour  program. As the flying hour program is the basis for resource training  requests and the Air Force factors unit training and accomplishments  into its flying hour program, the Air Force explained that it plans to  take no additional actions based on this recommendation. Further,  because the Air Force does not plan to take additional actions, it  estimated no resources were required to implement this  recommendation.

Partially Concur: In partially concurring with RAND recommendations  1, 3, 4, 6, and 9, Air Force officials explained that the mechanisms  reflected in its comments are sufficient to address the intent of the  RAND recommendations even though these mechanisms predated  RAND’s recommendations. Consequently, according to the Air Force  officials who briefed us, the Air Force plans no additional actions  based on the recommendations, obviating the need to estimate  resources in the case of these five recommendations.

Nonconcur: In not concurring with RAND Recommendation 7—to  consider changing how Air Force Flying Hour Program requirements  affect the flying hour program—the Air Force explained that the flying  hour program determination is standard across the Total Air Force,  affecting more than combat aircrews alone, and that the centrality of  the flying hour program to readiness and combat capability cannot be  overemphasized and must be defendable and auditable. Further,  based on its not concurring with this recommendation, the Air Force  did not estimate resources for this recommendation.

Notwithstanding the RAND study, the actions taken by the Air Force may  not fully implement our prior recommendations as described previously.  Specifically, we recommended that the Air Force comprehensively  reassess the assumptions underlying its annual training requirements— including for example the total annual training requirements by aircraft,  the criteria for designating aircrews as experienced or inexperienced, and  the mix between live and simulator training—and make any appropriate  adjustments in future aircrew training plans to ensure that its forces can  accomplish a full range of missions. While RAND accomplished such an  analysis as part of its review (see table 1, items 1(a) through 1(e) above),  RAND concluded in its analysis that the Air Force did not have the  objective measures of proficiency needed to determine the minimum and  optimum number of sorties and that the combat air forces aviation  community lacks consensus on how to define and measure an aircrew  member’s proficiency. While the Air Force has defined proficiency in an  Air Force manual issued in September 2019, the Air Force has not  reassessed its assumptions underlying training requirements and made  appropriate adjustments to future training plans per our recommendation.

Further, we recommended that the Air Force establish desired learning  objectives and training support elements needed to accomplish the  training expectations in its annual Ready Aircrew Program tasking  memorandums, and develop a process to collect data to assess the  effectiveness of annual training. In commenting on RAND’s  recommendations 2 and 8, the Air Force stated that, before the  recommendations were made, it had made available $5.15 million in fiscal  year 2020 funding to develop an Aircrew Readiness Training  Management Module as an upgrade to the Air Force’s Aviation Resource  Management System. The Air Force’s effort to upgrade this system, while  not a result of the RAND recommendations, may meet the intent of our  recommendation. For example, the Air Force’s development of the  Aviation Resource Management System is expected to centralize  management of the Air Force’s Ready Aircrew Program training  requirements data at the command level, transfer aircrew training data  whenever a member moves to a new station, and improve capability to  track the types of flight simulators used for training. When fully  implemented, these improvements may ultimately allow the Air Force to  assess the effectiveness of annual training, as we recommended.  However, it is too early to tell as the actions were under development or  had just begun at the end of September 2019, and sufficient data to  evaluate the results have not been collected.

Fully implementing both of our recommendations would better position  the Air Force to ensure that its aircrews receive effective training to  achieve a range of missions for current and emerging threats.

We provided a draft of this report to DOD for review and comment. In its  written comments (reprinted in appendix II), DOD stated that it saw great  value in our discussion. DOD added that its position on the two  recommendations we made in our 2016 report—with which DOD did not  concur—is fundamentally unchanged. However, we continue to believe  the recommendations should be implemented by the Air Force, as  previously discussed in this report.

In its comments, DOD also stated that it is addressing training  infrastructure and aircrew proficiency through two initiatives. First, in  response to the 2018 National Defense Strategy, the department's Joint  Operational Infrastructure Plan is framing the modernization effort of  DOD-wide Operational Infrastructure. The Joint Operational Infrastructure  Plan specifically addresses areas such as Live Virtual Constructive and  aircrew training. Second, the department is pursuing efforts to align  training events with the range of current and evolving threats. According  to DOD, both efforts will address the underlying assumptions for aircrew  training and proficiency with reportable readiness metrics.

However, the Joint Operational Infrastructure Plan is a draft and not yet  officially issued, according to an official at the Office of the Under  Secretary of Defense for Personnel and Readiness. Therefore, details  needed to assess this plan are not yet available for us to consider in  determining whether the change will help to address the  recommendations we made in 2016. Regarding efforts the department is  pursuing to better align training events, the DOD comments did not  include sufficient details to allow us to state whether the change could be  helpful in addressing our recommendations. Nonetheless, to the extent  that DOD is successful in completing, issuing, and implementing its new  Joint Operational Infrastructure Plan, or takes further actions related to  our prior recommendations, we will consider them as we continue to  analyze DOD efforts to address our recommendations.

We are sending copies of this report to the appropriate congressional  committees, the Secretary of Defense, the Under Secretary of Defense  for Personnel and Readiness, and the Secretary of the Air Force. In  addition, the report is available at no charge on the GAO website at  http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-5431 or russellc@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who made major contributions to this report  are listed in appendix III.

Appendix I: GAO’s Criteria for Generally Accepted Research Standards

To determine whether the RAND report followed generally accepted  research standards, we chose the following criteria, which are based on a  review of research literature and Department of Defense (DOD) guidance,  from which we identified frequently occurring, generally accepted  research standards that are relevant for defense studies, including those  related to the presentation of results. These standards have been used in  a number of our prior reports, modified as appropriate for each situation.

For the purposes of this engagement, we assessed the RAND study  against three major elements, listed below, that fall under generally  acceptable research standards, as based on our prior work. These  generally accepted research standards are consistent with Office of  Management and Budget (OMB) Guidelines  and DOD guidance on  ensuring and maximizing the quality of information disseminated by  federal agencies to the public.

We discussed these standards with RAND officials, who agreed that they  were generally consistent with their own quality standards for research  and were applicable to this study. We determined that these standards  are still current and relevant for the purposes of this report, based on their  consistency with OMB and DOD guidance, discussions with RAND  officials, and consideration of prior GAO work applying generally  accepted research standards with the assistance of GAO’s Applied  Research and Methods Team. The standards include the following:  Design—Study is well designed. For the RAND study, we focused on the  following elements for design:

The study plan, scope, and objectives follow existing guidance.

Assumptions and constraints are reasonable and consistent.

Execution—Study is well executed. For the RAND study, we focused on  the following elements for execution:

The methodology is successfully executed.

Data used to support study and analyses are validated.

Presentation—Results are well presented. For the RAND study, we  focused on the following elements for presentation:

Timely, complete, accurate, concise, and relevant to stakeholders.

Presentation of results supports findings.

Appendix II: Comments from the Department of Defense

In addition to the contact named above, Beverly Schladt (Assistant  Director), John Strong (Analyst in Charge), John Beauchamp, Vincent  Buquicchio, Martin De Alteriis, and Lillian Moyano Yob made key  contributions to this report.

In 2016, we reported that Air Force combat aircrew training requirements—such as the flight and simulator hours required for proficiency—needed a comprehensive review of their effectiveness. Congress then required the Air Force to commission an independent study of training requirements and to report how it planned to respond to any report recommendations.

The Air Force generally agreed with the study’s 9 recommendations on training but did not explain how it would specifically address them in its 1-page response to Congress. The Air Force said it did not have the resources for additional efforts and considers these recommendations “closed.”