Information Technology: DOD Needs to Fully Implement Program for Piloting Open Source Software

Open source software is code that is released under a license which grants users the right to modify, share, and reuse the software. Making code available for reuse as open source can have major benefits such as decreasing costs and improving efficiencies. The National Defense Authorization Act for Fiscal Year 2018 required DOD to submit a plan to Congress for initiating the open source software pilot program established by OMB memorandum M-16-21. DOD submitted its plan to Congress in June 2018.

The act includes a provision for GAO to report on DOD's implementation of the open source software pilot program. GAO's objectives were to (1) assess the extent to which DOD has implemented the open source software pilot program and other related requirements established by OMB; and (2) describe the views of responsible DOD officials on the use of open source software to achieve efficiency, transparency, and innovation at the department. To address these objectives, GAO compared DOD's plan for implementing the program to OMB's memo. GAO also interviewed defense officials at 11 DOD components including military departments, and defense agencies on their views about the benefits and risks of making code available as open source software.

The Department of Defense (DOD) has not fully implemented an open source software pilot program and related Office of Management and Budget (OMB) requirements as mandated by the National Defense Authorization Act for Fiscal Year 2018. OMB memorandum M-16-21 calls for agencies to implement a pilot program, which it defines as (1) releasing at least 20 percent of new custom developed code as open source, and (2) establishing a metric for calculating program performance. However, DOD has not fully implemented the program and has not established the metric. The OMB memorandum also requires agencies to implement other supporting activities. These include issuing policy on government-wide use of code, conducting analyses of software solutions, securing data rights and inventory code, and facilitating the open source community. DOD has not implemented the policy requirement and has partially implemented the remaining three requirements.

Regarding the policy and analysis requirements, DOD plans to issue a policy and conduct analyses by the end of the 2019 calendar year. If the department effectively implements these intended steps consistent with OMB direction, DOD should be able to fully address these requirements.

For the requirement of securing data rights and inventorying code, DOD issued a memorandum that directs contracting officers to secure data rights and to identify all source code created after August 2016. However, DOD's components have not executed these activities nor has DOD identified a milestone for when they will be completed.

For the facilitating community requirement, DOD issued a memorandum that encourages conversations to foster communities and allow others to contribute knowledge, among other initiatives. However, DOD has not fully engaged in open development, established a release schedule, or fully documented its source code to facilitate use and adoption. To address these areas, DOD's Chief Information Officer plans to issue guidance but has not established a milestone for doing so.

Until DOD fully implements the pilot program and develops milestones for two of the four OMB requirements (secure data rights and inventory code, and facilitate community), it will not be positioned to satisfy the mandate established in the law.

DOD officials from 11 components expressed their opinions that an open source pilot program would potentially result in financial benefits and increased efficiency. However, there were disparate views on how to manage the cybersecurity risk of using open source software. Specifically, officials from three components noted that security concerns could result in the sporadic use of OSS, whereas eight officials stated that the potential cybersecurity risks were managable.

GAO is making four recommendations to ensure DOD implements the program and develops milestones for completing requirements in the OMB memo. DOD agreed with two but did not agree with one and partially agreed with another. As discussed in this report, GAO maintains that all recommendations are needed to satisfy the act.

The federal government spends billions of dollars on software each year.  A significant proportion of software used by the Government is comprised  of either preexisting Federal solutions or commercial solutions. These  solutions include, among others, open source software (OSS). OSS is  software developed from source code that is obtained under a license  that allows it to be modified, shared, and reused. When Federal  agencies are unable to identify an existing Federal or commercial  software solution that satisfies their specific needs, they may choose to  develop a custom software solution on their own or pay for its  development. However, when agencies procure custom-developed  source code they do not necessarily make their new code broadly  available as OSS for government-wide reuse. According to the Office of  Management and Budget (OMB), even when agencies are in a position to  make their source code available on a government-wide basis, they do  not make such code available to other agencies in a consistent manner.  These challenges may result in duplicative acquisitions for substantially  similar code and an inefficient use of taxpayer dollars.

On August 8, 2016, OMB issued memorandum M-16-21 that established  requirements for federal agencies to improve the way they buy, build, and  deliver software solutions through the use of OSS code. The  memorandum called for the agencies to implement an OSS pilot program  and establish other associated requirements to implement the pilot  program.

Enacted on December 12, 2017, the National Defense Authorization Act  for Fiscal Year 2018 mandated that the Secretary of Defense initiate the  OSS pilot program established by the OMB memorandum within 180  days (by June 10, 2018). Further, the act required the department to  report to Congress within 60 days (by February 10, 2018) with details of  its plan. The plan was to identify candidate software programs, selection  criteria, intellectual property and licensing issues, and any other matters  determined by the Secretary.

The act also included a provision that we report to Congress no later than  June 1, 2019, on DOD’s implementation of the OSS pilot program. Our  objectives were to: (1) assess the extent to which DOD has implemented  the OSS pilot program and other related requirements established by the  OMB memorandum; and (2) describe the views of responsible DOD  officials on the use of OSS.

To address the first objective, we selected six requirements from the  OMB memorandum titled the Federal Source Code Policy: Achieving  Efficiency, Transparency, and Innovation through Reusable and Open  Source software (M-16-21, Aug.8, 2019) that we determined were needed  to establish an OSS pilot program. Two requirements establish the pilot  program: (1) releasing at least 20 percent of newly custom-developed  code each year as OSS for the term of the pilot program, and (2)  developing a metric to gauge the performance of the pilot program. The  other four requirements support the implementation of the pilot program:  (1) issuing an OSS policy, (2) conducting an OSS analysis, (3) securing  data rights and inventorying custom code, and (4) facilitating the OSS  community. We met with officials from the Office of the DOD Chief  Information Officer (CIO) and the Defense Digital Service to discuss the  status of the department’s implementation of the OSS pilot program. To  determine the extent to which the pilot program had been implemented,  we evaluated DOD’s efforts to implement the six pilot program  requirements.

To address the second objective, we conducted structured interviews with  DOD officials from across the department who are responsible for the  development and management of OSS. These officials included  representatives from the Office of the Under Secretary of Defense for  Acquisition and Sustainment, Office of the DOD CIO, Offices of the Navy  and Marine Corps CIOs, Army Communications-Electronics Command,  the Defense Information Systems Agency, the Defense Advanced  Research Projects Agency, and Office of the Assistant Secretary of the  Air Force for Acquisition, Technology, and Logistics.

We obtained their views on the benefits and concerns regarding the  general use of OSS. We also obtained their views on the specific  requirements about the implementation of an OSS pilot program  contained in OMB’s memorandum. See appendix I for a more detailed  discussion of our objectives, scope, and methodology.

We conducted this performance audit from August 2018 to September  2019 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.

OSS is software distributed under a license that provides broad rights to  use, modify, and redistribute the original source code. Open source  licenses impose certain obligations on users who exercise these rights.  Specific obligations vary among the many different open source licenses.  Common obligations include making the source code available,  publishing a copyright notice, or giving any recipient of the program a  copy of the license. Certain restrictive open source licenses allow users to  copy, modify and distribute software provided that modified versions (i.e.,  derivatives) are subject to the same license terms and conditions as the  original code. This is intended to prevent software that is derived from or  contains code issued under such a license from becoming a closed- source product that can be marketed and sold exclusively.

The reuse of OSS is viewed as a promising means to reduce  development costs while improving software quality. According to  software experts, software reuse has the potential to:  increase reliability because systems will be developed with thoroughly  tested and proven components,  increase productivity by reducing the time and effort needed to  develop software,  reduce costs by enabling the sharing of knowledge and practices  needed to develop and maintain software, and establish a more standard and consistent approach to software  development and maintenance by using common components and  procedures.

OMB Memorandum on Federal Source Code Policy

In August 2016, OMB issued a memorandum to the heads of  departments and agencies to ensure that new custom-developed source  code be made available for reuse across the federal government. The  memorandum was intended to improve the way federal agencies buy,  build, and deliver information technology and software, and required that  all agencies establish a pilot program under which at least 20 percent of  new custom-developed code would be released as OSS for 3 years. OMB  also required that agencies develop a metric to calculate the percentage  of code released as OSS to gauge its progress on implementing the pilot  program.

OMB’s memorandum also identified four supporting requirements, among  others, needed to implement an OSS pilot program:  Issue an OSS policy that ensures code is available for government- wide reuse.

Conduct an OSS three-step software solutions analysis that includes:  (1) a strategic analysis and analysis of alternatives, (2) consideration  of existing commercial solutions, and (3) consideration of custom  development. In addition, agencies must consider several factors  throughout each of the three-steps of the analysis such as cloud  computing and open standards.

Secure data rights to government-wide reuse and inventory new  custom code, in accordance with the guidance provided by the  code.gov website.

Facilitate the OSS community by developing and releasing the code in  a manner that (1) fosters communities around shared challenges; (2)  improves the ability of the OSS community to provide feedback on,  and make contributions to, the source code; and (3) encourages  federal employees and contractors to contribute back to the broader  OSS community by adding value to existing projects. In doing so,  agencies should comply with the following principles: (1) leverage  existing communities, (2) engage in open development, (3) adopt a  regular release schedule, (4) consider code contributions, and (5)  document source code to facilitate use and adoption.

DOD Implementation of OMB’s Open Source Software Requirements

DOD’s CIO is responsible for implementing OMB’s requirements for the  department’s OSS pilot program. The CIO reports directly to the  Secretary of Defense, and is responsible for the department’s information  technology (including national security systems and defense business  systems), information resources management, and efficiencies. The CIO  is also responsible for developing strategies and policy on the operation  and protection of all of the department’s information technology and  information systems. Other responsibilities include maintaining a  consolidated inventory of mission critical and mission essential  information systems, evaluating and monitoring performance  measurements, and other duties to manage the information environment  throughout the department.

In addition, the Defense Digital Service is responsible for assisting the  CIO in implementing the OSS pilot program, among other initiatives. The  Defense Digital Service is composed of commercially experienced  software developers, software designers, product managers, and problem  solvers within DOD. The organization works on specific projects or  programs in support of the DOD in a hands-on way to materially improve  digital services across the department. The Defense Digital Service also  works with the CIO to monitor the identified programs, facilitate the  process to open source the code, and populate the source code inventory  located on Code.mil.

DOD’s Efforts to Increase Use of Open Source Software

In June 2018, DOD’s CIO issued a report to Congress as directed by  section 875(b) of National Defense Authorization Act for Fiscal Year 2018  (Public Law 115-91). The report provided Congress with the department’s  plan to implement the OSS pilot program established by OMB’s  memorandum. In the report, the CIO committed to sharing its  unclassified, custom-developed source code as widely as possible in four  ways: (1) review and select software programs that have self-identified to  the Defense Digital Service as ready to open source its code; (2) query its  contracts database to identify contracts that contain appropriate data  rights language; (3) determine if source contained within existing source  code repositories can be made available; and (4) issue a department- wide data call to identify and select programs where new, custom code is  being developed.

The CIO also reported that the department would prioritize and assess  identified software programs and work with components to develop  mechanisms to report progress. The report included selection criteria for  identifying candidate software programs: (1) programs with contractually  secured government data rights; (2) programs with contractual rights to  enable, support, and enforce DOD and government-wide sharing and  reuse of custom-developed code; (3) programs that have appropriate  administration to ensure that government’s rights are maintained; and (4)  programs that utilize best practices to ensure custom-developed code,  documentation, and other associated materials are delivered in a  reusable manner.

The CIO also reported that the Defense Digital Service would assist  programs. Specifically, the Defense Digital Service is to develop  guidelines, processes, and answers to frequently asked questions  regarding the use of OSS.

In October 2018, the CIO issued a memorandum to the Chief  Management Officer, secretaries of the military departments, Chairman of  the Joint Chiefs of Staff, under secretaries of Defense, chiefs of the  military services, general counsel, Director of Cost Assessment and  Program Evaluation, Director of Operational Test and Evaluation, and the  Assistant Secretary of Defense for Legislative Affairs notifying them of the  need to implement an OSS pilot program in accordance with OMB’s 2016  memorandum. The CIO required these organizations to take four actions  within 30 days of issuing the memorandum:  Identify all unclassified custom-developed source code created or  paid for by the department—regardless of data rights and open  source status—created on or after August 2016 and provide the CIO  information required by the guidelines on the code.gov website;  Identify and provide a point of contact that can participate in open  source efforts;

Direct authorizing officials to rapidly review and approve unclassified  code for public, open source release after appropriate security, code,  and policy review; and

Direct contracting officers to secure the least restrictive data rights to  custom-developed source code in all future contracts in accordance  with DOD federal acquisition regulations.

DOD Has Not Fully Implemented an Open Source Software Pilot Program and Related OMB Requirements

DOD was mandated by law to initiate the OSS pilot program established  by OMB memorandum M-16-21 which required (1) releasing at least 20  percent of newly custom-developed code each year for the term of the  pilot program, and (2) collecting additional data concerning new custom  software to inform measures to gauge the performance of the pilot  program. Further, the OMB memorandum also required DOD to (1) issue  an OSS policy, (2) conduct an OSS analysis, (3) secure data rights and  inventory custom code, and (4) facilitate the OSS community.

As of late April 2019, DOD had not fully implemented the OSS pilot  program mandated by law. DOD had partially implemented the  requirement of releasing at least 20 percent of newly custom-developed  code as OSS. Specifically, as of July 2019, the Code.gov website  reported that the department had released less than 10 percent of its  custom developed code. The department was in the early stages of its  pilot program and had not determined when the pilot would be fully  implemented. The CIO reported that the size of the department makes it  nearly impossible to inventory all of its source code custom developed  since August 2016. As such, the CIO stated that it would be difficult to  meet the OMB memorandum’s goal of releasing at least 20 percent of its  new custom code as OSS.

In addition, DOD had not implemented the requirement to develop a  consistent measure to gauge the performance of the department’s pilot  program. DOD had not developed such a measure due to a lack of  consensus in the department about what data should be collected.  According to the CIO, if the measure is “lines of code,” then it unfairly  discounts projects that invest a significant amount on research, but are  small otherwise. If the measure is “project hours,” then it discounts those  projects that came about from sparks of innovation that took little time to  develop. If the measure is “project count,” then it ignores the other two  possible measures. The CIO noted that since the components will be  expected to collect and report the required data, the CIO office plans to  reach out to them to facilitate consensus around data needs and what  measure should be used to calculate and monitor performance.

Regarding the four OMB memorandum requirements supporting the  implementation of the pilot program, the department partially implemented  three and did not implement the remaining one. Table 1 describes the  extent to which DOD implemented the OMB supporting requirements, and  is followed by a description of DOD’s efforts on each requirement.

Issue OSS policy. DOD had not implemented the requirement to issue an  OSS policy. According to DOD’s CIO, the department has existing  acquisition policies applicable to OSS, such as the 5000 series and a  memorandum issued in October 2009 that clarifies OSS. However,  according to DOD officials, these policies are outdated and do not comply  with OMB’s memorandum. For example, while the department’s policies  indicated that programs must conduct an analysis of alternatives, trade  studies, or a business case analysis prior to initiating any technology  acquisition or custom code development, they did not require programs to  consider the value of publishing custom code as OSS and negotiate data  rights reflective of its value.

The department acknowledged that it does not have a policy that  addresses the OMB memorandum’s requirement. According to the CIO,  the department had been slow to develop a policy because these types of  changes require significant resources, coordination, and buy-in across the  department that will take additional time to address.

The CIO also stated that the department plans to update its existing OSS  memorandum by the end of the 2019 calendar year and issue it as policy.  In particular, DOD intends to work with acquisition and program  management officials to define approaches, processes, and best  practices to expand software reuse. If the department effectively  implements this intended step consistent with the OMB memorandum,  DOD should be able to fully address this requirement.

Conduct OSS analyses. DOD had partially implemented the requirement  to conduct analyses to consider alternative software solutions. According  to the DOD CIO, of the three elements required by OMB for a three-step  analysis, the department’s current 5000 series policy addresses some of  these elements. For example, as previously mentioned, the policy  required programs to conduct an analysis of alternatives, trade studies, or  a business case analysis prior to initiating any technology acquisition or  custom code development. However, according to the CIO, DOD’s policy  did not require programs to consider the value of publishing custom code  as OSS and negotiate data rights reflective of its value-consideration.  According to the CIO, the department has plans to address gaps in its  existing policy by the end of the 2019 calendar year. If the department  effectively implements this intended step consistent with the OMB  memorandum, DOD should be able to fully address this requirement.

Secure data rights and inventory new custom code. DOD had partially  implemented this requirement by initiating the process to secure data  rights for government-wide reuse and inventory its new custom code.  Specifically, the October 2018 memorandum called for defense  organizations to direct contracting officers to secure the least restrictive  data rights for custom-developed source code in all future contracts, and  identify all unclassified custom-developed source code created after  August 2016. However, when we discussed this matter with DOD  information technology and software officials responsible for the  management of software in December 2018—2 months after the October  2018 memorandum had been issued— seven of 11 officials were  unaware of the statutory mandate to initiate the pilot program in  compliance with the OMB memorandum. Further, DOD has not provided  a milestone for when it expects its inventory to be completed.

Facilitate OSS community. DOD had partially implemented the  requirement to establish an OSS community. According to the DOD CIO’s  Custom Developed Source Code Data Call memorandum, dated October  2018, the DOD CIO is working with Defense Digital Service to develop  guidelines and processes on when and how to open source code. In  February 2017, DOD announced the launch of Code.mil, an open source  initiative led by Defense Digital Service, that allows software developers  around the world to collaborate on unclassified code written by federal  employees in support of DOD projects. Finally, the Defense Information  Systems Agency established a website (Forge.mil) where community  members can collaborate on open source and DOD community source  software. The Forge.mil website also enables collaborative development  through services such as software version control, requirements  management, discussion forums and document repositories.

However, DOD had not yet fully engaged in open source code  development, established a regular release schedule for its software  code, or fully documented its source code to facilitate use and adoption  department-wide. According to the CIO, the department is in the early  stages of implementing the OSS pilot program and had not yet published  a revision to the existing OSS policy memorandum. The CIO stated that  the office plans to request collaboration and input from organizations  throughout DOD for improvement initiatives and identifying specific  processes and expectations for improving custom-developed software  within the Department. However, DOD has not provided a milestone for  when the requirements will be fully implemented and stated that achieving  100 percent compliance is not a realistic expectation.

Until DOD fully implements the OSS pilot program mandated in the  National Defense Authorization Act for Fiscal Year 2018 including the  requirements of OMB memorandum M-16-21, the department will likely  miss opportunities to achieve related cost savings and efficiencies.  Further, the department will not be effectively positioned to ensure  management oversight and implementation of the pilot program.

DOD Officials Shared Views of Expected Benefits and Risks Associated with the Use of Open Source Software

DOD Officials Agree That Using Open Source Software Could Result in Financial Benefits and Increased Efficiency

DOD officials representing 11 components reported that OSS can  potentially yield financial benefits and increase efficiency. Officials  provided the following examples of financial benefits:

Officials in the office of the Navy Chief Information Officer, the Army  Communications-Electronics Command, the Office of the Under  Secretary of Defense for Acquisition and Sustainment, and the Office  of the Assistant Secretary of the Air Force for Acquisition, Technology,  and Logistics stated that OSS is generally less expensive than  commercial off-the-shelf (COTS) leading to cost savings.

An Air Force official we spoke with stated that the increased use of  OSS may potentially result in cost savings. Further, the official noted  general criteria used by the Air Force to identify software projects that  may potentially be appropriate for the use of OSS. Specifically, OSS  should be considered if, among other things, the required  maintenance would not result in a reduction of cost savings or  efficiency if the maintenance is performed in-house.

An official we spoke with at the Defense Advanced Research Projects  Agency stated that OSS has positive benefits in terms of reducing  costs by reducing duplicative efforts. In addition, this official also  stated that OSS allows institutions of any size and budget to partake  in shared investments providing access to software capabilities at a  much lower cost.

A program manager from the Defense Information Systems Agency  reported that the agency had identified an OSS solution that provided  more functionality at less cost than the commercial solution provided  through a vendor. The program manager explained that when the  agency implemented the new OSS solution, it realized $20 million in  annual savings over the commercial solution that had been  maintained by a vendor.

Officials also shared examples of how OSS can increase efficiency in  software development. For example,

Officials from the offices of the Navy CIO and the US Marine Corps  CIO stated that OSS solutions may increase efficiency by providing a  rapid resolution to the needs and requirements of users. In contrast,  rapid development efforts are not conducive for COTS solutions  because of the long process required to obtain solutions that are  needed quickly.

Similarly, an Air Force official noted that the increased use of OSS  may result in increased efficiency by providing rapid responses to  user requirements.

A program manager from the Defense Information Systems Agency  reported that the selection of an OSS solution rather than a COTS  solution contracted through a vendor had resulted in increased  efficiency. The official explained that the use of the OSS solution  allowed the agency to develop and maintain in-house skills that would  not have been available had they opted to contract with a vendor  providing a skilled workforce.

DOD Officials Expressed Differing Views on the Cybersecurity Risk Posed by Open Source Software

Officials from the 11 components expressed mixed views on managing  cybersecurity risks that could be posed by using OSS. Specifically,  officials from three components expressed their views that security  concerns and the lack of a cybersecurity governance process could result  in the sporadic use of OSS. For example:

A Navy CIO official viewed insider threats, such as a disgruntled  employee embedding malicious code, as a factor that could  significantly limit the use and sharing of OSS. According to Navy  officials, without a process to verify that the software is free of  malicious code, the Navy would risk the assurance it requires to  increase the use of OSS. The official said that, in contrast, such  concerns are not an issue when it comes to COTS software because  of the test and verification process to ensure it is free from malicious  code.

An official in the office of the Marine Corps CIO stated that OSS is  used sporadically in their software development efforts because some  cybersecurity officials within the Marine Corps discourage its use due  to security concerns.

An official from the Army’s Communications-Electronics Command  noted that DOD lacks a governance process once the originating  entity releases the source code as open source. The originating entity  no longer retains control over redistributed versions of the source  code. According to Communications-Electronics Command officials,  Army project managers may be hesitant to utilize OSS because of this  perceived security risk.

On the other hand, DOD officials from eight components stated that the  potential cybersecurity risks posed by the use of OSS were manageable  and that the use of OSS should not be limited. For example:

The policy advisor from the Office of the Under Secretary of Defense  for Acquisition and Sustainment noted that scanning tools to analyze  and identify safe and reliable open source code are not being used.  Employing available scanning tool options could result in discovering  available OSS. The policy advisor also noted that building security  into software operations, rather than through the development of  software, would enable users to know if code has been subverted and  to react appropriately.

A program management official from the Office of the Under Secretary  of Defense for Acquisition and Sustainment suggested that security  concerns may be mitigated by establishing a secure repository for  trusted code.

An official in the Office of the Assistant Secretary of the Air Force for  Acquisition, Technology, and Logistics reported that, as long as OSS  is properly vetted to ensure it is secure and free from malware, it  offers an opportunity for the department to achieve cost savings and  efficiencies.

Pilot testing the use of OSS is an important way to ascertain and improve  the way DOD buys, builds, and delivers information technology and  software solutions. However, the department is in the early stages of  implementing its pilot program and had not determined when the pilot  would be fully implemented. Specifically, DOD has not made 20 percent  of its new code available for reuse nor has it identified a measure to  gauge the performance of its pilot program. Moreover, DOD has not yet  established milestones for securing data rights and conducting an  inventory or facilitating community. Until DOD fully implements its pilot  program and establishes milestones for completing the OMB  requirements, the department will not be positioned to take advantage of  significant cost savings and efficiencies.

We are making the following four recommendations to DOD:

The Secretary of Defense should ensure the department implements  the pilot program by releasing at least 20 percent of newly custom- developed code as OSS. (Recommendation 1)

The Secretary of Defense should ensure the department identifies a  measure to calculate the percentage of code released to gauge its  progress on implementing the pilot program. (Recommendation 2)

The Secretary of Defense should ensure the department establishes  milestones for completing the requirements of OMB memorandum M- 16-21 of securing data rights and conducting an inventory.  (Recommendation 3)

The Secretary of Defense should ensure the department establishes a  milestone for completing the OMB memorandum’s requirement of  facilitating an OSS community. (Recommendation 4)

DOD provided written comments on a draft of this report, which are  reproduced in appendix II. In its comments, the department did not concur  with our first recommendation, partially concurred with our second  recommendation, and concurred with the third and fourth.

DOD did not concur with the first recommendation on ensuring that the  department implements the pilot program by releasing at least 20 percent  of newly custom-developed code as OSS. The department stated that it  does not believe that the pilot program as described in the OMB  memorandum is implementable as proposed. For example, DOD asserts  that most of DOD’s custom developed software is created for weapons  systems and releasing the associated code is sensitive for national  security reasons. In addition, the size and complexity of DOD presents  unique challenges for the department compared to other federal agencies  such as inventorying all software development projects to establish a  baseline. DOD added, however, that the OMB memorandum explicitly  states that national security exceptions do not apply to the pilot program.  DOD also stated that it recognizes the value of collaborative software  development and has plans to release additional guidance on releasing  OSS and procedures for maintaining its inventory. Once DOD establishes  a baseline inventory of custom-developed software and the procedures  for maintaining it, the department states it will be able to determine if the  20 percent is an appropriate goal.

We understand the potential constraints DOD faces and that national  security considerations are to be factored into decisions DOD will need to  make about which custom developed software to include in the pilot.  However, DOD is mandated by law to implement the OSS pilot program  established by OMB memorandum M-16-21. Further, the OMB  memorandum instructs agencies to refrain from selecting code for release  that would fall under exceptions such as national security risk. As such,  DOD has flexibility on making decisions about which custom-developed  code to include in the pilot. While we agree that a baseline inventory is  needed, DOD must include at least 20 percent of new custom-developed  code each year for the term of the program to satisfy the mandate.

DOD partially concurred with the second recommendation on ensuring  that the department identifies a measure to calculate the percentage of  code released to gauge its progress on implementing the pilot program.  Specifically, the department stated that the additional guidance it plans to  release before the end of 2019 on OSS will include measures to gauge  how much code has been developed and how much has been released.  In addition, DOD noted that these measures will support good  management of the overall portfolio of information technology, even in the  absence of the mandated pilot program established by the OMB  memorandum. We believe that the measure to calculate the percentage  of code should be used to assist the department in meeting the OMB  memorandum’s requirements. We also agree with the benefits of  developing a measure to manage the portfolio of information technology.

DOD concurred with the third and fourth recommendations related to  establishing milestones for completing the OMB memorandum’s  requirements of securing data rights, conducting an inventory, and  facilitating an OSS community. According to the department, it has issued  a memorandum that directed contracting officers to secure the least  restrictive data rights to custom-developed source code, in furtherance of  the OMB requirements, and also included a data call that forms an initial  basis of an inventory of custom-developed software. Regarding facilitating  an OSS community, DOD stated that it has formed a community of  practice called DevSecOps that is open to all software development  organizations in the department and plans to use this forum to facilitate  collaboration on the use of OSS.

We are sending copies of this report to the appropriate congressional  requesters and the Secretary of Defense. In addition, the report is  available at no charge on the GAO website at http://www.gao.gov.

If you or your staffs have any questions about this report, please contact  me at (202) 512-4456 or at harriscc@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to: (1) assess the extent to which the Department of  Defense (DOD) has implemented the open source software (OSS) pilot  program and other related requirements established by the Office of  Management and Budget (OMB), and (2) describe the views of  responsible DOD officials on the use of OSS to achieve efficiency at the  department.

To address the first objective, we selected six requirements from the  OMB memorandum titled the Federal Source Code Policy: Achieving  Efficiency, Transparency, and Innovation through Reusable and Open  Source Software (M-16-21, Aug. 8, 2016) as criteria to assess the extent  to which DOD has implemented the OSS pilot program. Two  requirements establish the OSS pilot program: (1) releasing at least 20  percent of newly custom-developed code each year for the term of the  pilot program, and (2) developing a metric to gauge the performance of  the pilot program. The other four requirements support the  implementation of the pilot program: (1) issuing an OSS policy, (2)  conducting an OSS analysis, (3) securing data rights and inventorying  custom code, and (4) facilitating the OSS community. We met with  officials from OMB to collect background information on the selection of  requirements for the pilot program established in memorandum M-16-21.  We also met with officials from the Office of the DOD Chief Information  Officer (CIO) and the Defense Digital Service to discuss the status of the  department’s implementation of the OSS pilot program.

We reviewed DOD’s June 8, 2018 report to Congress and its October  2018 memorandum that details the department’s plans to implement the  pilot program and compared them to the six requirements. To determine  the extent to which the pilot program had been implemented, we  evaluated DOD’s efforts to address each of the requirements using a 3- stage gradient scale (implemented, partially implemented, and not  implemented). The requirement was assessed to be fully implemented if  DOD provided us with sufficient evidence that the requirement had been  fully met. We assessed a requirement to be partially implemented if DOD  provided us with documentation of initial plans or had initiated action  towards implementing the requirement. We determined that a  requirement was not implemented when DOD did not provide us with  documentation of planned or initiated actions to implement the  requirement.

To address the second objective on views of various responsible DOD  officials, using professional judgement, we selected components across  the department responsible for the management and development of  OSS. The scope of stakeholders selected represent department-wide  nongeneralizable views including military components, defense agencies,  and other offices. At least one representative was selected from the  following components: (1) Office of the Under Secretary of Defense for  Acquisition and Sustainment, (2) Office of the DOD CIO, (3) office of a  military service CIO, (4) Military Service Software Center or Command  Center, (5) the Defense Information Systems Agency, and (6) the  Defense Advanced Research Projects Agency. We conducted interviews  with DOD officials from the following entities: Office of the Under  Secretary of Defense for Acquisition and Sustainment; Office of the DOD  CIO; Offices of the Navy and Marine Corps CIOs; Office of the Air Force  Chief Technology Officer; Army Communications Electronics Command;  the Defense Information Systems Agency; the Defense Advanced  Research Projects Agency; and the Office of the Assistant Secretary of  the Air Force for Acquisition, Technology, and Logistics.

In order to summarize and report the views of the responsible DOD  officials, we conducted structured interviews with representatives from the  selected components. Each interview consisted of the same discussion  topics based on the pilot program requirements established in OMB’s  memorandum. The scope of this objective represents individual thoughts,  views, and opinions and is not intended to convey an official or  department response.

Prior to each interview, participants were provided with OSS discussion  topics, and the OMB memorandum, titled the Federal Source Code  Policy: Achieving Efficiency, Transparency, and Innovation through  Reusable and Open Source Software (M-16-21, Aug. 8, 2016). The  contents of each interview were reviewed and summarized to identify the  general views of OSS and on the anticipated implementation of the pilot  program requirements established in OMB’s memorandum. We noted  similarities and differences in the responses provided by the officials in  the use of OSS including, but not limited to, potential benefits of using  OSS, managing associated risk, and opinions on implementing the pilot  program in compliance with the OMB memorandum.

Discussions were split into two topic areas: practices on the use of OSS,  and OMB’s memorandum to establish an OSS pilot program. Specifically,  the discussion topics were presented during each meeting as follows:  Part I: Practices on the use of OSS

Your experience or your organization’s history with the practice of  using OSS as a means to achieve cost reduction or efficiencies when  buying, building, or delivering information technology and software  solutions.

The processes or practices that you or your organization perform to  leverage open source code for projects that require the acquisition or  development of custom source code.

The extent to which you or your organization shares or uses open  source software. For example, do you share or use open source  software: (1) within your organization only, (2) across the DOD with  other military services or defense agencies, (3) with other federal  agencies, or (4) outside the federal government with the public. Also,  how is the source code shared, leveraged, catalogued, stored, and  accessed.

Policies or guidance currently in use for OSS.

General views and opinions on the use of open source code.

Part II: OMB’s Memorandum to Establish an OSS Pilot Program

When and how you or your organization became aware of OMB’s  memorandum on Federal Source Code Policy.

Your opinions and views about the OMB memorandum.

Any specific concerns or reservations about the requirements  contained in the OMB memorandum.

The extent to which you or your organization may already be  performing the steps in OMB’s proposed Three-Step Software  Solutions Analysis.

Discuss the feasibility of the pilot program requirement to release at  least 20 percent of new custom-developed code as OSS.

Appendix II: Comments from the Department of Defense

Appendix III: GAO Contact and Staff Acknowledgments

In addition to the contact named above, Eric Winter (Assistant Director),  John Ortiz (Analyst-in-Charge), Rebecca Eyler, Franklin Jackson, and  Kate Nielsen made key contributions to this report.

Open source software is code released under a license that grants users the right to modify, share, and reuse the software. Making code available for reuse as open source can have major benefits such as reducing costs and improving efficiency.

Congress required the Department of Defense to start an open source software pilot program in accordance with requirements from the Office of Management and Budget. We found DOD hasn’t fully implemented a program that meets these requirements. We also found concerns among some DOD officials over open source cybersecurity.

We made 4 recommendations on how DOD could fully implement the pilot program.


Aviation Security: TSA Updated Screener Training to Address Risks, but Should Enhance Processes to Monitor Compliance

TSA is responsible for screening millions of airline passengers and their baggage each day at the nation's commercial airports for items that could threaten aircraft and passengers. In carrying out its mission, TSA requires its screener workforce to complete various trainings on screening procedures and technologies. TSA updated its security screening procedures and technologies in recent years to address risks identified through covert tests in 2015 and reports of emerging threats.

The TSA Modernization Act of 2018 included a provision for GAO to examine the effectiveness of TSA's updated screener training. This report addresses: (1) changes TSA made to screener training since 2015; (2) how TSA updates and evaluates screener training; and (3) how TSA ensures screener compliance with training requirements.

GAO analyzed TSA documentation on training development, compliance monitoring, and a non-generalizable sample of six recently updated training courses—selected to reflect a range of training types and topics. GAO also reviewed TSA data on airport screener training compliance rates from fiscal years 2016 through 2018, and interviewed TSA officials.

Since 2015, the Department of Homeland Security's (DHS) Transportation Security Administration (TSA) developed and updated screener training to address potential risks to commercial airports identified through covert testing and reports on emerging threats. From May 2015 through June 2019, TSA identified 62 potential risks that warranted review for a potential change in training. TSA made training changes in response to 56 of the identified risks—affecting 40 different training courses. TSA also responded to risks by developing or updating job aids or briefings for screeners.

TSA uses established models for developing, updating, and evaluating its screener training. The figure below shows TSA's process for updating and evaluating its screener training, in accordance with a training development model that is widely accepted and used across the federal government.

TSA relies on an online database to monitor screener compliance in completing required training at the nation's commercial airports. However, TSA has not documented its process for monitoring screener training compliance, including for analyzing compliance data and reporting and addressing instances of noncompliance at airports. Moreover, while TSA monitors airport compliance rates in a given year, it does not analyze the data across fiscal years for potential trends in noncompliance by individual airports that may warrant corrective action at the headquarters level. GAO found that in fiscal years 2016 and 2017, screeners at 435 commercial airports met TSA's 90 percent target compliance rate, while in 2018, five airports had compliance rates well below this target, dropping 15 to 26 percentage points from the prior year. TSA officials stated they were unaware of this development. By documenting its screener training compliance monitoring process and monitoring screener training compliance data across fiscal years, TSA would be better positioned to ensure it is aware of potential noncompliance trends warranting corrective action at the headquarters level.

GAO is making two recommendations, including that TSA (1) document its process for monitoring screener training compliance and (2) monitor screener compliance data across fiscal years. DHS concurred with the recommendations.

The screening of airport passengers and their belongings is a critical  component in securing our nation’s civil aviation system. Since the  terrorist attacks of September 11, 2001, the Transportation Security  Administration (TSA) has been responsible for screening airline  passengers and their carry-on and checked baggage for prohibited items  that could pose a threat to aircraft and passengers. According to TSA, in  fiscal year 2018, Transportation Security Officers (TSOs) screened more  than 804 million passengers at approximately 440 commercial airports  across the United States. TSOs are to follow standard operating  procedures that guide their screening processes and utilize technologies,  such as advanced imaging technology or walk-through metal detectors to  screen passengers, and x-ray and explosives detection technologies to  screen their belongings.

TSA has a variety of training and development programs to help ensure  the TSO workforce has the skills and expertise needed to fulfill security  responsibilities. In April and May of 2015, the Department of Homeland  Security (DHS) Office of Inspector General conducted covert tests that  raised questions regarding the effectiveness of aspects of TSA’s  passenger screening process. In response, in June 2015, the secretary  of DHS directed TSA to take a number of actions to address the risks  identified in the testing, including further training for all TSOs.

The 2018 TSA Modernization Act required that TSA establish a training  program for new TSOs located at the TSA Academy. The Act also  required TSA to establish, not later than 180 days after enactment,  recurrent training that covers updates to TSO screening procedures and  technologies, including, in response to weaknesses identified in covert  tests at airports; methods to detect false or fraudulent travel documents;  and training on emerging threats. Further, the Act states that the  recurrent training shall include internal controls for monitoring and  documenting compliance of TSOs with the requirements.

The TSA Modernization Act includes a provision for us to examine the  effectiveness of the new and updated security screening personnel  training. This report (1) describes changes TSA has made to its screener  training to reflect updates to its screening procedures and technologies  since 2015; (2) examines how TSA updates and evaluates its screener  training programs, and the extent to which these processes align with  leading practices; and (3) assesses how TSA ensures screener  compliance with training requirements.

To describe changes TSA has made to screener training to reflect  updates to its screening procedures and technologies in recent years, we  reviewed TSA training program documentation, including its National  Training Plan. We also reviewed information on TSA’s efforts to change  its TSO training from May 2015 to June 2019 in response to updates to  screening procedures and technologies, methods to detect false or  fraudulent travel documents, and emerging threats, such as standard  operating procedures and training curriculum. We also identified specific  TSO training changes TSA’s Training and Development office (T&D)  made from May 2015 through June 2019 in response to DHS Office of  Inspector General covert test results and emerging threat information. We  selected six of the 40 TSO courses that T&D updated during this time to  review specific curriculum changes. We selected the six courses to  provide a range of course type and topic. We then compared the course  materials from the six courses with the risks identified by covert tests and  reports of emerging threats, as well as standard operating procedure  documents, to examine how the courses aligned with the risks and  documented procedures. Findings from our review of the selected  courses cannot be generalized to all TSO courses, but provide illustrative  examples of how TSA updated curriculum in response to covert testing  results and emerging threat information. We also interviewed T&D  officials to obtain insight into the changes TSA made to its TSO training  programs.

To examine how TSA updates and evaluates its training programs and  the extent TSA’s processes for doing so align with leading practices, we  reviewed TSA documentation, such as TSA Management Directive  1900.14, Handbook, Training Standards, which outlines the process T&D  is to follow when developing and updating its training programs. We then  compared TSA’s process against selected leading practices that we have  identified in our prior work for training development in the federal  government. For our analysis, we selected eight leading practice  attributes that we identified in TSA’s Management Directive and our prior  work. With respect to TSA’s training evaluation process, we examined  TSA’s use of the Kirkpatrick Model—a training evaluation model endorsed  by the Office of Personnel Management (OPM)—as a means to assess  TSO training. To do this, we examined TSA documentation related to the  evaluation steps of its curriculum development process, including the  applicable evaluations for the six recently updated courses described  above. We also interviewed T&D officials about their processes to update,  develop, and evaluate training.

To assess how TSA ensures TSO compliance with training requirements,  we reviewed TSA documentation on its process for monitoring training  compliance, such as management directives, and assessed this process  against Standards for Internal Control in the Federal Government— specifically those related to monitoring. We also collected and analyzed  data on TSO training compliance rates from TSA’s Online Learning  Center database for fiscal years 2016 through 2018 (the most recent  complete year of data available at the time of our review). We analyzed  the data to determine trends in the annual training compliance rates for  each of 435 commercial airports that reported training compliance rates  during this timeframe. We assessed the reliability of these data by  reviewing TSA documentation related to the database, such as the user  manual, and interviewing knowledgeable officials about its use and  limitations. We also conducted electronic testing to check for missing  values and outliers and followed-up with TSA officials on the issues we  found. We determined the data to be sufficiently reliable for purposes of  determining trends in compliance rates. Additionally, we interviewed TSA  headquarters officials from T&D and Security Operations, as well as a  non-generalizable sample of seven Federal Security Directors (FSDs)  who oversee operations at 47 commercial airports, to understand their  roles ensuring training compliance.

We conducted this performance audit from May 2019 to February 2020 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

Background TSOs and Agency Roles for Their Training

TSA is the primary federal agency responsible for implementing and  overseeing the security of the nation’s civil aviation system and, in  general, is responsible for ensuring that all passengers and belongings  transported by passenger aircraft to, from, within, or overflying the United  States are adequately screened. Over 43,000 TSOs stationed across the  nation’s approximately 440 commercial airports are responsible for  inspecting individuals and belongings to deter and prevent passengers  from bringing prohibited items on board an aircraft or into the airport  sterile area.

Within TSA, two offices—T&D and Security Operations—are to work  together to manage TSOs and ensure their training is current and  relevant. T&D is responsible for developing initial and ongoing training  curricula for TSOs based in part on TSA’s standard operating procedures  that govern how TSOs screen passengers and baggage. Security  Operations is responsible for allocating TSO staff to airports, and  scheduling TSO work hours and training availability. Within Security  Operations, FSDs are responsible for overseeing security operations at  the nation’s commercial airports, many overseeing multiple airports within  a specific geographic area. FSDs report to one of three executive  directors, who in turn are responsible for annually assessing FSD  performance, including oversight of TSO training.

TSA’s screener training is comprised of a compendium of courses that  includes basic training for initial hires, recurrent training, remedial training,  and return-to-duty training. The National Training Plan specifies annual  training requirements and contains the core curriculum for TSOs,  including the classes and hours required for TSOs to complete. In  accordance with the Aviation and Transportation Security Act, screeners  must complete a minimum of 40 hours of classroom instruction and 60  hours of on-the-job training, and must successfully complete an on-the- job training examination. Until 2016, new TSOs completed these  training requirements at or near their home airports through the New Hire  Training Program. In January 2016, TSA centralized this training under  the TSO Basic Training Program at the TSA Academy in Glynco,  Georgia. Further, in August 2018, TSA launched the first phases of TSO  Career Progression, in which new hire screeners receive local training  and gain experience in a limited number of screening functions before  advancing to the next stage of training at the TSA Academy, roughly  around the four-month mark.

In 2015, in response to the DHS Office of Inspector General covert test  findings that highlighted areas of concern in the passenger screening  process, TSA implemented a TSO re-training effort, beginning with a  nationwide training called “Mission Essentials—Threat Mitigation.”  According to TSA, this training provided the opportunity for the TSO  workforce to become familiar with the threat information that underlies  TSA’s use of checkpoint technologies and operational procedures to  mitigate risks.

Federal Training Evaluation Requirements and Training Evaluation Models

In 2009, OPM developed and published regulations that require agencies  to evaluate training programs annually. According to the regulations,  these training evaluations are to help agencies determine how well such  plans and programs contribute to mission accomplishment and meet  organizational performance goals. One commonly accepted training  evaluation model, endorsed by OPM and commonly used in the federal  government to evaluate training, is known as the Kirkpatrick model. The  Kirkpatrick model consists of a four-level approach for soliciting feedback  from training course participants and evaluating the impact the training  had on individual development, among other things.

The following describes what each level within the Kirkpatrick model is to  accomplish:

Level 1: The first level measures the training participants’ reaction to,  and satisfaction with, the training program. A level 1 evaluation could  take the form of a course survey that a participant fills out immediately  after completing the training.

Level 2: The second level measures the extent to which learning has  occurred because of the training effort. A level 2 evaluation could take  the form of a written exam that a participant takes during the course.

Level 3: The third level measures how training affects changes in  behavior on the job. Such an evaluation could take the form of a  survey sent to participants several months after they have completed  the training to follow up on the impact of the training on the job.

Level 4: The fourth level measures the impact of the training program  on the agency’s mission or organizational results. Such an evaluation  could take the form of comparing operational data before, and after, a  training modification.

TSA Revised Screener Training to Address Risks Identified through Covert Tests and Emerging Threats

Since 2015, TSA’s T&D has developed and updated TSO training  programs in response to findings from covert tests and reporting on  emerging threats that identified risks to aviation security. T&D uses an  online database to track results from covert tests and reporting on  emerging threats, and any changes to training that T&D makes as a  result. According to T&D data from May 2015 through June 2019, T&D  officials reviewed 62 risks that warranted a review for a potential change  to training, and 56 of the risks led officials to make training changes  across its TSO curriculum. Overall, T&D made changes affecting 40  different training courses.

Based on our review of TSO training curriculum from May 2015 through  June 2019, we found that changes T&D made to its TSO training took  many forms. In some cases, T&D changed training to place additional  emphasis on a certain aspect of a current standard operating procedure  or provide context on the importance of following it. For example, in 2019,  T&D updated its instructor-led course—”Mission Essentials: Resolution  Tools and Procedures”—to address covert tests where TSOs failed to  detect simulated explosive devices hidden in bags or concealed on  individuals at checkpoints. The training included a review of methods  terrorists may use to plan and carry out attacks in order to emphasize the  importance of following the standard operating procedure. The updated  training also included leading practices for searching belongings and a  discussion of issues that may affect a TSO’s ability to detect threat items  hidden in belongings or on individuals. In fiscal year 2019, T&D also  updated instructor-led courses on its explosives detection system for  checked baggage to respond to covert test findings that TSOs failed to  detect certain simulated explosive devices. The updated training included  images of simulated explosives hidden in checked bags that replicated  scenarios similar to the failed covert tests.

In other cases, T&D developed TSO training in response to new or  updated standard operating procedures for using technologies. T&D  officials said that for this type of TSO training, they wait until TSA’s  Requirements, Capabilities, and Analysis office updates or establishes  new standard operating procedures for using new technologies and then  develops training based on the revisions. For example, T&D developed  TSO training to cover the differences between a prior and updated  version of the standard operating procedure for screening passengers  and belongings at security checkpoints. T&D included curriculum to cover  the major changes in the standard operating procedure and incorporated  additional training to address a covert test in which TSOs failed to detect  a simulated explosive device at a screening checkpoint. T&D also  developed training for TSOs who check passenger IDs and travel  documents. The training focused on updates to the standard operating  procedure and included procedures specific to the 2005 REAL ID Act,  which TSA will fully implement in 2020. Additionally, T&D incorporated  this new training to address covert tests that had found issues with  identifying false or fraudulent travel documents.

In addition to updating or developing new training involving instructor-led  courses, TSA responded to identified risks by developing or updating job  aids or briefings for TSOs. For example, TSA developed the “It’s Not the  Container” briefing in 2017 to address risks highlighted by an attempted  attack in Australia and included tactics used to conceal explosives in  benign items. The briefing provided best practices for using screening  technologies to identify concealed explosives, which aligns with current  standard operating procedures. T&D also developed the “Electronics vs.  Electrical Devices Job Aid”—covering how TSOs should handle the  devices at checkpoints—which instructors circulated during classroom  training and provided to TSOs at the screening checkpoints.

TSA Uses Established Models for Updating and Evaluating TSO Training and Has Followed Leading Practices

TSA uses established models and processes for updating and evaluating  TSO training, and these processes follow leading practices for training  and evaluation development. TSA updates its trainings using a training  development process that can be segmented into five broad, interrelated  elements, and is typically referred to as the ADDIE model. The elements  include (1) analysis, (2) design, (3) development, (4) implementation, and  (5) evaluation. In our prior work, we have found that these five elements  of the ADDIE model help to produce a strategic approach to federal  agencies’ training and development efforts. See figure 1 for how T&D  aligns its training development process with the ADDIE model.

T&D’s guidance and our prior work on federal agency training  development identify various leading practice attributes for developing  training. Such attributes include that the training development process: (1)  is formal and based on industry recognized standards; (2) provides the  ability to update training based on changing conditions and, if necessary,  quickly; (3) includes mechanisms to ensure programs provide training that  addresses identified needs; (4) ensures measures of effectiveness are  included in training program; (5) prevents duplication of effort and allows  for consistent message; (6) allows for stakeholder feedback; (7) provides  for continuous evaluation of effort; and (8) includes mechanisms to  ensure training programs are evaluated.

We found that T&D’s training development process incorporates all of the  identified leading practice attributes, as shown in table 1.

Two examples of TSA’s implementation of selected leading practice  attributes are that T&D (1) has methods for updating training quickly, if  needed, and (2) has mechanisms to ensure TSO training is evaluated.  Specifically:  Methods to quickly update training. In alignment with the leading  practice that agencies should have a process to enable quick updates to  training to respond to changing conditions, T&D has alternative processes  to develop and deliver training to TSOs faster than the approximately 6  months its standard process takes to develop or revise training. For  example, in 2018, T&D formalized a set of alternative processes to rapidly  develop and deliver training to TSOs. One such alternative is for T&D to  use its Rapid Response process, which allows for a response time to the  field of 72 hours. Additional options include the Rapid Update/Revision or  Rapid Development (Priority Training) processes to allow for a new  training to be issued in approximately 30 days. T&D officials said that the  rapid development processes are used when an issue, such as an  emerging threat, requires a response in days or weeks. T&D’s guidance  outlines situations when these processes are appropriate for use and  provides checklists to help T&D personnel follow key steps.

Mechanisms to help ensure evaluations of training effectiveness.  T&D has mechanisms for ensuring it evaluates the effectiveness of its  TSO training programs. In particular, T&D uses the Kirkpatrick model to  evaluate its training and, according to its policy, all of its courses are to be  evaluated at Level 1 of the model, which measures training participants’  reaction to, and satisfaction with, the training. T&D is also to plan course  evaluations for each training during the curriculum development process,  determine the formal review cycle, and include it in the curriculum  development paperwork. According to its policy, T&D must complete a  curriculum review at least once every 5 years, but may do so at shorter  intervals. During the curriculum review, T&D examines the training to  confirm the content is valid with respect to the applicable listing of tasks  and competencies, current law, policy, procedures, and equipment. As a  part of this process, T&D assesses participant evaluations to determine  whether changes to TSO training are needed.

As of October 2019, T&D’s efforts to evaluate new or updated TSO  training made from May 2015 through June 2019 are in line with its policy.  For example, T&D officials said they updated participant evaluations for  TSO training they changed during this time period to address risks  identified by covert testing and reports on emerging threats. These  officials told us that they had not yet formally analyzed the results of the  evaluations. This progress is in line with T&D policy, which requires a  review of each training every 5 years. We verified this by obtaining  evaluations T&D collected for the six selected sample courses we  reviewed. T&D provided us level 1 survey responses it had collected that  measure training participants’ reaction to, and satisfaction with, the  training programs for four of the courses. T&D implemented the four  courses from calendar years 2015 to 2019. Based on those dates and  T&D policy, T&D should complete curriculum reviews for the courses  between 2020 and 2024.

TSA Monitors Training Compliance, but Its Process Does Not Look for Trends across Fiscal Years and Is Not Fully Documented

TSA relies on a database that both field and headquarters staff use to  monitor TSO training compliance. According to TSA policy, TSA  documents and maintains the training status of all TSOs across  approximately 440 commercial airports through its Online Learning  Center database. Within the database, TSA records training completion in  three ways:  1.  TSOs self-certify they completed the training activity, such as reading  2.  A training staff member at a commercial airport will record training  completion on behalf of a TSO for instructor-led courses and on-the- job training;  3.  The database automatically records completion for training actions,  such as online training.

After recording training completion, the database calculates the  percentage of TSOs at a given airport who are on pace for completing  their required annual training. According to TSA guidance, the agency  has set its annual TSO target compliance rate at 90 percent per  commercial airport.

While TSA has guidance outlining roles and responsibilities for training  oversight at a high level, TSA headquarters and field officials told us their  processes for monitoring training compliance—including analyzing  training compliance data, reporting their results, and taking action to address the results—were not documented. Below are descriptions of  these roles and responsibilities at the field and headquarters levels,  based on what officials from each level told us.

TSA personnel in the field have various responsibilities for overseeing  training compliance:

FSDs. FSDs, who oversee operations at one or more airports, have  the primary responsibility for ensuring that TSOs within the airports  they oversee have fulfilled their training requirements. FSDs are  assessed on training compliance among TSOs at their respective  airports during their annual performance reviews. All seven FSDs we  interviewed said they use the Online Learning Center database to  verify that TSOs are on track for meeting their training requirements.  Further, these FSDs said they meet regularly with their on-site training  staff to discuss how training is going and whether TSOs are at risk of  not meeting their training requirements.

Executive Directors. Executive Directors oversee the FSDs who  work within their respective portfolios and discuss training compliance  with the FSDs during their annual performance review. To monitor  FSDs’ efforts, Executive Directors also review data from TSA’s Online  Learning Center database on TSO training compliance for airports  within their area of responsibility. According to an Executive Director  we spoke with, if an Executive Director notices that TSO training  compliance rates for an airport whose FSD they oversee are lower  than the 90 percent compliance target, he or she may reach out to the  FSD to obtain information on the causes and discuss an action plan to  improve training compliance.

TSA personnel at headquarters also have various responsibilities for  overseeing training compliance:

T&D. T&D officials said that on a monthly basis they analyze TSO  training compliance data from TSA’s Online Learning Center database  to identify how TSOs nationwide are meeting requirements and  whether there may be trends that indicate a need for changes to  training during the fiscal year. For example, officials told us that in  fiscal year 2019 they noticed that airports were generally behind in  meeting annual training requirements and determined this was due to  the effects of the federal government shutdown. In response, they  stated they adjusted the duration of some training courses to shorten  the amount of time it would take TSOs to complete the training within  the remainder of the fiscal year.

Security Operations. Security Operations tracks individual airport  progress toward meeting TSA’s annual 90 percent compliance target.  Security Operations officials said they receive and review monthly  training compliance reports from T&D. They are responsible for  analyzing the data to monitor whether airports are on pace toward  meeting the annual TSO training compliance target. For example,  TSA has set the required training completion pace goal at 8.3 percent  per month for each commercial airport—-so that by maintaining the  pace, by the end of the fiscal year, TSOs at each airport will have  completed their required annual training. Officials told us that if they  identify instances where an airport’s overall TSO training compliance  rate for a given month is below this goal during the course of a fiscal  year, they will reach out to the FSD responsible. They will provide the  FSD a point of contact at a comparable airport with higher compliance  rates to share best practices for addressing the issue.

While TSA headquarters officials from Security Operations and T&D are  responsible for analyzing and addressing TSO training compliance, they  focus on monthly airport progress toward the 90 percent TSO training  target, rather than annual changes in compliance rates. In particular, they  do not look back at prior year airport compliance data to assess whether  airports did not meet the compliance target across fiscal years, and  whether they require corrective action at the headquarters level.

However, we reviewed annual TSO training compliance data across fiscal  years for each of the 435 commercial airports that reported data from  fiscal years 2016 through 2018. We found that while all airports met  TSA’s 90 percent training compliance target in fiscal years 2016 and  2017, the compliance rates for five airports dropped well below 90  percent in 2018. These five airports’ TSO compliance rates dropped 15  to 26 percentage points from their reported compliance rate in 2017.

T&D and Security Operations headquarters officials said they were not  aware that five airports had not met TSA’s TSO training compliance target  in fiscal year 2018, nor the causes for it. Headquarters officials said that  they did not identify this development because their focus is on monthly  nationwide trends, rather than instances of noncompliance at individual  airports across fiscal years, which field officials would be responsible for  addressing. However, unlike headquarters officials, field officials do not  have the visibility to identify if or when such noncompliance may be  occurring across other commercial airports; and whether it may indicate a  broader issue. For example, the five airports whose TSO compliance  rates dropped significantly between fiscal years 2017 and 2018 varied by  size and location. As a result, FSDs and Executive Directors would  generally not have been aware that other airports experienced  noncompliance or been in a position to determine whether the  noncompliance was due to related reasons. Based on TSA’s process,  TSA headquarters officials from T&D and Security Operations are best  positioned to identify training compliance trends and their causes when  they occur, as they have visibility into training compliance data across the  agency in a way that field officials do not.

Headquarters officials from T&D and Security Operations told us the field- level processes for overseeing training compliance are not documented  because TSA has intentionally given field officials the flexibility to manage  TSO workload and training to meet the individual needs of their airports.  They said they did not document their processes at the headquarters  level because they already understood what to do and were not required  to document the analysis results. However, the headquarters officials said  there may be a benefit to documenting the headquarters process to  ensure consistency in how they carry out the process in the event of  attrition.

Standards for Internal Control in the Federal Government calls for  agencies to develop and maintain documentation of their internal control  system. This documentation allows management to retain  organizational knowledge and communicate that knowledge to external  parties. This documentation of controls is also evidence that controls are  identified, can be communicated to those responsible for their  performance, and can be monitored and evaluated by the entity.  Moreover, internal control standards state that internal control monitoring  should generally be designed to ensure that ongoing monitoring occurs in  the course of normal operations to ensure that known weaknesses are  resolved.

By documenting its headquarters process for monitoring TSO training  compliance—including its process for analyzing monthly training  compliance data, the results of its analyses, and actions taken in  response—TSA could better ensure its headquarters staff are aware of  their responsibilities for overseeing TSO training compliance and  consistently carry these responsibilities out as staff change over time.  Additionally, by monitoring for instances of TSO noncompliance at  individual airports across fiscal years in its analysis of training compliance  data, TSA headquarters would be better positioned to determine whether  they constitute a trend warranting corrective action at the headquarters  level.

TSOs’ ability to perform their duties effectively in screening passengers  and their belongings is crucial to the security of the nation’s aviation  system. While TSA has made updates to its TSO training programs to  address risks identified in covert testing, additional actions could improve  its processes for monitoring TSO training compliance so that the agency  can identify and address any potential training issues. In particular, by  documenting its process for monitoring TSO training compliance— including those for analyzing monthly training compliance data, reporting  the results of its monitoring efforts, and taking action to address potential  issues—TSA could help ensure that all of the various officials responsible  for monitoring training compliance, including new staff over time,  understand the process and can consistently implement it. Further, by  monitoring for instances of airport TSO non-compliance across fiscal  years in its analysis of training compliance data, TSA would be better  positioned to ensure that it is aware of potential trends so it may  determine whether corrective action at the headquarters level is  warranted.

We are making the following two recommendations to TSA:  The TSA Administrator should direct T&D and Security Operations to  document their processes for monitoring TSO training compliance— including those for analyzing training compliance data, reporting the  results from their analysis, and actions taken to address the results.  (Recommendation 1)

The TSA Administrator should direct T&D and Security Operations to  monitor for instances of TSO non-compliance by individual commercial  airports across fiscal years that could potentially warrant corrective action  at the headquarters level. (Recommendation 2)

We provided a draft of our report to DHS for review and comment. In its  comments, reproduced in appendix I, DHS concurred with both of our  recommendations. DHS also provided technical comments, which we  incorporated as appropriate.

With respect to our first recommendation that TSA document its process  for monitoring TSO training compliance, DHS stated that, among other  things, Security Operations will collaborate with T&D to develop and  maintain an internal control mechanism that will document responsibilities  at the field and headquarters level for monitoring TSO training completion  compliance, and actions taken to address the results.

With respect to our second recommendation that TSA monitor for  instances of TSO noncompliance by individual commercial airports across  fiscal years, DHS stated that T&D and Security Operations will begin  monitoring trends in non-compliance at individual airports and for specific  courses. Further, T&D has developed an internal website to share its  findings with Security Operations through monthly compliance reports.

We are sending this report to the appropriate congressional committees  and to the acting Secretary of Homeland Security. In addition, this report  is available at no charge on the GAO website at http://gao.gov.

If you or your staff members have any questions about this report, please  contact me at (202) 512-8777 or russellw@gao.gov. Contact points for  our Offices of Congressional Relations and Public Affairs may be found  on the last page of this report. GAO staff who made key contributions to  this report are listed in appendix II.

Appendix I: Comments from the Department of Homeland Security

Appendix II: GAO Contact and Staff Acknowledgments

In addition to the contact named above, Jason Berman (Assistant  Director), Julia Vieweg (Analyst-in-Charge), Benjamin Crossley, Elizabeth  Dretsch, Michael Dworman, Eric Hauswirth, Susan Hsu, Tom Lombardi,  and Heidi Nielson made key contributions to this report.

TSA screens millions of airline passengers and their bags each day at commercial airports. To do so effectively, it requires its screener workforce to complete trainings on screening procedures and technologies. TSA has updated these trainings in recent years to keep up with emerging threats.

However, we found that TSA has not documented its process for monitoring whether its screeners are taking these required trainings. The agency also hasn’t looked at its training data across years to identify trends that may require action.

We recommended that TSA document its process and analyze screener compliance data across years.