Weapon System Sustainment: DOD Needs a Strategy for Re-Designing the F-35's Central Logistics System

The F-35 is DOD's most ambitious and costly weapon system in history, with U.S. sustainment costs estimated at about $1.2 trillion over a 66-year life cycle. Central to the F-35 is ALIS—a complex system that supports operations, mission planning, supply-chain management, maintenance, and other processes. A fully functional ALIS is critical to the F-35's operational success. However, over the past 5 years GAO has reported on key risks associated with the system, such as challenges deploying the F-35 with ALIS, inaccurate data that reside in ALIS, and ineffective training for personnel who need to use ALIS.

GAO was asked to review DOD's efforts to improve ALIS. This report assesses the extent to which (1) improvements have been made over the past 5 years and challenges remain for ALIS users, and (2) DOD is taking actions to enhance the long-term viability of the system. GAO reviewed F-35 and ALIS program documentation and data, interviewed DOD officials and contractor employees, and visited five U.S. F-35 sites.

The Autonomic Logistics Information System (ALIS) is integral to supporting the F-35 fighter jet's operations and maintenance. F-35 personnel at 5 locations GAO visited agreed that ALIS is performing better in some aspects, such as faster processing speeds for some tasks. However, problems with ALIS continue to pose significant challenges for F-35 personnel (see figure).

The Department of Defense (DOD) has not (1) developed a performance measurement process for ALIS, which GAO recommended in 2014, or (2) determined how ALIS issues affect F-35 fleet readiness. Without efforts in these areas, DOD will be hindered in addressing ALIS challenges and improving aircraft readiness.

DOD and the prime contractor have a variety of initiatives underway for re-designing ALIS. However, these initiatives involve differing approaches and technical and programmatic uncertainties are hindering the re-design effort (see figure).

DOD has not developed a strategy for the future of ALIS that includes goals of the re-design, an assessment of key risks, or costs. Without this, DOD may not be able to coordinate various ALIS design-improvement initiatives that are under way or meaningfully enhance the system over the long term.

GAO is recommending that DOD track how ALIS is affecting readiness of the F-35 fleet and develop a strategy for the ALIS re-design. In addition, GAO believes that Congress should consider requiring DOD to develop a performance measurement process for ALIS. DOD concurred with both of GAO's recommendations.

The F-35 Lightning II aircraft (F-35) provides advanced tactical aviation  capabilities for the Department of Defense (DOD) and is intended to  replace a variety of legacy fighter aircraft in the Air Force, Navy, and  Marine Corps. The F-35 is also DOD’s most ambitious and costly weapon  system in history, with acquisition and sustainment costs for the three  U.S. military services estimated at over $1.6 trillion over a 66-year life  cycle. The Autonomic Logistics Information System (ALIS) is integral to  the more than 3,300 F-35 aircraft that the U.S. military services and  foreign nations plan to purchase. ALIS is a complex system that supports  operations, mission planning, supply-chain management, maintenance,  and other processes. ALIS—described as the critical information  technology element connecting the entire F-35 enterprise—is, according  to one DOD official, one of three major components that make up the F- 35, along with the airframe and engine. It comprises both software and  hardware. The F-35 program has been developing ALIS capabilities  incrementally. A fully functional ALIS is critical to the operational success  of the F-35.

ALIS is intended to enable holistic fleet management, improve speed,  enhance readiness, and reduce costs to the F-35 program. However, we  have reported over the past 5 years on key risks associated with ALIS.

In 2014, we found that ALIS was experiencing recurring problems, including user issues, which were contributing to time-consuming workarounds for maintainers and a backlogged issue-resolution process. Additionally, we found that the F-35 program did not have a process, with metrics and targets, to determine and address the most  significant performance issues with ALIS. As a result, we  recommended that DOD establish a performance measurement  process for ALIS.

In 2016, we reported on several key risks with ALIS, such as  challenges deploying, data accuracy and accessibility issues, and the  lack of a redundant infrastructure in the event of a system failure. We  recommended that DOD develop a plan to prioritize and address ALIS  issues. We also found that training for ALIS was largely ineffective  and lacked a standardized, common curriculum for teaching users  how to operate ALIS. We recommended that DOD develop a  standardized, program-wide plan for ALIS training.

In 2018, based on classified findings, we recommended that the F-35  program test the operation of the F-35 disconnected from ALIS for  extended periods of time in a variety of scenarios to assess the risks  related to operating and sustaining the aircraft.

DOD concurred with these recommendations and has taken some actions  in response, such as developing a training plan for ALIS users.

You asked us to review DOD’s efforts to address concerns from users of  ALIS, improve ALIS functionality, and determine the long-term viability of  the system. This report assesses the extent to which (1) improvements  have been made over the past 5 years and challenges remain for users of  the F-35’s ALIS, and (2) DOD is taking actions to enhance the long-term  viability of the system.

For each of our objectives, we reviewed relevant F-35 sustainment and  ALIS-related data, plans, program briefings, guidance, and other  documentation. We interviewed officials from the Office of the Under  Secretary of Defense for Acquisition and Sustainment; the F-35 Joint  Program Office; the Director, Operational Test and Evaluation; the U.S.  Air Force; the U.S. Navy; the U.S. Marine Corps; and the prime  contractor, Lockheed Martin. We conducted site visits to 5 of the 10 U.S.  F-35 locations—Luke Air Force Base, Edwards Air Force Base, Nellis Air  Force Base, Marine Corps Air Station Yuma, and Naval Air Station  Lemoore. We selected these locations to obtain perspectives from ALIS- users from all U.S. services participating in the F-35 program, and to  include operational, training, and testing locations. Additionally, we  developed and used a data collection instrument to collect ALIS-related  information from users (i.e. maintainers, pilots, supply personnel,  contractors) at all 10 U.S. F-35 locations. Finally, we met with officials  from the F-35 Joint Program Office, MIT Lincoln Labs, Lockheed Martin  Rotary and Mission Systems, Air Force Digital Service, Kessel Run (Air  Force), and others to discuss ALIS-related efforts.

In support of these objectives, we gathered data from the prime  contractor for fiscal year 2019 (October 2018–September 2019), the most  recent complete fiscal year information available for F-35 fleet  performance during our audit timeframes. To determine the reliability of  these data, we collected information from the prime contractor on how the  data were collected, managed, and used. Although we identified some  limitations in the way that the data were being collected and reported that  could potentially result in inaccuracies, we determined that the data were  sufficiently reliable for providing information on the progress and  challenges within the program. For a detailed description of our scope  and methodology, see appendix I.

We conducted this performance audit from August 2018 to March 2020 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

The F-35 Lighting II program is a joint, multinational acquisition program  intended to develop and field a family of next-generation strike fighter  aircraft for the U.S. Air Force, Navy, and Marine Corps (hereinafter  referred to as the services); seven international partners; and four foreign  military sales customers (collectively hereinafter referred to as program  participants). The program has developed and is delivering three variants  of the F-35 aircraft:

F-35A conventional takeoff and landing variant for the Air Force. (see  fig. 1)

F-35B short takeoff and vertical landing variant for the Marine Corps.

F-35C carrier-suitable variant for the Navy.

The characteristics of the services’ variants are similar in that each is  intended to be a multi-role, stealthy strike aircraft, but each service’s  variant also has unique operating requirements. For example, the Marine  Corps requires that the F-35B be capable of operating from aircraft  carriers, amphibious ships, and main and austere operating bases alike,  requiring the ability to conduct short take offs and vertical landings.

DOD initiated the F-35 program in October 2001. Since then, the Marine  Corps and Air Force declared initial operational capability in 2015 and  2016, respectively, while the Navy declared initial operational capability in  February 2019. Operational testing of the F-35 aircraft began in  December 2018 and is currently scheduled to be completed late 2020. At  that time, DOD will make a decision on whether to proceed with plans to  begin full-rate production of the aircraft. DOD has, concurrently, been  fielding and operating a growing fleet of aircraft as part of low-rate initial  production. As of October 2019, more than 435 U.S. and international  aircraft had been fielded and were operating from 19 sites worldwide. By  2023, the global F-35 fleet is expected to expand to more than 1,100  aircraft across 43 operational sites. In total, the program participants plan  to purchase more than 3,300 F-35 aircraft, with the U.S. services planning  to purchase nearly 2,500 of those aircraft. See Figure 2 for a timeline of  anticipated worldwide fleet growth in the F-35 program.

DOD has two primary contractors for the F-35 program: Lockheed Martin  for the overall aircraft system and Pratt & Whitney for the engine. As the  prime contractor for the overall aircraft system, Lockheed Martin  (hereinafter referred to as the prime contractor) is responsible for  managing the F-35 supply chain, depot maintenance, and pilot and  maintainer training, as well as for providing engineering and technical  support. Currently, DOD is contracting for this support with the prime  contractor largely through annual contracts. It plans to transition to  multiple-year, fixed-price, performance-based sustainment contracts  when the program achieves certain condition-based criteria, including the  establishment of critical sustainment capabilities and the government’s  ability to collect and more fully assess performance and cost data. In  addition, the U.S. Air Force, Navy, and Marine Corps have each  established an F-35 integration office or similar construct focused on how  the services will operate and afford the F-35, among other things. Figure  3 depicts how these key stakeholders provide support to the F-35  program participants across the three aircraft variants.

The Autonomic Logistics Information System (ALIS) is a system of  systems that serves as the primary logistics tool to support F-35  operations, mission planning, and sustainment. ALIS is intended to help  maintainers manage tasks including aircraft health and diagnostics,  supply-chain management, and other maintenance events. ALIS  functionality is intended to support many of the F-35 program’s key  performance parameters such as:  Increase sortie generation rate: Number of aircraft sorties launched  in a flight day.

Increase mission reliability: The probability that a system will  perform mission essential functions for a period of time.

Reduce logistics footprint: The size of in-theater logistics support  needed to move and sustain a warfighting force. The footprint  includes all the necessary support needed to maintain the force such  as fuels, parts, support equipment, transportation, and people.

According to DOD officials, ALIS is integral to supporting F-35 operations.  Figure 4 shows some of the key intended capabilities of ALIS. These  capabilities reside in multiple software applications within the system that  perform specific functions for maintainers, pilots, supply personnel, and  data analysts. Lockheed Martin is the prime contractor for ALIS and has  been responsible for developing and managing the capabilities of the  system, as well as developing training materials for F-35 pilots,  maintainers, and supply personnel.

ALIS is co-located with F-35 aircraft both at U.S. military installations and  in theater to support missions and assist with maintenance and resource  allocation. ALIS consists of the overarching system, the applications  housed within it, and the network infrastructure required to provide global  integrated and autonomic support of the F-35 fleet. It comprises both  hardware and software, and supports the flow of unclassified and  classified aircraft-related data. As a system of systems, major  components of ALIS consist of:

The Autonomic Logistics Operating Unit (ALOU). The ALOU is the  central computer unit that all F-35 data are sent through. As part of  the unit, the ALOU consists of two servers that process and store  classified and unclassified data respectively. There is only one ALOU,  and it is owned by the prime contractor.

The Central Point of Entry (CPE). The CPE is a server unit  configured to provide software and data distribution for a country’s  entire F-35 fleet. It is the node between the ALOU and each country’s  Standard Operating Units (generally housed at F-35 installations). The  CPE consists of two servers that process and store classified and  unclassified data respectively. There is typically one operational CPE  per country, although the United States has separate CPEs for its  operational commands and training sites.

The Standard Operating Unit (SOU). The SOU is a server that is  intended to provide all ALIS capabilities to support flying,  maintenance, and training at F-35 installations. Typically, each F-35  squadron has at least one SOU. It is the node local to each F-35  squadron. There are two types of SOUs: a classified SOU that  supports the flow of classified aircraft-related data and an unclassified  SOU that supports the flow of unclassified aircraft-related data.

The Portable Memory Device (PMD). The PMD is informally referred  to as the “brick” that F-35 pilots use to upload information such as  mission planning data. F-35 personnel use the PMD to store mission  and maintenance data generated during flight which may then be  downloaded into the ALIS SOU to support maintenance and mission  debrief activities.

The Portable Memory Device Reader (PMD Reader). The PMD  Reader is a device intended to be used to remove maintenance data,  including health-related codes, off of the Portable Memory Device and  load into the SOU.

The Portable Maintenance Aid (PMA). The PMA is an unclassified  ruggedized laptop used by F-35 maintainers and flight-line  supervisors to view unclassified technical data, and perform and  document maintenance activities.

According to the F-35 program office, the purpose of the server construct  is to support the exchange of information necessary to support the F-35  sustainment enterprise. As of September 2019, according to program  officials, there was one operational ALOU and CPE within the United  States. Each F-35 site in the United States has a varying number of  SOUs depending on the site’s number of aircraft and squadrons. The  SOU was designed to have its components fit into transit cases that can  be carried by two personnel, with each case weighing up to 200 pounds.  The PMDs, PMD Readers, and PMAs reside at the squadron and support  the collection and transfer of unclassified and classified aircraft-related  data. Figure 5 shows how unclassified ALIS data are collected and  transferred from component to component.

As we have previously reported, ALIS has experienced recurring  developmental issues and schedule delays. The development of ALIS  originated in 2002, a year after the start of the F-35 program. However,  the first major ALIS release was not fielded until October 2009, nearly 7  years after initial development began. DOD officials had originally  planned for the version of ALIS that would include all of the capabilities  required to complete developmental testing of the program to be finalized  in 2010. However, this milestone was reached in September 2018, nearly  8 years behind the original schedule. Figure 6 shows the timeline of major  ALIS software version releases and other significant ALIS-related  milestones.

DOD Has Made Some Improvements to ALIS, but Users Continue to Report Significant Challenges

ALIS users from all 5 F-35 locations we visited reported that ALIS has  improved in some aspects over the last 5 years. However, these users  continue to report significant challenges with ALIS that are affecting the  day-to-day operations of the aircraft. DOD is currently unable to assess  the overall performance of ALIS because it has not developed  performance metrics. Additionally, DOD is unaware of how challenges  with ALIS are affecting F-35 fleet-wide readiness.

Users Report Some Improvements with ALIS

According to pilots, maintainers, supply personnel, and contractors at 5  U.S. F-35 locations, ALIS is generally performing better than it was 5  years ago. Specifically, users at all 5 locations stated that data  processing, downloading of information, and screen navigation were  generally faster than previous years. According to users at 1 location, in  previous releases of ALIS, it could take several minutes to complete a  simple function like a screen download. Further, some users also  reported minor functionality improvements within certain ALIS  applications, such as the Computerized Maintenance Management  System, leading to reduced time required to perform actions within  those applications.

We reported in April 2016 that ALIS users had problems accessing data  in ALIS to produce service-specific reports for their squadrons. Users we  spoke to at 4 locations for this report stated that they can now access  some data within ALIS and can generate reports that they previously  could not. For example, users at 1 location said that it was now easier to  export aircraft-related maintenance information from ALIS and put it into  an external spreadsheet.

Additionally, in December 2015, the F-35 program began deploying  software “fixes” to address minor defects in ALIS at F-35 locations in  between major ALIS software version releases, which users at 1 location  said have made improvements to the system. According to the F-35  program office, these software releases, referred to as service packs,  have focused on improving user interface-related flaws that were  discovered during major releases. Service packs provide users more  frequent functionality fixes to the system, preventing them from having to  wait, in most cases, over a year for a major ALIS software release.

Users Continue to Report Significant Challenges Using ALIS

While users at all 5 F-35 locations we visited said that ALIS is performing  better than it was 5 years ago, they also stated that the system still posed  significant challenges to day-to-day F-35 operations. Specifically, users  across the 5 locations we visited stated that seven significant challenges  still exist with ALIS, as shown in table 1.

Many of the challenges cited above are similar to those we reported in  April 2016, including deployability, inefficient issue resolution process,  and data inaccuracies. We recommended at that time that DOD develop  a plan to prioritize and address ALIS issues. DOD concurred and in 2016  developed a plan that identified key areas for system modernization and  sustainment, which included prioritizing issues related to ALIS. While  DOD’s development of this plan is a positive step, significant user issues  persist today, which are discussed in more detail below. Continued  attention on ALIS is needed to make improvements to the system, reduce  the burden on its users, and mitigate risks to operations and  maintenance.

Users at all 5 F-35 locations we visited expressed concern about data  integrity issues related to inaccurate or missing data within ALIS. For  example, users at all the locations said they have had consistent  problems with data related to aircraft parts. Certain F-35 parts have an  associated electronic record, which is used to track the remaining time  before the part must be replaced, among other things. To be cleared for  flight, F-35 policy states that an aircraft must be electronically “complete”  in ALIS, meaning that all of the electronic records from each installed F- 35 part must be entered into ALIS. However, users at all 5 of the locations  we visited told us that electronic records are frequently incorrect, corrupt,  or missing, resulting in ALIS signaling that the aircraft should be  grounded, often in cases where maintainers know that the parts have  been correctly installed and are safe for flight. Users at 1 location said  that within a 6-month period in 2019, they experienced anywhere  between 0 and 400 issues per week related to inaccurate or missing  electronic records. These same users said that it is common for their  squadron leadership to elect to allow an aircraft to fly with over 20  inaccurate or missing electronic records that ALIS signals to ground.  According to users at all 5 locations we visited, squadron leadership (e.g.,  DOD personnel designated by maintenance squadron commanders) may  decide to fly an aircraft with inaccurate or missing electronic records, but  we found that this practice varies by location and type of part.

In June 2019, the Department of Defense Inspector General published a  report on missing electronic records on F-35 spare parts. The report  found that since 2015, F-35 locations have been consistently receiving  spare parts without requisite electronic records. For example, of the 263  spare parts delivered to one location in June 2018, 213 spare parts (81  percent) did not have electronic records.

Due in part to the unreliability of the data in ALIS, users at all 5 F-35  locations we visited have been collecting and tracking information outside  of the system that should be automatically captured in ALIS. Although not  a requirement, users said they need to track information outside of the  system because they do not always trust the data that reside in ALIS.  Users provided examples of critical aircraft data that they are tracking  outside of ALIS—such as aircraft performance data and maintenance  inspection deadlines—and said that manually tracking this information is  a time-intensive process that pulls maintainers away from completing  other aircraft maintenance-related responsibilities. For example, users at  1 location estimated that they spend an average of 5,000 to 10,000 hours  per year manually tracking information that should be automatically and  accurately captured within ALIS.

In addition, there may be risks associated with using information tracked  outside of the system of record to make decisions about the safety and  operational health of aircraft. For example, users at one location said that  there is a danger of overlooking a critical piece of information when key  aircraft data used to determine an aircraft’s status must be tracked  manually using Excel spreadsheets. Users also said that by continuously  ignoring alerts in ALIS caused by missing or inaccurate data, squadrons  could be at risk of ignoring an alert for a legitimate aircraft issue. Finally,  one commander we spoke with said that while his policy is to generally  require maintainers to resolve data issues before releasing an aircraft for  flight, in a wartime scenario, his squadron will carry out missions with  inaccurate or missing ALIS data and assume the subsequent risk that this  may entail.

Users at all 5 F-35 locations we visited cited challenges deploying with  ALIS to forward locations. Users stated that the required hardware for  ALIS is bulky, can be cumbersome to transport, and, when necessary,  difficult to store on a ship. For example, the unclassified and classified  Standard Operating Unit (SOU) servers that are required for collecting  and analyzing aircraft data in ALIS are broken up into a series of  transportable cases. These cases each weigh approximately 200 pounds  and require at least two people to lift. Users from 1 location told us that  they have taken several separate SOU-related cases to support ALIS on  deployments. These servers, as shown in figure 7, require dedicated  transportation to transport them to forward locations, and heavy-duty  equipment to load them on and off of ships. Some users stated that it was  challenging to find space on the ship to store these servers since they  typically require an entire room to function, as well as specific power and  environmental controls.

Additionally, users at all 5 locations stated that limited internet  connectivity can make deployments challenging. Although SOU servers  are critical ALIS hardware components, due to their size, squadrons will  not always take them on deployments. In these instances, internet  connectivity is important to access critical aircraft data from the forward  location and send it back to the squadron’s SOU for processing.  However, internet connectivity can be slow or non-existent at these  locations. In 2018, we recommended that the F-35 Program Executive  Officer should test operating the F-35 disconnected from ALIS for  extended periods of time in a variety of scenarios to assess the risks  related to operating and sustaining the aircraft. DOD concurred with the  recommendation, but as of December 2019, DOD had still not determined  how long the aircraft can safely fly without connectivity to ALIS.

Finally, users at 2 locations stated that contractor support is critical to  supporting deployments. For example, at one location, due to  inaccuracies with parts data in ALIS, the prime contractor prefers to  match every requisite electronic record with its respective spare part prior  to a deployment, which requires significant time and advanced planning.  Furthermore, according to users at another location, due to the  complexities and functionality issues related to ALIS, contractor support is  required on deployments; however, deploying with contractors could  become problematic in a combat scenario. Overall, users at all 5 locations  said that they have completed deployments using ALIS. However,  deployments are challenging and the current deployment preparation  process for ALIS inhibits a military service’s ability to deploy on short  notice.

Users at 4 of 5 F-35 locations we visited stated that ALIS requires more  contractor or military personnel support than originally planned.  According to the F-35’s Operational Requirements Document—the  document that outlines the overall requirements for the F-35 program— ALIS is supposed to help reduce the logistics footprint for the F-35.  However, a 2013 DOD-commissioned study on reducing F-35 costs  stated that the current ALIS support plan already uses 30 percent more  administrators across squadrons and bases than a similarly-scaled IT  implementation would normally require. In addition, current ALIS users  at these 4 locations are finding that as ALIS becomes more mature, even  more personnel are required to support the system’s operations. For  example, according to users at 1 Air Force location, the Air Force  currently relies on about 8 contractor employees to support each ALIS  SOU server, but has determined that this is not sufficient. Users at 2 Air  Force locations stated that until the Air Force can train more military  personnel to support ALIS-related issues, they will need to increase the  number of contractor employees per squadron to support F-35  operations.

Further, users from 1 Air Force location said they have had to assign full- time “ALIS Expeditor” responsibilities to military personnel within the  squadrons to keep track of ALIS-related issues and pressure the  contractor for resolution. Since these roles are not official billets, their  resulting responsibilities are adding to the military personnel’s existing,  non-ALIS related responsibilities on the flight line. Air Force users from 1  location reported that due to inconsistencies within ALIS, they now have  20 full-time ALIS Expeditors to track ALIS-related issues and help ensure  safety of flight for the aircraft. The Marine Corps had originally planned to  maintain ALIS using only military personnel; however, as the numbers of  aircraft and requisite SOUs increased, users at 1 Marine Corps location  said that it was too difficult to develop and retain personnel with ALIS- specific expertise. According to these users, this has resulted in the  Marine Corps needing increased numbers of contractor personnel to  support its squadron operations.

Inefficient F-35 Issue Resolution Process

Users at all 5 F-35 locations we visited said that the process for resolving  F-35 issues within ALIS remains problematic and inefficient. The Action  Request (AR) process requires personnel to use an application within  ALIS to submit an AR about any F-35 problem, including those about  ALIS itself, to the contractor for triaging and ultimate resolution.

In April 2016, we reported that ALIS users thought the AR process did not  allow for the effective reporting and resolution of F-35 aircraft and ALIS  issues. Specifically, users stated that the process did not provide  transparency to all ARs submitted across F-35 locations and placed  responsibility for resolving the requests primarily on the contractor. ALIS  users at 4 locations stated that this remains the case. Users from 3  locations stated that the overall process would be more efficient if they  were able to search ARs submitted by other squadrons across the fleet to  determine if a solution to the problem already exists. Without this ability,  users must submit an AR for every issue and wait for a response that can  sometimes take months. For example, 1 location reported that from  October 2018 through September 2019, F-35 aircraft were grounded for  9,262 hours or 9 percent of possible flight hours, due to unresolved ALIS- related ARs attributed mainly to missing and inaccurate electronic parts  records. Officials from another location reported that during a 6-month  period they had to ground aircraft for 2,200 hours as a result of waiting for  contractors to resolve parts-related ARs. Users from a third location  stated that more transparency in the AR process could reduce reliance on  contractor support, provide a way to address F-35 problems more  efficiently, and reduce costs to the program since DOD incurs a fee each  time an AR is submitted.

Users at all 5 F-35 locations we visited stated that ALIS is not user- friendly or intuitive. While users stated that there have been some limited  improvements to ALIS over the past years, as previously discussed, in  general, users at all 5 locations described ALIS applications as difficult to  navigate. For example, users from 1 location stated that it is more difficult  and time-consuming to search for information on parts in ALIS than in  legacy logistics systems because the information is located in multiple  locations within ALIS.

Additionally, users from all 5 locations said that some of the applications  within ALIS have very slow processing speeds. According to users at 1  location, in some instances, ALIS’s slow applications require maintainers  to work additional hours to complete required maintenance tasks. During  a demonstration of ALIS and its Joint Technical Data application at one of  the locations we visited, we observed maintainers deal with a slow log-in  process, problems filtering and searching for data in an application, and  ultimately having the application freeze and kick them out. Figure 8 shows  a maintainer using a PMA to work in ALIS.

Users at all 5 F-35 locations we visited stated that the training and  mission planning applications within ALIS remain immature. Users at all 5  locations said they are not using the Training Management System  (TMS), an application designed for pilots and maintainers to track training  qualifications and assign personnel to carry out specific tasks, for its  intended purpose. Users from 4 locations said that because of the  ongoing issues with TMS, they are using legacy systems in its place. For  example, one Air Force command released a memorandum in January  2018 allowing some squadrons to use an external legacy system in place  of the TMS application due to shortfalls in TMS functionality, which it  stated had caused excessive work to execute normal operations and  become an unacceptable burden. Marine Corps and Navy users from 2  locations we visited said that they are using other legacy systems to  circumvent the TMS application as well.

Additionally, pilots at 4 locations stated that the Off-Board Mission  Support (OMS) application within ALIS is immature and remains non- intuitive, time consuming, and difficult to navigate. The OMS application is  a key application for pilots to conduct mission planning and debriefing.  Pilots at 2 locations said that they rely on contractors to help them  complete tasks in the application.

Users at all 5 F-35 locations we visited stated that training to learn how to  use ALIS does not provide adequate knowledge or information to fully  prepare users to operate the system. Specifically, users at 3 locations we  visited stated that the training for ALIS does not reflect a realistic  operational environment. Instead, users at all 5 locations stated that  training materials are usually in the form of PowerPoint slides and that  knowledge of ALIS and its functionality is primarily obtained at the  squadron level through on-the-job-training.

In April 2016, we reported that almost every user in the F-35-related focus  groups we conducted at that time noted that they did not learn how to  operate any ALIS applications until on-the-job training began on the flight  line. Users stated that this remains true today. Users at 1 of the locations  we visited stated that learning how to use ALIS in this manner has caused  people to develop their own unique way of operating the system, which  creates an F-35 fleet environment that is using its primary logistics tool in  different ways.

DOD Is Unable to Assess the Performance of ALIS or How the System Is Impacting F-35 Fleet Readiness

DOD Has Not Developed Performance Metrics for ALIS

Although DOD and F-35 program officials agreed that ALIS continues to  provide challenges for users and is generally not performing well, DOD  still has not determined how it wants the system to perform. For example,  officials from the Joint Strike Fighter Integrated Test Force told us that  testing for individual ALIS software version releases focuses primarily on  whether the new version is performing “better” than the previous version.  Specifically, ALIS testers have developed criteria to determine if the  newest version of ALIS is functioning more efficiently than the previous  version by comparing such tasks as screen download times. However,  according to these officials, these tests are not determining if the ALIS  system is performing to a specified standard because DOD has not  defined this standard.

In September 2014, we recommended that DOD develop a performance- measurement process for ALIS that includes, but is not limited to,  performance metrics and targets that (1) are based on the intended  behavior of the system in actual operations and (2) tie system  performance to user requirements. The DOD Systems Engineering  Guide for Systems of Systems states that to fully understand performance  of systems of systems (such as ALIS), it is important to have a set of  metrics that assess the system’s performance and trace back to user  requirements because the system will likely evolve based on incremental  changes—similar to ALIS’s incremental fielding. These metrics should  measure the intended behavior and performance of the system in actual  operations versus the progress of the development of the system,  allowing an assessment of system capabilities based on user  requirements.

After over 5 years, and more than 400 aircraft fielded, DOD has not yet  established a performance-measurement process for ALIS. DOD  concurred with our 2014 recommendation, and repeated its commitment  to develop performance metrics for ALIS after the release of our 2016  report on ALIS risks. In September 2019 program officials told us that  DOD remains in the process of developing these metrics and has no set  timeline for their completion. Without a performance-measurement  process, the F-35 program does not have critical information about ALIS  performance across F-35 locations. Such information could help address  current and future ALIS performance issues and systematically measure  ALIS functionality compared to intended performance.

Problems with ALIS Could Be Affecting Overall F-35 Fleet Readiness

Users at all 5 F-35 locations we visited also stated that problems with  ALIS are affecting the overall readiness of the F-35 fleet; however, they  were unable to tell us the degree to which this is the case. Overall F-35  fleet-wide performance has been falling short of warfighter  requirements—that is, aircraft cannot perform as many missions or fly as  often as required. Figure 9 shows F-35 fleet aircraft performance from  October 2018 through September 2019. Full mission capability, or the  percentage of time during which the aircraft can perform all of its tasked  missions, was 31.6 percent across the fleet, as compared with the  warfighter minimum target of 60 percent. Mission capability, or the  percentage of time during which the aircraft can safely fly and perform at  least one tasked mission, was 59.5 percent across the fleet, as compared  with the warfighter minimum target of 75 percent. Furthermore, citing less  than desirable aircraft performance, in September 2018, the Secretary of  Defense directed the military services to achieve and maintain 80 percent  mission capability rates for their critical aviation platforms, including the F- 35 fleet, by the end of fiscal year 2019.

Two F-35 locations have started tracking information on how ALIS is  affecting F-35 aircraft performance at their locations. Officials from one  location told us that from October 2018 through September 2019, F-35  aircraft were grounded and thus non-mission capable for 16,221 hours, or  2 percent of possible flight hours, as a direct result of issues with ALIS— such as inaccurate or missing electronic records. However, according to  officials at this location, this number does not capture all scenarios in  which ALIS is affecting aircraft performance because sometimes  squadron commanders make decisions to fly an aircraft when ALIS  signals that they should not, in order to fulfill mission requirements.  Officials from another location reported that in fiscal year 2018, ALIS- related issues caused the F-35 aircraft to be non-mission capable for  3,246 hours, or .5 percent of possible flight hours; however, as was the  case with the previous location, officials said that this number also did not  capture all scenarios in which ALIS is affecting aircraft performance.

These limited efforts represent squadron-specific initiatives, as no other  F-35 location has tracked similar ALIS-related data. Further, the data  collected by the two locations only capture non-mission capability rates  when ALIS signals to ground the aircraft and makes the aircraft incapable  of completing a mission. The data do not account for the workarounds  users said they are routinely performing to circumvent a non-functioning  aspect of ALIS in order to get an aircraft ready to fly, or the times when  squadron leadership decides to fly the aircraft when ALIS signals  otherwise.

Different factors can play a role in reducing F-35 aircraft readiness. For  example, in April 2019, we reported that reduced aircraft performance  was due largely to spare parts shortages. This conclusion was drawn  from data that had been collected and tracked by both the contractor and  DOD across the entire fleet to determine non-mission capability rates due  to supply issues. Further, the F-35 program collects data on the degree to  which maintenance issues are affecting F-35 mission capability. And,  there are ongoing efforts to improve F-35 fleet readiness that are  specifically targeted at supply and maintenance issues that are causing  the significant mission-capability degradation. However, users and  program officials stated that recurring issues with ALIS could also be  affecting aircraft performance and noted that data on these issues are not  being collected by the contractor or DOD. Although users reported  multiple instances when ALIS-related issues grounded aircraft, these  issues are being captured and categorized as either supply or  maintenance-related issues, thus masking ALIS’s effect on fleet-wide  readiness.

DOD Instruction 5000.02T, “Operation of the Defense Acquisition  System,” states that the program manager will use technical performance  measures and metrics to assess program progress. It further states that  the analysis of technical performance measures and metrics, in terms of  progress against established plans, will provide insight into the technical  progress and risk of a program like the F-35. In the case of ALIS, the F-35  program does not have a fleet-wide process for measuring, collecting,  and tracking information on how ALIS is affecting the performance of the  F-35 aircraft, such as fleet-wide mission capability rates. Without such a  process, the F-35 program may be limited in its ability to identify all of the  drivers of reduced aircraft performance and appropriate target solutions.

Further, as we previously reported, DOD plans to enter into multi-year,  performance-based F-35 sustainment contracts with the prime contractor,  but may not be well positioned to enter into such contracts because, in  part, it does not fully understand the technical characteristics of the  aircraft. ALIS may or may not be having a notable effect on mission  capability rates for the F-35 fleet. However, without understanding how or  the extent to which ALIS is affecting the performance of the aircraft, DOD  risks entering into long-term, performance-based logistics contracts  without fully understanding all of the factors currently affecting aircraft  operations. This could hinder DOD’s ability to effectively negotiate  performance-related terms of the contract. Finally, without understanding  how ALIS is affecting the performance of the aircraft, DOD risks  developing a performance-measurement process for ALIS that is not tied  to the overall performance goals of the program.

DOD Is Pursuing Actions to Enhance the Long-Term Viability of ALIS, but It Has Not Established a Strategy for the Future System Re- Design

DOD is taking actions to enhance the long-term viability of ALIS. Limited  DOD attention on ALIS has resulted in a troubled history with the system.  As a result, multiple efforts are currently underway to re-design and  attempt to improve ALIS. However, key technical and programmatic  uncertainties hinder these efforts. Furthermore, DOD does not have an  overarching strategy for the future redesign of ALIS.

Limited DOD Attention Has Resulted in a Troubled History with ALIS

As originally envisioned, ALIS was intended to be a first-of-its-kind, fully  autonomic system that would provide users access to data on a range of  capabilities—including operations, maintenance, prognostics, supply  chain, customer support services, training, and technical data—in one  logistics system to support aircraft operations. According to Joint Strike  Fighter Integrated Test Force officials, previous DOD aircraft logistics  systems were much simpler, not fully autonomic, and generally included  data related to fewer major capabilities.

However, the F-35 program office did not clearly specify what it required  from ALIS from the warfighter’s perspective beyond the broad capabilities  to be included in the system. Air Force officials stated that instead, the F- 35 program office relied on the prime contractor to take the lead in  managing the development of the system. For example, the F-35  Operational Requirements Document provides only overarching, high- level requirements for ALIS and does not include specific, user-related  requirements or requirements to adapt and modernize the system over  time. DOD officials acknowledged that historically, DOD has prioritized  other aspects of the F-35 program, such as the development of the  airframe, over its logistics system.

In addition, DOD’s focus with ALIS development over the last 5 years has  largely centered on adding capabilities required to complete  developmental testing for the F-35. As issues with the fielded system  have arisen, DOD and the prime contractor’s approach has generally  been to resolve these issues on a case-by-case basis as available  resources allowed, as opposed to making more costly and time-intensive  improvements to the system’s underlying design and functionality. DOD  contracting officials and prime contractor representatives stated that the  need to balance a limited number of software development personnel  between efforts to stabilize the current system and add new features has  negatively affected the development of ALIS. In a 2017 report, the Air  Force Digital Service recommended that the F-35 program office cease  adding new capabilities in order to re-evaluate ALIS-related design  choices and improve software development processes and procedures.  According to the report, many of the issues with ALIS have known root  causes that are directly related to software and hardware design choices  that are 15 years old. For example, ALIS is made up of siloed applications  that each have their own, sometimes conflicting, databases. Further,  according to the Air Force Digital Service report, efforts to upgrade ALIS  from an out-of-date operating system have not been prioritized by the F- 35 program office. Finally, ALIS hardware is cumbersome, consisting of  heavy servers as well as laptops that were originally designed in the mid- 1990s.

The current approach to developing ALIS has generally led to scheduling  delays and challenges addressing a backlog of ALIS deficiencies. For  example, the ALIS version required to complete developmental testing for  the F-35 was not released until 2018—8 years after the originally planned  release date. F-35 program office officials emphasized that in general, the  timeframe for releasing major software updates for ALIS—up to 18  months—has been long. Further, based on data from the prime  contractor, as of September 2019, there were about 4,700 open ALIS  deficiencies, which are used by the prime contractor to track and manage  issues with the system. According to an F-35 program office official,  ALIS deficiencies may be identified in the field by F-35 users, in the prime  contractor’s testing laboratory, or during DOD-led developmental and  operational testing of the F-35 and ALIS. Of these 4,700 deficiencies,  about 34 percent were identified in 2017 or earlier and 22 percent were  category 1 or category 2 deficiencies. Category 1 deficiencies are  considered critical and could jeopardize safety, security, or another  requirement; category 2 deficiencies are those that could impede or  constrain successful mission accomplishment. As shown in figure 10,  the total number of open deficiencies has generally increased over the  last 2 years. In addition, the number of open category 1 through category  3 deficiencies, which are considered critical or have an adverse effect on  mission accomplishment, generally increased during this period. While  the rate at which the prime contractor closed deficiencies during this  period increased, the rate of increase was generally lower than the rate at  which new deficiencies were identified.

Officials from the Joint Strike Fighter Integrated Test Force and Office of  the Director of Operational Test and Evaluation expressed concerns  about the number and nature of the ALIS-related deficiencies they have  identified during developmental and operational testing. For example, F- 35 testers identified a number of deficiencies with the most recent ALIS  software version, ALIS 3.5, including eight category 1 deficiencies. ALIS  3.5 is referred to as the “stabilization” release because it was intended to  address longstanding issues with ALIS. In addition, F-35 testers stated  that since 2016, they have identified a number of cyber-related ALIS  deficiencies, most of which remain open today. While officials said that  the number of cyber deficiencies is consistent with other DOD weapons  systems, they stressed that a vulnerable ALIS is particularly problematic  because of how interconnected the system is with the F-35 aircraft and its  operations.

Multiple Efforts Are Underway to Re-Design ALIS

DOD and the prime contractor have acknowledged ALIS’s troubled  history and have established three initiatives to re-design and fix ALIS. At  a November 2019 congressional hearing, the F-35 Program Executive  Officer stressed that significant additional work is required to improve  ALIS functionality and that this work cannot be done in old and outdated  ways. Table 2 summarizes the three initiatives, led by the F-35 program  office, Air Force, and prime contractor respectively.

Key Technical and Programmatic Uncertainties Hinder Efforts to Re-Design ALIS

According to the F-35 program office, the three initiatives are  complementary and will eventually be integrated in a final redesign of  ALIS. However, we found that DOD lacks clarity on how it will address  key technical and programmatic uncertainties about the future of the  system (see figure 11). These uncertainties relate to complex aspects of  ALIS that will significantly impact the future design of the system and how  it will be managed. Further, there are divergent views among officials  involved with the various initiatives in terms of how DOD should approach  key aspects of the re-design, highlighting the uncertainty that exists about  the future of ALIS.

DOD has not fully determined what capabilities will be included in the  ALIS re-design. After years of focusing on adding new capabilities with  each major ALIS software version release, DOD officials agreed that that  their current goal is to streamline and simplify ALIS. For example, the  Mad Hatter initiative is designing applications based on the minimum  capabilities required by maintainers to quickly release an aircraft for flight.

Similarly, the ALIS Next initiative is working to optimize functions in ALIS  by identifying aspects of the current design that could be slowing down  the system—for example, transferring an aircraft’s entire digital history  each time the jet is transferred from one SOU to another. However,  officials from the Office of the Director of Operational Test and Evaluation  indicated that there continues to be uncertainty about the capabilities— both classified and unclassified—that will be included in the re-design.

Further, as discussed previously, the F-35 program office has not formally  established how it expects ALIS to perform in operations or developed a  performance-measurement process for ALIS. Program officials indicated  the need for discussions with the services and international partners  about aspects of the current system that are not consistently being used  and may therefore not be required (such as the Training Management  System) through an updated process for establishing ALIS-related  requirements. This process, which requires coordination across all  military services and international partners, has proven to be challenging  in the past. According to a 2017 Air Force Digital Service report, the F-35  program office faces challenges identifying and prioritizing ALIS  capabilities across multiple services and international partners, and this  has negatively affected the development of the system.

DOD is unclear about the extent to which it can adopt a more flexible  software development model known as Agile. As we reported in April  2019, the F-35 program as a whole is pursuing a faster and more  incremental approach for delivering new aircraft capabilities to the  warfighter in order to more flexibly address evolving threats. One  approach to software development that helps facilitate such incremental  delivery is Agile, which calls for the delivery of software in small, short  increments rather than in the typically long, sequential phases of a  traditional software development approach. More a philosophy than a  methodology, Agile emphasizes early and continuous software delivery,  as well as using collaborative teams, and measuring progress with  working software. According to some F-35 program office officials,  adopting Agile could result in a more secure system because it involves  continually testing software for security vulnerabilities. Further, we have  previously reported that following an incremental development approach,  such as Agile, gives agencies the opportunity to obtain additional  feedback from users, which increases the probability that each  successive increment will meet user needs. The Mad Hatter initiative is  experimenting with an Agile approach and has had some initial successes  using this model. For example, in July 2019, we observed a  demonstration of a Mad Hatter-developed application that allows the user  to quickly and easily search through Joint Technical Data, an application  within ALIS that has been reported by some users as being extremely  difficult to navigate. However, the Mad Hatter initiative has operated  outside of F-35 program office policies and processes and its applications  are currently not integrated with the fielded ALIS system. Further, Mad  Hatter and F-35 program office officials said that they have faced  challenges communicating the value of their approach with one another,  and according to a senior Air Force official associated with the Mad Hatter  initiative, the F-35 program office has not clarified the role of Mad Hatter  representatives in current planning efforts aimed at scaling the results of  the Mad Hatter initiative to the entire F-35 enterprise. Separately, as part  of its own ALIS initiative, prime contractor officials said that their company  recently began taking steps to adopt best practices for delivering new  ALIS software using an Agile model. However, these efforts are new, and  the F-35 program office has not developed standards for software  developed by the prime contractor using this model.

DOD officials we spoke with expressed differing views on the extent to  which DOD should adopt an Agile software delivery model for ALIS. For  example, in a 2018 memorandum establishing the Mad Hatter pilot, a  senior Air Force acquisition official stated that the F-35 program should  embrace the tenets of this type of model in order to innovate and rapidly  deliver useful capability through ALIS. Similarly, Air Force, Office of the  Secretary of Defense, and some F-35 program office officials stated that  modernizing ALIS will require DOD to adopt industry best practices by  making decisions quickly, delivering usable products early and often, and  revising plans to reflect experience from completed software iterations. In  contrast, Marine Corps and some F-35 program office officials indicated  that DOD should carefully consider different commercially-available  software tools, as well as DOD-specific constraints, before delivering new  ALIS capabilities. For example, F-35 program office officials associated  with the ALIS Next initiative stated that they conducted an assessment of  the commercial software tools that could be used for new ALIS software  development. These officials said that some of the tools that were initially  being used by the Mad Hatter initiative to develop applications make  software development easier in the short-term but more difficult to switch  toolsets and/or contractors in the long-term. Marine Corps and some F-35  program officials also noted that current DOD processes and  procedures—such as the software certification and cost-estimating  processes—may not be able to support quick software releases. While an  Agile software delivery model has been identified as having the potential  to improve the way in which the federal government develops and  implements IT, we previously reported that this type of model requires  significant procedural and organizational changes in order to be  implemented successfully.

DOD has not made a decision about the extent to which the ALIS re- design will be hosted in the cloud as opposed to onsite servers at the  squadron level. In April 2019, we reported that cloud computing allows  federal agencies to access on-demand, shared computing resources with  the goal of delivering services more quickly and at a lower cost. More  specifically, purchasing IT services through a provider enables agencies  to avoid paying for all of the computing resources (e.g., hardware,  software, networks) that would typically be needed to provide such  services. This approach offers federal agencies a means to buy the  services faster and possibly at less cost than building, operating, and  maintaining these computing resources themselves. However, National  Institute of Standards and Technology guidance states that public cloud  computing represents a significant shift from the norms of on-site data  centers and should therefore be approached carefully with consideration  to the sensitivity of data. While the Mad Hatter initiative has embraced  hosting ALIS in the cloud, including at the squadron level, ALIS Next is  conducting an assessment of the extent to which a cloud-based system is  the best option for ALIS. Further, as part of its internal ALIS investment,  the prime contractor has designed an alternative model to the current  system that includes an onsite server at each F-35 squadron.

Office of the Secretary of Defense, Air Force, and F-35 program office  officials we talked to agreed that the ALIS re-design will involve migrating  some portions of ALIS from onsite servers to the cloud. For example,  these officials agreed that DOD should explore options for migrating the  ALOU and U.S. CPE to the cloud. However, these officials disagreed  about how much of the future system should be cloud-based at the  squadron level. For example, Air Force, Office of the Secretary of  Defense, and some F-35 program office officials stressed that for day-to- day maintenance at U.S. bases, F-35 squadrons should be able to  access ALIS using Wi-Fi, and that the reliance on onsite servers should  therefore be minimal and limited to deployed scenarios. According to  these officials, DOD can achieve significant cost savings by moving ALIS  to the cloud. These officials also indicated that DOD’s hesitation about  moving from onsite servers to the cloud is mostly cultural and the result of  a lack of understanding about what the cloud is. One senior Office of the  Secretary of Defense official with software expertise stated that  warfighters should be able to deploy with a minimal amount of ALIS  hardware (for example, only a high-powered laptop). In contrast, other F- 35 program office officials told us that the F-35 program office is restricted  in the extent to which it can migrate to cloud-based SOUs due to  connectivity and security restrictions. Further, at an ALIS Next  conference, some partner country representatives expressed concerns  about hosting ALIS in the cloud, stating that stringent security  requirements would likely prevent their governments from accepting a  cloud-based solution for ALIS.

DOD does not have a plan for incorporating users early and often in the  development of new ALIS software across the F-35 enterprise. Previous  GAO reports as well as other DOD studies have found that giving users  the opportunity to provide feedback on actual working software early and  often in the software development process, and incorporating that  feedback in subsequent development, is critical to the success of any  software development effort. For example, in March 2019, we reported  that obtaining frequent feedback is linked to reducing risk, improving  customer commitment, and improving technical staff motivation.  Historically, user feedback has not been prioritized in the ALIS software  development process. According to users we talked to, working groups do  exist that serve as a venue for voicing user-related issues; however,  users stated that these working groups meet infrequently and often do not  lead to desired changes. Further, prime contractor representatives told us  that while they recently began soliciting user feedback as part of their  ALIS initiative, the F-35 program office has not contractually required  incorporating user feedback in the ALIS software development process.

The Mad Hatter initiative is currently incorporating user feedback into new  software development for ALIS and has established a process whereby  F-35 users and Mad Hatter software developers can communicate  directly about the Mad Hatter applications that are in development. As  part of this process, Mad Hatter product teams develop simple  applications, field the applications to users, and then use feedback from  users—obtained by email or videoconferences—to adjust and enhance  the applications. Although Mad Hatter’s process for incorporating user  feedback aligns with the practice of incorporating feedback early and  often, the initiative is being executed at one F-35 installation, with one  military service. Further, while the F-35 program office intends to  eventually scale the results of Mad Hatter’s experimentation to the rest of  the F-35 enterprise, it has not formally outlined how it will institutionalize  the initiative’s process for incorporating user feedback across multiple  services and international partners.

DOD has not determined the roles of DOD and the prime contractor in  future ALIS development and management. DOD officials stressed that  historically, the department has relied heavily on the prime contractor to  develop and manage ALIS. Officials also said that moving forward, DOD  will need to play a more active role in the management of ALIS. For  example, Air Force, Office of the Secretary of Defense, and F-35 program  office officials all said that DOD should serve as the primary owner of the  ALIS software system, with the prime contractor and other firms  developing applications that will feed into DOD’s software pipeline.  However, the F-35 program office has not officially named DOD as the  prime ALIS owner, or specified how it will coordinate software  development across these multiple entities. Further, while one of the long- term objectives of the Mad Hatter initiative is to build DOD’s capacity to  manage and develop new ALIS software itself, Air Force officials involved  in this initiative stated that DOD has not yet fully developed this capacity.

As the original ALIS developer, prime contractor representatives stated  that their company is in the best position to modernize ALIS. F-35  program office officials acknowledged that because the prime contractor  plays such a critical role in the development and sustainment of the F-35,  it will be necessary for DOD to work closely with the contractor,  regardless of the direction DOD decides to take. For example, DOD  officials said they have faced challenges obtaining key technical data  from the prime contractor that would be required by DOD to lead ALIS  software development, such as the underlying source code for current  ALIS software, and that they were uncertain about the extent to which  they would be able to obtain these data in the future. At a November  2019 congressional hearing, the Under Secretary of Defense for  Acquisition and Sustainment stressed that many of the challenges with  ALIS stem from the fact that ALIS data are fed back through prime- contractor computers, and there is resulting ambiguity over the ownership  of that data. As we previously reported, DOD continues to lack clarity  about the technical data it owns and the additional data it would require to  maintain flexibility in the sustainment of the F-35.

DOD has not agreed on the extent to which the ALIS re-design will  incorporate current ALIS software—consisting of 8 million lines of code.  As part of the ALIS Next initiative, F-35 program office officials said they  intend to review the underlying source code for ALIS to determine which  aspects of the current software should be integrated in the re-design.  These officials explained that redesigning ALIS software from scratch will  take too long and the future ALIS system will therefore need to  incorporate, to some extent, current ALIS software. In contrast, a senior  Air Force official associated with the Mad Hatter initiative stated that the  initiative intends to replace most current ALIS applications with  commercial or new custom applications, retaining only those ALIS  applications that can be cost-effectively modernized. Further, officials  from the Air Force, Office of the Secretary of Defense, and F-35 program  office indicated that because most of the ALIS source code has not been  updated in years and contains numerous security vulnerabilities, the  software should be completely re-designed.

DOD Does Not Have a Strategy for the Future Re- Design of ALIS

DOD is unclear about how it will approach the key technical and  programmatic uncertainties surrounding ALIS because the department  has not developed a strategy for the future re-design of the system. DOD  guidance for program managers states that a sound strategy requires,  among other things, a clear articulation of program goals as well as an  understanding of the risks or uncertainties and costs associated with  achieving those goals.

While DOD and the prime contractor have established various initiatives  to re-design ALIS, DOD has not developed a strategy for the future of  ALIS that clearly identifies and assesses goals, key risks or uncertainties,  and associated costs. For example, as discussed previously, DOD lacks  clarity about the goals of the re-design, such as the capabilities that will  be included in the future system and the extent to which ALIS will be  hosted in the cloud. In addition, DOD has not fully assessed key risks or  uncertainties, including the extent to which DOD can adopt an Agile  software development approach or manage the system itself. Finally,  because it has not answered key questions about the future of the  system, such as the extent to which the re-design will incorporate current  ALIS software, DOD has not been able to develop accurate cost  estimates for the ALIS re-design.

In the past, DOD has faced challenges estimating and tracking ALIS  costs. For example, in 2016 we reported that while DOD had estimated  that ALIS would cost approximately $17 billion, the estimate was not fully  credible because DOD had not performed uncertainty and sensitivity  analyses as part of the cost-estimating process. Further, for this review,  the F-35 program office was not able to provide us with historic costs  showing how much the department has spent on ALIS over the years.

DOD officials stated that historically, the department has faced challenges  allocating scarce resources across competing priorities, and that the F-35  air vehicle has generally been prioritized over ALIS. With the completion  of F-35 developmental testing in April 2018, program officials said they  are now in a better position to focus on ALIS and address long-standing  issues with the system. However, efforts to correct ALIS are relatively  new and have not been fully developed.

Without a strategy to guide the re-design of ALIS, DOD will not be able to  effectively plan for the transition from the current system to a future one.  For example, according to F-35 program office officials, DOD recently  procured additional hardware for the current system, which officials said  may not be required if DOD is able to develop and field a re-designed  ALIS in the near term. Officials from the Office of the Director of  Operational Test and Evaluation stressed that effectively transitioning  from the current system to a future one will be particularly challenging for  DOD given the need to continue sustaining the more than 400 aircraft that  have already been fielded with current ALIS. Further, as discussed  above, there are divergent views in terms of how DOD should approach  key technical and programmatic aspects of the re-design, and integrating  the different efforts that are underway to fix ALIS—led by the F-35  program office, Air Force, and prime contractor—will therefore require significant direction and leadership. Without a strategy, DOD may not be  able to effectively coordinate and leverage the different ALIS initiatives  that are underway, potentially leading to inefficiencies. DOD also risks  repeating history by failing to clearly articulate what it expects from ALIS  and how it will play a more active role in the management of the system  going forward.

The F-35 aircraft, with its advanced warfighting capabilities, provides  critical tactical aviation for the Department of Defense. However, DOD will  need to overcome substantial challenges related to ALIS if it wants to find  successes in both sustainment and operations of the aircraft. Current  ALIS users continue to report significant challenges with the system that  are affecting day-to-day operations of the aircraft, adding additional flight  line-related responsibilities, and, in some instances, causing squadron  leadership to assume the risk of flying aircraft when ALIS tells them to  stay on the ground. Although ALIS is not currently performing well, over 5  years after we recommended it, DOD has yet to establish a performance- measurement process that would define how ALIS should perform. In the  absence of such a process, DOD will be challenged to address current  and future ALIS-performance issues because it cannot measure ALIS  functionality compared to intended system performance. Furthermore,  ALIS users collectively agree that the issues with ALIS are affecting the  readiness of the aircraft; however, the degree to which this is true  remains unknown. Fleet-wide mission capability rates for the F-35 are still  below the warfighter’s minimum targets, but DOD does not have a  process for measuring, collecting, and tracking information on how ALIS  is affecting these rates. Without such a process, DOD may not  understand all of the factors behind the reduced aircraft performance,  thus limiting its ability to target appropriate solutions.

DOD officials have acknowledged the ongoing challenges with ALIS and  know that the system, as it stands today, cannot be sustained into the  future; therefore, it is positive that the department has embarked on  efforts to re-design and fix ALIS, as well as take on a more active role in  the management of the system. However, DOD faces a significant  challenge as there are several complex technical and programmatic  uncertainties that will need to be resolved before any future ALIS solution  can be realized. Additionally, there are divergent views among ALIS  stakeholders about how to go about addressing these complex issues.  The future of ALIS remains unclear because the department has not  developed a strategy for the re-design of the system that would identify,  among other things, what the system should look like, how will it be  developed and managed, how it will address key risks, and how much it  will ultimately cost. Without such a strategy, DOD will not be able to  effectively plan for the transition from the current ALIS system, which is  already embedded in over 400 aircraft across the global F-35 fleet, to  whatever solution is determined. Furthermore, a strategy would help align  what is currently a chorus of divergent views within the department on  how to address the future of ALIS. With the worldwide fleet expected to  grow to over 1,000 aircraft over the next four years, and with the U.S.  services becoming increasingly reliant on the F-35’s capabilities to  support their operational strategies, it will be imperative for DOD to  address the ongoing issues related to the F-35’s logistics system.

Congress should consider legislation requiring the Department of  Defense to establish a performance-measurement process for ALIS that  includes, but is not limited to, performance metrics and targets that (1) are  based on intended behavior of the system in actual operations and (2) tie  system performance to user requirements. (Matter for Consideration 1)

We are making the following two recommendations to DOD:  The Secretary of Defense should ensure the Under Secretary of Defense  for Acquisition and Sustainment, in consultation with the F-35 Program  Executive Officer, develops a program-wide process for measuring,  collecting, and tracking information on how ALIS is affecting the  performance of the F-35 fleet to include, but not be limited to, its effects  on mission capability rates. (Recommendation 1)

The Secretary of Defense should ensure the Under Secretary of Defense  for Acquisition and Sustainment, in consultation with the F-35 Program  Executive Officer, develops and implements a strategy for the re-design  of ALIS. The strategy should be detailed enough to clearly identify and  assess the goals, key risks or uncertainties, and costs of re-designing the  system. (Recommendation 2)

We provided a draft of this report to DOD for review and comment. In its  written comments, reproduced in appendix II, DOD concurred with our  recommendations and identified actions that it was taking or planned in  response. We agree that DOD is taking positive steps in addressing  issues with ALIS, including the decision to replace ALIS with a future  system that it has named the F-35 Operational Data Integrated Network  (ODIN). According to DOD, the department is currently developing a  strategy that will guide ODIN’s development. As DOD proceeds with  replacing ALIS with ODIN, it will be imperative for the department to  carefully consider and assess the key technical and programmatic  uncertainties discussed in this report. These issues—including how much  of ALIS will be incorporated in ODIN and the extent to which DOD has  access to the data it needs to play a more active role in the management  of the system—are complex, and will require significant direction and  leadership to resolve.

As agreed with your office, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 7 days from the  report date. At that time, we will send copies of this report to  congressional requesters; the Secretary of Defense; the Under Secretary  of Defense for Acquisition and Sustainment; the F-35 Program Executive  Officer; the Secretaries of the Air Force and Navy; and the Commandant  of the Marine Corps.  In addition, the report will be available at no charge  on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-9627 or maurerd@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. Staff members making key contributions to  this report are listed in appendix III.

For each of our objectives, we reviewed relevant F-35 sustainment and  the Autonomic Logistics Information System (ALIS)-related data, plans,  program briefs, guidance, and other documentation and collected  information by interviewing officials from the Office of the Under Secretary  of Defense for Acquisition and Sustainment, the F-35 Joint Program  Office, the Director, Operational Test and Evaluation, the Defense  Contract Management Agency, the U.S. Air Force, the U.S. Navy, the  U.S. Marine Corps, the Air Force Digital Service, and the prime  contractor, Lockheed Martin. To interview officials and observe ALIS- related operations, we conducted site visits to five F-35 locations—Luke  Air Force Base, Arizona; Edwards Air Force Base, California; Nellis Air  Force Base, Nevada; Marine Corps Air Station Yuma, Arizona; and Naval  Air Station Lemoore, California. We selected these locations to obtain  perspectives from ALIS-users (i.e. maintainers, pilots, supply personnel,  contractors) from all U.S. services participating in the F-35 program,  including from operational, training, and testing locations. Additionally, we  developed a data collection instrument to collect ALIS-related inputs and  data from ALIS-users (i.e. maintainers, pilots, supply personnel,  contractors) at all 10 U.S. F-35 locations—Luke Air Force Base, Arizona;  Edwards Air Force Base, California; Nellis Air Force Base, Nevada;  Marine Corps Air Station Yuma, Arizona; Naval Air Station Lemoore,  California; Hill Air Force Base, Utah; Naval Air Station Patuxent,  Maryland; Eglin Air Force Base, Florida; Marine Corps Air Station  Beaufort, South Carolina; and Marine Corps Air Station Iwakuni, Japan.  Finally, we met with officials from the F-35 Joint Program Office,  Massachusetts Institute of Technology (MIT) Lincoln Labs, Lockheed  Martin Rotary and Mission Systems, Air Force Digital Service, Kessel Run  (Air Force), and others to discuss ALIS-related improvement efforts.

In support of our objectives, we gathered data from fiscal year 2019 (the  most recent full fiscal year of data available at the time of our review) from  the prime contractor on the performance of the F-35 fleet such as the full  and mission capability rates. We also collected the most recent available  information on ALIS software deficiencies. To determine the reliability of  these data, we collected information on how the data were collected,  managed, and used through a questionnaire and interviews. Although we  identified some limitations in the way that certain data are being collected  and reported—such as data related to aircraft performance like mission  capability rates—we determined that they are sufficiently reliable for the  way in which we reported them and our purposes of providing information  on the progress and challenges within the program. All the performance  data presented in our report are sufficiently reliable to provide a general  comparison of capabilities to minimum targets.

To assess the extent to which there have been improvements as well as  key challenges with ALIS over the last 5 years, we interviewed officials  and examined guidance and briefing documents from the Office of the  Under Secretary of Defense for Acquisition and Sustainment, the U.S.  Services, the F-35 Joint Program Office, the Defense Contract  Management Agency and Lockheed Martin Rotary and Mission Systems  officials to discuss the current status of the system and plans for  mitigating risks.

To determine user views on risks to (or issues with) ALIS, we interviewed  officials at our 5 selected bases, conducted a short data collection  instrument of the other 5 bases, interviewed officials at Air Force  headquarters and the contractor, and reviewed relevant documents. At  the 5 bases, we interviewed groups of pilots, maintainers and supply  personnel about ALIS performance, challenges, and possible  improvements. In addition, we posed several targeted questions based on  risks found in our last report. In total, we received input from more than  160 users at the 5 bases we visited through group discussions or  interviews.

We analyzed the responses provided in these group interviews, and  identified the issues/risks that at least one set of users reported at each of  the 5 bases. We also considered any improvements that were described  as having occurred during the last few years. We also compared the  responses from the interviews at the 5 bases with our data collection  responses, and the other testimonial and documentary evidence we  obtained. The list of issues/risks we identified contains some that were  reported in our 2016 report as well as some new ones. While this list  summarizes the types of issues/risks described at the 5 bases, and also  in other interviews and document review, individual user views and  experiences could vary by base and user group.

We also interviewed officials and reviewed reports from the Air Force  Audit Agency, the Director, Operational Test and Evaluation, and the  Department of Defense Inspector General to identify improvements as  well as any functionality issues with ALIS. We interviewed and gathered  information from DOD officials on testing for ALIS, metrics on ALIS’s  performance, and the operations of the system.

As discussed previously, we collected and analyzed data for fiscal year  2019 that we obtained from the prime contractor on the overall aircraft  performance such as the full mission capability and mission capability  rates. We analyzed and compared information obtained from interviews,  site visits, data collection instruments, and documents with guidance such  as DOD’s System Engineering Guide for System of Systems to determine  the extent to which DOD has an effective procedure for addressing and  mitigating specific risks and challenges that may be associated with a  major weapon system. We also compared this information with previous  GAO reports from 2014, 2016, and 2018 to determine the extent to which  DOD has addressed our prior recommendations on ALIS-related issues.

To assess the extent to which the F-35 program has addressed issues  with ALIS, we gathered and analyzed data from the prime contractor on  open and closed ALIS deficiencies identified from November 2017  through October 2018. We selected this timeframe because it included  the most recent data on ALIS deficiencies at the time of our review and  also allowed us to observe trends in ALIS deficiencies over a two-year  period. The data we received included summary information on the total  number of open deficiencies, the total number of closed deficiencies, the  number of newly closed deficiencies, the number of newly identified  deficiencies, and the total number of open category 1 through category 3  deficiencies (considered critical or adverse) for each month during the  two-year period. To determine the reliability of these data, we conducted  electronic tests to identify any internal inconsistencies with the data. We  also reviewed documentation from the prime contractor on the  management of ALIS deficiency data and collected information on how  the data were collected, managed, and used through a questionnaire.  Specifically, we asked questions about inconsistencies we identified  through electronic testing of the data, the extent to which the prime  contractor’s system for collecting deficiency information includes edit  checks or controls to help ensure the data are entered accurately, and  limitations related to the accuracy or completeness of the data. As a  result, we determined the data to be sufficiently reliable for the purpose of  reporting trends in the number of open and closed ALIS deficiencies over  time.

To determine the extent to which DOD is taking actions to enhance the  long-term viability of the system, we interviewed officials and reviewed  guidance and/or planning documents from the Office of the Under  Secretary of Defense for Acquisition and Sustainment, the F-35 Joint  Program Office, and the Office of the Assistant Secretary of the Air Force  for Acquisition, Technology, and Logistics. We interviewed officials from  the prime contractor to determine their role in helping DOD mitigate risks  regarding the long-term viability for ALIS. Additionally, we examined  briefing documents from the MIT-Lincoln Labs, a federally-funded  research and development center assisting the F-35 Joint Program  Office, on plans, timelines, and risks for modernizing the hardware and  software.

We interviewed officials from the Air Force’s Kessel Run team to discuss  their Mad Hatter initiative (intended to improve ALIS functionality), the  viability of current ALIS software, and any risks associated with the future  of ALIS. We conducted a site visit to Nellis Air Force Base to observe the  Mad Hatter initiative and discuss its results and the future of ALIS  software. Further, as discussed previously, we analyzed data from  November 2017 through October 2019 on ALIS deficiencies. We  reviewed reports and interviewed officials from the Air Force Digital  Service and the Director, Operational Test and Evaluation on the future  viability of these long-term initiatives for ALIS. Finally, we analyzed and  compared information obtained from interviews, site visits, and  documents with applicable guidance to determine the extent to which  DOD has an effective long-term plan for ALIS that addresses operational  and financial risks.

In support of our work, we interviewed officials from the following DOD  organizations and other organizations during our review. We selected  these organizations based on their oversight, planning, and/or execution  roles related to F-35 ALIS operations.

Office of the Under Secretary of Defense for Acquisition and  Sustainment, Arlington, Virginia

Office of the Director for Operational Test and Evaluation, Arlington,

Defense Contract Management Agency Lockheed Martin, Orlando,

F-35 Joint Program Office, Arlington, Virginia

Office of the Assistant Secretary of the Air Force for Acquisition,

Air Force F-35 Integration Office, Arlington, Virginia

Kessel Run Team, Hanscom Air Force Base, Massachusetts

Navy F-35 Integration Office, Arlington, Virginia

Air Force Digital Service, Arlington, Virginia

Lockheed Martin Rotary and Mission Systems, Orlando, Florida

MIT Lincoln Laboratory, Lexington, Massachusetts  We conducted this performance audit from August 2018 to March 2020 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides and reasonable basis for our findings and  conclusions based on our audit objectives.

Diana Maurer, (202) 512-9627, maurerd@gao.gov  In addition to the contact named above, Alissa Czyz (Assistant Director),  Matthew Bader, Vincent Buquicchio, Tracy Burney, Juana Collymore,  Martin De Alteriis, Michael Holland, Jeff Hubbard, Clarice Ransom, and  Elisa Yoshiara made key contributions to this report.

F-35 Aircraft Sustainment: DOD Faces Challenges in Sustaining a  Growing Fleet. GAO-20-234T. Washington, D.C.: November 13, 2019.

Space Command and Control: Comprehensive Planning and Oversight  Could Help DOD Acquire Critical Capabilities and Address Challenges.  GAO-20-146. Washington, D.C.: October 30, 2019.

F-35 Joint Strike Fighter: Action Needed to Improve Reliability and  Prepare for Modernization Efforts. GAO-19-341, Washington, D.C.: April  29, 2019.

F-35 Aircraft Sustainment: DOD Needs to Address Substantial Supply  Chain Challenges.GAO-19-321. Washington, D.C.: April 25, 2019.

Cloud Computing: Agencies Have Increased Usage and Realized  Benefits, but Cost and Savings Data Need to Be Better Tracked.  GAO-19-58, Washington, D.C.: April 4, 2019.

DOD Space Acquisitions: Including Users Early and Often in Software  Development Could Benefit Programs. GAO-19-136. Washington, D.C.:  March 18, 2019.

F-35 Joint Strike Fighter: Development Is Nearly Complete, but  Deficiencies Found in Testing Need to Be Resolved. GAO-18-321.  Washington D.C.: June 5, 2018.

Warfighter Support: DOD Needs to Share F-35 Operational Lessons  Across the Military Services. GAO-18-464R, Washington D.C.: April 25,  2018.

Military Aircraft: F-35 Brings Increased Capabilities, but the Marine Corps  Needs to Assess Challenges Associated with Operating in the Pacific.  GAO-18-79C. Washington, D.C.: March 28, 2018.

Information Technology Reform: Agencies Need to Improve Certification  of Incremental Development. GAO-18-148. Washington, D.C.: November  7, 2017.

F-35 Aircraft Sustainment: DOD Needs to Address Challenges Affecting  Readiness and Cost Transparency. GAO-18-75. Washington D.C.:  October 26, 2017.

F-35 Joint Strike Fighter: DOD’s Proposed Follow-on Modernization  Acquisition Strategy Reflects an Incremental Approach Although Plans  Are Not Yet Finalized. GAO-17-690R. Washington, D.C. August 8, 2017.

F-35 Joint Strike Fighter: DOD Needs to Complete Developmental  Testing Before Making Significant New Investments. GAO-17-351.  Washington, D.C.: April 24, 2017.

F-35 Joint Strike Fighter: Continued Oversight Needed as Program Plans  to Begin Development of New Capabilities. GAO-16-390. Washington,  D.C.: April 14, 2016.

F-35 Sustainment: DOD Needs a Plan to Address Risks Related to Its  Central Logistics System. GAO-16-439. Washington, D.C.: April 14, 2016.

F-35 Joint Strike Fighter: Preliminary Observations on Program Progress.  GAO-16-489T. Washington, D.C.: March 23, 2016.

F-35 Joint Strike Fighter: Assessment Needed to Address Affordability  Challenges. GAO-15-364. Washington, D.C.: April 14, 2015.

F-35 Sustainment: Need for Affordable Strategy, Greater Attention to  Risks, and Improved Cost Estimates. GAO-14-778. Washington, D.C.:  September 23, 2014.

F-35 Joint Strike Fighter: Slower Than Expected Progress in Software  Testing May Limit Initial Warfighting Capabilities. GAO-14-468T.  Washington, D.C.: March 26, 2014.

F-35 Joint Strike Fighter: Problems Completing Software Testing May  Hinder Delivery of Expected Warfighting Capabilities. GAO-14-322.  Washington, D.C.: March 24, 2014.

F-35 Joint Strike Fighter: Restructuring Has Improved the Program, but  Affordability Challenges and Other Risks Remain. GAO-13-690T.  Washington, D.C.: June 19, 2013.

F-35 Joint Strike Fighter: Program Has Improved in Some Areas, but  Affordability Challenges and Other Risks Remain. GAO-13-500T.  Washington, D.C.: April 17, 2013.

F-35 Joint Strike Fighter: Current Outlook Is Improved, but Long-Term  Affordability Is a Major Concern. GAO-13-309. Washington, D.C.: March  11, 2013.

Software Development: Effective Practices and Federal Challenges in  Applying Agile Methods. GAO-12-681. Washington, D.C.: July 27, 2012.

Joint Strike Fighter: DOD Actions Needed to Further Enhance  Restructuring and Address Affordability Risks. GAO-12-437. Washington,  D.C.: June 14, 2012.

Joint Strike Fighter: Restructuring Added Resources and Reduced Risk,  but Concurrency Is Still a Major Concern. GAO-12-525T. Washington,  D.C.: March 20, 2012.

Joint Strike Fighter: Implications of Program Restructuring and Other  Recent Developments on Key Aspects of DOD’s Prior Alternate Engine  Analyses. GAO-11-903R. Washington, D.C.: September 14, 2011.

Joint Strike Fighter: Restructuring Places Program on Firmer Footing, but  Progress Is Still Lagging. GAO-11-677T. Washington, D.C.: May 19,  2011.

Joint Strike Fighter: Restructuring Places Program on Firmer Footing, but  Progress Still Lags. GAO-11-325. Washington, D.C.: April 7, 2011.

Joint Strike Fighter: Restructuring Should Improve Outcomes, but  Progress Is Still Lagging Overall. GAO-11-450T. Washington, D.C.:  March 15, 2011.

The F-35 is the Department of Defense’s most ambitious and costly weapon system. It’s also supposed to be one of the smartest—relying on its Autonomic Logistics Information System (ALIS) to manage missions, maintenance, training, and more.

However, after years of development and testing, the system does not work properly—with inaccurate or missing data sometimes leading ALIS to ground flight-ready aircraft, among other problems.

DOD knows the system needs to be re-designed, but has not answered critical questions about the future system. We recommended that DOD clearly identify the associated goals, risks, and costs of re-designing ALIS.