Defense Acquisitions: Senior Leaders Should Emphasize Key Practices to Improve Weapon System Reliability

DOD invests tens of billions of dollars each year in major defense acquisition programs, designing and developing technologically advanced weapon systems that warfighters expect will meet specific performance requirements, including reliability requirements. Systems that are not reliable make it more difficult for warfighters to perform their missions.

GAO was asked to examine DOD weapon system reliability. This report addresses (1) how selected companies in the commercial sector address reliability, (2) how selected DOD acquisition programs addressed reliability, and (3) the extent to which DOD leadership has highlighted key reliability practices.

GAO collected information on leading commercial practices at the 2019 Reliability and Maintainability Symposium and from four commercial companies known for delivering reliable products. GAO also assessed how seven DOD acquisition programs—both older and newer, and representing all the military services—addressed reliability; reviewed key documents and interviewed knowledgeable officials; and reviewed reliability-related guidance and policy from senior DOD leaders.

The commercial companies GAO reviewed proactively address reliability. They strive to identify reliability issues at the component level early in the development process to avoid expensive rework after producing an entire system. GAO found these companies focus on the following key practices:

1. Leveraging reliability engineers early and often

2. Establishing realistic reliability requirements

3. Emphasizing reliability with their suppliers

4. Employing reliability engineering activities to improve a system's design throughout development

GAO found that the seven Department of Defense (DOD) acquisition programs it reviewed did not consistently adhere to these key practices (see figure). These programs often prioritized schedule and cost over incorporating the key reliability practices, and these systems generally were not as reliable as promised.

In 2019, DOD highlighted in a policy memorandum the importance of emphasizing reliability with contractors. However, the other three key practices have not been similarly highlighted. DOD has taken steps to accelerate weapon system development, and decision-making authority has been delegated to the military services. In an environment emphasizing speed, without senior leadership focus on a broader range of key reliability practices, DOD runs the risk of delivering less reliable systems than promised to the warfighter and spending more than anticipated on rework and maintenance of major weapon systems.

GAO recommends the Secretaries of the Air Force, Army, and Navy highlight the importance of three key reliability practices: leveraging reliability engineers, establishing realistic reliability requirements, and employing reliability engineering activities to improve a system's design throughout development. DOD agreed with GAO's recommendations.

The Department of Defense (DOD) invests tens of billions of dollars each  year designing and developing technologically advanced major defense  acquisition systems which are expected to meet specific performance  requirements. These performance requirements focus on capabilities like  range and survivability, but law and DOD policy also mandate that  defense weapon system acquisition programs address the performance  requirement for reliability. Reliability is the probability that a system will  perform without failure over a particular timeframe and under specified  conditions. For example, reliability requirements often address how long  an aircraft or land vehicle should operate before needing repair. A  weapon system’s reliability directly affects a warfighter’s ability to  complete a mission, and how much DOD must spend to operate and  support the weapon system over its lifetime, which often spans decades.  Nonetheless, DOD acquisition programs continue to struggle to deliver  reliable weapon systems. For example, DOD’s Director, Operational Test  and Evaluation (DOT&E) has identified poor reliability as the reason for  some acquisition programs not delivering suitable weapon systems to the  warfighter.

Decisions made early in the acquisition process influence reliability  throughout a system’s life cycle, and members of Congress have  expressed concerns that DOD does not focus adequate attention on  reliability when it is designing and developing weapon systems. Congress  has passed legislation related to weapon system reliability. The fiscal  year 2017 National Defense Authorization Act (NDAA) contained a  provision that established an Under Secretary of Defense for Acquisition  and Sustainment (USD(A&S)). Weapon system reliability has a  significant impact on sustainment efforts. The following year, the fiscal  year 2018 NDAA included a provision mandating that DOD program  managers include certain reliability requirements in weapon system  engineering and manufacturing development and production contracts.  For its part, DOD has also demonstrated an increased focus on weapon  system reliability recently. In January 2019, the USD(A&S) working with  the Under Secretary of Defense for Research and Engineering  (USD(R&E)), according to a DOD official, issued a policy memorandum to  senior acquisition leaders at the military services addressing several  reliability-related concerns involving requirements, contract solicitations,  and data collection.

While there has been an increased focus on reliability, Congress provided  DOD with additional tools to enable DOD to accelerate weapon system  acquisitions. As we reported in June 2019, the 2016 NDAA devolved  much of the decision-making authority for major defense acquisition  programs from the Office of the Secretary of Defense (OSD) to the  military services. The 2016 NDAA required DOD to issue guidance  establishing two new streamlined acquisition pathways for DOD—rapid  prototyping and rapid fielding—under the broader term “middle tier of  acquisitions.” According to the Joint Explanatory Statement  accompanying the 2016 NDAA, the guidance was to create an expedited  and streamlined “middle tier” of acquisition programs intended to be  completed within 5 years. Programs using this authority are generally to  be exempt from DOD’s traditional acquisition and requirements  development policies.

You asked us to complete a body of work examining DOD weapon  system sustainment issues, including weapon system reliability. In this  report, we discuss (1) how selected companies in the commercial sector  address reliability, (2) how selected DOD acquisition programs addressed  reliability, and (3) the extent to which DOD leadership has highlighted key  reliability practices.

To examine how companies in the commercial sector address reliability,  we collected information including presentations, papers, and tutorials  from leading reliability engineers at the 2019 Reliability and  Maintainability Symposium, an annual premier event in the area of  reliability engineering. We attended in-depth sessions at the Symposium  on a number of reliability-related topics. Symposium participants included  representatives from commercial industry, academia, and government.  We also selected a non-generalizable sample of four companies with  known success in demonstrating reliability based on literature searches  and information obtained at the 2019 Reliability and Maintainability  Symposium. These companies won awards in the areas of reliability,  dependability, performance, or quality or have been recognized as  experts in reliability skill development. We met with representatives from  the four commercial companies to discuss reliability engineering and  collect documentation about the practices they use to develop reliable  products. We identified themes mentioned from these sources and used  these to select the practices most frequently mentioned across all  sources as key practices. The four commercial companies we spoke with  are listed below in table 1.

To identify how selected DOD acquisition programs addressed reliability,  we assessed a non-generalizable sample of seven major defense  acquisition programs. We selected both older programs that have publicly  reported reliability problems, including problems we have previously  identified, as well as programs that started more recently, to see how they  were addressing reliability during acquisition. We reviewed key  documentation, including operational test reports, Reliability, Availability,  and Maintainability and Cost Rationale Reports, Life-Cycle Sustainment  Plans, Systems Engineering Plans, Test and Evaluation Master Plans,  and interviewed knowledgeable officials, including reliability engineers,  testing officials, and program managers. We selected a mix of older and  more recent acquisition programs representing all of the military services  to account for changes in the acquisition environment. We excluded Navy  shipbuilding programs because we have ongoing work in this area. Figure  1 shows the DOD acquisition programs included in our review.

To understand the extent to which DOD leadership has highlighted key  reliability practices, we examined and compared the following documents  to the key reliability practices identified in the commercial sector: DOD  Instruction 5000.02, Operation of the Defense Acquisition System; the  Systems Engineering Plan Preparation Guide; the Life Cycle Sustainment  Plan Guide; DOD Product Support Manager Guidebook; DOD Operating  and Support Cost Management Guidebook; DOD Reliability, Availability,  Maintainability, and Cost Rationale Report Manual; DOD Guide for  Achieving Reliability, Availability, and Maintainability; DOD Handbook on  Reliability Growth Management; the 2018 Joint Capabilities Integration  and Development System Manual; the January 2019 USD(A&S)  Memorandum on Sustainment Factors in Weapon System Design; and  service level reliability guidance. We also met with DOD officials— including the primary proponent for weapon system reliability from OSD  and each of the military services—to discuss the extent to which these  documents reflected key reliability practices and DOD’s ongoing efforts to  improve the reliability of its weapon systems.

We conducted this performance audit from July 2018 to January 2020 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

DOD acquires new weapon systems for its warfighters through a  management process known as the Defense Acquisition System. This  system is implemented by two key acquisition policies: DOD Directive  5000.01, which establishes the overarching framework for the Defense  Acquisition System; and DOD Instruction 5000.02, which provides  detailed procedures for the operation of the Defense Acquisition System  and the management of acquisition programs. These policy documents  establish the guiding principles for all aspects of the DOD acquisition  process. Additionally, each of the military services has its own acquisition  policies which incorporate and enhance the DOD acquisition guidance.  Figure 2 depicts DOD’s acquisition process beginning with Milestone A in  general terms.

Several entities in the Office of the Secretary of Defense and the military  departments play a role in the oversight of DOD weapon system  acquisition programs, including the following:

The Under Secretary of Defense for Research and Engineering is  responsible for establishing policies on and supervising all aspects of  defense research and engineering, technology development,  technology transition, prototyping, experimentation, and  developmental testing activities and programs, including the allocation  of resources for defense research and engineering. DOD’s Reliability  and Maintainability Engineering lead reports to this Under Secretary.

The Under Secretary of Defense for Acquisition and Sustainment is  responsible for establishing policies on and supervising all matters  relating to acquisition (including (1) system design, development, and  production; and (2) procurement of goods and services) and  sustainment (including logistics, maintenance, and materiel  readiness). This organization has certain oversight responsibilities for  major defense acquisition programs throughout the acquisition  process, such as collecting and distributing performance data. The  Under Secretary is the Defense Acquisition Executive and serves as  the milestone decision authority for certain major defense acquisition  programs, meaning the Under Secretary authorizes these programs to  proceed through the DOD acquisition process’s major milestones.

At the military department level, the service acquisition executive, also  known as the component acquisition executive, is a civilian official  within a military department who is responsible for all acquisition  functions within the department and can serve as the milestone  decision authority. Congress has recently devolved much of the  decision making authority for major defense acquisition programs  from OSD to these service acquisition executives. According to a  DOD official the service acquisition executive will normally assign a  relevant program manager who will then assign a chief engineer or  lead systems engineer and team members with responsibility for the  engineering effort of a program, including the reliability engineering  effort. The following officials serve as the service acquisition executive  for the military departments:  the Assistant Secretary of the Air Force (Acquisition, Technology,  and Logistics);  the Assistant Secretary of the Army (Acquisition, Logistics and  Technology); and  the Assistant Secretary of the Navy (Research, Development and  Acquisition) for both the Navy and the Marine Corps.

Major defense acquisition program managers, who can be either  civilian or military, are tasked with developing and delivering new  weapon systems while balancing factors that influence cost, schedule,  and performance and ensuring that systems are high quality,  supportable, reliable, and effective.

According to DOD guidance, reliability is the probability of an item to  perform a required function under stated conditions for a specified period  of time. DOD’s acquisition environment has changed over time and this  has affected the way the Department addresses reliability. Until the late  1990s, DOD’s goal was to achieve good reliability by focusing on specific  reliability engineering tasks during design and manufacturing, and early  testing to prevent, detect, and correct design deficiencies. In the late  1990s, in response to various NDAAs, DOD implemented certain  acquisition reforms, eliminating and consolidating acquisition functions,  and reducing the number of personnel assigned to the remaining  functions.

According to the Defense Science Board Task Force on Developmental  Test & Evaluation, these reforms altered several aspects of the military  acquisition process and DOD’s acquisition workforce. As a result, DOD  lost experienced acquisition management and technical personnel. DOD  officials stated this loss included reliability personnel who contributed to  developmental testing and evaluation. DOD also canceled the Military  Standard pertaining to reliability at this time. DOD officials explained  that, after acquisition reform in the late 1990s, the department shifted  much of the responsibility for reliability issues to contractors, and  government personnel primarily focused on how systems performed  during operational tests at the end of their development program.

In the mid to late 2000s, Congress and DOD took actions to increase the  focus on reliability engineering during weapon system design and  development. Both Congress and DOD took steps to elevate the  importance of reliability, which has continued through 2019. Figure 3  depicts selected laws related to reliability and DOD reliability efforts over  time.

Impacts of Poor Reliability on Warfighters

Poor reliability can negatively affect the warfighters through low  operational availability; that is, the amount of time a system is available to  execute its mission. For example, the MV-22 aircraft was less reliable  than intended, and required many more spare parts than expected. When  the Marine Corps deployed to Iraq, MV-22 maintainers had to cannibalize  parts from some MV-22s to keep other MV-22s flying, and as a result, the  Marine Corps had fewer aircraft available to fly missions.

Impacts of Poor Reliability on Operating and Support Costs

Reliability can significantly influence a weapon system’s operating and  support costs, which we have previously reported account for  approximately 70 percent of a weapon system’s total life-cycle cost.  Operating and support costs are a reflection of how programs achieve  operational availability for weapon systems. Programs can achieve  operational availability by building highly reliable weapon systems or, if  the systems are not highly reliable, supporting them with an extensive  logistics system that can ensure spare parts and other support items are  available when needed. DOD has previously reported that deficiencies in  DOD weapon systems—such as high failure rates and an inability to  make significant improvements in reliability—have historically limited  program performance and increased operating and support costs.

Impacts of Poor Reliability on Commercial Companies

In the commercial world, the manufacturer carries most of the risks that  would result from developing a product with poor reliability. Such risks  include increased warranty expenses that decrease profits. For example,  reliability personnel from Ford, Cummins, and Thermo Fisher Scientific  explained that more reliable products cost their companies less because  they do not have to dedicate as many resources to fixing systems that  fail, which would lead to warranty claims.

In addition to increased costs, poor reliability can also negatively  influence a company’s reputation. Ford representatives said that failures  and product recalls are not just financial costs; recalls are highly  publicized. A Thermo Fisher Scientific product manager explained that a  customer’s bad experience can be shared in the media and negatively influence a company’s reputation. This may alter future buying behavior,  especially in industries with relatively small customer bases in closely  linked professional communities. This person shared a prior experience at  a different company, where a design risk was identified during  development. Instead of addressing the risk effectively, a standard cycle  test was done to prove or disprove the risk. However, the test did not  apply the stress necessary to cause the failure. The product was released  to the market based on this successful but inadequate test. In the field,  the components failed, and the company had to remove the product from  the market. This damaged the company’s reputation and sales. We have  previously reported that poor reliability is a concern for commercial  companies because their customers demand products that work, or are  reliable and do not experience failure, and the companies must develop  and produce high-quality products to sustain their competitive position in  the marketplace.

Commercial Companies Proactively Address Reliability

In the commercial sector, reliability engineers told us their companies  proactively address reliability from the beginning of the development  process. We reviewed documentation from these companies and the  2019 Reliability and Maintainability Symposium and found engineers  strive to identify reliability issues at the component and sub-system level  early in the development process to avoid expensive rework after  producing an entire system. We identified the following key practices in  the commercial sector:  leveraging reliability engineers early and often, establishing realistic reliability requirements—for example, not  expecting a product to operate twice as long as its predecessor before  failing, emphasizing reliability with their suppliers, and employing reliability engineering activities to improve a system’s  design throughout development.

Figure 4 shows some of the activities involved with these key practices.

Leverage Reliability Engineers Early and Often

We found commercial companies in our review include reliability  engineers as part of their development teams. In this role, reliability  engineers implement reliability tools and methods that integrate statistics,  physics, and engineering principles to help develop a reliable product. For  example, HBM Prenscia identified that reliability engineers from several  commercial companies said it was important to initiate their assessments  early in the development life cycle when there is greatest opportunity to  influence product design.

According to leading reliability engineers, engineering activities can add  value to decision-making by providing direction and feedback that helps  development teams refine designs that lead to more reliable and cost  effective systems. Researchers have reported reliability engineers  should be empowered to influence decisions, such as delaying overall  project schedule or negotiating for more resources when necessary. In  addition, our analysis of reliability engineers’ documentation from the  Reliability and Maintainability Symposium and commercial companies  found it important that management provide sufficient resources and time  dedicated specifically to improving reliability by discovering failures,  implementing corrective actions, and verifying their effectiveness. Our  analysis found that cost and schedule constraints can negatively  influence reliability testing, which can limit development teams’ ability to  discover failures and improve designs through corrective actions.

Our analysis of documentation from the Symposium also highlighted the  importance of having experienced reliability engineers. For example, Ford  representatives told us they have a dedicated reliability engineering  community that coaches the members of the company’s different product  development teams. Ford’s reliability engineers said they focus on  teaching development team members to ask the right questions at the  right point in time with the right people in the room.

Establish Realistic Reliability Requirements Based on Proven Technologies

We found companies in our review emphasize that reliability requirements  should be realistic, be based on proven technologies, and reflect  customer usage and the operating environment. To determine feasibility  of meeting a requirement, reliability engineers we spoke with at Cummins  and Thermo Fisher Scientific recommend conducting comparative  analysis with historical data and assessing risk due to new, unique, or  difficult technology. In addition, an independent reliability engineer with  over 40 years of experience told us programs should provide justifications  for how reliability requirements were established to demonstrate they are  within the realm of technological possibility.

If the reliability requirement turns out not to be technically feasible, it  could have broad implications for the intended mission, life-cycle costs,  and other aspects of the system. We have previously reported on the  importance of making informed trade-offs when considering requirements  to reduce program risk or total ownership costs. HBM Prenscia  representatives told us the commercial companies they work with  regularly make trade-offs involving capability, reliability, and cost  requirements. Reliability representatives at Ford told us it is important to  have the right people involved in these trade-off decisions, and that they  work with user representatives and reliability engineers to define their  systems’ reliability requirements.

Systems produced by commercial companies in our review include parts  or components produced by suppliers, and reliability engineers repeatedly  told us the reliability of those parts or components directly impacts the  reliability of the overall system. According to a leading reliability engineer,  vendor quality can affect a part’s reliability, so it is critical that the  reliability of vendors’ parts be evaluated before being approved for use.  To emphasize reliability with suppliers, commercial companies in our  review engage with suppliers early, clearly specify requirements with the  supplier, and evaluate and monitor the supplier.

Cummins representatives stated engaging the supplier early is critical.  They explained that they engage the supplier early, during concept  development, and ask the supplier to demonstrate it can meet  requirements. According to Cummins representatives, this is to ensure  the supplier is able to meet quality standards and to ensure there is  enough lead time and testing of components. Reliability engineers at the  Reliability and Maintainability Symposium also emphasized that reliability  requirements must be clearly specified with suppliers, and product teams  must actively monitor suppliers and assess their deliverables. Cummins  representatives explained their engineers work directly with the supplier  and hold it responsible for meeting reliability requirements. Ford  representatives told us they evaluate and monitor the supplier to ensure  the components it is providing are reliable. For example, they visit their  suppliers’ testing facilities and evaluate their testing programs, focusing  specifically on their failure analysis and reliability activities. We have  previously reported that leading commercial companies use disciplined  quality management practices to hold suppliers accountable for high  quality parts through such activities as regular supplier audits and  performance evaluations.

A Thermo Fisher Scientific product manager provided a scenario where  relying on an external supplier’s quality assurances would be insufficient.  For example, a compressor is a critical – and commonly outsourced— component in complex industrial equipment. The product manager  recommended in-house testing for critical components like a compressor  rather than relying on a supplier’s testing that may not factor in real-world  operating conditions. In house testing is recommended to avoid finding a  failure after the product is brought to market. Post-sale failures result in  dissatisfied customers, reputation damage, warranty claims and similar  issues. The Thermo Fisher Scientific product manager said, in some  cases, a company should establish a dedicated test facility for vital  outsourced components provided by suppliers.

Employ Reliability Engineering Activities to Improve a System’s Design throughout Development

Based on our review of commercial sector practices, we found companies  use reliability engineering activities to identify potential product failures  and their causes. They also use these activities to improve a system’s  design early and often throughout development to avoid surprises that  lead to expensive rework or excessive repairs after integrating  components and subsystems. For example, HBM Prenscia  representatives told us that failures should be identified early, and that  identification should be viewed as an opportunity to improve the design  and make the product better. According to leading reliability engineers,  the earlier changes are made to designs, the less costly they are to the  program. It is expensive, time consuming, and risky to make changes late  in development, as late changes jeopardize product reliability. The  commercial company representatives we spoke with also emphasized the  need to conduct reliability engineering activities iteratively until the design  is optimized. For example, HBM Prenscia has identified that a common  mistake is establishing a reliability plan but not actively utilizing it  throughout development.

Reliability engineers use various reliability engineering activities to  increase system reliability, and generally refer to these activities as  design for reliability tools. These tools can be tailored to meet the specific  needs of a particular development project, and can complement one  another and increase reliability prior to any testing. These tools can help  identify how long a part or component will work properly, how a part or  component’s failure will affect a system, and what actions are needed to  correct failures. See table 2 for some examples of design for reliability  tools that can be used to help meet reliability goals.

We have previously reported that leading commercial companies use a  knowledge-based development process that enables decision makers to  be reasonably certain that product quality, reliability, and timeliness are  assured. Our analysis of documentation from reliability engineers found  that reliability engineering activities should be integrated into the product  development process, and their outputs should be reviewed at  development milestones. These reviews can help ensure that reliability is  a robust process rather than a paper exercise by providing an opportunity  to assess data from reliability analysis or testing. For example, Cummins  incorporates reliability reviews into its product development processes to  ensure products meet reliability goals prior to moving to the next phase of  development. This helps ensure the company is on track to fulfill its  reliability commitments and will be able to deliver the promised product  reliability to customers.

The leading commercial practices we reviewed highlight the importance  of consistently collecting, sharing, and analyzing data from reliability  engineering activities to inform development efforts. Commercial  companies we spoke with recognized the value of reliability data. For  example, Cummins representatives stated they capture reliability data  and share it across different product development teams to help inform  estimates of reliability for new product development efforts. In addition,  Cummins representatives noted that they are moving to an interactive  database that personnel throughout the entire company can access.  Similarly, HBM Prenscia representatives told us that failures and lessons  learned from previous projects should be captured and shared within a  company, and that doing so could help inform future product development  efforts.

Selected Major Defense Acquisition Programs Did Not Consistently Reflect Key Reliability Practices

We reviewed seven major defense acquisition programs and found they  often reactively addressed reliability after identifying issues later in  development. As shown below, these programs did not consistently  reflect key practices we identified in the commercial sector, and instead  prioritized other activities intended to have positive acquisition cost and  schedule impacts. However, DOD officials noted that there has recently  been a greater emphasis on reliability, and the three programs that  started development in 2012 and 2014 reflected more of the key practices  than the older programs. See figure 5, which notes a distinction between  commercial companies’ suppliers and DOD contractors. For more  detailed information on each program, see appendix I.

Two of the Seven Selected Programs Did Not Leverage Government Reliability Engineers in Decision Making Early

The Expeditionary Fighting Vehicle (EFV) and F-22 programs did not  involve reliability engineers early during system development. Instead,  these programs leveraged engineers after reliability problems arose,  including after they integrated components and subsystems and during  system-level testing. At the end of system development, the programs  brought in additional engineers and established more concerted reliability  growth efforts.

In one example, the EFV program did not have an overall systems  engineer. Marine Corps acquisition officials stated that reliability was not  a priority during the original system development process, and we have  previously reported the program was instead focused on achieving other  performance parameters, including water speed, survivability, and  lethality. Prime contractor representatives identified some of their design  engineers who lacked experience and did not comply with engineering  standards as a root cause for problems discovered late in the  development process. We also reported the lack of early systems  engineering discipline and knowledge undermined the EFV program’s  ability to develop informed and reasonable reliability requirements,  delayed the identification of potential failures until integration, and  contributed to poor vehicle reliability. In addition to frequent hydraulic  system failures, leaks, and pressure problems, the EFV also suffered  main computer failures that froze steering while operating in water.

As we have previously reported, the EFV program was subsequently  restructured. The program office hired additional engineers and  consulted with Army reliability engineers to institute a reliability growth  program. This program was intended to mitigate previously identified  vehicle design issues related to reliability and other risks before  proceeding into a second development and demonstration phase.  However, the EFV program never got to fully realize the benefits of its  new reliability approach, as less than 3 years after restarting development  it was canceled due to continuing technology problems, development  delays, and affordability concerns.

For the F-22 program, officials stated that at points during development  the program did not have a leadership position focused on reliability, and  the official who oversaw reliability was also responsible for supply chain  management. The officials noted that at the time these were not focus  areas because the Air Force expected the contractor to conduct the  needed reliability engineering. In 2004, we reported that, as early as low- rate initial production, however, the Air Force identified 68 parts that had  a high rate of failure and needed to be removed or replaced, requiring  additional contractor work. We also reported the F-22 canopy also  experienced failures during testing, allowing it to achieve only about 15  percent of its expected lifetime. In 2014, we reported that later reliability  maturation projects intended to address reliability deficiencies had a  positive effect on availability over time, but as of 2018 the F-22 still had  not met its availability target.

Four of the Seven Selected Programs Initially Pursued Unrealistic Operational Requirements for Reliability

As we have found in our prior reports as well as in this review, the EFV,  F-22, F-35, and V-22 programs set unrealistic operational requirements  for reliability. These requirements were, therefore, unachievable during  development and before fielding the systems to warfighters. As we have  previously reported, when programs overpromise a weapon’s prospective  performance and deliver systems that cannot achieve their requirements,  such as reliability goals, the warfighter receives less capability than  originally promised.

In one example, as we reported in 2019, more than 11 years after the  start of F-35 production, none of the three aircraft variants (Air Force,  Marine Corps, and Navy) had met the minimum targets for two of the  program’s five reliability metrics. These include mean flight hours  between part removals for replacement and mean flight hours between  critical failures. We found that only the Navy variant had achieved the  minimum target for a third goal, mean flight hours between maintenance  events. As we reported, while the program has instituted an effort  intended to improve reliability, the effort does not align improvement  projects with the F-35’s reliability requirements. That is, the reliability  improvement projects being funded may not improve the F-35’s  performance against its reliability metrics. Ultimately, the program does  not expect to achieve the unmet reliability metrics by full aircraft maturity,  and program officials have acknowledged that the requirements should  be reevaluated. As a result, the warfighter may not receive an aircraft  that is as reliable as was expected.

In a review of the V-22 program, DOD found that the program integrated  complex technologies and unprecedented capabilities into its weapon  system without accounting for unknown reliability risks. Specifically,  these capabilities included a conceptually new design and multiple  service and mission needs. However, officials stated that the program  derived its reliability requirements from antecedent helicopters, systems  that were not representative of the V-22 given its increased complexity.  With a limited understanding of the V-22’s mission profile, program  officials stated that they also underestimated the amount of time the  system would be used in helicopter mode and its operating time on the  ground. Subsequently, when the Marine Corps variant of the V-22—the  MV-22—was deployed in Iraq from 2007 to 2009, a number of  components experienced high rates of failure, affecting systems such as  the engines and engine housing. This situation, combined with an  immature parts supply chain, reduced the system’s availability  significantly below minimum levels. At the time, as we reported in May  2009, the MV-22 had a stated minimum mission capability rate of 82  percent, but the three MV-22 squadrons in Iraq demonstrated an average  of 62 percent.

The development and integration of new technologies on the F-22— stealth, supersonics, and integrated avionics—were critical to achieving  operational success, but also presented significant reliability risks.  Officials told us that the F-22 was initially expected to cost less to acquire  and operate than one of its predecessors, the F-15, and be more reliable  as well. However, they also stated this was an unrealistic expectation. We  have previously reported that the immaturity of technologies at the start of  and throughout development weaken a system’s ability to achieve  reliability requirements. Since 2005, when full rate production of the F- 22 began, the program has made substantial additional investments in  increasing the system’s reliability through various improvement programs.  But the program also changed its mean time between maintenance  reliability requirement to an operational availability metric, a target that as  of 2018 it had yet to meet and may need to reevaluate, according to  program officials. If the F-22 cannot achieve its current reliability  requirement, warfighters will have to execute their missions with a less  capable aircraft than expected.

Four of the Seven Selected Programs Did Not Effectively Emphasize Reliability with Contractors

The AMPV, EFV, F-35, and V-22 programs did not effectively emphasize  reliability with DOD contractors. Specifically, according to DOD, the  AMPV, EFV, and V-22 did not effectively incentivize reliability with the  contractor and one program, the F-35, did not include all of the program’s  reliability metrics in the contract.

Each F-35 aircraft variant is measured against five reliability metrics, two  of which are in part of the contract. Contractors are not responsible for  achieving reliability requirements if programs do not include them in  contracts. As of August 2018, two of the F-35’s three variants had not met  minimum targets for any of the three metrics that are not in the contract.  The last variant (Navy) has met the minimum target for only one of the  three metrics. As we have previously reported, the warfighter may have to  accept F-35 aircraft that are less reliable and more costly than originally  expected.

As we have reported, the F-35 program tried to encourage the aircraft’s  manufacturer to improve reliability through an incentive fee in sustainment  contracts. These contracts, for sustainment services, included  incentives for meeting aircraft availability. Reliability of parts is one of the  factors that influences aircraft availability, because broken parts prevent  aircraft from flying. Program officials told us they hoped the incentive fee  in the sustainment contract would incentivize the contractor to invest in  and implement additional reliability activities, which would help improve  aircraft availability, but according to the program office, the incentive has  not been effective. Program officials told us the contractor has not  pursued the incentive fee in the sustainment contract through efforts to  improve aircraft reliability because it would have to invest significant  resources to design and incorporate changes into production aircraft in  order to do so. F-35 aircraft, especially early production aircraft, continue  to face challenges related to parts that are failing more often than planned  and are in short supply. For example, we have previously reported that  DOD found the special coating on the F-35 canopy that helps maintain  the aircraft’s stealth failed more frequently than expected and that the  manufacturer could not produce enough canopies to meet demand,  ultimately degrading system capability.

According to program officials, to ensure that reliability growth was on  track, the AMPV program offered an incentive fee of up to $16 million if  the contractor could demonstrate at least 80 percent of the system’s  reliability before low rate production. But officials stated that the AMPV  contractor did not achieve the goal. The AMPV was a derivative system of  the Army’s Bradley Fighting Vehicle with an accelerated development  schedule, and officials stated that for this reason the contractor assumed  the government would accept much of the Bradley’s initial design and  changes to the AMPV’s performance resulting from legacy reliability  issues. As a result of these expectations, officials stated that the  contractor did not put enough resources, including a robust reliability  team, toward the work that was eventually needed to improve reliability,  and the contractor understaffed in this area.

Five of the Seven Selected Programs Deferred Key Reliability Engineering Activities until Later in Development

The AMPV, EFV, F-22, F-35, and V-22 programs deferred key reliability  engineering activities, intended to improve system designs, until later in  development. As a result, they missed opportunities to identify,  understand, and mitigate reliability issues early in the development  process. After realizing reliability shortfalls late in development, some  programs initiated expensive redesign efforts that continued well into  production and deployment, while others accepted degraded  performance.

Based on our prior reporting, we found the EFV program did not  implement a proactive reliability approach, which would include identifying  challenges early and designing reliability into the system in a cost- effective manner. Instead, the program used a test-fix-test approach  that relied on identifying failure modes after the system-integration phase.  Early in the acquisition process, officials noted in program documentation  that the program had conducted little reliability growth planning before  starting development, and officials stated that the EFV program did not  plan for or conduct dedicated reliability testing. Then, the program  prematurely conducted its critical design review, a key review during the  development phase which confirms the system’s design is stable and is  expected to meet system performance requirements, before the EFV  prototype’s system-integration work was complete. The program did not  have the time necessary to demonstrate design maturity as scheduled  and officials stated that they did not schedule long enough corrective  action periods to allow for proper failure mitigation. As a result, during a  2006 operational assessment, the EFV demonstrated very low reliability  and failed to complete amphibious, gunnery, and land mobility tests.

F-22 program officials stated that many of the aircraft’s components and  subsystems had to be tested as part of an integrated system. This limited  the discovery of reliability issues early in the development phase. DOD  reliability experts told us programs should not use integrated system  testing to demonstrate individual component reliability, and should instead  use it to focus on how components work together and identify more  complex system failure modes. F-22 officials also stated the program  office frequently continued with development and other testing before  implementing corrective actions for critical reliability issues. As we have  previously reported, the F-22 program started a program to improve its  reliability in 2005, near the start of full rate production, to mitigate  hundreds of known reliability issues deferred from earlier in development.  Nonetheless, we reported in 2012—nearly 3 years after DOD announced  the end of F-22 production—that reliability deficiencies had increased  support costs, and continued to prevent the aircraft from meeting its  reliability requirement.

According to program officials, the Army selected a derivative of the  Bradley Fighting Vehicle to meet the AMPV requirements, even though  that vehicle’s transmission had known reliability problems. According to  AMPV program officials, the Army selected this vehicle because it had  prioritized controlling costs and accelerating schedule. Program officials  stated that the focus on cost and schedule caused the contractor to  underestimate the necessary reliability work at the start of development  and led to a backlog of test incident reports and deferred corrective  actions. According to 2018 program documentation we reviewed, the  AMPV’s reliability growth did not track to targets during development and  the vehicle did not achieve its pre-production reliability goal. Moreover,  some of the AMPV’s deferred work may need to be addressed during a  future corrective action period that could continue through fiscal year  2021.

DOD Acquisition Policy and Guidance Documents Identify, but Do Not Emphasize, Key Reliability Practices

Although there are differences between the DOD and commercial sector  stemming from the statutory and regulatory structures that govern DOD’s  acquisition processes, DOD has had long-established policy and  guidance at both the department and service level that recognize the four  key reliability practices we found in the commercial sector. For example,  the Defense Acquisition Guidebook encourages acquisition programs to  involve reliability engineers early and often, and DOD Instruction 5000.02  identifies the need for establishing realistic reliability requirements.  Additionally, the 2005 DOD Guide for Achieving Reliability, Availability,  and Maintainability addresses the importance of emphasizing reliability  with contractors, and the service-level policies at all three military  departments establish the importance of reliability engineering activities.

However, most of these documents cover a wide range of acquisition  issues or many aspects of reliability engineering, and they do not  specifically emphasize the four key practices we identified in our review of  the commercial sector. For example, the DOD Instruction 5000.02 is an  overarching policy document covering the entire acquisition life cycle at a  high level, from concept development to live fire test and evaluation, and  only one section provides significant detail and direction on reliability. The  service level instructions and Defense Acquisition Guidebook similarly  cover the entire acquisition life cycle, and reliability is one of dozens of  characteristics addressed in each document. The DOD Guide for  Achieving Reliability, Availability, and Maintainability is largely focused on  achieving reliability, but the reliability proponents at OSD, the Army, and  the Navy said the guide is not consistently used throughout DOD, noting  that it was issued in 2005 and has not been updated since.

DOD policy provides decision makers flexibility to tailor regulatory  activities that acquisition programs perform when developing weapon  systems. The process is inherently complex, and these decision makers  must balance many factors when overseeing and executing the  programs. In the absence of an emphasis on the key reliability practices  we identified, we found decision makers for the programs we reviewed  prioritized other activities intended to have positive acquisition schedule  and cost impacts. For example, AMPV program officials told us the  program eliminated 7,500 miles of contractor reliability testing in order to  proceed to the next development phase more quickly, believing that there  would be sufficient time later to complete corrective actions.

Recently, DOD has begun employing the Middle Tier Acquisition  pathway—an alternative acquisition pathway with an objective of  beginning production within 6 months and completing fielding within 5  years. This emphasis may encourage decision makers to prioritize  activities that promise to reduce schedule. We found that for the  programs we reviewed, however, such an approach can come at the  expense of other activities, such as implementing effective reliability  practices.

DOD has recently taken steps that could introduce more balance when  decision makers consider trade-offs between schedule and reliability.  Specifically, DOD has highlighted the importance of one of the four key  reliability practices we identified: emphasizing reliability with contractors,  and Congress has passed legislation related to reliability. The NDAA for  fiscal year 2018 included a provision mandating DOD program managers  to include certain reliability requirements in weapon system engineering  and manufacturing development and production contracts. In January  2019, the USD(A&S) implemented the NDAA by issuing a policy  memorandum to Service Acquisition Executives and other DOD Directors  echoing this key practice. However, USD(A&S) has not similarly  emphasized the three other key reliability practices we identified in the  commercial sector, nor have the Secretaries of the Air Force, Army, and  Navy, who now have ultimate responsibility for most of DOD’s major  acquisition programs. Specifically, these senior leaders have not  emphasized the value of  leveraging reliability engineers early and often, establishing realistic reliability requirements, and employing reliability engineering activities to improve a system’s  design throughout development.

As a result, it is less likely that acquisition programs will take the actions  necessary to recognize and address potential reliability problems early in  the development process. Without senior leadership emphasis on a  broader range of key reliability practices, DOD runs the risk of delivering  less reliable systems than promised to the warfighter and spending more  than anticipated on rework and maintenance of major weapon systems.  This risk is exacerbated in an environment where decision makers are  striving to deliver systems in an accelerated manner.

The best opportunity to influence the reliability of a weapon system is  early on during the design of the system. Decisions and tradeoffs made at  that time can increase the weapon system’s reliability, help warfighters  execute their missions, and decrease operating costs for years to come.  However, these decisions and tradeoffs are not easy, as acquisition  decision makers are tasked with managing competing priorities such as  cost, schedule, and performance. Many of the DOD acquisition program  examples in this report illustrate what can happen when reliability is not  prioritized. The programs often approached reliability in a reactive  manner, discovered problems late in the development process, and then  tried to fix them through costly and time-consuming rework. The programs  did not consistently adhere to key practices we identified in the  commercial sector:  reliability engineers were not leveraged early in the development  process,  reliability requirements were not realistic,  reliability was not emphasized with contractors, and  reliability engineering activities were not utilized throughout design  and development.

Recent DOD actions have highlighted the importance of emphasizing  reliability with contractors. DOD senior leaders can help improve reliability  by highlighting the importance of the three other key reliability practices  we identified in the commercial sector. In light of the current focus on  accelerating the acquisition process, balancing the desire for speed with  reliability considerations is critical. Given the delegation of acquisition  decision authority to the military services, the Secretaries of the Air Force,  Army, and Navy are in the best position to do so.

We are making a total of three recommendations: one each to the Air  Force, the Army, and the Navy.

We recommend the Secretary of the Air Force issue policy emphasizing  the following three key reliability practices when planning and executing  acquisition programs:  leveraging reliability engineers early and often, establishing realistic reliability requirements, and employing reliability engineering activities to improve a system’s  design throughout development. (Recommendation 1)

We recommend the Secretary of the Army issue policy emphasizing the  following three key reliability practices when planning and executing  acquisition programs:  leveraging reliability engineers early and often, establishing realistic reliability requirements, and employing reliability engineering activities to improve a system’s  design throughout development. (Recommendation 2)

We recommend the Secretary of the Navy issue policy emphasizing the  following three key reliability practices when planning and executing  acquisition programs:  leveraging reliability engineers early and often, establishing realistic reliability requirements, and employing reliability engineering activities to improve a system’s  design throughout development. (Recommendation 3)

We provided a draft of this report to DOD for review and comment. DOD’s  written comments are reprinted in appendix II. DOD stated that the Air  Force, Army, and Navy concur with our recommendations to their  respective Departments. The comments also state that the Air Force and  Navy plan to update their policies in response to our recommendations.  As for the Army, the comments state that the Army Acquisition Executive  will issue direction emphasizing the three key reliability practices and  highlight an existing Army regulation focused on reliability engineering.

In addition to the responses to our recommendations, DOD’s written  comments included technical comments that we addressed as  appropriate. For example, we provided additional detail on an existing  DOD policy, and clarified how a program engaged with a contractor.

We are sending copies of this report to the appropriate congressional  committees and the Secretary of Defense. In addition, the report is  available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-4841 or mackinm@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix III.

Appendix I: Key Characteristics of Selected Major Defense Acquisition Programs’ Approach to Reliability

This appendix summarizes key characteristics of seven selected major  defense acquisition programs’ approach to reliability. The four key  characteristics are categorized as: did not leverage government reliability engineers in decision making  initially pursued unrealistic operational requirements for reliability; did not effectively emphasize reliability with contractors; and, deferred key reliability engineering activities until later in development.

These summaries do not address all the reliability actions taken by each  program; rather they focus on key characteristics we identified in our  review of commercial companies and associated deficiencies. See figure  6, which notes a distinction between commercial companies’ suppliers  and DOD contractors.

Appendix II: Comments from the Department of Defense

In addition to the contact named above, Nathan Tranquilli (Assistant  Director), Julie A. Clark (Analyst-in-Charge), Lori Fields, Laura Greifner,  Brendan K. Orino, LeAnna Parkey, Christine Pecora, Andrew N. Powell,  Timothy Saunders, and Michael J. Sullivan made key contributions to this  report.

DOD invests billions of dollars in its weapons systems and expects them to be reliable.

We spoke with commercial companies known for creating reliable products to find out how they achieve reliability, and found that they often focus on four key practices—such as involving reliability experts early in development. We looked at 7 DOD programs and found that they didn’t consistently follow these practices. Instead, they often prioritized producing systems faster.

We recommended that the Air Force, Army, and Navy emphasize key reliability practices when developing weapons systems.