Global Positioning System: Updated Schedule Assessment Could Help Decision Makers Address Likely Delays Related to New Ground Control System

The U.S. military and the public depend daily on GPS data. OCX, the ground system that will command and control next generation GPS satellites, is one of several interdependent systems the Air Force is developing to modernize GPS. OCX has been hampered by delays and $2.5 billion in cost growth since the program started in 2012. The Air Force set a new baseline for cost and schedule in 2018 after OCX breached its cost threshold in 2016.

The National Defense Authorization Act for Fiscal Year 2016 contained a provision that the Air Force provide quarterly reports to GAO on the next generation GPS acquisition programs, and a provision that GAO brief the defense committees as needed. GAO provided numerous briefings from 2016 through 2018 and issued reports in 2016 and 2017. Continuing this body of work, this report focuses on the extent to which schedule risks may affect OCX delivery, acceptance, and approval for operation.

GAO reviewed the Air Force's baseline review results, schedule risks, and progress, and applied selected best practices for cost and schedule management. GAO also reviewed OCX monthly management briefings and quarterly assessments, and interviewed officials from the OCX program office and Raytheon (the prime contractor), among others.

The Global Positioning System's (GPS) next generation operational control system's (OCX) program schedule continues to be optimistic and, with significant development remaining, more delays are likely for delivery, acceptance, and operation. See the figure below for previous delays, cost growth, and the current baseline.

Completing the full OCX program schedule requires (1) timely delivery by the contractor and acceptance by the Air Force and (2) an efficient completion of a planned 7-month government-run post-acceptance developmental testing. GAO found that there is potential for significant delays on both fronts. While there has been some improvement to the pace of software development, the rollout of the new development methodology has been delayed to a point where most of the contractor's schedule reserve has been used. Assumed improvements in how long it takes to repair software defects has not occurred as planned, placing additional pressure on the contractor's delivery date. Additionally, Air Force officials have acknowledged that the government developmental test period after acceptance could double in duration and delay operations further because of concurrency, test plan uncertainty, and risks of late discovery of problems.

With approximately 2 years of work remaining before delivery, there is no plan to have the full schedule independently assessed. For complex programs, such as OCX, best practices state an independent view is necessary and that a periodic schedule assessment should be performed as progress is made and risks change. Such an assessment would help inform congressional and DOD decision makers as they consider what steps may be taken to address delays to the start of OCX operations and ensure the investments in needed new receivers are properly aligned.

GAO recommends that DOD conduct an independent schedule assessment of the full program schedule at the end of 2019. DOD did not concur with the recommendation. GAO believes the recommendation remains valid.

The U.S. military and civilian users throughout the world depend daily on  the Global Positioning System (GPS). The GPS next generation  operational control system (OCX) is one of the critical interdependent  systems required to sustain and modernize GPS. Without OCX or  additional upgrades to the current GPS ground system, the Air Force  cannot fully command and control the next generation of GPS satellites  and fully modernize the system to continue to provide positioning,  navigation, and timing data to the military and the public. OCX will replace  the current ground system, which lacks modern cybersecurity protections  and cannot currently control the next generation of GPS satellites.

Over the past 7 years, the Air Force has repeatedly underestimated the  time required for the development of the ground system, establishing  unrealistic schedules and then revising them when they were not  achieved. As a result, OCX development is currently projected to take  approximately 5 years longer and cost at least $2.5 billion more than  originally estimated. In early 2018, before the Department of Defense  (DOD) approved the latest revision to the cost and schedule baseline, the  OCX program office conducted a review of OCX’s schedule for the  remainder of development. Subsequently, DOD’s Under Secretary for  Acquisition and Sustainment approved the new baseline in September  2018.

The National Defense Authorization Act for Fiscal Year 2016 contained a  provision that the Air Force provide quarterly reports to GAO on next  generation GPS acquisition programs. The Act also contained a  provision that GAO brief congressional defense committees on the first  report, and at GAO’s discretion, on subsequent quarterly reports. We  addressed the first quarterly report provision in 2016, have continued to  brief congressional defense committees on GPS acquisition progress,  and issued a comprehensive GPS report in December 2017. This report  focuses on the extent to which schedule risks may delay OCX delivery,  acceptance, and approval for operation.

To conduct our work, we reviewed the Air Force’s OCX 2018 integrated  baseline review results and the full program schedule—which includes  the contractor’s delivery schedule, government acceptance, and post- acceptance government-run developmental testing. We also reviewed  monthly management briefings, senior executive briefings, and quarterly  Defense Contract Management Agency reports. We reviewed GAO’s best  practice guides for cost estimating and assessment and schedule  assessment to identify best practices for assessing a program’s cost and  schedule and applied selected best practices. We also interviewed  officials from the Air Force OCX program office; Raytheon Company  (Raytheon), the prime contractor; Defense Contract Management  Agency; Air Force Cost Analysis Agency; DOD’s Office of Cost  Assessment and Program Evaluation; and the Office of the Director,  Operational Test & Evaluation, among others. Appendix I contains a more  detailed description of our scope and methodology.

We conducted this performance audit from November 2017 to May 2019  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis of our findings and  conclusions based on our audit objectives.

Collectively, the ongoing GPS acquisition efforts aim to (1) modernize and  sustain the existing GPS capability and (2) enhance the current GPS  system by adding a more cybersecure ground system that enables M- code. M-code is a stronger, encrypted, military-specific GPS signal  designed to meet military positioning, navigation, and timing needs. It will  help military users overcome GPS signal jamming by using a more  powerful signal and protect against false GPS signals, known as  spoofing, by encrypting the signal. Figure 1 below shows how GPS  satellites, ground control, and user equipment—in the form of receiver  cards embedded in systems—function together as an operational system.

The Air Force’s OCX program is primarily a software development effort  to replace the current ground system, the operational control system  (OCS), with a modernized and more cybersecure system. OCS lacks  modern cybersecurity protections and cannot currently control—or  enable—modernized features of the three latest generations of GPS  satellites now in orbit, including M-code and three new civilian signals.  Because existing military receivers were not designed to work with the  new M-code signal, military users will have to make investments in new  receiver development and procurement timed to when the new signal will  be available before they can use it.

Raytheon is the prime contractor working to deliver OCX in a series of  blocks that enable additional capabilities. Block 0, which is a subset of  block 1 broken out after development started, was delivered in September  2017. It helped to successfully enable the launch and initial testing of the  first GPS III satellite, which was launched in December 2018, and will  continue to support subsequent GPS III satellite launches. Blocks 1 and  2, originally planned as separate deliveries, have been combined into a  single delivery and are currently in development. This combined delivery  enables OCX to command and control each satellite and begin using the  full M-code signal, as well as control new civilian signals, among other  capabilities.

Because of significant delays to OCX, the Air Force initiated two  additional programs to modify OCS to deliver some of the planned  capabilities before OCX is operational. The first program is Contingency  Operations (COps)—which will enable the control of GPS III satellites to  operate with the same capabilities as current GPS satellites without the  additional military and civilian signals. The second program is M-code  Early Use (MCEU)—which will permit some functions of M-code to be  used before OCX is delivered. Neither COps nor MCEU will enable the  additional civilian signals or the full M-code functionality that is expected  with OCX.

DOD is required by statute to establish and approve cost and schedule  baselines for major defense acquisition programs before those programs  enter system development, also known as the engineering and  manufacturing development phase. As part of program planning,  including for major defense acquisition programs, DOD policy requires  program managers to establish program goals for cost, schedule, and  performance parameters. Approved program baseline parameters are  reported in the program’s acquisition program baseline as objective and  threshold values. The objective values represent goals in terms of what  the user—in the case of GPS, the Air Force—desires and expects. The  threshold values represent the limit of what is acceptable—meaning cost  or schedule growth above threshold values are outside of the approved  cost or schedule limits.

For OCX, the cost and schedule objective and threshold dates in the  baseline are tied to an event called “ready to transition to operations,”  which will be the completion of the OCX acquisition program schedule.  For the OCX program, this is a decision within the Air Force to switch  control of the GPS constellation from the current GPS ground system,  OCS—at this future point with COps and MCEU modifications already  added—to OCX. The delivery date of the system by Raytheon and  acceptance date by the Air Force will both come before the ready to  transition to operations decision. These two dates are important because  their timing may influence when OCX operates.

What is a critical Nunn-McCurdy unit cost  breach?

For major defense acquisition  programs, a critical Nunn-McCurdy  unit cost breach of a unit cost  threshold is triggered by cost  increases of at least 25 percent or  more of a program’s current cost  baseline or at least 50 percent or  more of a program’s original cost  baseline.

As an acquisition program works to achieve its objective and threshold  values, the original baseline goals may become unachievable. When this  occurs, a revised baseline, or rebaseline, is created so the program’s cost  and schedule goals are updated to more realistically reflect the program’s  current status. If the increase from the cost baseline meets certain  thresholds, DOD is required to notify Congress in writing. This is known  as a Nunn-McCurdy breach. This notification assists Congress with  monitoring program progress, especially on troubled programs. A critical  Nunn-McCurdy unit cost breach is the most serious type of breach and  requires a program to be terminated unless the Secretary of Defense  submits a written certification to Congress that certain criteria have been  met, including that the new estimate of the program’s cost has been  determined to be reasonable by the Director of DOD’s Office of Cost  Assessment and Program Evaluation, and takes other actions, including  restructuring the program.

History of Increasing OCX Cost and Schedule Baselines

As we have previously reported, the Air Force has had significant  difficulties developing OCX. The program’s cost and schedule baselines  have been unstable and unexecutable since the first baseline was  established in 2012. In total, there have been three OCX program  baselines:  1.  November 2012 original baseline at development start,  2.  October 2015 rebaseline due to a schedule breach, and  3.  September 2018 rebaseline prompted by a critical Nunn-McCurdy unit  cost breach.

Since 2012, reflecting the newest baseline and additional cost and  schedule growth since the Nunn-McCurdy breach, the schedule has more  than doubled and the costs have grown by approximately 68 percent.  Figure 2 shows the three OCX baselines with their schedule and cost  growth since the start of development.

The National Defense Authorization Act for Fiscal Year 2017 required an  independent assessment of OCX. The Act required an assessment of  the Air Force’s ability to complete blocks 0 through 2 on a schedule  necessary to transition OCX to full operation and an estimate of the cost,  among other issues. The MITRE Corporation conducted the study and  DOD provided it to congressional defense committees in December 2017.

As a result of the 2016 Nunn-McCurdy unit cost breach, the program  repeated the milestone associated with system development start and  established new cost and schedule objectives and thresholds, conducted  a baseline review of the schedule to verify the work necessary to  complete the program, and received approval of the acquisition program  baseline by the milestone decision authority—the Under Secretary of  Defense for Acquisition and Sustainment.

To support certification of OCX’s new baseline, in May 2017 the Air Force  produced an $8.7 billion OCX service cost position for development,  sustainment, and disposal. The Air Force service cost position was  subsequently reaffirmed in 2018 by the Air Force and supported by an  additional independent cost estimate from DOD’s Office of Cost  Assessment and Program Evaluation in June 2018, which was  approximately 3 percent higher in cost for the development portion. The  Under Secretary of Defense for Acquisition and Sustainment selected the  Air Force service cost position for the OCX baseline.

In 2014, the Air Force identified root causes for OCX cost and schedule  growth and concluded that the problems were driven by (1) incomplete  systems engineering, (2) inadequate process discipline, and (3)  difficulties implementing cybersecurity due to its complexity.

We reported in 2015 that the program office paused development in late  2013 to fix what it believed were the root causes of development issues,  and significantly increased the program’s cost and schedule estimates.  Despite the pause to address root causes, problems persisted and in the  same report we questioned whether all root causes—such as a  persistently high software development defect rate—had been adequately  identified, let alone addressed, and whether realistic cost and schedule  estimates had been developed. We also found that the program was not  following various acquisition best practices, such as the completion of a  preliminary design review prior to development start. In 2015, we  recommended that DOD assemble a task force to assess the OCX  program and provide concrete guidance for addressing program  problems, to determine root causes for OCX defects, and to establish a  high confidence schedule and cost estimate, among other  recommendations. DOD concurred with our four OCX-related  recommendations and has taken some steps to implement some of them.  However, to date, none have been fully implemented and they remain  open.

In 2016, DOD’s Director of Performance Assessments and Root Cause  Analyses concluded that the root causes for OCX’s Nunn-McCurdy unit  cost breach were (1) an unrealistic schedule driven by the need to sustain  the GPS constellation, (2) an underestimation of the cost to fully  implement information assurance, or cybersecurity, and (3) poor  performance by both the government, caused by a lack of requisite  software expertise, and Raytheon, caused by poor systems engineering  that led to significant rework. We found and DOD’s 2016 root cause  analysis has shown a significant and recurring cause of delays on the  OCX program has been a lack of mutual understanding of the work  between the Air Force and Raytheon.

In December 2017, we found risks to the latest proposed (but not yet then  approved) OCX schedule, noting that the schedule to which the program  was working at that time (1) was built on certain unproven assumptions  regarding planned coding and testing improvements, (2) had not yet  undergone a baseline review to verify that the schedule incorporated all of  the work required for program completion, and (3) did not yet include a  number of changes that the Air Force needed to incorporate into the  contract with Raytheon as modifications, which may lead to additional  schedule slips. In 2017, we did not make additional recommendations  for OCX because the Air Force had undertaken the COps and MCEU  programs to provide interim capabilities to mitigate OCX delays.

Changes in Software Development Methodology During the Program

In 2016, Defense Digital Service—a DOD office established by the  Secretary of Defense—engaged with the OCX program to suggest  improvements to Raytheon’s software development practices. The office  recommended that Raytheon change its software development approach  to use an incremental development approach. This approach uses a  continuous integration and testing process, where the software code is  frequently integrated and tested so that defects are detected and  addressed sooner. This is done through automation of the software  development process, version control tools, and coordination between  different teams building software. Traditional software development  methods entail a more linear approach whereby each process is  completed before proceeding to the next process in the sequence. By  such an approach, the software development processes are completed  prior to the testing of a full product before the product’s release to the end  user.

In 2016, DOD told the Air Force and Raytheon to utilize the new  approach, which Raytheon began implementing in a series of seven  phases. The first phase began in late 2016 and the last phase is  scheduled to be in place by the end of 2019. According to the Air Force  and Raytheon, through this new approach, the program aims to introduce  efficiencies building software in several ways:  1.  discovering defects in software code earlier;  2.  reducing the number of defects;  3.  reducing the amount of time it takes to repair defects; and  4.  reducing the overall time to code, integrate, and test OCX software  through automation for some aspects of the software development.

OCX Schedule at Risk for Additional Delays to Delivery and Operation

OCX delivery, acceptance, and the ready to transition to operations  decision will likely be delayed, potentially exceeding the April 2023  threshold date for completing the program. Actual development progress  has been mixed, with some improvement to the pace of software  development. However, the majority of the schedule reserve has been  consumed and defect repairs are taking longer than assumed with  significant work remaining. In addition, a number of new cost and  schedule risks to OCX delivery have arisen since the program schedule  was established. GAO’s schedule and cost estimating best practices  recommend that the schedule assessment be periodically updated to  reflect actual progress and new risks. To mitigate program optimism,  GAO’s cost estimating best practices also state it is important to have an  independent view of cost estimates and schedules. While the Air Force  and the contractor periodically update their schedule estimates, no plans  currently exist for further independent analysis of the full program  schedule within DOD, and there is no requirement to do so.

Significant Development and Testing Remains Before OCX Is Operational

The OCX program has significant work remaining before OCX is  operational, including years of integration and testing. Achieving the full  program schedule requires two interrelated steps. First, in order to meet  the program schedule there must be timely delivery by Raytheon and  acceptance of the system by the Air Force. Second, there must be timely  completion of government-run post-acceptance developmental testing.  Once the Air Force determines that the developmental testing is  completed, OCX will be ready to transition to operations, which ends the  full program schedule. GPS operations will then be transferred from OCS  to OCX. Figure 3 shows the major activities until the ready to transition to  operations decision.

OCX development is expected to continue for approximately 2 more  years, after which Raytheon will submit a Material Inspection and  Receiving Report (Form DD 250) at delivery. The Air Force will then  evaluate OCX for acceptance. Air Force acceptance will be informed by  numerous contractor-run developmental tests conducted to help the Air  Force understand the maturity of the system. Air Force officials will use  information from these contractor tests to inform their approval and  complete acceptance. For example, the Air Force will review data and  demonstrated system capabilities from the tests to determine whether  OCX is ready for integration into the overall GPS. These tests have  formal entrance criteria to demonstrate the system is ready for testing and  exit criteria to ensure tests are successful before proceeding to the next  activity. At the conclusion of contractor testing and delivery to the Air  Force, the Air Force will inspect OCX over approximately 2 months before  OCX is officially accepted. The Air Force will indicate acceptance by  signing the Form DD 250. Currently, the period of performance under the  contract ends June 30, 2021. Consequently, acceptance of the delivered  OCX would need to occur prior to that date.

Air Force Developmental Testing and Rehearsals

After acceptance, Air Force program officials said OCX will go through  government-run developmental testing—currently scheduled to last 7  months—that includes operator transition exercises and rehearsals of the  system. According to OCX program officials, Raytheon will provide interim  contractor support to address any defects or incomplete work as well as  address any additional issues found during the planned 7 month post- acceptance developmental testing. According to program officials, the  ground control operators—who have already been working and providing  feedback—and training and readiness oversight personnel will continue to  work with the new ground system to assess the system’s readiness  through hands-on engagement with the installed system.

At the end of this 7 month period, the Air Force will determine whether the  system is ready to transition to operations. To make the ready to  transition to operations decision, Air Force officials said the system must  receive approval from different groups, including senior leadership within  the Air Force. Once the decision has been made, the Air Force will  transition ground control of the GPS satellite constellation from OCS to  OCX. Additionally, after this transition, which completes the program  schedule, OCX will undergo an operational test and evaluation period,  which will support the Air Force’s separate operational acceptance  decision for OCX.

Delivery: Contractor’s Date Remains Optimistic Compared to Other Estimates

The OCX contractor’s delivery date is optimistic and much earlier than Air  Force and independent projections. All government and independent  analyses project OCX delivery will exceed June 2021 by at least 6  months, but still deliver in time to support the April 2023 threshold (latest  acceptable) date for the full program schedule. However, meeting the  ready to transition to operations threshold date depends on acceptance of  OCX by September 2022, at the latest. This will allow for a planned 7  months of government-run developmental testing that must occur before  April 2023.

Numerous OCX schedule estimates were produced between December  2017 and January 2019. Table 1 indicates the estimator, date of the  estimate, and the reason the estimate was completed.

Figure 4 shows the results of the varying estimates for the start of OCX  operations in months as measured from the beginning of calendar year  2019.

The most recent independent OCX assessment of the delivery date is  from the Defense Contract Management Agency in January 2019. That  assessment estimates that Raytheon’s projected delivery and the cost at  completion are both unrealistic based on staffing profiles, task movement,  completion rates, baseline execution, and schedule performance metrics.  The Defense Contract Management Agency projects that there are not  enough cost and schedule reserves left to cover its own estimate to  complete the work plus all of the identified risks. In fact, the Defense  Contract Management Agency estimates Raytheon will need over $400  million more in cost reserves and that OCX will likely be delivered 11  months after June 2021.

Delivery: Actual Development Progress Is Mixed

Actual development progress has been mixed. While the pace of software  development has improved, implementing the new software development  approach has been slower than expected. As a result, Raytheon has  used the majority of its schedule reserve and delayed planned staff  reductions, indicating that work is not being completed as quickly as  planned. In addition, the schedule assumes improvements to software  defect discovery have not all come to fruition and repair rates have not  been achieved.

Pace of Software Development Has Improved

Under its new software development approach, Raytheon is building and  testing OCX software more quickly than under its previous approach. In  2018, the software development rate to build and test software was  reduced in duration from 4 weeks or more to less than 7 hours on  average—better than planned. Defense Contract Management Agency  officials said that software development has improved compared to block  0 by having a better software development process in place. These  officials cited in particular the improvement that has occurred with the  introduction of software testing automation in some areas. The pace of  software development is one area of many that is necessary to improve  overall performance and achieve the delivery schedule.

Implementing the New Software Development Approach Took Longer than Planned

OCX program officials told us that the full implementation of the new  software approach is foundational to the success of the program; failure  to successfully implement the new approach on time would lead to cost  growth and schedule delays. However, implementation of the new  software approach has taken longer than planned, using a majority of the  available schedule reserve. Defense Contract Management Agency  officials found that since the current baseline was established, Raytheon  consistently takes 5 months to perform 4 months of planned work. This  has not yet delayed the delivery schedule because the program has been  able to use cost and schedule reserves to cover the delays. Between  April 2017, when the current schedule baseline was established, and  January 2019, Raytheon used 4 of the 6 months of total schedule  reserve. As of April 2019, Raytheon had approximately 26 months of work  remaining until June 2021, but only 2 months of schedule reserve. As a  result, there will not be enough time to complete OCX development and  have it accepted by June 2021 unless the contractor significantly reduces  its use of schedule reserve.

Raytheon started using the new software approach on April 1, 2018 to  improve software development, but implementation took longer than  planned for six of the seven initial adoption phases, with two completing  more than a year late. Some of the subsequent expansion phases are  also experiencing delays. For example, phase 3 expansion was  completed more than a year behind the planned schedule. Three other  expansion phases are still in progress and scheduled to complete in mid-  to late-2019.

Raytheon’s divergence from the baseline staffing plan indicates that work  is not being completed as quickly as planned, and more staff have been  needed to prevent additional delivery delays. Raytheon had planned to  reduce the number of staff working on the program from approximately  1,000 to 700 between the autumn of 2017 and the end of 2018. However,  to maintain schedule, Raytheon delayed those reduction plans and  increased staff by approximately 10 percent from January to August  2018. Figure 5 shows the difference between the staffing baseline and  actuals for OCX between January 2018 and December 2018.

Our analysis shows a gap between the January 2018 baseline planned  staffing reduction and actual contractor staffing levels in each month from  January to December 2018, collectively indicating an increase of  approximately 29 percent above the plan. According to DOD’s Office of  Cost Assessment and Program Evaluation officials, this increase is likely  to continue through mid-2019. These officials estimated in their June  2018 independent cost estimate that contractor staffing levels will be  higher than planned through May 2019 so that Raytheon can complete  key software coding events. OCX program officials told us that the  program has been able to afford the additional contractor staff as there  are cost risks to support higher than anticipated staffing levels. They said  that continued increases too far into 2019, however, will result in a breach  of the cost threshold. Further, they said the increased contractor staffing  is consistent with their priority on achieving the delivery schedule.

The new software approach implementation will remain a cost and  schedule risk until at least late 2019. At this time, the final expansion  phase for the new software development approach is planned to be  completed in order to support final testing of the entire system.

Assumed Earlier Defect Discovery Shows Mixed Results and Reduction of Time to Repair Defects Has Not Occurred

Progress finding software defects sooner in development is also mixed.  Raytheon officials told us that cost reductions are possible if they are able  to find defects earlier, as this approach would lead to earlier defect  resolution and reduce any backlog of defects. Further, they said there is  efficiency in having the same developers repair software code that they  created instead of different developers repairing the code later. In March  2018, Raytheon reported increasing the percentage of defects found in  the phase of development where the defect was created from 27 percent  in block 0 to 66 percent in blocks 1 and 2. However, an independent DOD  assessment contradicted this improvement. DOD’s Office of Cost  Assessment and Program Evaluation analyzed Raytheon’s defect  discovery progress a few months into 2018 and found that after showing  some initial improvement, the defect discovery rate dropped from  approximately 53 percent to 24 percent.

In addition, assessing progress discovering defects is now more difficult  to compare with earlier development since Raytheon changed how it  tracks and counts defects in 2018. According to OCX program officials,  Raytheon now only counts a defect if it is repaired in a later phase.  Therefore, if a defect is found and repaired in the same phase, it is not  counted. As of November 2018, Raytheon officials said the predictive  measure they are now using estimates the total number of defects  expected while measuring the actual defects discovered. From this data,  Raytheon found fewer total defects than it predicted, which Raytheon  officials said will result in fewer defects likely to be discovered later in  subsequent phases.

Further, the defect repair-rate—or how many hours it takes to find and  repair a defect—is currently projected to be higher than planned, placing  additional pressure on the delivery schedule. According to Defense  Contract Management Agency officials, the delivery schedule included  defect repair assumptions that were unrealistic. That schedule assumed  30 hours to repair each defect. However, as of November 2018,  Raytheon projects it will need 52 hours to repair each defect on average.  For example, in one area of the program, defects required 61 hours to  repair on average as of December 2018. Defense Contract Management  Agency officials told us that they had concerns that the complexity of the  defects was driving the time needed to repair them. They said that the  more mature software created under the new software approach may be  creating much more complex defects to repair than originally planned.  This may lead to additional schedule delays as the time to repair these  more complex defects may continue to be significantly higher than the  delivery schedule assumed.

More complete data on defects and defect repair rates will likely be  available by the end of 2019, when the final expansion phases of the new  software approach and more software development are completed.

Delivery: Risks Have Changed Substantially Since Schedule Established

How do programs track risk as progress is  made and risks evolve?

Programs track risks to help manage and  mitigate their effect on cost and schedule.   completion requires knowing potential  risks and identifying ways to respond to  them before they happen—using risk  management to identify, mitigate, and  assign resources to manage risks so that  their effects can be minimized.

Raytheon’s estimate that OCX will be accepted by the end of June 2021  is further challenged because of significant identified risks that remain in  the schedule and changeover in those risks in 2018. As of January 2019,  Raytheon was tracking 48 risks it has identified with cost effects—26 with  a moderate likelihood of occurring. For example, a moderate risk includes  the possibility of finding more defects than planned, which could have   both cost and schedule consequences. Other moderate risks include the  possibility of software development taking longer to complete or needing  to create more software code than planned. If realized, both of these risks  have cost effects to pay for additional work and schedule effects to allow  additional time to complete work. As of January 2019, Raytheon has no  high risks that it tracks. There was also a significant amount of change in  the risks themselves in 2018, as Raytheon added 27 new risks while  closing 30. The majority of the risks that are currently tracked will not be  realized or retired until late 2019, with at least one key risk of concern to  the Air Force not realized or retired until 2020.

As the program progresses, risks can (1)

According to OCX program officials, approval to transition OCX to  operations assumes a 7-month developmental test schedule after  acceptance. As currently formulated, this period will be used to prepare  for the transition from OCS to OCX via (1) transition exercises to train  operators, (2) transition rehearsals to practice the actual handover from  OCS to OCX, and (3) a 156-day integrated system test to verify OCX’s  requirements, operational suitability, and readiness to enter operational  testing.

However, that 7-month duration may not be sufficient to conduct all of the  activities that are necessary to verify OCX is ready to transition to  operations. First, the head of the GPS Directorate’s Lead Development  Test Organization, which plans and executes the 7-month developmental  testing, said that there is some schedule risk because of concurrent  activities that need to be accomplished, including crew rehearsals and  other test events. Second, the content of the test period has not yet been  finalized. The planned testing events will be reviewed and refined about 6  months before beginning the test as it becomes clearer what can be  tested and what data will be available from the system. The test director  and the OCX program manager are considering combining some test  events and, if possible, starting some testing prior to acceptance. Third,  the test director and the OCX program manager described a number of  risks that could delay completion of developmental testing including (1)  the late identification of issues requiring significant new software coding  and retesting and (2) identification of new requirements that are not in the  scope of the current effort.

In addition, OCX program officials stated that neither they nor senior Air  Force leadership would transition OCX to operations if the operators are  not ready or requirements have not been verified. They also stated that  there are numerous levels of review within the Air Force, and any of these  decision makers can refuse to approve the transition of OCX to  operations. As a result, according to OCX program officials it could take 5  to 7 months longer than planned, or potentially 14 months total, to  complete developmental testing.

In addition, experience with prior upgrades to the current GPS ground  system indicates that the completion of developmental testing may  require more time than the 7 months assumed in the schedule. Air Force  Cost Analysis Agency officials provided us with data for two upgrades that  were made to OCS, the existing operational ground system. Those  upgrades took 11 and 8 months, respectively. The 11-month upgrade to  OCS from 2006 to 2007 was for an effort that was significantly smaller in  software size in comparison to the size of OCX, but similarly brought new  capabilities to OCS related to the command and control of satellites. The  8-month upgrade to OCS from 2009 to 2010 also provided command and  control of a new type of GPS satellite and enhanced security for the  current GPS receiver cards. Figure 6 shows the different forecasts with 7-  and 14-month developmental test periods as measured from the  beginning of calendar year 2019.

If the time doubles for the completion of post-acceptance government-run  developmental testing, most OCX schedule estimates would exceed the  program schedule threshold.

Air Force and Contractor Updating Schedule Assessments in Accordance with Best Practices, but No Independent Assessments Are Planned of the Full Program Schedule

GAO’s Cost Estimating and Assessment Guide (Cost Guide) and  Schedule Assessment Guide (Schedule Guide) identify best practices for  managing a program’s cost and schedule. According to these best  practices, a well-planned schedule is a fundamental management tool  that can help government programs use public funds effectively by  specifying when work will be performed and measuring program  performance against an approved plan. Typically, schedule delays are  followed by cost growth. When this occurs management tends to respond  to schedule delays by adding more resources or authorizing overtime.  Therefore, a reliable schedule can contribute to an understanding of the  cost effect if the program does not finish on time. Moreover, an integrated  and reliable schedule can show when major events are expected, as well  as the completion dates for all activities leading up to them, which can  help determine whether a program’s parameters are realistic and  achievable.

Further, the Cost Guide states that, too often, programs overrun costs  and schedule because estimates fail to account for the full technical  definition, unexpected changes, and risks. The Cost Guide states that  one of many challenges program managers face is too much optimism in  the original estimate. The Cost Guide also states that because optimism  is often prevalent, organizations will encourage goals that are  unattainable by accentuating the positive. Because over-optimism  potentially affects both cost estimates and schedules, an independent  view and analysis is important to properly overcome this bias. An  independent view also allows decision makers to react sooner and take  steps to minimize any identified risks, like schedule delays. The following  best practices recommend that the schedule estimate should be  periodically updated to reflect (1) actual progress and (2) newly identified  risks.

Periodic Updates and Actual Progress: GAO’s Schedule Guide  states that updating a schedule to reflect actual progress is important  when assessing the realism of the initial schedule duration  assumptions. Programs should make adjustments, if necessary, to the  forecast of the remaining effort.

Periodic Updates and Risk: GAO’s Schedule Guide states that  prudent organizations recognize that uncertainties and risks can  become better defined as the program advances and should conduct  periodic reevaluations of risks. GAO’s Cost Guide states that program  managers often do not sufficiently account for risks because they tend  to be optimistic and because they believe in the original estimates for  the plan without allowing for additional changes in scope, schedule  delays, or other elements of risk.

Since the current schedule was approved in September 2018, Raytheon  has updated its delivery schedule estimate quarterly or as needed to  reflect changes, and modifies the delivery and acceptance dates  accordingly. Raytheon does not update the full program schedule  because the government-run developmental testing is not included in its  schedule estimate. OCX program officials said they are currently updating  their program schedule estimate by incorporating Raytheon’s data  through the end of 2018.

No plans currently exist to conduct another OCX independent cost  estimate—which would include a full, independent program schedule  assessment—at the DOD-level, and currently there is no requirement to  do so. An independent assessment of the schedule would normally be  produced in conjunction with the statutory requirement to conduct another  independent cost estimate at the next major program milestone.  However, in September 2018 the milestone decision authority waived the  requirement to hold the next major program milestone. DOD’s Office of  Cost Assessment and Program Evaluation conducts independent cost  estimates which account for a full program schedule when statutorily  required. In addition, according to an official in that office, they also  conduct only schedule assessments, without completing a full  independent cost estimate, when requested by a program’s milestone  decision authority. In June 2018, the Office of Cost Assessment and  Program Evaluation provided the last full, independent cost estimate with  a schedule assessment to the Under Secretary of Defense for Acquisition  and Sustainment, the milestone decision authority, to support the decision  to approve the OCX baseline. Officials from the Office of Cost  Assessment and Program Evaluation said that they have not been asked  by the OCX milestone decision authority to conduct another independent  assessment. Without an independent schedule assessment, decision  makers may lack updated information when determining whether to take  new steps to avoid or mitigate additional delays.

It is still unknown when OCX will be ready to support the command and  control of the next generation of GPS satellites. While Raytheon has  improved the pace of building and testing software, the majority of  schedule reserve has already been consumed and work is not being  completed as quickly and efficiently as the delivery schedule predicted.  Once software development is complete, it must go through  developmental testing. The schedule for this phase may also be optimistic  as risks associated with competing activities have the potential to double  the amount of time needed for testing.

DOD will be in a better position to assess OCX’s progress and the  potential for additional delays when the majority of its changes to its  software development approach are completed at the end of 2019. At this  time, however, while the program plans to continue assessing schedule  progress, there are no plans in place for an independent schedule  assessment. The program’s history has consistently shown program and  contractor estimates to be optimistic and that independent assessments  have provided useful insights about risks as well as past experience with  similar activities. Our best practice guidance also emphasize that  independent assessments are a necessary step to counter balance  schedule optimism. Decision makers in DOD and Congress could use  realistic knowledge about the schedule to either request or provide  additional funds to complete the acquisition of OCX or develop  contingency plans for delays.

We are making the following recommendation to DOD:  The Secretary of Defense should direct the Director, Office of Cost  Assessment and Program Evaluation to conduct an independent  schedule assessment of the full program schedule for the Global  Positioning System’s next generation operational control system based  on progress made through the end of calendar year 2019.  (Recommendation 1)

We provided a draft of this report to DOD for review and comment. In its  written comments (reproduced in appendix II), DOD did not concur with  our recommendation to conduct an independent assessment of the full  OCX program schedule based on progress made through the end of  calendar year 2019. DOD said that the Office of Cost Assessment and  Program Evaluation conducted an independent cost and schedule  estimate supporting the OCX program’s September 2018 system  development milestone and that DOD subsequently funded OCX  consistent with that estimate. Further, DOD said that the Office of Cost  Assessment and Program Evaluation as well as the Defense Contract  Management Agency continually assess the program’s ability to meet  cost, schedule, and performance objectives. DOD also said the OCX  forecast is currently holding to the government schedule, which is ahead  of the Office of Cost Assessment and Program Evaluation’s independent  cost estimate. Finally, DOD said senior executive reviews continue on a  bi-annual basis to monitor the program’s progress.

We continue to believe the recommendation is necessary. As stated in  our report, DOD has not conducted an assessment of the full schedule  since June 2018, since which time program risks have evolved. In  addition, the other potential sources for schedule oversight suggested by  DOD are limited in scope. The Defense Contract Management Agency  does not look at the full OCX program schedule, as it examines the  schedule only until contractor delivery. Officials from the Office of Cost  Assessment and Program Evaluation said they do some programmatic  monitoring of OCX, including on selected metrics, to inform DOD’s annual  program and budget submission. But those metrics do not examine the  full schedule that includes the developmental test period after delivery.  The Office of Cost Assessment and Program Evaluation is in a position to  independently assess the full OCX program schedule, as it has previously  done, but only if DOD requests that it do so.

We maintain that for complex programs, such as OCX, best practices  state an independent view is necessary and that a periodic schedule  assessment should be performed as progress is made and risks change.  Given the mixed progress developing software, the number of new  contractor risks discovered in 2018, the limited remaining schedule  reserve held by the contractor (with at least two years of significant work  remaining), and the potential for doubling the time frame for the planned  7-month post-acceptance government-run developmental testing period,  we determined that the recommendation remains a prudent step. Such an  assessment would help inform congressional and DOD decision makers  as they consider what steps may be taken to address delays to the start  of OCX operations and ensure the investments in needed new receivers  are properly aligned.

We are sending copies of this report to the appropriate congressional  committees, the Acting Secretary of Defense, the Secretary of the Air  Force, and other interested parties. In addition, the report is available at  no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions concerns this report, please  contact me at (202) 512-4841 or by email at chaplainc@gao.gov. Contact  points for our Offices of Congressional Relations and Public Affairs may  be found on the last page of this report. Key contributors to this report are  listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

To determine the extent to which schedule risks may delay the delivery,  acceptance, and approval for the operation of the Global Positioning  System (GPS) next generation operational control system (OCX), we  reviewed information relevant to OCX from Air Force GPS quarterly  reports, senior management reviews, the program acquisition baseline,  software development plans, monthly program management reviews that  included schedule risks and progress updates, Air Force monthly  acquisition reports, Air Force service cost position documentation,  independent cost estimate documentation and analysis, earned value  management data, Defense Contract Management Agency performance  assessment reports, and slides and information provided by Raytheon  Company (Raytheon), the prime contractor, on topics of our request. We  reviewed the Air Force’s 2018 integrated baseline review results of the  period until government acceptance and assessed the full program  schedule—which includes the contractor’s schedule, government  acceptance, and post-acceptance government-run developmental  testing—until OCX is ready to transition to operations. We reviewed  GAO’s best practice guides for cost estimating and assessment and  schedule assessment to identify best practices for assessing a program’s  cost and schedule and applied selected best practices. We also  reviewed relevant reports and assessments focused on OCX completed  by the government or required by Congress. We interviewed officials from  the OCX program office and GPS Directorate, Defense Contract  Management Agency, DOD’s Office of Cost Assessment and Program  Evaluation, Air Force Cost Analysis Agency, the Lead Development Test  Organization for the GPS Directorate, Defense Digital Services, Office of  the Director, Operational Test and Evaluation, the MITRE Corporation,  and Raytheon.

We conducted this performance audit from November 2017 to May 2019  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

Appendix II: Comments from the Department of Defense

Appendix III: GAO Contact and Staff Acknowledgments

In addition to the contact named above, the following staff members  made key contributions to this report: David Best (Assistant Director),  Patrick Breiding (Analyst-in-Charge), Marie P. Ahearn, Pete Anderson,  Brian Bothwell, Jonathan Mulcare, Andrew Redd, Karen Richey, Roxanna  Sun, and Robin Wilson.

Americans rely on the Global Positioning System daily. The Defense Department is working to upgrade the systems that make up GPS, including completing a critical new ground control system known as OCX.

We found that the OCX upgrade—already delayed approximately 5 years—will likely be delayed further. At least 2 years of work remain. The OCX contractor has already used most of the extra time built into its schedule, leaving little room to fix any problems that may arise.

We recommended an independent assessment of the OCX schedule, which would allow the Defense Department and Congress to better plan how to deal with potential delays.