Medicare Plan Finder: Usability Problems and Incomplete Information Create Challenges for Beneficiaries Comparing Coverage Options

Medicare beneficiaries—more than 60 million as of 2019—have a series of decisions to make when selecting their Medicare health and prescription drug coverage. Beneficiaries must first choose between two main options for their Medicare coverage: either original fee-for-service Medicare or MA. Within these two options, beneficiaries have many additional choices, and they are permitted to change their coverage at least annually. These selections can be difficult due to the Medicare program's complexity and can have important implications for beneficiaries' out-of-pocket costs and access to providers.

According to CMS, the MPF website is intended to help beneficiaries make informed decisions regarding their health care and prescription drug coverage. However, some stakeholders have raised concerns that beneficiaries experience challenges using MPF to compare their Medicare coverage options.

GAO was asked to review MPF. This report examines what is known about the usability of MPF and the completeness of its information. GAO reviewed research and CMS documentation on MPF, and surveyed 51 directors of SHIP offices that have counselors who assist beneficiaries with Medicare decisions. Forty SHIP directors completed the survey, resulting in a 78 percent response rate. GAO also interviewed CMS officials and officials with 13 stakeholder groups, including seven beneficiary advocacy groups.

GAO provided a draft of this report to the Department of Health and Human Services. The department provided technical comments, which GAO incorporated as appropriate.

The Medicare Plan Finder (MPF) website—a primary resource for comparing Medicare coverage options—is difficult for beneficiaries to use and provides incomplete information, according to stakeholders and research studies. These sources and directors of State Health Insurance Assistance Programs (SHIP) GAO surveyed—who assist beneficiaries with their Medicare coverage choices—reported that beneficiaries struggle with using MPF because it can be difficult to find information on the website and the information can be hard to understand. For example, MPF

requires navigation through multiple pages before displaying plan details,

lacks prominent instructions to help beneficiaries find information, and

contains complex terms that make it difficult for beneficiaries to understand information.

In response to GAO's survey, 73 percent of SHIP directors reported that beneficiaries experience difficulty finding information in MPF, while 18 percent reported that SHIP counselors experience difficulty.

Stakeholders and SHIP directors reported that MPF provides incomplete estimates of costs under original Medicare, making it difficult to compare original Medicare and Medicare Advantage (MA), the program's private heath plan alternative. Specifically, MPF's plan results pages do not integrate information on Medigap plans. (These plans help cover some of beneficiaries' out-of-pocket costs.) Seventy-five percent of the SHIP directors surveyed reported that the lack of Medigap information in MPF limits the ability of beneficiaries to compare original Medicare to MA.

The Centers for Medicare & Medicaid Services (CMS)—the agency that administers MPF—is aware of the difficulities beneficiaries face using MPF and is planning to launch a redesigned website in August 2019. According to CMS, redesigning MPF involves multiple iterations of changes and ongoing user testing, and CMS will know more about how well the redesigned MPF addresses user needs after it is used by beneficiaries.

Due to the complexity of the Medicare program, choosing among  Medicare coverage options can be difficult and overwhelming for the  growing number of Medicare beneficiaries. An estimated 10,000 people a  day become eligible for Medicare, adding to the over 60 million Medicare  beneficiaries in 2019. Beneficiaries have two main options for their  Medicare health coverage: original fee-for-service Medicare or Medicare  Advantage (MA), the private plan alternative. Within the two main  Medicare options, beneficiaries have many choices; for example, in 2019  a beneficiary could choose from an average of 24 MA plans, each with  different coverage, costs, and providers. Beneficiaries may need to  consider some key differences when deciding among these options. For  example, beneficiaries in original Medicare may receive covered services  from any Medicare provider while beneficiaries who enroll in MA plans  may have a limited choice of providers. Beneficiaries in MA plans may  receive coverage for additional items and services, such as vision or  dental care, which are generally not covered under original Medicare.  Under either original Medicare or MA, beneficiaries may also choose to  obtain prescription drug coverage. Beneficiaries may change their  coverage during annual open enrollment periods, among other times.

The Centers for Medicare & Medicaid Services (CMS) administers  Medicare Plan Finder (MPF)—an interactive website, www.medicare.gov/find-a-plan—to help beneficiaries and those assisting  them compare and choose their Medicare health coverage options and  drug plans. As a primary source of information to compare Medicare  coverage options, CMS’s goal is for MPF to be a trusted and neutral  resource for Medicare beneficiaries, according to agency officials. Due to  the complexity of choices within Medicare and the implications for  beneficiaries’ costs and access to care, beneficiaries need MPF to be  easy to use and understand. However, some stakeholders have raised  concerns that beneficiaries experience challenges using MPF to compare  their Medicare coverage options.

You asked us to review MPF as a resource to assist beneficiaries in  deciding among their Medicare coverage options. This report examines  what is known about the usability of MPF and the completeness of its  information.

To determine what is known about the usability of MPF and the  completeness of its information, we reviewed CMS-sponsored research  and studies published by beneficiary advocacy groups regarding  beneficiaries’ experiences navigating and understanding MPF. We  reviewed federal law related to the Medicare program and obtained  documentation from CMS on its management of MPF, including past and  proposed changes to MPF. We interviewed officials from CMS and  officials with 13 stakeholder groups knowledgeable about Medicare  beneficiaries’ health insurance needs. Stakeholders included seven  beneficiary advocacy groups, five organizations representing the private  insurance industry, and the State Health Insurance Assistance Program  (SHIP) steering committee.

We also surveyed 51 SHIP directors, who represent the SHIP in their  respective states and are knowledgeable about user experiences with  MPF. We developed the survey based on a review of federal standards  for website design, studies on CMS health insurance websites, including  MPF, and interviews with stakeholders. We pretested the survey with four  selected SHIP directors to help ensure that our questions were  understandable and clear. We conducted the survey between December  2018 and January 2019. The survey asked the SHIP directors for their  opinions on MPF usability and the completeness of its information— based on their interactions with the website and their perceptions of  beneficiaries’ experiences with it. Forty of the 51 SHIP directors submitted  completed surveys, resulting in a 78 percent response rate. Results for  survey questions are expressed as a percentage of all 40 responses  received from SHIP directors. For some questions, a few SHIP directors  either did not respond or reported no basis to respond. We also  interviewed CMS officials about CMS’s processes for managing and  verifying the information posted to MPF and CMS’s plans for future  changes to the website.

We conducted this performance audit from February 2018 to July 2019 in  accordance with generally accepted government auditing standards.  These standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings based on audit objectives. We believe that the evidence obtained  provides a reasonable basis for our findings based on our audit  objectives.

Beneficiaries have several Medicare options from which to select, which  can have important consequences for their out-of-pocket expenses and  access to care. These decisions include the following:

What type of coverage? The first coverage decision faced by  Medicare beneficiaries is choosing between original Medicare or MA.  Original Medicare includes coverage for Medicare Part A services,  such as inpatient hospital stays, and for Medicare Part B services,  such as outpatient hospital care and physician office visits. Under  MA—the private plan alternative to original Medicare—beneficiaries  enroll in MA plans that generally must provide coverage for all the  services included under original Medicare, and may also offer extra  benefits. MA plans generally establish a network of health care  providers to provide services to enrolled beneficiaries.

Add prescription drug coverage? Beneficiaries in original Medicare  and those in certain MA plans may also choose whether to add  prescription drug coverage (Medicare Part D). Prescription drug plans  are administered by private insurance companies that contract with  CMS. Beneficiaries in original Medicare obtain drug coverage by  purchasing a separate prescription drug plan (PDP), while those in  MA generally obtain coverage by selecting a MA plan that offers  prescription drug benefits. MA prescription drug plans and separate  PDPs vary in the amount beneficiaries need to pay and in the drugs  that are covered.

Add supplemental coverage? Beneficiaries in original Medicare can  also purchase Medicare supplemental insurance—known as Medigap  plans—offered by private insurance companies. These plans help pay  for Medicare’s required cost sharing and some out-of-pocket costs not  covered under original Medicare, such as emergency health care  during international travel.

Figure 1 illustrates the decisions beneficiaries have to make when  selecting their Medicare coverage options.

Two research studies we reviewed indicate that cost is a key  consideration for Medicare beneficiaries when selecting Medicare  coverage. Beneficiaries may want to know what their likely out-of-pocket  costs will be monthly, annually, or both. Beneficiaries may also want to  know what their costs may be if they have a change in health status, such  as by experiencing an illness. Beneficiaries may be responsible for  several specific types of health care costs, including the following:

Premiums—Beneficiaries generally make monthly payments to  purchase coverage. Medicare Part A generally does not require  beneficiaries to pay a premium. Part B premiums are established by  statutory formula and are means-tested so that beneficiaries with  higher incomes pay higher premiums. The premiums charged by MA  plans and Part D plans are established by each plan and can vary  widely. Beneficiaries in original Medicare who opt to purchase  Medigap will also pay a monthly premium for coverage, with the  amount of the premium varying across the 10 standardized plans and  by the different companies offering these plans.

Cost sharing—Beneficiaries are typically responsible for paying a  portion of the costs for the services they receive as either a  copayment or coinsurance. A copayment is a fixed dollar amount for  each doctor visit, medical service, or medication. With coinsurance, a  beneficiary pays a percentage of the allowed charge for each health  care service or medication.

Deductibles—Beneficiaries must pay out-of-pocket a specified annual  amount of expenses before Medicare will begin paying for approved  services or medications.

MA plans establish out-of-pocket maximums or set limits on the amount a  beneficiary will have to spend a year. In contrast, original Medicare has  no limit on beneficiary out-of-pocket costs. In 2019, two Medigap plans  provide maximum out-of-pocket limits, and beneficiaries with these plans  do not have to pay costs above the limits.

The same two research studies identified access to particular health care  providers as another key consideration for beneficiaries when selecting  Medicare coverage. Beneficiaries in original Medicare may see any  doctor or use any facility that accepts Medicare payment, and referrals  are not needed to see specialists. In contrast, MA beneficiaries must  typically use the MA plan’s network of health care providers, including  doctors, hospitals, and outpatient facilities, and referrals are generally  needed to see specialists. Further, beneficiaries in MA plans that allow  access to out-of-network providers may be required to pay more when  receiving services from such providers. MA provider networks can change  during the year and from year to year.

According to CMS officials, MPF was launched in 1998 in response to the  Balanced Budget Act of 1997, which required the Department of Health  and Human Services—the agency responsible for overseeing CMS—to  maintain MA plan information on the internet, among other things.  According to CMS, MPF is a primary CMS resource for beneficiaries to  compare costs and coverage of different Medicare health and prescription  drug coverage options in their area, including comparing original  Medicare to MA plans, and Part D plans. As illustrated in figure 2, the  MPF landing page—the first web page users see when accessing MPF— includes a section where beneficiaries start the process of searching for  and comparing coverage options (see A in fig. 2), and a section providing  links to additional decision support tools for beneficiaries (see B in fig. 2).

Beneficiaries begin searching in MPF by entering their zip code and  following a 4-step process that moves them through different MPF  website pages.

Step 1—Basic search: Beneficiaries provide responses to requested  information, including identifying whether they have Medicare coverage  and whether they would like to add prescription drug coverage to their  search.

Step 2—Enter drugs: Beneficiaries may add a list of prescription drugs,  along with the dosage and dosing frequency, to identify which plans cover  these drugs and the cost sharing amount under each plan.

Step 3—Select pharmacies: Beneficiaries select up to two pharmacies  that they prefer for obtaining their medications.

Step 4—Refine plan results: Beneficiaries see a list of available coverage  options—original Medicare, MA plans, and separate PDPs—based on the  zip code they entered. Beneficiaries can filter these search results by  variables such as monthly premium or deductible amounts, and then they  can sort those results by variables such as lowest estimated annual costs  or lowest plan deductible. Beneficiaries can then select up to three  choices, view specific coverage and cost details for each, and do a  detailed side-by-side comparison of each. The plan results page shows  this comparison and includes beneficiaries’ estimated annual out-of- pocket costs for each coverage option they choose to review.

The additional decision support tools available on the MPF landing page  that beneficiaries may use to help select their Medicare health and drug  coverage include the following:  1.  Help with Your Medicare Choices, which uses filtering questions to  help new beneficiaries understand their Medicare coverage choices;  2.  Estimate Medicare Costs, which helps beneficiaries compare the  average estimated costs of original Medicare options, such as original  Medicare with a prescription drug plan and a Medigap plan, to the  costs of MA with prescription drug coverage; and  3.  Find and Compare Medigap Policies, which helps beneficiaries find  information on the different standardized Medigap plans offered by zip  code.

Since its inception, MPF has undergone many modifications as new parts  were added to the Medicare program, such as the addition of Medicare  Part D. According to CMS officials, the agency has also taken steps to  make additional changes to improve the website, including technology  updates to improve system stability and performance, such as page load  times and error rates. In addition, CMS seeks feedback from  stakeholders, such as the customer service representatives at the 1-800- MEDICARE help line, SHIP personnel, and others, which according to  agency officials, has resulted in additional changes. Changes have  included allowing beneficiaries to log into their Medicare account to  access some of their existing data, such as their prescription drugs, and  the addition of a help feature that can connect beneficiaries to 1-800- MEDICARE customer service representatives for live help.

Stakeholders and Research Indicated Medicare Plan Finder Is Difficult to Use and Provides Incomplete Information, and CMS Is Redesigning It to Make Improvements

Challenges Navigating and Understanding Information in Medicare Plan Finder Make It Difficult for Beneficiaries to Use, According to Stakeholders and Research

Stakeholders, research studies, and SHIP directors responding to our  survey generally indicated that MPF is difficult for beneficiaries to  navigate and understand. All 13 stakeholder groups we interviewed  reported that MPF is challenging for Medicare beneficiaries to use.  Specifically, most stakeholders cited difficulty navigating as beneficiaries  click through multiple complex pages in order to find and compare  coverage options. For example, two stakeholders noted that beneficiaries  must answer questions about their current Medicare health and drug  coverage and then go through a series of pages and steps before they  can view detailed information on their coverage options. One of these  stakeholders also told us that MPF navigation is cumbersome because  users cannot jump directly to certain pages or sections that address their  needs, such as viewing the availability of preferred pharmacies. One of  the stakeholders we interviewed also noted the lack of prominent  instructions on how to use MPF contributed to difficulties navigating the  four steps. Finally, in our interviews two stakeholders also noted that  navigation is difficult because beneficiaries are uncertain of the  information needed to make different comparisons or identify specific  plans. For example, the ability to filter and sort plan information does not  appear until later in the plan search process, where users are refining  plan results. This makes it hard for users to narrow options specific to  their needs because they first must go through all the options presented.  Specifically, beneficiaries will first see a list of plans available in their zip  code—on average 24 plans—and then must narrow down that list before  they can compare up to three selected plans.

A 2018 report conducted jointly by two advocacy groups cited difficulties  locating the filter and sort functions in MPF, which contributed to  navigation problems. CMS user testing conducted on MPF found that  overall beneficiaries are confused about how to find a MA plan on MPF.  For example, this testing showed that some users had difficulties with the  steps for refining plan results because they overlooked or ignored the  filters. A 2017 CMS study noted that MPF navigation is difficult and is  better suited for specialist users who assist beneficiaries in determining  their coverage options, such as 1-800 MEDICARE customer service  representatives and SHIP counselors. Further, CMS officials said the  study found that beneficiaries would benefit if navigation through the site  were more tailored to the tasks they were undertaking.

Our survey of SHIP directors, who provide assistance to Medicare  beneficiaries and therefore are familiar with MPF usability, also found that  it is difficult for beneficiaries to navigate and find information. Specifically,  73 percent (29 of 40) of the SHIP directors who responded to our survey  reported that it is difficult or very difficult for beneficiaries to find  information in MPF. While SHIP directors reported that it is easier for  SHIP counselors to find information, they noted that some also  experience difficulty. Eighteen percent (7 of 40) SHIP directors reported  that it is difficult for SHIP counselors to find information in MPF. (See fig.  3.)

In addition to website navigation, it is also difficult for beneficiaries to  understand the information in MPF, according to stakeholders, research  studies, and SHIP directors responding to our survey. All seven  beneficiary advocacy groups interviewed reported that beneficiaries find it  challenging to understand information in MPF. For example, some  stakeholders noted that beneficiaries do not always understand  terminology, such as the differences between cost sharing, copayment,  and out-of-pocket costs. Most stakeholders also noted that beneficiaries  struggle to understand cost estimates and interpret how much they will  have to pay. CMS user testing of MPF in 2018 found that beneficiaries  were overwhelmed by the number and complexity of options from which  they had to choose. According to a 2018 research study conducted by two advocacy groups, the website explains health coverage terminology  poorly and does not use plain language. As a result, users with low health  insurance literacy may not understand, for example, the cost differences  between generic versus brand-name drugs.

Sixty-five percent (26 of 40) of the SHIP directors we surveyed reported  that the information in MPF is difficult or very difficult for beneficiaries to  understand, while 23 percent (9 of 40) reported that it is difficult for SHIP  counselors to understand information (see fig. 4). SHIP directors  identified health coverage terminology as a challenge, with 38 percent (15  of 40) reporting that MPF does a poor or very poor job explaining health  coverage terminology, such as non-network providers, drug formularies,  and drug tiers to beneficiaries.

According to Stakeholders, Medicare Plan Finder Provides Incomplete Information on Costs and Coverage, Making it Difficult to Compare Medicare Options

MPF provides incomplete estimates of beneficiaries’ costs under original  Medicare, making it difficult to compare coverage options, according to  stakeholders and SHIP directors responding to our survey. The cost  estimates on the plan results pages are incomplete because they do not  include the effect of Medigap—which helps cover beneficiaries’ cost  sharing responsibilities under original Medicare. As a result, beneficiaries  who want to use MPF to compare original Medicare with a Medigap plan  to specific MA plans are unable to do so. Most—4 of 7—beneficiary  advocacy group stakeholders that we interviewed noted that beneficiaries  must leave MPF to obtain information about Medigap plans, such as the  specific benefits covered under those plans and their estimated costs.  Six of seven beneficiary advocacy groups that we interviewed noted that  MPF’s incomplete information on estimated beneficiary costs is a concern  because beneficiaries need this information for understanding and  comparing their Medicare options. CMS’s other coverage decision  support tools—Help with Your Medicare Choices and Estimate Medicare  Costs—provide general information intended to help beneficiaries  understand and compare their Medicare options. However, these tools  are separate links; their information is not included on the plan results  pages in MPF.

The SHIP directors we surveyed also noted lack of information as a  concern, with 75 percent (30 of 40) reporting that the lack of Medigap  information in MPF limits the ability of beneficiaries to compare original  Medicare and MA plans. Further, SHIP directors surveyed reported more  general concerns with MPF’s cost estimates, with 80 percent (32 of 40)  reporting that improvements are needed to better estimate total annual  beneficiary costs, and 63 percent (25 of 40) of the SHIP directors  reporting that MPF does a poor or very poor job comparing the costs of  original Medicare to MA.

Stakeholders and SHIP directors responding to our survey reported that  MPF also provides incomplete information on MA plan provider networks.  According to a CMS-sponsored study, determining if specific providers  are in an MA plan provider network is a key factor for beneficiaries when  making coverage decisions, and beneficiaries stated in user testing that  they must have this information. However, to obtain information on the  providers in specific MA plans, MPF users must exit the website and go to  the individual plan websites. Most stakeholders—10 of 13—cited the lack  of information on provider networks as a shortcoming for beneficiaries in  using MPF to select a plan, with one group stating that MPF users may  need to call individual plans to determine if providers are in a plan’s  network. SHIP directors also cited this issue as a problem, as 85 percent  (34 of 40) who responded to our survey reported that the lack of a  provider directory limits MPF as a resource for beneficiaries to compare  MA plans. Without provider information, beneficiaries are not able to use  MPF to narrow their options to MA plans that include desired providers or  make comparisons among these plans.

CMS Is Redesigning MPF in an Effort to Improve its Usability and the Completeness of Cost Information

According to CMS officials, the agency is redesigning MPF to make it  more usable for beneficiaries and is planning to release the redesigned  MPF in early August 2019. With the redesign, CMS plans to improve the  navigation of MPF by providing more prominent explanations on how to  use MPF; reducing the steps users must take to get to more detailed  coverage information; configuring MPF so users can more easily switch  between different topics inside MPF, such as switching between MA plan  information and Part D plan information; and improving the filter and sort  functions so users can narrow down their coverage options more quickly.  CMS also plans to make information easier to understand by simplifying  and reducing the volume of information on the pages and revising  frequently misunderstood terms with more user-friendly language.

As part of the redesign, CMS is also taking steps to provide more  complete cost information in MPF to help compare coverage options,  according to agency officials. CMS plans to provide more information to  the redesigned MPF to help beneficiaries understand their coverage  options and decide whether original Medicare or MA is right for them.  CMS officials also told us in June 2019 that the redesigned MPF will allow  beneficiaries to do estimated cost comparisons of MA to all their original  Medicare options, such as original Medicare with a Medigap plan and a  prescription drug plan. Officials also told us that CMS is incorporating the  functionality of the additional decision support tools currently available on  the MPF landing page—Help with Your Medicare Choices and Estimate  Medicare Costs—into the redesigned MPF to help beneficiaries  understand their coverage options and compare their estimated costs  across these options. In June 2019, CMS officials stated these additional  tools will also continue to appear as separate links on the MPF landing  page. CMS officials also told us that they are currently examining how to  integrate MA plan provider information, but this is not part of the  redesigned MPF being released in August 2019. The officials said they  are working with the plans to develop requirements to help support the  integration of provider directories into future versions of MPF.

According to CMS, the redesign of MPF is not finalized and CMS will  continue to evaluate the extent to which the changes will make MPF  easier for beneficiaries to use and whether it provides complete  information for making coverage decisions. As of June 2019, CMS  officials told us they are continuing to gather feedback from stakeholders,  such as 1-800-MEDICARE customer service representatives and SHIP  personnel, and conduct user testing on a redesigned MPF model. CMS  then plans to publicly launch the redesigned MPF to a subset of users in  early August 2019. Once launched, CMS plans to incorporate feedback  from this subset of users to confirm the core features that will be released  in the redesigned MPF prior to the Medicare open enrollment period  starting October 15, 2019. According to CMS officials, the development of  the redesigned MPF is an incremental process that will involve  continuous changes based on feedback and user testing. According to  the agency, CMS will know more about how well the redesigned MPF  addresses user needs after it is used by beneficiaries.

We provided a draft of this report to the Department of Health and Human  Services for review and comment. The Department of Health and Human  Services provided technical comments, which we incorporated as  appropriate.

As agreed with your office, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies of this report to the  Secretary of Health and Human Services and other interested parties. In  addition, the report will be available at no charge on GAO’s website at  http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-7114 or cosgrovej@gao.gov. Contact points for our  Office of Congressional Relations and Office of Public Affairs can be  found on the last page of this report. GAO staff who made key  contributions to this report are listed in appendix I.

Appendix I: GAO Contact and Staff Acknowledgments

In addition to the contact named above, Tim Bushfield, Assistant Director;  Maggie G. Holihan, Analyst-in-Charge; Sylvia Diaz Jones; Anne  Hopewell; Dennis A. Antonio; and Dan Ries made key contributions to  this report. Also contributing were Cathy Hamann, Krister Friday, Ethiene  Salgado-Rodriguez, Julie Flowers, and Jennifer Rudisill.

Medicare beneficiaries have many decisions to make when selecting their health and prescription drug coverage. Their choices affect their out-of-pocket costs and which providers they can see. The Medicare Plan Finder website is a primary source for comparing options.

Many officials who assist beneficiaries in selecting coverage—about three-quarters of those we surveyed—told us beneficiaries struggle with the website. They and others said it is difficult to navigate, contains complex terms, and lacks information needed to compare coverage options.

Medicare’s administrator plans to launch a redesigned website in August.


Cost-Type Contracts: Procedures Needed for Sharing Information on Contract Choice among Military Departments

When acquiring major weapon systems, DOD can choose between several different contract types. One of these is cost-type, under which DOD pays allowable costs incurred by the contractor. Historically, DOD has struggled to manage its major acquisition programs. The result has been billions in cost growth and schedule delays in providing systems to the warfighter.

GAO was asked to review DOD's use of cost-type contracts for its major acquisition programs. This report addresses the use of and range of cost and schedule outcomes for cost-type contracts for major weapon system acquisitions, and how military departments share information about contract choice.

GAO analyzed government contracting data on obligations by contract type for fiscal years 2011 through 2019 on contracts in DOD's portfolio of major acquisition programs. GAO compared contract types for 21 major acquisition programs with their cost and schedule outcomes; reviewed seven recently awarded cost-type contracts for major acquisition programs, selected to reflect the different military departments and appropriation types; and interviewed contracting officials.

To acquire new major weapon systems, such as aircraft, ships, and satellites, the Department of Defense (DOD) uses a variety of contract types including cost-type contracts, under which the government assumes more risk. DOD is required to document its risk assessment in choosing contract types for major programs. Risks assessed can include use of new technologies and stability of system costs and requirements. Once awarded, cost-type contracts have additional reporting requirements to help monitoring of cost and schedule performance.

GAO analyzed program cost and schedule outcomes for 21 major acquisition programs, and did not find a clear relationship between these outcomes and contract types used. However, programs that completed certain knowledge-based acquisition practices generally had better cost and schedule outcomes than programs that did not implement those practices. These practices include completing preliminary design review before the start of system development and releasing at least 90 percent of design drawings by critical design review.

From fiscal years 2011 through 2019, DOD used cost-type contracts for a small proportion—under one-fifth on average—of obligations for its major acquisition programs. This proportion varied across the military departments (see figure).

A change to DOD's peer review process for its largest contract awards reduced a means for sharing best practices and lessons learned about contract choice across the military departments. In 2019, the Office of the Secretary of Defense announced the end of its peer reviews for most competitive procurements above $1 billion. While these contracts will instead be reviewed through the military departments' own processes, DOD currently does not require the departments to collect and share their findings. DOD has an online compendium of peer review findings; however, this was last updated in 2013. Using an existing centralized resource such as the compendium could help contracting officials learn from the experiences of peers across DOD by exposing them to good practices for structuring contracts.

GAO recommends that DOD establish procedures requiring the military departments to collect and share findings from their reviews of contracting approaches, such as by updating the existing online compendium. DOD agreed with GAO's recommendation.

The Department of Defense (DOD) has historically provided its new major  weapon systems—including aircraft, ground vehicles, missiles, ships, and  satellites—to the warfighter through its major defense acquisition  programs (MDAP). DOD expects its current portfolio of 85 MDAPs to  cost $1.8 trillion in total. Despite some improvements in recent years,  DOD has historically struggled to meet cost and schedule expectations  for its MDAPs. This has resulted in billions of dollars of cost growth and  delays in providing systems to the warfighter. As a result, weapon  systems acquisition has been on GAO’s High Risk List since 1990. DOD  typically contracts with private-sector companies in order to acquire these  systems, using a variety of contract types. These include cost-type  contracts that shift risk onto the government and away from the  contractor, particularly for complex weapon systems development work  that may lack precise specifications and accurate cost estimates.

You asked us to review DOD’s use of cost-type contracts for MDAPs.  This report addresses: (1) the extent to which DOD uses cost-type  contracts for MDAPs; (2) how DOD chooses among cost-type and other  contract types for MDAPs and monitors their cost and schedule  performance; (3) the range of cost and schedule outcomes across  MDAPs that used cost-type contracts; and (4) the extent to which DOD  shares information about choosing MDAP contract types across the  military departments.

To assess the extent to which DOD uses cost-type contracts for MDAPs,  we analyzed Federal Procurement Data System-Next Generation (FPDS- NG) data regarding obligations by contract type from fiscal year 2011  through fiscal year 2019 on contracts for programs in DOD’s MDAP  portfolio awarded from fiscal year 2010 through fiscal year 2018. We  assessed data reliability by comparing the contract types identified in  FPDS-NG for each contract with information on contract types contained  in two DOD databases—Defense Acquisition Management Information  Retrieval (DAMIR), and Earned Value Management-Central Repository— and determined the data were sufficiently reliable for the purposes of  analyzing the extent of DOD’s use of cost-type contracts for MDAPs.

To assess how DOD chooses among cost-type and other contract types  for MDAPs and monitors their cost and schedule performance, we  analyzed documentation and interviewed officials regarding contract  choice and monitoring from DOD, military departments, and selected  contracting commands. As illustrative examples of contract choice and  monitoring under a variety of conditions, including different military  departments and appropriation types, we also selected a  nongeneralizable sample of seven MDAP contracts. Specifically, we  selected for each of the three military departments the most recently  awarded cost-type MDAP Research Development, Test, and Evaluation  contract and the most recently awarded cost-type MDAP Procurement  contract as reported in the December 2017 Selected Acquisition Reports.  We also selected the most recently awarded cost-type MDAP contract for  the Marine Corps. We interviewed contracting officials and reviewed key  documentation such as acquisition strategies relating to each one of  these contracts. We also reviewed our past work related to contract types  used for MDAPs, including DOD’s use of incentive contracts and the  Navy’s use of fixed-price-incentive contracts for shipbuilding.

To assess the range of cost and schedule outcomes across MDAPs that  used cost-type contracts, we identified the 21 non-shipbuilding MDAPs in  DOD’s current portfolio that as of January 2019 had completed system  development, held a critical design review, and started production. We  then compared the unit cost and schedule changes between each  program’s first full estimate and our most recent in-depth assessment of  the program as of May 2019 with the types of contracts each program  used.

Finally, to assess the extent to which DOD shares information about  choosing MDAP contract types across the military departments, we  reviewed DOD and military department documentation related to  contracting review processes. We compared this information to DOD  memorandums establishing practices and policies for sharing of  acquisition information across DOD. We also interviewed officials from  offices including Defense Pricing and Contracting (DPC) within the Office  of the Under Secretary of Defense for Acquisition and Sustainment  (USD(A&S)), and the cognizant Deputy Assistant Secretaries of the  military departments. See appendix I for more information on our  objectives, scope, and methodology.

We conducted this performance audit from February 2019 to May 2020 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

Background Contract Types Described by the Federal Acquisition Regulation

The government can choose from a wide selection of contract types to  acquire the variety and volume of supplies and services agencies require  to meet their needs. Contract types vary according to the degree and  timing of the responsibility assumed by the contractor for the costs of  performance, and the amount and nature of the profit incentive offered to  the contractor for achieving or exceeding specified standards or goals.

The primary contract types described by the Federal Acquisition  Regulation (FAR) fall into two broad categories—cost-type and fixed- price-type—and table 1 summarizes key features of each.

As illustrated in figure 1, within these categories the specific contract  types range from cost-plus-fixed-fee, in which the contractor has minimal  responsibility for the performance costs and the negotiated fee (profit) is  fixed, to firm-fixed-price, in which the contractor has full responsibility for  the performance costs and resulting profit (or loss). In between are the  various incentive contracts, under which the contractor’s responsibility for  the performance costs and the profit or fee incentives offered are tailored  to the uncertainties involved in contract performance. For contracts with  incentive fees or profits, the amount of fee or profit payable is related to  the contractor’s performance, and generally involves an objective  evaluation by the government of the contractor’s performance toward  cost, schedule, or technical goals. Award fees, on the other hand,  typically emphasize multiple aspects of contractor performance that are  more subjectively assessed, such as the contractor’s responsiveness,  technical ingenuity, or cost management. Furthermore, the basic types of  contracts may be used in combination, with both fixed-price-type and  cost-type contract line item numbers, unless otherwise prohibited. For  example, a firm-fixed-price contract may have a cost-type line item for  travel.

The FAR states that selecting the contract type is generally a matter for  negotiation and requires the exercise of sound judgment by the  contracting officer. Negotiating the contract type and negotiating prices  are closely related and should be considered together. The objective is  for the government to negotiate a contract type and price (or estimated  cost and fee) that will result in reasonable contractor risk and provide the  contractor with the greatest incentive for efficient and economical  performance. As also noted in the FAR, the government usually  assumes greater risk in its contracts for more complex requirements,  particularly those unique to the government. This is especially true for  complex research and development contracts, where performance  uncertainties or the likelihood of changes make it difficult to estimate  performance costs in advance. Cost-type contracts are suitable for  instances when uncertainties about contract performance do not allow  accurate enough cost estimates to use a fixed-price-type contract—in  other words, when programs choose to accept more risk. The level of risk  drives the contract type chosen, with the contract then reflecting the risk  of the work.

DOD programs may use different contract types across the life of the  MDAP. For example, DOD guidance notes that the preferred contract  type for development efforts is cost-type, and requires particular  consideration of fixed-price-incentive contracts for acquisitions moving  from development to production. Consistent with the FAR, DOD guidance  also notes that firm-fixed-price production contracts may be in the  government’s best interest once costs have become stable.

DOD and Congress have encouraged use of fixed-price-type contracts  where appropriate. For example, DOD’s Better Buying Power initiative,  which started in 2010, called for increased use of fixed-price-incentive  contracts for programs transitioning from development to production. In  addition, the National Defense Authorization Act (NDAA) for Fiscal Year  2017 required DOD to establish a preference for fixed-price-type  contracts in the determination of contract type and specified approval  requirements for use of cost-type contracts above certain dollar  thresholds. Congress has also limited DOD’s ability to use cost-type  contracts to acquire production units absent congressional notification.  Our prior work contains many recommendations related to incentive-type  contracts. For example, in March 2017 we recommended that the Navy  remind contracting officials to follow guidance on documenting the  rationale for using fixed-price-incentive contracts, and in April 2017, the  Navy issued a memorandum addressing this issue. In July 2017 we  recommended that DOD collect and analyze data to determine the extent  to which incentive contracts achieved desired outcomes. While DOD  agreed with the recommendation and developed a template for the  military departments to use to collect relevant information, it is still  gathering updates from the military departments about the status of this  effort.

Contracting for Major Defense Acquisition Programs

DOD acquires MDAPs through the Defense Acquisition System, which  implements an adaptive acquisition framework that allows DOD officials  to develop acquisition strategies and employ acquisition processes that  match the characteristics of the capability being acquired. The pathway  for acquiring major capabilities generally includes four phases, three of  which we focus on in this report: (1) technology maturation and risk  reduction; (2) engineering and manufacturing development; and (3)  production and deployment. Programs typically complete a series of  milestone reviews and other key decision points that authorize entry into  a new acquisition phase, as illustrated in figure 2.

These milestones also typically mark critical contract award decisions.  For example, the Milestone B decision commits the resources, including  authorizing award of the program’s development contract, needed to  conduct development leading to production. Milestone C represents the  decision to move forward with initial production, including award of the  initial production contract. A number of officials and agencies are  involved in DOD’s choice and monitoring of MDAP contracts.

Milestone decision authority: The designated individual with overall  responsibility for the program who, at the time of key milestone reviews,  approves the acquisition strategy with specified contract types. In  approving the acquisition strategy, this individual must ensure that the  strategy considers how to manage risk and how the contract type  selected relates to the level of program risk in each acquisition phase.  This individual is to use the acquisition strategy to assess the viability of  the proposed approach, ensuring that it clearly explains how it is to be  implemented with available resources, and is tailored to address program  requirements and constraints.

Milestone decision authority for most MDAPs now resides with the military  departments following a reform enacted in the NDAA for Fiscal Year  2016. Prior to this reform going into effect, a position within the Office of  the Secretary of Defense typically served as the milestone decision  authority for MDAPs until they entered the production and deployment  phase. Following a reorganization of the Office of the Secretary of  Defense enacted in the NDAA for Fiscal Year 2017, the USD(A&S) now  serves as milestone decision authority for a small number of MDAPs,  such as the F-35 program. For other MDAPs, the following officials  serve as milestone decision authority within the military departments:  the Assistant Secretary of the Air Force (Acquisition, Technology, and  Logistics);  the Assistant Secretary of the Army (Acquisition, Logistics, and  Technology); and  the Assistant Secretary of the Navy (Research, Development, and  Acquisition).

Program manager: The designated individual with responsibility for and  authority to accomplish program objectives for development, production,  and sustainment to meet user operational needs. The program manager  plans acquisition programs, prepares programs for key decisions, and  executes approved acquisition and product support strategies.

Contracting officer: The individual with the authority to enter into,  administer, or terminate contracts and make related determinations and  findings. Contracting officers are responsible for ensuring performance of  all necessary actions for effective contracting, ensuring compliance with  the terms of the contract, and safeguarding the interests of the United  States in its contractual relationships. In order to perform these  responsibilities, contracting officers are allowed wide latitude to exercise  business judgement.

Defense Contract Management Agency (DCMA): The entity that  provides contract administration services for most DOD buying activities.  Its contract management offices work with defense contractors to help  ensure they deliver goods and services that meet performance  requirements on time and at projected cost.

Supervisor of Shipbuilding, Conversion and Repair (SUPSHIP): The  entity that is the Navy’s on-site technical, contractual, and business  authority for the construction of Navy ships. SUPSHIPs are co-located  with the nation’s major shipbuilders and oversee the construction of every  Navy ship, from patrol craft to the Navy’s most complex surface  combatants and nuclear submarines and aircraft carriers.

In addition to serving as milestone decision authority for certain MDAPs,  USD(A&S) is responsible for improving outcomes by gathering and  distributing best practices and lessons learned across the military  departments. One such mechanism related to contract type choice,  established in 2008, was mandatory preaward peer review—conducted  by DPC, an office within USD(A&S)—for solicitations and contracts  valued at over $1 billion and noncompetitive procurements over $500  million. For these competitive procurements, DPC conducted phased  peer reviews prior to three events—issuance of the solicitation, issuance  of the request for final proposal revisions, and contract award. The peer  review teams—composed of senior DOD contracting leaders and officials  from other military departments, and whenever possible comprising the  same personnel across the three phases—discussed contract type and  structure, and reviewed key program documentation such as acquisition  strategies. Upon completion of a review, the team provided its findings  and recommendations to the contracting officer, among other officials.  However, in August 2019, DPC announced that it would no longer  conduct peer reviews for most competitive procurements above $1 billion.  Further details of this change are discussed later in this report.

While the individual military departments have distinct requirements for  the weapon systems they acquire, they also on occasion procure similar  types of platforms, and use the same relatively small pool of contractors.  For example, the Air Force and Navy both purchase fighter aircraft, and  all three military departments buy missile systems. In 2019, we analyzed  the 183 major development and procurement contract awards for MDAPs  reported by DOD at that time, and found that almost half went to five  corporations and entities connected with them, constituting 72 percent of  the dollars associated with those contracts.

Small Proportion of Obligations for Major DOD Acquisitions Since 2011 Was on Cost-Type Contracts and Level Varied across Military Departments

From fiscal year 2011 through fiscal year 2019, a small proportion—an  average of less than one-fifth—of obligations for programs in DOD’s  portfolio of MDAPs was on cost-type contracts, although this proportion  varied across the military departments. The remainder were on fixed- price-type contracts, split between firm-fixed-price and fixed-price- incentive, as illustrated in figure 3.

Figure 4 illustrates the proportion of obligations by contract type for each  of the military departments across the 9-year period. The Air Force made  the most use of cost-type contracts, at an average of around one-quarter  of obligations. While the Army made the least use of cost-type contracts,  it made the most use of firm-fixed-price contracts. The Navy made the  most use of fixed-price-incentive contracts. We have previously reported  that the Navy has generally used cost-type contracts for lead ships and  fixed-price-incentive contracts for follow-on ships.

Choice of Cost-Type Contracts Informed by Program Risk and Subject to Additional Risk-Based Monitoring

We found that the choice of cost-type contracts for MDAPs by contracting  officers is based on assessments of program risk and uncertainty,  underpinned by a number of statutory, regulatory, and policy provisions.  Risk assessment also drives the application of additional reporting and  surveillance requirements—designed to help the program office monitor  cost and schedule performance—once DOD has awarded a cost-type  contract for an MDAP.

Choice of Cost-Type Contracts Is Based on Consideration of Program Risk and Uncertainty

A range of statutory, regulatory, and policy provisions emphasize the  importance of considering program risk and uncertainty when planning  acquisitions and determining contract types for MDAPs. These provisions  guide the decisions of contracting officers when choosing contract type  and establish documentation requirements such as acquisition strategies.  Table 2 describes key provisions related to program risk and uncertainty.

Contracting and program officials, among others, collaborate and  determine the appropriate contract type based on assessments of risk,  considering factors such as availability of historical contract information,  use of new technologies, cost stability, and the level of definition of  requirements, such as software. In arriving at these determinations,  officials we met with noted the importance of contracting officers having  experience using a range of contract types.

The seven MDAP cost-type contracts included in our review had  documented rationales for their choice that all indicated areas of risk and  uncertainty, addressing provisions noted in table 2. For example, four  were development contracts, and FAR Part 35 states that the use of cost- type contracts for research and development is usually appropriate given  the absence of precise specifications and difficulties in accurately  estimating costs. The other three cost-type contract rationales noted that,  consistent with the FAR, uncertainties in contract performance did not  allow for costs to be estimated with sufficient accuracy to use a fixed- price-type contract. Table 3 summarizes these rationales.

Additional Risk-Based Reporting Requirements for Cost-Type Contracts Designed to Help Programs Monitor Cost and Schedule Performance

Contract types that shift more risk onto the government—including cost- type contracts—and exceed certain dollar thresholds have additional  contractual reporting requirements. These requirements are designed to  help the program office to monitor cost and schedule performance. In  order to receive a cost-type or incentive contract valued at $20 million or  more, a contractor must have an earned value management (EVM)  system that complies with certain guidelines. These systems integrate the  scope of work with cost, schedule, and performance elements to support  project planning. They also provide program offices with monthly contract  performance reports that include cost and schedule status and risks. Our  prior work contains recommendations related to DOD’s use of EVM. For  example, in 2009 we recommended that DOD modify policies governing  EVM to ensure they addressed a number of weaknesses we had  identified. In response, DOD developed and incorporated into its program  management curricula a new EVM training course.

Among the duties of two specialized government contract administration  agencies—DCMA and SUPSHIP—are the review and approval of  contractor EVM systems, and ongoing surveillance of data generated by  the systems. The regular reports provided to program offices by these  agencies include EVM data and analysis and highlight areas of concern  and contract performance risk.

In addition to use of EVM data, contracting officials from the seven cost- type MDAP contracts included in our review noted the importance of  regular interactions between DOD—whether the program office, DCMA,  or SUPSHIP—and the contractor in order to proactively identify drivers of  cost or schedule overruns. These interactions can range from day-to-day  tracking to comprehensive quarterly reviews. Several officials also noted  the importance of having DCMA and SUPSHIP representatives on-site at  contractor facilities, overseeing the contract and communicating with the  contractor.

Program Outcomes Vary Regardless of Contract Type but Correspond to the Use of Knowledge to Reduce Risk

Our analysis of program cost and schedule outcomes for 21 MDAPs did  not find a clear relationship between these outcomes and the contract  type used. DOD’s current portfolio of MDAPs contains a total of 85  programs. The 21 MDAPs in our review are the non-shipbuilding subset  of the 85 that, as of January 2019, had completed system development,  held a critical design review, and started production. Thus, these 21  programs are sufficiently far along the acquisition process that we can  analyze their cost and schedule outcomes. We found that they  demonstrated a range of cost and schedule performance, regardless of  contract type chosen. Table 4 notes the contract types used for these  MDAPs as well as unit cost and schedule change between each  program’s first full estimate and our most recent in-depth assessment of  the program as of May 2019. As reflected in the table, all but four of the  MDAPs used some mix of cost-type and fixed-price-type contracts.

Performance varied widely for programs using cost-type contracts at  some stage, with unit cost change varying from 44 percent reduction to  183 percent growth, and schedule change varying from zero to 146  percent growth. In addition, while two of the three programs that used  only fixed-price-type contracts had unit cost reductions, they also  experienced schedule growth of over 40 percent. Programs generally  made greater use of cost-type contracts than fixed-price-type contracts  during development, and greater use of fixed-price-type contracts during  procurement, as knowledge built over time.

While we did not find a clear relationship between contract type and cost  and schedule performance, we have found a relationship between  improved outcomes and implementation of certain knowledge-based  acquisition practices on these 21 programs. These are practices identified  in our body of prior work that ensure a high level of knowledge is  achieved at key junctures in development. We apply these practices as  criteria in weapon system reviews, including our annual assessment of  weapon systems. As shown in table 5 and based on analysis of the 21  programs, in general MDAPs that implemented certain knowledge  practices—thus reducing risk—before the start of system development  and critical design review had better unit cost and schedule outcomes  than those that did not. The first such practice—completing preliminary  design review before system development start—means that a program  has held a review that assesses the maturity of the preliminary design,  supported by the results of activities including prototyping and critical  technology demonstrations. The second practice—release of at least 90  percent of drawings by critical design review—refers to the design  drawings released or deemed releasable to manufacturing by that point.

Our prior work has shown that establishing a sound business case is  essential to achieving better program outcomes. A solid, executable  business case provides credible evidence that the warfighter’s needs are  valid and can best be met with the chosen concept. The business case  should also demonstrate that the chosen concept can be developed and  produced within existing resources such as technologies, design  knowledge, funding, and time. At the heart of a business case is a  knowledge-based approach, in which knowledge supplants risk over time.  Establishing a business case calls for a realistic assessment of risks and  costs; doing otherwise undermines the intent of the business case and  invites failure. Over the years, we have identified a number of factors that  undermine business cases and drive cost and schedule overruns, several  of which are illustrated in figure 5.

Undesirable outcomes such as cost and schedule growth reflect  decisions made to move forward with programs before the knowledge  needed to reduce risk and make those decisions is sufficient. For  example, we have previously found that the majority of cost growth  occurs after production start, which may be a sign that programs are  entering production without attaining key knowledge about technology  maturity, design stability, and production readiness in preceding phases  of development. The primary consequences of risk are often more time  and money, and these consequences flow through the acquisition  phases, with unplanned overlap—known as concurrency—in  development, testing, and production.

Our annual assessment of weapon systems has identified numerous  examples of programs proceeding without sufficient knowledge to reduce  risk, and their subsequent cost and schedule growth. These examples  have included the following from among the 21 MDAPs reviewed in this  report:

The F-35 program started development without a match between  resources and requirements and without a stable design. Critical  technologies were immature, development and production occurred  concurrently, and critical deficiencies were still not resolved well into  production. As of May 2019, the program had experienced unit cost  growth of 75 percent and schedule growth of 35 percent since its first  full estimate in October 2001.

The MQ-4C program did not achieve technology maturity or design  stability prior to development start and critical design review,  respectively, and developmental challenges delayed production start.  As of May 2019, the program had experienced unit cost growth of 10  percent and schedule growth of 70 percent since its first full estimate  in February 2009.

The CH-53K program failed to demonstrate technology and design  maturity at appropriate points earlier in system development. As of  May 2019, the program had experienced unit cost growth of 21  percent and schedule growth of 60 percent since its first full estimate  in December 2005.

A year after the production decision for the Ground/Air Task Oriented  Radar program, the Marine Corps revised the program’s reliability  requirements in response to an expert panel finding that the existing  requirements did not reflect operational needs, contributing to delayed  full-rate production. As of May 2019, the program had experienced  unit cost growth of 168 percent and schedule growth of 146 percent  since its first full estimate in August 2005.

We have identified and recommended solutions to these issues, including  that MDAPs establish firm and feasible requirements, mature  technologies, incremental acquisition approaches, and realistic cost  estimates. While DOD has agreed with most of our recommendations in these areas, it has not always implemented them. As we noted in our  most recent High Risk List report, as of November 2018, 88  recommendations related to DOD weapon systems acquisition remained  open. Furthermore, while we had previously reported better cost  performance on newer programs initiated after implementation of major  acquisition reforms in 2010, more recently we found cost growth on those  programs. We attributed the deteriorating performance of newer  programs to the inconsistent implementation of knowledge-based  acquisition practices, as the negative effects of entering development with  insufficient knowledge cascade throughout the acquisition cycle.

Peer Review Change in 2019 Reduced a Means for Sharing Information about Contract Choice across DOD

In August 2019, DPC announced that it would no longer conduct  mandatory peer reviews for competitive procurements above $1 billion,  except for the small number of MDAPs for which USD(A&S) remains  milestone decision authority, and other programs of special interest to  USD(A&S). As part of the same announcement, DPC stated that it  planned to continue to perform peer reviews for noncompetitive  procurements of $500 million or more. DPC officials expect that the  procurements no longer covered by DPC’s peer review will instead be  covered by the military departments’ own review processes, which  already address competitive procurements up to $1 billion. While these  review processes exist within the military departments, there is not an  active mechanism for sharing across the departments any best practices  and lessons learned—including about contract choice—found in the  course of the reviews. DPC does not currently have plans to address the  reduced potential for information sharing resulting from this change.

Figure 6 depicts key developments related to the DPC peer reviews since  their establishment in 2008, including the last update to an online  compendium—a tool designed to share best practices, lessons learned,  and recommendations from peer reviews across DOD—in 2013.

According to DPC officials, updates to the compendium stopped as  personnel became more familiar with the peer review process. They also  noted that the change to peer reviews in 2019 resulted from resource  constraints and staff reductions associated with recent acquisition  reforms. The officials expect this change to reduce the number of DPC  peer reviews by half to approximately 50 per year, consisting primarily of  the reviews for noncompetitive procurements of $500 million or more.

The peer review process was established with the following objectives:  1.  to ensure that contracting officers across DOD consistently and  appropriately implement policies and regulations;  2.  to continue to improve the quality of contracting processes across  3.  to facilitate cross-sharing of best practices and lessons learned across  DOD.

In support of this third objective, procedures for conducting peer reviews  stated that the predecessor office to DPC would look for common trends  and issues to be shared with the broader DOD contracting community,  and maintain information about best practices and lessons learned on its  website. This public website currently houses the online compendium,  although, as noted above, the last update was in 2013.

Contracting officials we met with noted the value of being able to learn  from the experiences of officials in other military departments through  peer reviews. For example, contracting officials on an Air Force program  that had a peer review involving Navy officials stated that lessons shared  by those officials reduced the time it took to subsequently execute a  contract. Officials from across the military departments cited benefits that  resulted from these opportunities to learn from the real-world experience  of peers across DOD, including the ability to share contracting information  and expertise, review cost-sharing arrangements, and recalibrate  contracting decisions.

The online compendium is a spreadsheet with a row for each example of  feedback, with the program and officials concerned kept anonymous.  Columns include the category of feedback (e.g., source selection, terms  and conditions), the type of feedback (e.g., recommendation, lesson  learned, best practice), and the phase of review (e.g., issuance of the  solicitation). Our analysis of the compendium found that it captures  practices and recommendations related to contract type, as illustrated by  the following examples:

Use of incentives: Consider development of cost and performance  incentives, rather than use of an award fee.

Different contract type: Reconsider plan to award a fixed-price- incentive contract, given historical use of a cost-plus-incentive-fee  arrangement under which contractor delivered at or around target  cost.

Source selection: Throughout solicitation for an award combining  firm-fixed-price and cost-type line items, tell offerors what they are  expected to provide and how they will be evaluated, and document  that evaluation occurred in this exact way.

Officials from the military departments confirmed that they are aware that  they will now be expected to perform the reviews that DPC previously  conducted. They have taken steps to adjust procedures accordingly,  including updating their acquisition regulations as necessary. However,  DPC does not currently have plans to encourage sharing of findings from  military department-level reviews across the departments. For example,  there are no plans to solicit updates to the online compendium or a similar  centralized resource. USD(A&S) is responsible for improving acquisition  results—including cost, schedule, and performance—by gathering and  distributing data, best practices, and lessons learned across the military  departments. Without a centralized resource for sharing findings, and as  most reviews transition to the military departments, it will become more  difficult for USD(A&S) to identify contracting trends across DOD and  perform this assigned role. An updated compendium or other centralized  resource could help contracting officials continue to learn from the  experiences of peers across DOD—including when acquiring similar  platforms and from similar contractors—by exposing them to good  practices for structuring contracts and prompting consideration of  alternative contract types.

With DPC conducting fewer peer reviews and no updates to the  compendium since 2013, contracting officials might not have insight into  how other programs across DOD structure contracts. As the reviews will  now primarily occur within the military departments, these officials could  lose exposure to alternative contracting approaches suitable for their  programs. A centralized resource such as the compendium takes on a  new significance as a means for sharing information between the military  departments as they proceed with their own peer reviews. USD(A&S)  remains well-positioned to facilitate information exchange and contribute  to positive program outcomes by requiring the military departments to  share the findings of their peer reviews.

The Under Secretary of Defense for Acquisition and Sustainment should  establish procedures requiring the military departments to collect and  share findings from their peer reviews of MDAP contracting approaches— including choice of contract type—such as by updating the existing online  compendium of best practices and lessons learned as they complete their  reviews.

We provided a draft of this report to DOD for review and comment. DOD  concurred with our recommendation and provided written comments,  which are reprinted in appendix II. DOD also provided technical  comments, which we incorporated as appropriate.

We are sending copies of this report to the Secretary of Defense. In  addition, the report is available at no charge on the GAO website at  http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-4841 or oakleys@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who made key contributions to this report  are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

This report addresses: (1) the extent to which the Department of Defense  (DOD) uses cost-type contracts for major defense acquisition programs  (MDAP); (2) how DOD chooses among cost-type and other contract types  for MDAPs and monitors their cost and schedule performance; (3) the  range of cost and schedule outcomes across MDAPs that used cost-type  contracts; and (4) the extent to which DOD shares information about  choosing MDAP contract types across the military departments.

To assess the extent to which DOD uses cost-type contracts for MDAPs,  we analyzed Federal Procurement Data System-Next Generation (FPDS- NG) data regarding obligations by contract type from fiscal year 2011  through fiscal year 2019 on contracts for programs in DOD’s MDAP  portfolio awarded from fiscal year 2010 through fiscal year 2018. These  data reflect programs that were part of DOD’s MDAP portfolio and  contracts that were reported in Selected Acquisition Reports at any point  during this period. The basic types of contracts may be used in  combination, with both fixed-price-type and cost-type contract line item  numbers, unless otherwise prohibited. Per the Defense Federal  Acquisition Regulation Supplement (DFARS) Procedures, Guidance, and  Information, when entering contract type information info FPDS-NG, the  data entrant is to choose the contract type that is applicable to the  predominant amount of the contract action, based on the value of the line  items; the selected contract type automatically populates any subsequent  contract action reports for modifications. We aggregated obligations on  orders under indefinite-delivery contracts and basic ordering agreements  by contract type for each fiscal year.

We used the Defense Acquisition Management Information Retrieval  (DAMIR) system to identify those contracts reported in Selected  Acquisition Reports for programs in the MDAP portfolio awarded from  fiscal year 2010 through fiscal year 2018. Our dataset includes only  obligations on MDAP contracts awarded since fiscal year 2010 due to  problems identified in a prior GAO report regarding how data on contract  types were reported in FPDS-NG for contracts awarded prior to that  date. Specifically, prior to fiscal year 2010, data entrants could select the  contract types “combination” and “other”, or not enter a contract type at all. The Office of Federal Procurement Policy subsequently removed  those contract types as options in FPDS-NG, and made completion of the  field mandatory. Contracts retain their original designation in FPDS-NG  when modifications to those contracts are subsequently made. Therefore,  in order to avoid including contracts coded as “combination” or “other”, we  limited our analysis to contracts awarded since fiscal year 2010.

We assessed data reliability by comparing the contract types identified in  FPDS-NG for each contract with information on contract types contained  in DAMIR and in another DOD database—Earned Value Management- Central Repository—and determined the data were sufficiently reliable for  the purposes of analyzing the extent of DOD’s use of cost-type contracts  for MDAPs. Contractors for programs with earned value management  (EVM) reporting requirements submit EVM data to Earned Value  Management-Central Repository. EVM reporting is generally required for  cost-type or incentive contracts valued at $20 million or more. We  included obligations associated with contract types contained in FPDS- NG if they matched contract types contained in either DAMIR or Earned  Value Management-Central Repository. When there was no match with  either source, we reviewed the narrative discussion of contract types  contained in Selected Acquisition Reports in order to determine the most  appropriate contract type with which to label those obligations.

To assess how DOD chooses among cost-type and other contract types  for MDAPs and monitors their cost and schedule performance, we  reviewed relevant statutes, regulations, and policies. We analyzed  documentation and interviewed officials regarding contract choice and  monitoring from the following DOD and military department offices and  selected contracting commands:

Under Secretary of Defense for Acquisition and Sustainment

Deputy Assistant Secretary of the Air Force for Contracting

Deputy Assistant Secretary of the Army for Procurement

Deputy Assistant Secretary of the Navy for Procurement

Naval Information Warfare Systems Command

Naval Sea Systems Command  As illustrative examples of contract choice and monitoring under a variety  of conditions, including different military departments and appropriation  types, we also selected a nongeneralizable sample of seven MDAP  contracts. Specifically, we selected for each of the three military  departments the most recently awarded cost-type MDAP Research  Development, Test, and Evaluation contract and the most recently  awarded cost-type MDAP Procurement contract as reported in the  December 2017 Selected Acquisition Reports. We also selected the most  recently awarded cost-type MDAP contract for the Marine Corps. Table 6  notes the selected MDAPs and contracts, as well as the milestone  decision authority responsible for approving the acquisition strategy  associated with that contract.

We interviewed contracting officials for these programs and reviewed key  documentation such as acquisition strategies relating to each one of  these contracts. We also reviewed our past work related to contract types  used for MDAPs, including DOD’s use of incentive contracts and the  Navy’s use of fixed-price-incentive contracts for shipbuilding.

To assess the range of cost and schedule outcomes across MDAPs that  used cost-type contracts, we identified the contract types as reported in  DAMIR or GAO’s April 2018 and May 2019 annual assessments of  weapon systems for 21 non-shipbuilding MDAPs that as of January 2019  had completed system development, held a critical design review, and  started production. Table 7 notes the 21 MDAPs, as well as the dates of  their first full estimate, and their most recent individual assessment by  GAO as of May 2019.

We compared the contract types reported in DAMIR or GAO’s annual  assessments of weapon systems with the percentage unit cost and  schedule change between the first full estimate and our most recent in- depth assessment of each program as of May 2019. Since 2018, as part  of our annual assessment of weapon systems, we have conducted a  statistical analysis evaluating programs’ completion of knowledge-based  acquisition practices and corresponding performance outcomes. Our  report cites results of this analysis as it pertains to these 21 MDAPs. We  reviewed prior GAO work on the drivers of cost and schedule overruns for  MDAPs.

To assess the extent to which DOD shares information about choosing  MDAP contract types across the military departments, we reviewed DOD  and military department documentation related to contracting review  processes. We compared this information to DOD memorandums  establishing practices and policies for sharing of acquisition information  across DOD. We also interviewed officials from offices including Defense  Pricing and Contracting within the Office of the Under Secretary of  Defense for Acquisition and Sustainment, and the cognizant Deputy  Assistant Secretaries of the military departments.

We conducted this performance audit from February 2019 to May 2020 in  accordance with generally accepted government auditing standards.

Appendix II: Comments from the Department of Defense

Appendix III: GAO Contact and Staff Acknowledgments

Shelby S. Oakley, (202) 512-4841 or oakleys@gao.gov  In addition to the contact named above, Raj Chitikila (Assistant Director),  Robert Bullock, Jenny Chanley, Jasmina Clyburn, Andrea Evans, Lori  Fields, Suellen Foth, Kurt Gurka, Stephanie Gustafson, and Grace Haskin  made key contributions to this report.

The 2 main contract types that DOD uses to acquire its major weapon systems are:

Cost-type contracts—DOD pays allowable contractor costs (e.g., labor) and risks paying more if costs increase

Fixed-price-type contracts—DOD pays a fixed price, at which the contractor must deliver the item or service

We analyzed 21 weapon acquisition programs and found that the type of contract DOD used didn’t affect how well programs stuck to cost or schedule estimates.

DOD used to centrally review acquisitions and share what it learned about contract types. Now, the military departments do the reviews; we recommended requiring them to share their findings.