Movement of Household Goods: DOD Should Take Additional Steps to Assess Progress toward Achieving Program Goals

DOD, through its DP3, arranges for the movement and storage of about 400,000 personal property shipments of servicemembers and their families annually—40 percent of them during peak moving season. DOD has identified problems meeting peak season demand and addressing long-standing quality-of-service issues.

GAO was asked to evaluate matters related to DOD's plans to implement the Global Household Goods Contract. GAO assessed the extent to which TRANSCOM has (1) determined the cost implications of moving to a DP3 that incorporates the Global Household Goods Contract and (2) developed metrics to assess program activities and that relate to overarching DP3 goals. GAO evaluated TRANSCOM's cost estimates against the GAO Cost Estimating and Assessment Guide and a DOD business case analysis against GAO's Assessment Methodology for Economic Analysis.

The U.S. Transportation Command (TRANSCOM) has developed cost estimates to assess the cost implications of adjusting the Defense Personal Property Program (DP3), its program to move and store servicemembers' household goods, to incorporate a single move manager approach through the Global Household Goods Contract. However, TRANSCOM may not have accurately calculated some Department of Defense (DOD) costs because of unanswered questions about how tasks related to counseling servicemembers and overseeing contractor performance will be performed. DOD plans to conduct a manpower study in the third year of the contract to determine the number and cost of government personnel required to perform these tasks. However, TRANSCOM does not have a process in place to track data over the initial years of the contract to inform its manpower study, such as the number and associated cost of military service personnel needed to perform contract oversight. We have reported that organizations should determine their personnel requirements by identifying the minimum number and type of personnel needed to fulfill their missions, functions, and tasks by conducting a workforce analysis. Without a way to track key data, DOD risks conducting a manpower study that would result in less than a full understanding of the personnel and cost implications of the move to the Global Household Goods Contract.

TRANSCOM has developed performance metrics for assessing some, but not all, DP3 activities. For example, TRANSCOM has developed indicators for assessing contractor performance, including the timeliness of household goods deliveries under the Global Household Goods Contract. However, TRANSCOM has not developed metrics for other activities that DOD personnel will continue to perform at least partially once the contract is in place, such as servicemember counseling. Further, TRANSCOM has not articulated how existing metrics link to TRANSCOM's program goals that relate to servicemembers' household goods movement and storage experience (see fig.).

Without developing performance metrics for all DP3 activities, and articulating the linkage between metrics and goals, TRANSCOM will have limited ability to assess whether a DP3 incorporating the new contract is an improved program for servicemembers.

GAO makes three recommendations–that DOD collect and track data to more precisely determine DP3's manpower needs and costs, develop performance metrics for DP3 activities not part of the contract, and articulate the linkage between performance metrics and program goals. DOD concurred with all three GAO recommendations.

The Department of Defense (DOD) is the single largest customer in the  nation’s personal property moving and storage industry, representing  about 15 percent of all domestic and international moves. Each year,  through its Defense Personal Property Program (DP3), DOD arranges for  the movement and storage of approximately 400,000 personal property  shipments of servicemembers and their families. DOD reports that about  40 percent of these shipments occur during DOD’s peak moving season  of mid-May through late-August. The U.S. Transportation Command  (TRANSCOM), located at Scott Air Force Base, Illinois, manages DP3,  which provides privately owned vehicle shipment and storage as well as  household goods relocation, transportation, and warehouse services  worldwide for servicemembers.

In 2018, servicemembers and their spouses aired grievances on social  media about loss and damages to their personal property resulting from  the quality of work of moving companies. In response to the social media  attention and concerns about DP3 and its ability to meet the demand for  personal property shipments during the peak moving season, DOD  established a Military Personnel Relocation/Household Goods Movement  Cross-Functional Team in October 2018. The cross-functional team, co- chaired by the Under Secretaries of Defense for Acquisition and  Sustainment and for Personnel and Readiness, respectively, was  established with the goal of significantly improving the likelihood that  servicemembers and their families will experience high-quality, on-time  moves that are free from loss and damage to property. In conjunction with  TRANSCOM, the cross-functional team identified a number of short- and  long-term actions to improve DP3, including awarding a multiyear Global  Household Goods Contract. Under the contract, a single commercial  move manager would oversee certain DP3 activities related to the  movement and storage-in-transit of servicemembers’ and their families’  household goods.

We have previously reported on DOD’s efforts to address long-standing  quality-of-service issues associated with DOD’s household goods  movement and storage process, such as late pick-up and deliveries and  high claims costs. On January 6, 2019, the DOD Office of the Inspector  General issued a report that described similar quality-of-service issues  involving the timeliness of household goods deliveries and the claims  resolution process associated with moves during fiscal year 2018.

The Chair and Ranking Member of the Subcommittee on Readiness and  Management Support, Senate Committee on Armed Services requested  that we evaluate DOD’s plans to implement the Global Household Goods  Contract. Subsequently, the National Defense Authorization Act for Fiscal  Year 2020, enacted in December 2019, included a provision for us to  similarly review matters related to DOD’s proposed single move manager  approach. In this report, we assess the extent to which TRANSCOM has:  (1) determined the cost implications associated with moving to a DP3 that  incorporates the Global Household Goods Contract; and (2) developed  metrics to assess program activities and that relate to overarching DP3  goals.

For objective one, we assessed preliminary cost estimates that  TRANSCOM developed in connection with the move to a DP3 approach  that incorporates the Global Household Goods Contract, including  servicemember counseling that will be partially implemented under the  contract. Using the GAO Cost Estimating and Assessment Guide, we  reviewed the validity of the assumptions TRANSCOM used in developing  these preliminary cost estimates. We also evaluated updated cost  estimates developed by the Logistics Management Institute (LMI), which  developed a business case analysis (BCA) of the move to the Global  Household Goods Contract for TRANSCOM. The BCA examines, among  other things, the costs associated with the movement and storage-in- transit of household goods under DP3 as currently configured and DP3 as  it would be configured with the proposed Global Household Goods  Contract. We evaluated these cost estimates using criteria from our  Assessment Methodology for Economic Analysis, which provides criteria  for evaluating economic analyses like LMI’s BCA and outlines key  elements of an economic analysis. For the purposes of this objective, we  focused on those DP3 activities that would be moved entirely under the  Global Households Goods Contract, such as the movement and storage- in-transit of household goods, and those activities that would be partially  implemented under the contract and partially implemented by the military  services, such as servicemember counseling. We also focused on the  costs associated with elements of DP3 that would be performed by DOD,  such as contract oversight.

For objective two, we reviewed relevant DOD guidance, including DOD  Instruction 4500.57, Transportation and Traffic Management, which  requires TRANSCOM, in coordination with the DOD components, to  annually evaluate the effectiveness of DP3. We collected information on  TRANSCOM’s overarching program goals for DP3 and compared  TRANSCOM’s plans for assessing DP3 activities, including those that  would be performed under the Global Household Goods Contract, against  those program goals. We evaluated DOD’s actions against the Standards  for Internal Control in the Federal Government, which calls for  management to define objectives (or metrics) in terms of what is to be  achieved, who is to achieve it, how it will be achieved, and the time  frames for achievement so that performance toward achieving those  objectives (or metrics) can be assessed.

To inform both objectives, we met with officials from the Office of the  Secretary of Defense (OSD), TRANSCOM, the military services, the  Coast Guard, and the DOD Office of the Inspector General. We also met  with external stakeholders, including associations representing the  movement and storage industry, officials from the Small Business  Administration and the American Federation of Government Employees,  and members of TRANSCOM’s Personal Property Relocation Advisory  Panel. More detailed information on our scope and methodology can be  found in appendix I of this report.

We conducted this performance audit from May 2019 to April 2020 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

DOD currently relies on more than 900 commercial industry transportation  service providers (TSPs) to move and store servicemembers’ and their  families’ household goods. About 40 percent of DOD’s annual household  goods moves occur during the 14-week annual peak moving season,  which runs from May 15 through August 31. As figure 1 shows,  servicemember survey satisfaction scores declined in 2018 during the  peak moving season. According to TRANSCOM officials, this decline in  satisfaction scores during the peak moving season has persisted for  years.

Under the current DP3, the military services own, operate, and staff most  of the infrastructure involved in managing and overseeing the movement  and storage-in-transit of DOD’s household goods, including the personal  property processing offices and personal property shipping offices. The  military services’ processing offices, among other things, ensure that the  servicemembers’ reassignment orders and paperwork authorizing a move  are in order, and that their application is accurate and complete. The  processing offices also provide counseling, including advising  servicemembers on the DP3 process, entitlements, and restrictions so  they can make informed decisions about their moves. The military  services’ shipping offices work with TSPs to schedule moves, monitor  TSP performance, and take or recommend punitive action against poor  performing TSPs.

Recent DOD Efforts to Address Long-Standing DP3 Performance Issues

DOD has conducted and sponsored several studies to help address  persistent DP3 performance issues, including those related to  servicemember satisfaction and challenges with meeting capacity  demands during the peak summer months. For example, LMI and the  Institute for Defense Analyses (IDA) produced reports on behalf of DOD  in 2012 and 2018, respectively, to analyze DP3 performance issues and  make recommendations to improve DP3. In its report, LMI outlined the  impact of a range of options for contracting out some or all of the  management of DOD’s household goods movement and storage  activities. IDA recommended that DOD create consistent performance  metrics, unify its DP3 operating structure, and improve servicemember  support.

Concurrent with pursuing its Global Household Goods Contract,  TRANSCOM has several ongoing initiatives intended to improve  servicemember satisfaction in general, and household goods loss and  damage specifically. According to survey data, loss of and damage to  household goods is the number one reason servicemembers cite for  dissatisfaction. Ongoing TRANSCOM initiatives to improve  servicemembers’ moving experiences that are independent of the Global  Household Goods Contract include the following:  Increasing the use of shipping containers in domestic household  goods moves to reduce the number of servicemember loss and  damage claims. According to TRANSCOM officials, in the past  TRANSCOM has primarily used shipping containers only for  international moves.

Replacing the internet-based system used to manage DOD household  goods moves, the Defense Personal Property System, with a new,  mobile-friendly information technology operating system platform  called MILMOVE. MILMOVE is intended to provide servicemembers  and their families with more reliable information about their planned  and ongoing household goods moves through access to all phases of  the move process via their personal smartphones or tablets.

Tracking whether quality inspections occur in person or via telephone,  and requiring that at least 50 percent of all shipments have an in- person quality inspection.

Increasing the liability limits for damaged or lost household goods  shipments to $6 per pound and the total loss cap to $75,000 per  shipment, when claims are filed within a specified time.

TRANSCOM Efforts to Develop the Global Household Goods Contract

In November 2018, TRANSCOM, in coordination with OSD and the  military services, began planning and developing requirements for the  Global Household Goods Contract. According to TRANSCOM officials,  the goal was to have the contract in place in time for the 2021 peak  moving season. TRANSCOM announced that in April 2020 it would  award a Global Household Goods Contract to a single commercial move  manager to oversee DP3 activities that relate to the movement and  storage-in-transit of household goods. The proposed contract includes an  initial 9-month transition period that will commence in May 2020, a 3-year  base period, three 1-year option periods, two 1-year award terms, and an  option to extend the contract 6 months. Including all option periods and  award terms, the contract is expected to be completed at the end of  January 2029 or, if the option to extend services is exercised, by the end  of July 2029. The 9-month transition period is intended to give the Global  Household Goods contractor sufficient time to, among other things,  integrate its information technology systems with existing DOD  information technology systems. During the initial years of the contract,  the contractor’s volume of workload will incrementally increase, including  the volume of household goods moves that it will handle.

The Global Household Goods Contract is not intended to replace DP3 in  its entirety. Activities that are planned to be a part of the contract include  the movement and storage-in-transit of household goods, direct  procurement method shipments, and some level of servicemember  counseling. DP3 activities that are not to be a part of the Global  Household Goods Contract include the long-term storage of household  goods (referred to as non-temporary storage) and the movement and  storage of servicemembers’ privately owned vehicles. Additionally, while  the contract is expected to include counseling to servicemembers, the  military services will also perform some counseling. According to  TRANSCOM and military service officials, they are working to determine  the amount of counseling that will be retained by the military services.

Key Differences in How Household Goods Are Moved and Stored in DP3 Currently and under the Planned Global Household Goods Contract

The movement and storage-in-transit of household goods in the current  DP3 differs in key ways from the approach under the planned Global  Household Goods Contract, as shown in table 1.

TRANSCOM Has Taken Steps to Determine the Cost Implications Associated with Moving to a DP3 That Incorporates the Global Household Goods Contract, but Plans Not to Determine Some Cost Estimates for Years

TRANSCOM has developed preliminary and refined cost estimates for  determining the cost implications associated with moving to a DP3 that  incorporates the Global Household Goods Contract. According to  TRANSCOM officials, they began developing preliminary cost estimates  in December 2018 in order to (1) create a baseline cost estimate for the  movement and storage of household goods under the current DP3, (2)  create a cost estimate for those DP3 activities that will be a part of the  Global Household Goods Contract, and (3) serve as a point of  comparison between the two estimates to determine the cost implications  of moving to the Global Household Goods Contract.

TRANSCOM’s preliminary cost estimates for the current DP3 and for the  activities that will be a part of the Global Household Goods Contract  included costs associated with the movement and storage of household  goods and government personnel costs associated with the military  services’ personal property processing and shipping offices. TRANSCOM  developed these preliminary cost estimates in part by requesting and  collecting information from the military services about their current costs  under DP3. Some DP3 costs were not included in the preliminary cost  estimates, such as infrastructure and vehicle costs because these assets  are used to support multiple programs and could not be easily isolated.

However, our assessment of the preliminary DP3 cost estimates  associated with moving to the Global Household Goods Contract found  that TRANSCOM may not have accurately calculated some costs  because of unanswered questions about how certain activities will be  performed. TRANSCOM’s cost estimates associated with the number of  government personnel that (1) will counsel servicemembers and (2)  oversee contractor performance under the contract had weaknesses.  Specifically, these estimates relied on assumptions that may have  resulted in over- or underestimating some costs.

First, TRANSCOM’s preliminary DP3 cost estimates associated with  moving to the Global Household Goods Contract assume that DOD’s  costs related to government personnel will be equivalent to those under  the current DP3. However, TRANSCOM officials acknowledge that this  assumption is based on discussions with military service officials that  occurred before the Global Household Goods Contract draft request for  proposals was issued in April 2019. Furthermore, military service officials  told us they have not decided how much of the counseling function their  service has provided will be moved to the planned contract, and the  services’ approaches will likely differ. TRANSCOM officials pointed out  that while the Global Household Goods contractor will perform some level  of servicemember counseling, the contract will allow the individual military  services to decide how much of the counseling responsibility to retain.

Second, TRANSCOM’s cost estimates do not account for the number of  DOD contracting officer’s representatives and quality assurance  evaluators required to oversee contractor performance under the  contract. According to TRANSCOM officials, the command’s preliminary  cost estimates assume that government personnel who have been  relieved of servicemember counseling responsibility will perform contract  oversight and quality assurance responsibilities under the planned  contract. For example, TRANSCOM officials told us that quality  assurance inspectors under the current DP3 will transition to serve as  quality assurance evaluators once the contract is in place. However,  TRANSCOM officials acknowledge that it is possible that workload (and  personnel costs) for the military services under the current DP3 will  change under the planned contract. Moreover, if government personnel  do switch roles and responsibilities, TRANSCOM did not take into  account the cost to transition and train personnel to execute their  contracting officer’s representative and quality assurance evaluator  responsibilities under the contract.

TRANSCOM officials told us that they were unable to determine with any  precision the number and associated costs of government personnel  required to counsel servicemembers and oversee the contract because of  the fractured nature of the current DP3. Nonetheless, in September 2019,  TRANSCOM tasked LMI with developing a BCA with refined cost  estimates. LMI finalized the BCA and we received it on January 17,  2020. When we assessed the BCA, we found that like us LMI  determined that it did not have complete information to fully calculate the  cost of a DP3 that incorporates the Global Household Goods contract.  Specifically, LMI acknowledged some of the same limitations in its cost  estimates that we identified in TRANSCOM’s preliminary cost estimates,  such as uncertainty about the number of government personnel required  to oversee the Global Household Goods Contract. For example, LMI  states in its BCA that the Global Household Goods Contract may reduce  DOD staffing requirements for functions such as counseling; however,  roles such as quality assurance evaluators and contracting officer’s  representatives to oversee contractor performance may increase DOD  staffing requirements.

When we raised concerns about these unanswered questions and their  potential cost implications with TRANSCOM officials, they told us that the  move to the Global Household Goods Contract is less about saving  money than it is about representing the best value to DOD when both  cost and program performance are considered. They also told us they  have developed a plan of action for the phase-in of the Global Household  Goods Contract, and that plan includes conducting a manpower study  during the third year of the contract. As described earlier in this report, the  initial 3 years of the contract involve a gradual phase-in of household  goods move volume for the contractor, and in the third year the contractor  will be responsible for all of DOD’s household goods move volume.  According to these officials, by waiting until year 3 of the contract to  conduct a manpower study, DOD will have data to more precisely  determine the number (and cost) of government personnel required to  counsel servicemembers and oversee the contract, such as contracting  officer’s representatives and quality assurance evaluators.

However, we have determined that TRANSCOM does not have a process  in place to track data that would inform its manpower study in year 3,  such as how many personnel within each military service are needed to  perform contract oversight duties and the costs associated with these  personnel. We have reported that organizations should determine their  personnel requirements as part of a systematic requirements- determination process that includes (1) identifying an organization’s  mission, functions, and tasks and (2) determining the minimum number  and type of personnel—military, civilian, and contractor—needed to fulfill  those missions, functions, and tasks by conducting a workforce  analysis. Without a way to track key data, DOD risks conducting a  manpower study that does not allow it to fully understand the personnel  and cost implications of its move to a DP3 that incorporates a Global  Household Goods Contract. Notably, in its BCA LMI recommended that  DOD reexamine the BCA when additional relevant information becomes  available, such as when DOD completes its manpower study.  TRANSCOM officials told us they are in agreement with this  recommendation. However, if TRANSCOM does not have a process in  place to collect key pieces of data during the first 3 years of the contract,  a reexamination of the BCA will be less fruitful than it otherwise might be.

TRANSCOM Has Developed Performance Metrics to Assess Some but Not All DP3 Activities, and Not All of TRANSCOM’s Overarching Program Goals Have Clearly Associated Performance Metrics

First, TRANSCOM has developed performance metrics—referred to as  performance indicators—for its Global Household Goods Contract, and  has developed performance metrics for some, but not all, activities that  fall outside of the contract.

TRANSCOM’s draft quality assurance surveillance plan for the Global  Household Goods Contract, which was developed in coordination with the  military services, outlines how the contractor’s performance will be  assessed against performance indicators. These performance indicators  set measurable standards for, among other things, information technology  systems’ availability, claims settlement timeliness for lost and damaged  goods, and the timeliness of household goods pick-up and deliveries.  Examples of performance indicators include:  settling 90 percent of all loss and damage claims valued at less than  $1,000 within 30 days and settling 95 percent of all claims, regardless  of value, within 60 days; delivery of household goods on the scheduled date 95 percent of the  time per month; and overall customer satisfaction rating of satisfactory at least 95 percent  per month.

Based on a DP3 briefing TRANSCOM provided us, the command has  also developed metrics for assessing other contracted DP3 activities,  such as the transport of privately owned vehicles and non-temporary  storage, which TRANSCOM captures under contracts separate from the  Global Household Goods Contract. For example, TRANSCOM tracks the  number and cost of non-temporary storage lots maintained annually to  support its storage requirements.

However, TRANSCOM officials acknowledge that the command has not  developed metrics for other activities that government personnel will  continue to perform in DP3, once a Global Household Goods Contract is  in place, such as servicemember counseling and claims resolution.  Although TRANSCOM’s draft quality assurance surveillance plan includes  a performance indicator to assess the contractor’s performance with  respect to servicemember counseling, TRANSCOM officials told us the  command has not developed commensurate metrics to assess the  military services’ performance in providing servicemember counseling.  TRANSCOM officials also noted that while the military services’ claims  offices will be responsible for handling unresolved loss and damage  claims between the servicemember and the contractor, the command has  not developed metrics associated with this DP3 activity.

Second, we found that, while TRANSCOM has articulated overarching  DP3 goals, it has not clearly articulated how its performance metrics align  with each of these goals. According to TRANSCOM officials, they decided  to move certain DP3 activities to the Global Household Goods Contract  because doing so could positively impact the program’s five goals  regarding cost, quality, capacity, accountability, and responsibility, terms  described in LMI’s BCA as shown below:

Cost considers the potential financial impact, including either  opportunities or risks to the future-state costs and any savings the  program is likely to achieve.

Quality encompasses the value of a move, typically measured by  customer (i.e., servicemember) satisfaction. Quality may improve with  on-time performance and minimized loss or damage.

Capacity is the availability of industry providers to meet the program’s  demands at a given time. This can be during peak or non-peak  season.

Accountability refers to the government’s ability to affix responsibility  to the contractor and their supplier network for performance.

Responsibility encompasses the authority of the government and  how each government stakeholder will be held responsible for  accomplishing their assigned tasks.

Planned contract performance indicators and metrics are intended to  assess contractor performance, but they may also provide information  that TRANSCOM could use to more broadly assess the extent to which  DP3 is meeting its overarching program goals. However, TRANSCOM  has not clearly articulated which of the performance metrics it has  established under the Global Household Goods contract align with which  program goals. For example, one performance indicator for the Global  Household Goods Contract is for contractor delivery of household goods  to be on-time at least 95 percent of the time. While it appears that this  performance indicator could relate to the quality and capacity goals,  TRANSCOM has not established linkage between this performance  indicator and either of these goals. Determining how it will assess  improved capacity is particularly important. When we spoke with them  about the move to the Global Household Goods contract, some military  service officials, representatives of the moving and storage industry, and  members of TRANSCOM’s Personal Property Relocation Advisory Panel  told us that they doubt whether the Global Household Goods Contract  will, indeed, improve capacity.

Moreover, none of the existing performance indicators appear to relate to  TRANSCOM’s responsibility goal. It is not surprising that the Global  Household Goods Contract does not include performance indicators on  how government stakeholders are to be held responsible for various program activities, because indicators under the contract are intended to  evaluate contractor performance. However, we would expect to see  performance metrics outside of the contract to measure government  stakeholders’ performance.

When we spoke with TRANSCOM officials about the gaps we identified in  their performance assessment approach, they agreed that clarifying the  linkage between performance metrics and the overarching program goals  and ensuring there are performance metrics for assessing each of the  goals would improve the command’s ability to assess overall program  performance. While the performance indicators and metrics related to the  various contracts, including the Global Household Goods Contract, are  intended to assess contractor performance, they could also provide  information that TRANSCOM could use to more broadly assess the  extent to which DP3 is meeting its overarching program goals. DOD  Instruction 4500.57, Transportation and Traffic Management, calls for  TRANSCOM, in coordination with DOD components, to conduct annual  program reviews to ensure the overall effectiveness of the DP3. The  instruction calls for these reviews to include a metrics-based evaluation of  the program, and assessments of TSP and service-provider cost and  performance, information technology systems and contracts that support  DP3, and external factors that impact the program such as industry  capability and changes to servicemember shipping entitlements.  Moreover, the Standards for Internal Control in the Federal Government  call for management to define objectives (or metrics) in terms of what is to  be achieved, who is to achieve it, how it will be achieved, and the time  frames for achievement so that performance toward achieving those  objectives (or metrics) can be assessed. We have also previously  reported on the importance of linking lower-order performance metrics  and higher-order strategic goals to achieve desired outcomes.

Without performance metrics that account for those DP3 activities that fall  outside of the Global Household Goods Contract and other DP3 contracts  and without clearly articulating the linkage between performance metrics  and program goals, TRANSCOM’s ability to assess progress toward, and  take actions to help achieve, its overarching program goals will be  hindered. The need to take these steps is particularly important, given  some of the skepticism that TRANSCOM faces with its move to a DP3  that incorporates the Global Household Goods Contract. Further,  because TRANSCOM officials have determined that the move to the  Global Household Goods contract is about delivering better value, rather  than just saving money, it is imperative that they have a robust  performance assessment approach in place.

DOD has experienced long-standing quality issues moving and storing  the household goods of servicemembers and their families, despite  numerous reform efforts. To address persistent quality-of-service issues,  such as late pick-up and deliveries and high claims costs for lost and  damaged goods, TRANSCOM intends to award a multi-year Global  Household Goods Contract, under which a single commercial move  manager would oversee the movement and storage of household goods  shipments. DOD has been working to award its Global Household Goods  Contract in time to meet the fiscal year 2021 peak season demand, and  will award the contract without precise information on the number and  cost of government personnel required to counsel servicemembers and  oversee the contract. Given this determination, it is particularly important  that DOD put in place a process to track key data during the first 3 years  of the contract to inform its planned manpower study, so that it can fully  determine the cost implications of this shift in DP3. Moreover, because  DOD has stated that the purpose of the move to the Global Household  Goods contract is to provide better value for the customers, DOD should  ensure that it has in place performance metrics to assess all DP3  activities, including those that fall outside of the Global Household Goods  contract, and clearly articulate the linkage between performance metrics  and overarching program goals. Without doing so, TRANSCOM will be  hindered in its ability to assess whether a DP3 that incorporates the  Global Household Goods Contract is an improved program, particularly  as it relates to the moving experiences of servicemembers and their  families.

We are making the following three recommendations to the Secretary of  Defense:  The Secretary of Defense, in coordination with the Chairman of the Joint  Chiefs of Staff, should ensure that the TRANSCOM Commander, in  coordination with the military services and the Coast Guard, develop a  process for tracking data during the first 3 years of the Global Household  Goods Contract to inform its planned manpower study during the third  year of the contract to more precisely determine DP3 manpower needs  and associated costs. (Recommendation 1)

The Secretary of Defense, in coordination with the Chairman of the Joint  Chiefs of Staff, should ensure that the TRANSCOM Commander develop  performance metrics for those DP3 activities that will not be a part of the  Global Household Goods Contract, such as servicemember counseling  and claims resolution that will, at least in part, continue to be performed  by the military services. (Recommendation 2)

The Secretary of Defense, in coordination with the Chairman of the Joint  Chiefs of Staff, should ensure that the TRANSCOM Commander  articulate the linkage, where appropriate, between DP3 performance  metrics, including Global Household Goods Contract performance  indicators, and overarching program goals. (Recommendation 3)

We provided a draft of this report to DOD for comment. In its comments,  which are reproduced in Appendix III, DOD concurred with all of our  recommendations and described ongoing and planned actions to address  them.

We are sending copies of this report to the appropriate congressional  committees; the Secretary of Defense, the Chairman of the Joint Chiefs of  Staff, and the Secretaries of the Army, the Navy, and the Air Force; and  the Commandant of the Marine Corps. In addition, the report will be  available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-2775 or fielde1@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who made key contributions to this report  are listed in appendix IV.

Our review focused on household goods movement and storage-in-transit  for servicemembers in the Army, the Navy, the Air Force, the Marine  Corps, and the Coast Guard. We did not evaluate Department of Defense  (DOD) Defense Personal Property Program (DP3) activities that will not  be a part of the Global Household Goods Contract, which include the  long-term storage of household goods and the movement and storage of  servicemembers’ privately owned vehicles. We did, however review  metrics being tracked for these activities to determine the U.S.  Transportation Command’s (TRANSCOM) ability to assess performance  for the broader DP3.

During our review, we coordinated with officials from the DOD Office of  the Inspector General to gain an appreciation for the objectives, scope,  and methodology of their ongoing audit on the timeliness of household  goods deliveries and claims resolution under the current DP3. The  resulting audit report contained several recommendations, including that  TRANSCOM issue warnings or letters of suspension to transportation  service providers (TSPs) within 14 days of missing the agreed-upon  delivery date from storage, and that TRANSCOM help servicemembers  and their families file inconvenience claims with TSPs within 14 days of a  missed delivery date.

To inform both of our objectives, we met with officials from the Office of  the Secretary of Defense (OSD), TRANSCOM, the military services, the  Coast Guard, and the DOD Office of the Inspector General. We also met  with external stakeholders, including associations representing the  moving and storage industry, the Small Business Administration, the  American Federation of Government Employees, and members of  TRANSCOM’s Personal Property Relocation Advisory Panel.

For objective one, we assessed preliminary cost estimates TRANSCOM  developed in connection with the move to a DP3 approach that  incorporates the planned Global Household Goods Contract against best  practices in the GAO Cost Estimating and Assessment Guide. The guide  states that valid and useful historical data are important in developing  sound cost estimates, and that risk and uncertainty and sensitivity  analyses should be performed to mitigate the effects of changing  assumptions.

We analyzed TRANSCOM’s preliminary cost estimates, and discussed  with TRANSCOM officials assumptions that were used to develop the  estimates and techniques that were applied to account for variability in  the assumptions. Given that TRANSCOM’s cost estimates were  preliminary and likely to change based on the Logistics Management  Institute’s (LMI) ongoing business case analysis (BCA), we did not  conduct a reliability assessment of TRANSCOM’s cost estimates. We  instead focused on TRANSCOM’s treatment of evolving assumptions in  its preliminary cost estimates. We met with representatives from LMI to  discuss their process for updating TRANSCOM’s preliminary cost  estimates as part of a BCA that it was preparing for the command. We  evaluated these cost estimates, including the cost estimates in the BCA  that was issued in January 2020, using criteria from GAO’s Assessment  Methodology for Economic Analysis that outlines five key elements for an  economic analysis. Our assessment of LMI’s BCA is included in   appendix II.

For objective two, we reviewed relevant DOD guidance, including DOD  Instruction 4500.57, Transportation and Traffic Management. The  instruction requires TRANSCOM, in coordination with the DOD  components, to annually evaluate the effectiveness of the DP3. We also  reviewed the Global Household Goods request for proposals and  TRANSCOM’s draft quality assurance surveillance plan. The draft quality  assurance surveillance plan outlines how DOD will oversee the move  manager’s performance under the Global Household Goods Contract and  describes the performance indicators that will be used to assess contractor performance. Using the Standards for Internal Control in the  Federal Government, which calls for management to define objectives (or  metrics) in terms of what is to be achieved, who is to achieve it, how it will  be achieved, and the time frames for achievement so that performance  toward achieving those objectives can be assessed, we also assessed  DOD’s DP3 performance metrics. Additionally, we discussed with  TRANSCOM officials their approach for overseeing the Global Household  Goods Contract, including the linkage between performance measures for  those DP3 activities that will and will not be a part of the contract and  DP3’s overarching goals.

Appendix II: GAO’s Assessment of the Logistics Management Institute’s (LMI) Business Case Analysis (BCA)

Appendix II: GAO’s Assessment of the  Logistics Management Institute’s (LMI)  Business Case Analysis (BCA)

We assessed the cost and benefit information in LMI’s BCA against our  Assessment Methodology for Economic Analysis. Our assessment  methodology identifies the following five key components of an economic  analysis:  We found that the BCA informs decision-makers and stakeholders, with  caveats, about the economic effects of the proposed Global Household  Goods Contract. Further, we found that the BCA fully met four and  partially met one of the five key components of an economic analysis. A  summary of our rationale for these assessments is outlined in table 2.

Appendix III: Comments from the Department of Defense

Appendix IV: GAO Contact and Staff Acknowledgments

Elizabeth A. Field, (202) 512-2775 or fielde1@gao.gov.

In addition to the contact named above, GAO staff who made key  contributions to this report include Marc Schwartz, Assistant Director;  Pedro Almoguera, John Bumgarner, William Cordrey (retired), Tim  DiNapoli, Jennifer Echard, Christopher Gezon, Mae Jones, Jason Lee,  Ned Malone, Tara Porter, Oliver Richard, Mike Shaughnessy, Susan  Tindall, Nate Tranquilli, John Van Schaik, and Mary Weiland.

DOD is the largest customer in the nation’s personal property moving and storage industry—accounting for about 15% of all moves (about 400,000 shipments a year). U.S. Transportation Command (TRANSCOM) manages the DOD program.

DOD proposed putting a single commercial move manager in charge of moving and storage, in part to address customer complaints.

TRANSCOM plans to award a multi-year contract, and determine, in the contract’s third year, the number and cost of the DOD oversight personnel needed.

We made 3 recommendations, including that TRANSCOM develop a process to track data during the first 3 years of the contract to inform its manpower study.


U.S. Postal Service: Addressing Policy Gaps Could Improve Pilot Design and Evaluation for Postal Innovations

USPS faces a challenging business environment that has led to reduced demand for its traditional services and significant financial losses. USPS aims to address this challenge by offering innovative products and services. The success of these efforts will depend, in part, on how effectively USPS tests each innovation's performance on a small scale to determine whether, how, and when to launch an innovation more broadly—a practice known as “piloting.”

GAO was asked to review USPS's efforts to develop postal innovations. This report (1) describes key innovations that USPS recently piloted and (2) examines the extent to which USPS's policies reflect leading practices for pilot design and evaluation. GAO analyzed information on USPS pilots from fiscal years 2013 through 2017; compared USPS policies for piloting innovations to leading practices for pilot design and evaluation in prior GAO work and relevant standards for internal control; and selected four key innovations based on various characteristics (e.g., innovation type) to serve as illustrative examples of USPS's piloting efforts.

From fiscal years 2013 through 2017, the U.S. Postal Service (USPS) piloted 24 key innovations intended primarily to generate revenue or improve customers' experience. The following four selected innovations illustrate these efforts:

Same-Day Delivery: USPS delivered goods consumers bought online or in stores. The pilot sought to test the product's feasibility and revenue potential.

Grocery Delivery: USPS delivered groceries to consumers in metropolitan areas. The pilot sought to test the product's feasibility and revenue potential.

Informed Delivery: USPS emailed customers an advance image of the mail they would receive. The pilot sought to test the service's potential benefits, such as generating new revenue from advertisers that may use the service.

Keyless Parcel Lockers: USPS is testing lockers where customers can independently pick up packages at post offices. The pilot seeks to test the service's operation and potential benefits for USPS and customers.

USPS's policies for piloting innovations do not fully reflect the five leading practices for pilot design and evaluation identified in GAO's prior work. The policies fully reflect two of the leading practices because they require articulating a methodology for evaluating pilot performance and documenting lessons learned. The policies do not fully reflect the other three practices because they do not require: (1) linking pilot objectives to identified performance measures; (2) documenting conclusions based on pilot results; or (3) communicating with key external stakeholders, as appropriate. These policy gaps limit the extent to which USPS can ensure that it is making good resource allocation decisions based on pilot experiences. For example, GAO found that USPS did not document its conclusions based on the results of its pilots of same-day delivery, grocery delivery, and Informed Delivery. Documenting conclusions can be especially important when USPS continues to offer the product or service after the pilot has concluded, even though the pilot did not achieve all of its objectives, as was the case with these three innovations. Further, while USPS's policies require documenting lessons learned from its pilots, USPS did not do so for some pilots GAO reviewed. Senior USPS officials said that USPS did not consistently follow this policy because it had not developed tools or training that could help ensure such consistency. As a result, USPS risks losing information that could be relevant to future innovation efforts.

GAO recommends that USPS (1) develop policies that fully reflect leading practices for pilot design and evaluation and (2) develop tools or training to ensure consistent documentation of lessons learned from pilots. USPS neither agreed nor disagreed with the recommendations but described actions it plans to take related to each.

The U.S. Postal Service (USPS) faces a challenging business  environment that has reduced demand for traditional postal services and  led to significant financial losses. Revenue from USPS’s most profitable  postal-service category—First-Class Mail—has declined over the last  decade, from $38.2 billion in fiscal year 2008 to $25 billion in fiscal year  2018. According to USPS, offering innovative products and services is  critical to adapting to evolving business and customer needs and  improving its financial condition. The success of any particular innovation  will depend, in part, on how effectively USPS tests the innovative  product’s or service’s performance on a small scale to determine whether,  how, and when to launch the innovation more broadly—a practice known  as “piloting.” Effectively piloting innovations can help ensure that USPS  invests its limited resources on innovations that are most likely to improve  its long-term viability.

You asked us to review USPS’s efforts to develop innovative products  and services. This report (1) describes key innovations USPS has  recently piloted and (2) examines the extent to which USPS’s policies for  piloting key innovations reflect leading practices for pilot design and  evaluation.

To identify key innovations that USPS has recently piloted, we first  requested information from USPS on all such innovations piloted between  fiscal years 2013 and 2017. In response, USPS identified 24 key piloted  innovations. We compared the list provided by USPS against information  in USPS, USPS Office of Inspector General, and Postal Regulatory  Commission (PRC)—an independent agency with regulatory oversight  over USPS—documents to help corroborate the completeness and  accuracy of USPS’s response. We conducted an in-depth review of 4 of  the 24 innovations to provide illustrative examples of USPS’s piloting  efforts. We selected the 4 innovations—Metro Post (i.e., same-day  delivery), Customized Delivery (i.e., grocery delivery), Informed Delivery,  and keyless parcel lockers—to represent a range of the innovations’  primary characteristics, such as innovation type (e.g., new product or new  service); primary innovation goal (e.g., generating revenue or improving  customer service); and investment level.

To evaluate the extent to which USPS’s policies for piloting key  innovations reflect leading practices for pilot design and evaluation, we  compared policies identified by USPS officials as applicable to its efforts  to pilot innovative products and services against five leading practices  we identified in our prior work, as well as standards for internal control  related to documenting key information: The five practices we identified  include: establish appropriate and measurable objectives linked with identified articulate a methodology for evaluating pilot performance; evaluate pilot performance and identify and document lessons draw and document conclusions about scalability (i.e., determining  whether, how, and when to launch the innovation more broadly)  based on pilot results; and ensure appropriate two-way communication at all stages of the pilot  with key internal and external stakeholders in order to understand and  address their views.

We discussed these leading practices with senior USPS officials, and  they agreed that the practices were reasonable and relevant to USPS’s  efforts to pilot innovative products and services.

We examined how USPS followed its policies and applied these leading  practices for the four selected key innovations by reviewing the pilot  proposals and revenue and other performance data. Our findings related  to these four selected key innovations are not generalizable to all  innovations piloted by USPS but provide illustrative examples of how  USPS has followed its policies and applied the leading practices that we  identified.

For both objectives, we also interviewed USPS officials and  representatives from four postal associations knowledgeable about  USPS’s efforts to develop innovative products and services, as well as  two mailers directly affected by USPS’s Informed Delivery innovation.

We conducted this performance audit from December 2017 to March  2019 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.

USPS’s Financial Position and Strategic Goals

USPS’s mission is to provide universal postal service while operating as a  self-financing entity, but USPS’s current financial position is not  sustainable. To achieve its mission, USPS must cover its expenses  through revenues generated from the sale of its products and services.  However, USPS’s total operating expenses have exceeded total  operating revenue each year since fiscal year 2007, including a $2.6  billion loss from operations in fiscal year 2017 alone (see fig. 1).  Moreover, we have reported that USPS’s overall financial condition is  deteriorating. For example, in August 2018 we reported that USPS had  about $149 billion in unfunded liabilities and debt at the end of fiscal year  2017. As a result, USPS’s financial condition remains on our list of high- risk areas needing attention by Congress and the executive branch.

According to USPS financial documents, its ability to sell innovative  products and services will be a key factor in improving its financial  condition. Thus, USPS established a strategic goal to “innovate faster to  deliver value” to its customers, by making investments in innovations that  respond to rapidly evolving customer needs. A key element of this effort  is to accelerate testing of innovative products and services to better serve  these needs, according to USPS.

While USPS is allowed to develop certain new postal products and  services, there are statutory restrictions that currently limit the range of  innovations USPS can offer. For example, under current statute, USPS is  not permitted to ship alcoholic beverages. Similarly, although USPS is  explicitly authorized to provide services to federal executive agencies  (e.g., passport services), such authorization does not include services to  state, local, and tribal governments. Legislation has been introduced in  previous sessions of Congress that would permit USPS to deliver  alcoholic beverages and allow USPS to provide property and services to  state, local, and tribal governments under certain conditions. According  to USPS officials, USPS supports these legislative proposals, which could  enhance its ability to offer innovative products and services.

USPS Postal Innovation Pilot Policies and Responsibilities

According to USPS officials, two USPS handbooks include policies  applicable to piloting key innovations. Specifically, the first handbook  includes requirements, procedures, and responsibilities for all types of  investment programs and projects undertaken by USPS, regardless of  size, cost, or complexity. This handbook, for example, requires the  identification and documentation of lessons learned for all investments  and projects. The second handbook includes requirements and  procedures specifically for major operating expense investments. This  handbook, among other things, establishes requirements and procedures  meant to ensure that new and enhanced products and services  consistently meet customer needs, generate new revenue, and  strengthen USPS as a business. This handbook further states that USPS  has a responsibility to subject new initiatives to rigorous financial analysis,  testing, and measurement, to determine whether these initiatives will  make a positive financial contribution to the organization and ensure that  USPS’s leadership has appropriate information for effective decision- making.

USPS’s Office of Product Innovation generally has lead responsibility for  piloting innovations. According to USPS’s policies, a project manager is  responsible for establishing and coordinating a cross-functional team to  design and evaluate the pilot. This team typically includes officials from a  variety of USPS departments, such as finance, general counsel,  information technology, marketing, and operations. The project manager,  with support from the cross-functional team, is responsible for preparing a  proposal for the pilot that includes key information, such as the pilot’s  objectives and performance measures, and overseeing pilot  implementation and communication with key stakeholders.

In some cases, PRC has a role in overseeing postal innovation pilots.  For example, USPS must notify PRC before it pilots any postal product  innovation for which it will impose a price (i.e., a pilot that generates  revenue for USPS) and must subsequently report quarterly revenue,  volume, and cost data. PRC also ensures that certain safeguards are  maintained during the pilot, such as limitations on the pilot’s duration and  revenue. However, according to PRC officials, the commission has limited  involvement in other areas of USPS’s efforts to develop innovations.

USPS Piloted Key Innovations Are Intended Primarily to Generate Revenue or Improve Customers’ Experience

USPS piloted 24 key innovations from fiscal years 2013 through 2017.  For example, USPS piloted an Identity Verification Service that allows  users to verify their identity either remotely (i.e., online) or in person at a  postal facility. Similarly, USPS piloted an innovation to allow mailers to  print shipping labels, track packages, and schedule package pick-ups by  accessing USPS data. The primary goal of the majority of these key  innovations (16 of 24) was to generate revenue, while the primary goal for  the remaining innovations was generally to improve customers’  experience using USPS products or services (see fig. 2). Appendix I  includes a complete list of key innovations USPS piloted from fiscal years  2013 through 2017.

The following discussion provides additional information about the 4 key  innovations we selected as illustrative examples of USPS’s efforts to pilot  innovative products and services.

Same-Day Delivery: From December 2012 to December 2015, USPS  piloted same-day delivery for consumer e-commerce purchases (see  fig. 3). According to the pilot proposal, this innovation was intended to  generate revenue for USPS by allowing it to leverage its existing  delivery infrastructure to capture part of the growing e-commerce  market. To determine the potential scalability of same-day delivery,  USPS first tested its operational feasibility and potential demand in  several major metropolitan areas, including San Francisco, New York,  and Phoenix. During the pilot, USPS delivered photos, chocolates,  water, electronics, and other goods from 38 participating mailers to  consumers in these areas. At the pilot’s conclusion, USPS decided to  continue offering same-day delivery to interested participating mailers  under Priority Mail contracts.

Grocery Delivery: From November 2014 to October 2017, USPS  piloted a grocery delivery product in nine selected metropolitan  areas. According to USPS, the innovation was intended to generate  additional revenue by taking advantage of the growing market for  grocery delivery. To test the innovation’s operational feasibility, USPS  required the pilot’s sole participating mailer to bring totes containing  groceries and other prepackaged goods ordered by customers directly  to post offices (see fig. 4). USPS was then responsible for sorting the  totes and delivering them to customers. According to its proposal for  this pilot, USPS expected grocery delivery to provide a substantial  revenue generation opportunity. At the pilot’s conclusion, like same- day delivery, USPS decided to continue offering grocery delivery with  the participating mailer under a Parcel Select contract.

Informed Delivery: From spring 2014 through July 2016, USPS piloted  a notification service called Informed Delivery in Northern Virginia and  New York. According to USPS, this innovation is intended to bridge  the gap between the physical and digital worlds by, for example,  emailing customers with a scanned image of the exterior address side  of letter-sized mail they should receive later that day (see fig. 5).  Informed Delivery can also allow mailers to conduct marketing  campaigns by integrating other elements—such as hyperlinks to  mailers’ websites—into the email and other notifications that  customers receive. In its proposal to pilot Informed Delivery, USPS  stated the pilot was intended to help USPS understand the service’s  business opportunity and increase the certainty of its potential  benefits, which included retaining mail volume and generating new  revenue from large advertisers. In addition, the pilot aimed to  generate “statistically valid data” on how subscribers respond to  marketing campaigns that mailers conduct. According to USPS, more  than 70,000 customers were actively using the service at the pilot’s  conclusion. In July 2016, USPS decided to end the pilot and launch  the service nationally. According to USPS, about 13 million customers  were subscribed to the service as of October 2018. USPS aims to  have 40 million customers subscribed to the service by 2020.

Keyless Parcel Lockers: Since October 2013, USPS has piloted  keyless parcel lockers that allow customers to independently pick up  packages in 98 selected post offices. According to USPS, among  other things, this innovation is intended to reduce the number of  missed package deliveries to customers’ post office boxes and  thereby reduce USPS’s delivery costs (see fig. 6). The purpose of  the pilot is to assess the performance and use of the lockers and to  assess their performance. In October 2013, USPS began pre-testing  the technical performance of 10 prototype keyless parcel locker units  at post offices in New York City and Northern Virginia. Following this  pre-test, in February 2015, USPS approved the installation of 50  made-to-order locker units in selected post offices across the country.  Finally, in May 2016 USPS expanded the pilot to include an additional  50 units, including 2 units that a senior USPS official told us were not  yet installed. As of November 2018, the pilot is still ongoing.

USPS’s Policies for Piloting Key Innovations Reflect Some but Not All Leading Practices for Pilot Design and Evaluation

USPS’s policies applicable to piloting key innovations fully reflect two of  the five leading practices for pilot design and evaluation that we identified  in prior GAO work and relevant standards for internal control (see table  1). These policies do not, however, fully reflect the other three leading  practices due to policy gaps. Further, we found that USPS had not  consistently followed its policies to document lessons learned at the  conclusion of each pilot, as discussed below.

Senior USPS officials acknowledged that gaps exist in its policies for pilot  design and evaluation because they were not developed by USPS to fully  reflect all leading practices. These policy gaps limit the extent to which  USPS can ensure that it is making good resource allocation decisions  based on pilot experiences. Below we further discuss the extent to which  USPS’s policies reflect the five leading pilot practices we identified as well  as how USPS applied these leading practices among the four piloted  innovations that we reviewed.

Establish appropriate and measurable objectives linked with identified  performance measures: We found USPS’s policies do not fully reflect this  leading practice. While USPS policies require that project managers  establish pilot objectives and performance measures, they do not require  that each objective be linked with identified performance measures. As a  result, some pilots may have objectives without an associated  performance measure.

For example, although USPS established both objectives and  performance measures for each of the four innovations we selected for  review, it did not consistently link each established objective to  performance measures. USPS’s proposal to pilot same-day delivery, for  example, had objectives of generating new revenue and improving  customers’ experience. However, while the proposal included  performance measures associated with generating new revenue—i.e.,  package volume, gross revenue, and net revenue—it did not identify and  link any performance measures with its objective of improving customer  experience. Similarly, USPS’s proposal to pilot keyless parcel lockers  included improving customers’ experience as one of its objectives.  However, while the proposal included a variety of performance  measures—reduction in the number of missed deliveries to post office  boxes, locker rate utilization, and on-time locker installation—it did not  identify and link any performance measures with its improving customers’  experience objective. Absent such measures, USPS may not know  whether customers have experienced an improvement using keyless  parcel lockers compared to using manual, keyed parcel lockers. Linking  all objectives to performance measures could help ensure that USPS has  the performance information to assess the extent to which a pilot has  achieved all of its objectives.

USPS officials told us that it can be difficult to measure performance for  some objectives related to customer experience. While measuring  customers’ experience can be challenging, it is important to understand  the extent to which a pilot has achieved all of its objectives. Further,  USPS has demonstrated that it can measure improvement in customers’  experience. For example, during its Informed Delivery pilot, USPS  conducted a consumer survey with approximately 5,500 Informed  Delivery subscribers to collect data on consumer adoption and  satisfaction. In the survey, USPS found that over 80 percent were  satisfied or very satisfied with the service. According to USPS officials,  this data helped USPS to measure the pilot’s success in meeting its  objective of improving customers’ experience.

Articulate a methodology for evaluating pilot performance: We found that  USPS’s policies fully reflect this leading practice because the policies  require officials to develop and communicate a methodology for  evaluating pilot performance. Articulating such a methodology helps  managers to identify the types and sources of performance information  necessary to evaluate the pilot. USPS’s policies require project  managers to work with the pilot’s cross-functional team to develop and  reach consensus on the methodology. These policies also require the  project manager and cross-functional team to identify data needs, data  sources, and how the data will be evaluated.

For the four innovations we reviewed, we found that USPS articulated a  methodology for evaluating the pilot’s performance. For example, for the  Informed Delivery pilot, USPS identified its customer registration system  as the method for tracking progress toward performance measures  related to the number of Informed Delivery subscribers. Similarly, in its  proposal to pilot keyless parcel lockers, USPS identified its central parcel  locker monitoring system as a method of tracking progress toward  performance measures related to utilization of keyless parcel lockers.

Evaluate pilot performance and identify and document lessons learned:  We found that USPS’s policies fully reflect this leading practice, but USPS  did not consistently follow its policy that requires documenting lessons  learned. Specifically, the policies require project managers to evaluate  performance and document lessons learned at the conclusion of each  pilot. Doing so can enable USPS to identify information needed to make  conclusions about the pilot’s scalability and ensures that such information  will be accessible to inform future related efforts. However, among the  key innovations we selected for review USPS had not consistently  documented lessons learned (see table 2).

USPS officials told us that they discussed lessons learned during ongoing  monitoring of pilot performance for these innovations, but had only  documented lessons learned for its Informed Delivery pilot. Specifically,  USPS identified lessons learned in its July 2016 proposal to launch its  Informed Delivery service nationally. In this proposal, we found that USPS  identified some lessons learned about the pilot related to user satisfaction  and adoption rates. USPS officials told us that this information helped to  inform USPS’s decision to launch the service nationally. However, USPS  officials acknowledged that USPS did not document lessons learned for  the other two concluded pilots that we selected for review (same-day  delivery and grocery delivery).

Senior USPS officials told us that USPS had not consistently documented  lessons learned at the conclusion of pilots across the 24 key innovations  because it had not developed tools, such as a template, or training that  could help ensure such consistency. Without consistently documenting  lessons learned for all of its pilots, USPS risks losing information  garnered during pilot implementation that could be relevant to future  innovation efforts. Doing so can be particularly important because,  according to a senior USPS official, officials responsible for pilot projects  sometimes retire or leave USPS for employment elsewhere, creating a  gap in knowledge of pilot experiences. Standards for internal control  underscore the importance of maintaining documentation in order to  retain organizational knowledge and mitigate the risk of having knowledge  limited to a few personnel.

Draw and document conclusions about scalability based on pilot results:  USPS’s policies do not fully reflect this leading practice. These policies  require that project managers draw conclusions based on the results and  lessons learned from the pilot. According to USPS officials, conclusions  may include determining scalability—i.e., whether, how, and when to  integrate pilot activities into overall efforts. However, USPS’s policies do  not specifically require that officials document these conclusions.  Documenting conclusions about scalability based on pilot results helps to  ensure retention of organizational knowledge related to the pilot that may  inform future decisions.

Among the three innovations that we selected for review for which the  pilots had concluded (i.e., same-day delivery, grocery delivery, and  Informed Delivery), USPS officials told us that senior leadership  discussed the results and lessons of the pilots and made determinations  regarding whether, how, and when to launch them more broadly, but that  they did not document these decisions or the rationale for them. By not  documenting conclusions, USPS risks losing information that could affect  the success of future related efforts and that could inform future USPS  leadership of the rationale for maintaining investments in activities upon  which pilots were based.

Documenting conclusions for innovation pilots can be especially important  in cases in which USPS decides to continue or expand pilot activities  even when the pilots do not meet all of their intended objectives. For  example, USPS’s same-day delivery and grocery delivery pilots had  revenue objectives, along with associated performance measures;  however, neither pilot achieved these objectives. For the same-day  delivery pilot, costs exceeded revenue in 12 of the 13 fiscal year quarters  in which the pilot was conducted, according to data USPS reported to  PRC. Likewise, USPS data indicate it did not reach its annual revenue  target for its grocery delivery pilot. Similarly, USPS’s pilot of Informed  Delivery was intended to generate “statistically valid data” on how  consumers respond to mailer marketing campaigns. However, according  to a senior USPS official, the pilot did not generate the data as intended,  because no such campaigns were conducted during the pilot.

As discussed earlier in this report, USPS did not discontinue any of these  three selected innovations when their pilots concluded. Although USPS  may have had good reasons to continue with, or more broadly launch,  these innovations despite the pilots not meeting all of their objectives, the  lack of documentation regarding its reasoning and decisions limits  information relevant to whether USPS is making judicious use of limited  resources.

Ensure appropriate two-way communication at all stages of the pilot with  key internal and external stakeholders in order to understand and address  their views: USPS’s policies do not fully reflect this leading practice.

USPS’s policies require the involvement of key internal stakeholders in  pilots. Specifically, USPS’s policies require the involvement of cross- functional teams—which include legal, finance, and other departments— and varying levels of review during the design and implementation of pilot  proposals. However, USPS’s policies do not address communication with  key external stakeholders. According to USPS officials, some pilot  projects may be confidential or have limited or no direct effect on external  stakeholders and, thus, communication with external stakeholders may  not be appropriate. While external stakeholder communication may not be  appropriate with some pilots, such communication, as appropriate, can  help to ensure that issues critical to the success of a pilot activity are  identified and addressed.

Among the innovations that we selected for review, USPS officials  explained various steps taken to involve internal stakeholders in the  design and evaluation of the pilots, such as the involvement of cross  functional teams to develop pilot proposals. Further, while USPS’s  policies do not address external stakeholder communication, we found  that USPS employed strategies for some of the innovations we selected  for review to communicate with some external stakeholders—i.e., industry  associations and mailers. For example, a representative of a postal  association told us that USPS shared information and sought input about  its Informed Delivery pilot during a quarterly meeting with industry groups.  Similarly, a mailer we interviewed told us that USPS had shared  information and sought input on the Informed Delivery pilot through direct  outreach with the mailer.

However, USPS did not consistently employ strategies to communicate  with some key external stakeholders among the innovations that we  selected for review. Specifically, USPS did not design or implement  strategies to obtain feedback from consumers on its pilots for same-day  delivery, grocery delivery, or keyless parcel lockers, despite the fact that  each of these innovations directly affected consumers. In contrast, as  previously discussed, for its Informed Delivery pilot, USPS planned and  conducted a survey to obtain consumer feedback, the results of which  helped USPS project managers support the proposal to expand the  service nationally. Absent communication with all key stakeholders,  USPS risks not having a complete understanding of perspectives that  could inform the viability of its innovations.

In recent years, USPS has sought to compete in a challenging business  environment by piloting innovations intended primarily to generate  revenue and enhance customers’ experience. The policies that USPS  uses for piloting key innovations fully reflect some leading practices for  pilot design and evaluation, such as articulating a methodology for  evaluating pilot performance. However, addressing gaps between  USPS’s policies and leading practices related to linking objectives and  performance measures, documenting conclusions, and communicating  with key external stakeholders would enable USPS leadership to better  assess the outcomes of its pilots, understand the rationale for conclusions  about scalability based on pilot results, and gauge customers’ reactions to  innovative products and services. Moreover, developing tools or training  to ensure that USPS consistently implements its policy of documenting  lessons learned from pilots would provide USPS with key information to  inform future related efforts.

We are making the following two recommendations to USPS:  The Postmaster General should direct the Vice President of Product  Innovation to develop policies that fully reflect leading practices for pilot  design and evaluation in areas such as linking objectives and  performance measures; documenting conclusions about scalability based  on pilot results; and communicating with key external stakeholders, as  appropriate. (Recommendation 1)

The Postmaster General should direct the Vice President of Product  Innovation to develop tools, such as a template, or training to help ensure  USPS consistently documents lessons learned at the conclusion of pilots,  as required by USPS policies. (Recommendation 2)

We provided a draft of this product to USPS and PRC for comment.  USPS provided a written response, which is reproduced in appendix II of  this report. In its response, USPS did not state whether it agreed with our  recommendations, but described actions that it plans to take related to  each. These actions, if fully implemented, would meet the intent of our  recommendations. For example, USPS stated that it would develop  policies specifically for pilot design, and would reflect leading practices for  pilot design and evaluation based upon best practice research. USPS  also noted that it would develop training to ensure consistent  documentation of lessons learned from its pilots. USPS added that this  planned training would cover best practices for pilot tests.

Regarding our first recommendation USPS said that pilots are only one  step in a larger process for developing innovations. We agree with this  and noted in our report that piloting is one key element of USPS’s efforts  to innovate. Nonetheless, given USPS’s financial position, effectively  piloting innovations is a critical step to ensure that USPS invests its  limited resources on innovations that are most likely to improve its long- term viability. USPS also stated that flexibility is important in innovation  pilots, particularly as it pertains to linking pilot objectives with performance  measures. We continue to believe that linking objectives with  performance measures is key to effectively evaluating pilots. In so doing,  however, there is flexibility to adjust pilot objectives and performance  measures as new information is gleaned during the pilot. Finally, with  regard to communication with external stakeholders during pilots, USPS  said that it communicates consistently with external stakeholders  regarding pilots at Mailers’ Technical Advisory Committee meetings  (MTAC). In our report, we noted that USPS employed strategies to  communicate with some external stakeholders—i.e., industry associations  and mailers.  We continue to believe, however, in the importance of  communication with all key external stakeholders, which may include  stakeholders, such as consumers, that do not participate in MTAC  meetings.

USPS and PRC also provided technical comments, which we  incorporated as appropriate.

We are sending copies of this report to the appropriate congressional  committee, the Postmaster General, Chairman of PRC, and other  interested parties. In addition, the report is available at no charge on the  GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-2834 or RectanusL@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix III.

Appendix I: List of U.S. Postal Service Key Piloted Innovations, Fiscal Years 2013-2017

The U.S. Postal Service (USPS) piloted 24 key innovations from fiscal  years 2013 through 2017 (see table 3).

Appendix II: Comments from the U.S. Postal Service

Appendix III: GAO Contact and Staff Acknowledgments

In addition to the individual named above, Derrick Collins (Assistant  Director); William Colwell and James Leonard (analysts in charge);  Barbara El Osta; Geoffrey Hamilton; Gina Hoover; Anthony Jackson; and  Laurel Voloder made key contributions to this report.

Revenue from USPS’s most profitable product—First Class Mail—has declined over the last decade. To help stem its significant financial losses and sustain its operations, USPS has been piloting innovative products and services, such as same-day delivery and grocery delivery.

However, with ever-shrinking financial resources, USPS needs to make sure that its new and innovative ideas generate revenue and enhance customer experience.

We found that the pilot process doesn't consistently give USPS the information it needs to effectively assess and draw conclusions about its pilots. We made 2 recommendations to improve the process.