Army Marketing: The Army Has Taken Recent Actions That Could Improve Program Oversight, Effectiveness, and Workforce Practices

The Army requested nearly $335 million for fiscal year 2020 to conduct marketing and advertising activities intended to increase awareness of Army service and ultimately generate leads for potential recruits.

In April 2018, AAA made recommendations in two reports to improve the contract oversight and return on investment of the Army's marketing and advertising program. Further, in May 2018 and October 2018, respectively, the Army and OPM made recommendations to improve the workforce practices and organizational structure of the program.

The John S. McCain National Defense Authorization Act for Fiscal Year 2019 included a provision for GAO to review the actions taken to implement AAA's recommendations and the effects of these actions on AMRG's leadership, workforce and business practices, and return on investment.

This report assesses the extent to which the Army has taken steps to address recommendations (1) from AAA to improve the contract oversight and measurement of the effectiveness of the Army's marketing and advertising program and (2) from the Army and OPM to improve the workforce practices and organizational structure of the marketing and advertising program.

GAO analyzed Army marketing and advertising data from fiscal year 2018; reviewed marketing and advertising plans and guidance; conducted focus groups with AMRG personnel; interviewed cognizant officials; and compared the Army's efforts to GAO-identified best practices.

The Army provided technical comments, which GAO incorporated as appropriate.

The Army has recently taken steps to improve the oversight of its primary marketing and advertising contract and measurement of the effectiveness of its marketing and advertising program in response to two U.S. Army Audit Agency (AAA) reports. In April 2018, AAA found that the Army Marketing and Research Group (AMRG)—the component responsible for conducting the Army's national-level marketing and advertising program—did not fully evaluate the performance of its contracted advertising agency or track the effectiveness of its marketing and advertising efforts. GAO found that AMRG has taken or is taking actions to address AAA's recommendations:

Contract Oversight . AMRG has developed processes for overseeing the advertising agency's performance and services. For example, AMRG developed a form program managers use to validate that proposed advertising services are not already provided through other contracts.

Program Effectiveness . AMRG has taken steps in several areas related to revising its strategic marketing goals to support Army recruiting, updating how it assesses marketing and advertising effectiveness, and improving the reliability of data systems. AMRG's steps are consistent with commercial best practices that GAO identified for assessing the effectiveness of advertising, such as identifying outcomes from advertising.

The Army has also taken steps to improve the workforce practices and organizational structure of its marketing and advertising program in response to two workforce reviews. The two reviews—by an Army organization and the U.S. Office of Personnel Management (OPM)—found that AMRG, among other things, did not have regular communication throughout its workforce and with its stakeholders, and had a poor workforce climate. AMRG took initial steps to address the reviews' recommendations. The Army then established a new office effective August 2019—the Office of the Chief Army Enterprise Marketing—to replace AMRG and to assume all marketing and advertising activities. Some of the Army's early steps to establish the new office are consistent with key practices for agency reform efforts identified previously by GAO. For example, the Army outlined a three-phased plan with specific tasks and associated dates to fully establish the new office by early 2020 consistent with the key practice to establish implementation goals and a timeline.

Each year, the Army seeks to achieve its recruitment goals by targeting  the small percentage of the U.S. population able to meet the education,  physical fitness, and age requirements for entry into military service. The  Army did not meet its recruitment goals in fiscal year 2018 and reported  meeting them in fiscal year 2019. To help achieve these goals, the Army  requested nearly $335 million for fiscal year 2020—an increase of about  31 percent over the prior year—to conduct marketing and advertising  activities. These activities include television and radio advertisements,  internet and other digital marketing, and direct mail to potential recruits.

Over the past few years, we and other organizations have identified  challenges within the Army’s marketing and advertising program and  recommended actions to improve the program’s operations. In May 2016,  we found that, among other things, the military services, including the  Army, did not fully measure the effectiveness of their advertising  activities. We recommended that the Department of Defense (DOD)  ensure that the military services fully measure the performance of  advertising activities. DOD partially concurred with this recommendation.  DOD has taken steps to address this recommendation by issuing  guidance requiring the military services to establish metrics, including  measurable goals, by which the success of a service or component’s  overall marketing plan or events can be measured.

Subsequent to our report, organizations within and outside of the Army  have reviewed the Army’s marketing and advertising program and made  recommendations to improve its operations. In two reports issued in April  2018, the U.S. Army Audit Agency (AAA) found that the Army Marketing  and Research Group (AMRG)—the Army component that was  responsible for conducting the Army’s national-level marketing and  advertising program—could improve its processes and procedures to  demonstrate that its activities provided what AAA termed a positive  “return on investment” and to increase management controls and  oversight of its marketing and advertising contracts. AAA made seven  recommendations in each of the two reports, and the Army concurred  with all 14 recommendations.

In addition, in May 2018, the Office of the Assistant Secretary of the Army  for Acquisitions, Logistics, and Technology (ASA(ALT)) identified several  high-risk issues that contributed to organizational inefficiencies within  AMRG and made recommendations to improve performance  transparency, mission-to-function alignment, and program accountability.

Further, in accordance with section 599 of the John S. McCain National  Defense Authorization Act for Fiscal Year 2019 (hereafter NDAA for  Fiscal Year 2019), the Army reported that AMRG requested support from  the U.S. Office of Personnel Management (OPM) in reviewing AMRG’s  organization. In October 2018, OPM issued a study identifying  organizational design issues contributing to challenges with AMRG’s  reputation, trustworthiness, work environment, and organizational  performance, and identified actions that AMRG could take to improve its  workforce practices.

Section 599 of the NDAA for Fiscal Year 2019 included a provision that  we conduct a review of the actions taken to implement AAA’s  recommendations and of the effects of these actions on AMRG  leadership, workforce and business practices, and return on investment.

In this report, we assess the Army’s response to recommendations from  AAA, ASA(ALT), and OPM. Specifically, we assess the extent to which  the Army has taken steps to: (1) improve contract oversight within its  marketing and advertising program; (2) improve how AMRG measures  the effectiveness of its marketing and advertising program; and (3)  improve the workforce practices and organizational structure of the  marketing and advertising program. In appendix II, we provide a  consolidated summary of the 14 recommendations from the AAA reports  on contract oversight and return on investment, the actions that AMRG  has taken to address them, and the status of their implementation,  according to AAA, as of September 2019.

For the first objective, we reviewed documentation and interviewed  AMRG, AAA, and U.S. Army Mission Installation and Contracting  Command officials on AMRG’s actions to address deficiencies identified  in the AAA report on contract oversight. We analyzed the number of task  orders issued and amounts obligated from fiscal year 2011 through 2018  on AMRG’s primary marketing and advertising contract. We obtained  this data from the Federal Procurement Data System-Next Generation,  which is a primary source of federal procurement data. We chose data  starting in fiscal year 2011 because the Army awarded the previous  contract in March 2011 and ending at the end of fiscal year 2018 because  it was the last complete fiscal year of data at the time of our review. To  assess the reliability of the Federal Procurement Data System-Next  Generation data that we used, we reviewed the data dictionary, data  validation rules, data outputs, and other documentation. Based on these  steps, we determined that the data were sufficiently reliable for the  purposes of identifying the number of task orders issued and amounts  obligated on AMRG’s primary marketing and advertising contract.

For the second objective, we reviewed documentation on the Army’s  process for developing its annual marketing plan, goals, and performance  measures, and we conducted interviews with officials at organizations  with responsibilities for implementing the Army’s marketing and  advertising program, including AMRG; the U.S. Army Training and  Doctrine Command (TRADOC); Office of the Chief, Army Reserve; and  Army National Guard. We also conducted a site visit to Fort Knox,  Kentucky, to meet with officials from U.S. Army Recruiting Command  (USAREC), U.S. Army Cadet Command (USACC), and the U.S. Army  Marketing and Engagement Brigade to discuss their efforts to measure  the effectiveness of their marketing and advertising activities and to  observe Army marketing and advertising facilities and assets. We  compared this information with the three categories of commercial best  practices for assessing the effectiveness of advertising identified in our  prior work.

In addition, we analyzed data from the Enterprise Marketing Management  (EMM) system—the Army’s marketing system of record—on key  performance indicators for measuring the effectiveness of the Army’s  marketing and advertising activities conducted in fiscal year 2018—the  last complete fiscal year of data at the time of our review. To assess the  reliability of the EMM data, we met with AMRG officials to discuss the  processes for entering, maintaining, and reviewing the quality of the data,  and we conducted electronic testing to identify inconsistencies and  outliers. Based on these steps, we determined that the data from fiscal  year 2018 were not sufficiently reliable for the purposes of identifying key  performance indicators associated with the Army’s marketing and  advertising activities. Later in this report, we describe the actions the  Army is taking to address limitations in the reliability of the EMM data.

For the third objective, we reviewed Army general orders, directives, and  other related guidance and documentation on changes to the  organizational structure of the Army’s marketing and advertising program.  In addition, we reviewed the recommendations from two reviews—one  conducted by ASA(ALT) on AMRG’s business processes and the other  conducted by OPM on AMRG’s organizational design. We requested  information and documentation of the Army’s steps to reorganize the  workforce and organizational structure of the marketing and advertising  program and compared this information to applicable key practices for  agency reform efforts we identified in our prior work. Because of the  timing of the reorganization of the Army’s marketing and advertising  program during our review—which we describe later in this report, we did  not assess the Army’s steps against all of the key practices.

We also conducted three focus groups in May and June 2019 with non- supervisory personnel from AMRG to gain their perspectives on  leadership and workforce changes. We conducted two of these focus  groups at AMRG Headquarters in Arlington, Virginia, and randomly  selected participants from non-supervisory personnel who had been at  AMRG since September 2017. We chose this date because the  personnel would be able to comment on changes in the leadership and workforce practices since officials reported receiving a draft of the AAA  reports, as well as changes in AMRG leadership that occurred in  November 2017. We conducted the other focus group at an AMRG office  in Fort Knox, Kentucky, with non-supervisory personnel who had been at  AMRG since September 2017. In addition, we interviewed five  supervisory personnel who had been at AMRG since September 2017.  We met with non-supervisors separately from supervisors to encourage  candid discussions about the Army’s marketing and advertising program.  We completed written summaries of each focus group and interview, and  then multiple GAO analysts independently coded the responses to  identify themes across the focus groups and interviews and worked to  ensure agreement on the themes.

We conducted this performance audit from December 2018 to November  2019 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.

Roles and Responsibilities in the Army’s Marketing and Advertising Program

The Secretary of the Army has the responsibility to recruit personnel,  subject to the authority, direction, and control of the Secretary of  Defense. The Assistant Secretary of the Army for Manpower and  Reserve Affairs serves as the principal advisor to the Secretary for the  Army’s management of its manpower and personnel and provides overall  governance for marketing, advertising, and research. The Deputy Chief  of Staff of the Army, G-1, is the principal military advisor to the Assistant  Secretary of the Army for Manpower and Reserve Affairs and the Chief of  Staff of the Army for all matters related to manpower across the Army. As  of August 2019, the Deputy Chief of Staff of the Army, G-1, has  responsibility for overseeing the new Office of the Chief Army Enterprise  Marketing (AEMO) once it is fully established, as described later in this  report.

In addition, multiple other Army organizations from across the accessions  enterprise—the collection of Army organizations involved in efforts to  recruit and train soldiers for the Army—have roles and responsibilities in  carrying out the Army’s marketing and advertising program, as  summarized in table 1.

The Army contracts with a primary advertising agency to develop and  implement its marketing and advertising program. The advertising agency  is responsible for providing a range of services from the development of  the Army’s marketing and advertising strategy to the production of  marketing and advertising activities, including television and print  advertisements, event marketing, and social media. In November 2018,  the Army awarded a contract to a new advertising agency for up to 10  years with a value not to exceed $4 billion. The contract with the  previous advertising agency was awarded in March 2011, and from  March 2011 through the end of fiscal year 2018, the Army issued 702  task orders and obligated about $1.6 billion on this contract, according to  our analysis of data from the Federal Procurement Data System-Next  Generation.

Types of Marketing and Advertising Activities

The Army conducts a variety of marketing and advertising activities at the  national and local levels in support of the Army’s recruiting goals. Figure  1 shows examples of the various types of Army marketing and advertising  activities, such as mobile assets used at recruiting events and digital  advertising on social media.

Different marketing and advertising activities are useful for supporting the  three phases—awareness, engagement, and activation—of an  individual’s decision-making process, sometimes referred to as the  consumer journey. The goal of marketing and advertising is to move a  potential recruit through each of the three phases and, ultimately, to the  decision to enlist:

Awareness: Individuals learn about the opportunity to serve in the  Army and the distinct characteristics of serving in the Army. The Army  pursues awareness through marketing and advertising activities such  as television commercials, print advertisements, banners at events,  and billboards.

Engagement: Individuals who are aware of the opportunities for  service in the Army begin considering the possibility of joining the  Army. During this phase, the Army seeks to provide recruits with  additional information to aid in their decision-making process. Often  this phase of advertising takes place in the digital environment, as the  Army seeks to provide informative social media posts and use banner  advertisements to attract individuals to visit its website for more  information.

Activation: Individuals have considered the Army and are ready to  talk to a recruiter about enlistment. As such, activation activities seek  to encourage these individuals to provide their contact information to  schedule an opportunity to meet with a recruiter. Activation is often  conducted in person, such as through recruiters’ presence at events  like career fairs. Activation may also be conducted through other  means—such as direct mail and online or print classified  advertisements—as long as the advertisement prompts viewers to  provide their contact information. Further, the Army employs the use  of mobile assets, such as large trucks and trailers fitted with  equipment and activities intended to draw crowds and encourage and  facilitate public interaction with a recruiter at an event to generate  leads.

Figure 2 shows the awareness, engagement, and activation phases and  examples of marketing and advertising activities that are used in support  of each phase.

The Army Has Taken Steps to Develop Processes to Improve Its Oversight of Its Primary Marketing and Advertising Contract

To implement AAA’s recommendations, the Army has developed  processes designed to improve its oversight of the primary contract for  executing the Army’s marketing and advertising program. In April 2018,  AAA reported that AMRG did not sufficiently evaluate the performance of  its primary contractor, effectively oversee deliverables included in its three  main marketing and advertising contracts, or effectively oversee the  negotiation process of task orders for its primary marketing and  advertising contract. AAA made seven recommendations to AMRG to  improve its contract oversight, with which AMRG agreed. As shown in  table 2, as of September 2019, AAA considered three of the  recommendations implemented but not closed, with AMRG still taking  steps to address the other four recommendations; as a result, it is too  soon to assess the extent to which the Army’s steps have improved  contract oversight within the marketing and advertising program. AMRG  officials stated that the implementation of these recommendations has  been slowed, in part, because of the recent award of its primary  marketing and advertising contract to the new advertising agency.

Among AMRG’s actions to improve its contract oversight are steps to  develop processes for overseeing contractor performance, deliverables,  and price negotiations. Specifically, AMRG has taken steps in the  following areas:  Created a new project management office. AMRG established a  project management office to help address the challenges identified by  AAA and to serve as a coordinating body that centralizes contract  oversight. According to its charter, the office is responsible for maintaining  cost, schedule, and performance for Army marketing and advertising  programs to help ensure that they are completed on time and within  reasonable costs to support the accessions mission.

Documents outlining the contract management process indicate that the  project management office reviews key contract documents, such as the  statement of objectives and the quality assurance surveillance plan,  before the documents are submitted to the contracting officer who is  responsible for administering the contract. After the contractor submits its  proposal for providing the requested product or service, the project  management office reviews the proposal to ensure it meets the  requirements of the statement of objectives. Also, the project  management office coordinates and submits the technical evaluation form  for review by the contracting officer’s representative and the contracting  officer.

Within the newly-formed AEMO, there will be a project management  office with six authorized personnel, according to the organizational chart  for AEMO. In July 2019, AMRG officials stated that they were developing  standing operating procedures and continuity plans that AEMO could use  as it establishes its project management office.

Implementing training and tools to evaluate contractor performance.  AAA found that AMRG did not sufficiently evaluate the performance of its  primary contractor and recommended that AMRG require that all program  managers receive contracting officer’s representative training and ensure  that individual quality assurance surveillance plans are developed for  each task order over $150,000. As of September 2019, AAA reported  that this recommendation had not been fully implemented. In response to  AAA’s recommendations, in April 2019, AMRG reported that all program  managers had completed contracting officer’s representative training.  Further, as of September 2019, AMRG reported that the Army had  developed individual quality assurance surveillance plans for each task  order issued in fiscal year 2019. According to AAA officials, AMRG  provided them with examples of these quality assurance surveillance  plans, and AAA provided AMRG with feedback on additional information  that should be included, which AMRG officials stated they were taking  actions to address.

In September 2019, AMRG also issued standing operating procedures for  program managers to provide internal policy and instruction for executing  the Army’s marketing and advertising program as well as evaluating its  performance, including overseeing contractor performance. We found that  the standing operating procedures require contracting officer’s  representative training for program managers and that the Director of  Marketing, Director of Research, and AMRG contracts team are to  monitor compliance quarterly. In addition, the procedures include steps  outlining contract oversight mechanisms, such as information on the  purpose and contents of quality assurance surveillance plans.

Implementing a standardized technical evaluation form. AAA found  that AMRG had not effectively negotiated prices for its primary marketing  and advertising contract; AAA recommended that AMRG (1) define and  implement a well-structured policy for conducting technical evaluations of  contractor proposals and (2) establish a standardized form to ensure  consistency during the evaluation process. As of September 2019, AAA  reported that these two recommendations were implemented but not  closed. The standing operating procedures that we reviewed outline the  process for program managers to complete a standardized form for  evaluating contractor proposals. The form is intended to ensure that  program managers are consistently evaluating contractor proposals for  performing work under the contract. For example, the form requires  program managers to conduct a comparative price analysis by comparing  the contractor’s proposed price to total task order cost in prior years.  According to the instructions on the form, program managers are to  submit the completed form to their supervisor or director for approval. The  project management office then coordinates and submits the technical  evaluation to the contracting officer’s representative and contracting  officer for review.

Overseeing contract deliverables. AAA found that AMRG did not  effectively oversee deliverables in its marketing and advertising contracts  and recommended that AMRG develop procedures to ensure that  contracts or task orders do not contain deliverables already provided in  other contracts. As of September 2019, AAA reported that this  recommendation had not been fully implemented. AMRG and U.S. Army  Mission Installation and Contracting Command officials have  implemented processes to prevent duplicative deliverables (i.e., services  or products to be provided through a contract) in the future, such as the  standardized technical evaluation form noted above. We found that the  standardized technical evaluation form requires program managers to  certify that they have reviewed other tasks and contracts within their  purview and to validate that the task order being requested does not  duplicate existing or other requested work. In addition, AMRG and U.S.  Army Mission Installation and Contracting Command officials stated that  they consolidated all contract actions under one team at the U.S. Army  Mission Installation and Contracting Command and that both AMRG’s  contracting officer’s representative and budget office must verify that  contract requests are not duplicative.

AAA also found overlapping deliverables between AMRG’s primary  marketing and advertising contract and a contract for creative technology  support and recommended that AMRG use the creative technology  support contract for all of the creative technical services within its scope.  As of September 2019, AAA reported that this recommendation had not  been fully implemented. AMRG officials told us that they plan to issue a  modification to the contract for creative technology support to remove  services duplicated in the primary marketing and advertising contract. As  of September 2019, an AMRG official told us that the Army expected to  issue the modification in November 2019.

Revising process for contract award fees. AAA found that AMRG had  minimal support to justify its higher award fee ratings for its primary  contractor and recommended that AMRG update its award fee plan and  award fee review process to include soliciting feedback from program  managers, maintaining supporting documentation, and obtaining objective  performance data, among other things. As of September 2019, AAA  reported that this recommendation had not been fully implemented. We  found that AMRG revised its process in fiscal year 2018 for determining  award fee incentives for its advertising agency. For example, according to  the documentation associated with the award fee decision for the  agency’s performance from April 2017 through April 2018, AMRG  reported that it, among other things, included feedback from program  managers on the advertising agency’s performance and obtained  objective performance data from an independent entity, DOD’s Joint  Advertising, Market Research & Studies.

In future work with the new advertising agency, AMRG officials stated that  the Army plans to offer award fees for specific task orders rather than one  fee for performance in a given year. According to AMRG officials, this  change allows greater flexibility in deciding which programs should be  incentivized with an award fee. As of September 2019, AMRG officials  stated that they had not issued any task orders with an award fee under  the new marketing and advertising contract.

The Army Has Taken Steps to Improve Its Approach for Measuring the Effectiveness of Its Marketing and Advertising Program

To implement AAA’s recommendations, the Army has taken steps to  improve how it measures the effectiveness of its marketing and  advertising program; these steps are consistent with commercial best  practices for assessing the effectiveness of advertising identified in our  prior work. In its 2018 report on return on investment, AAA found that  AMRG had deficiencies in how it measured the effectiveness of its  marketing and advertising efforts and made seven recommendations to  AMRG, with which AMRG agreed. Of these recommendations, AAA  considered four implemented but not closed as of September 2019, with  AMRG still taking steps to address the other three recommendations, as  shown in table 3. Since the Army’s steps were recently implemented or  are ongoing, it is too early to determine if they will achieve their desired  results.

Based on our analysis of the Army’s actions, the Army’s steps to  implement AAA’s recommendations fall into the following five areas: (1)  revising strategic goals, (2) updating and documenting its assessment  process, (3) improving the reliability and capabilities of data systems, (4)  integrating national and local marketing and advertising efforts, and (5)  obtaining new tools to determine required marketing and advertising  resources. The steps the Army has taken in these areas thus far are  consistent with commercial best practices for assessing the effectiveness  of advertising we identified in our prior work. As the Army takes  additional steps to establish the newly-formed AEMO, it will be important  for the Army to continue to align its efforts with these commercial best  practices for assessing the effectiveness of advertising to ensure  advertising dollars are used efficiently to help meet stated recruiting  goals.

Revising strategic goals. AAA found that AMRG did not have specific  goals to measure the long-term effects of investments in marketing and  advertising efforts to support the Army’s accessions mission and  recommended that AMRG develop such goals. As of September 2019,  AAA reported that this recommendation was implemented but not closed.  AMRG has revised its strategic marketing goals from tracking changes in  individuals’ attitudes toward the Army, such as support for the Army  among the general population, to tracking the behaviors of these  individuals, such as the number of visits to GoArmy.com. For fiscal year  2018, AMRG had seven strategic marketing goals that tracked the  attitudes of the general population and prospects toward the Army. For  fiscal year 2019, AMRG revised the goals to four that track attitudes, two  that track behaviors, one that tracks effectiveness, and one that tracks  efficiency. Goals that track attitudes are aligned with the awareness  phase of the consumer journey, whereas goals that track behaviors are  more aligned with the engagement and activation phases. Figure 3 shows  how the fiscal year 2018 and 2019 strategic marketing goals align with  the three phases of the consumer journey.

Looking ahead, AMRG officials told us that, consistent with feedback they  received from marketing industry experts, the strategic goals in the fiscal  year 2020 marketing plan will all be behavioral and will target the different  stages of what AMRG refers to as a lead nurturing funnel. AMRG  officials stated that their goal is to use information so they can quickly  shift attention and funding to different stages of the funnel that are not  meeting their goals, so as to ensure that those stages get the attention  needed to reach mission success. AMRG’s recent and ongoing steps to  revise its marketing goals are consistent with the commercial best  practice to develop an evaluation framework that identifies the target  audience and includes measurable goals.

To institutionalize AMRG’s updated processes, AAA recommended that  AMRG update Army Regulation 601-208 to reflect the new goals and  processes it would implement to improve its program effectiveness. As  of September 2019, AAA reported that this recommendation had not been  fully implemented. The Army is in the process of revising its marketing  and advertising regulation to reflect the updated strategic marketing goals  and process. AMRG had drafted a revision to the regulation; however,  AMRG officials had put this process on hold while senior Army leaders  were making the decision about AMRG’s placement within the Army. Now  that AMRG has been redesignated as AEMO and reassigned within the  Army, as of August 2019 AMRG officials stated that the Army was  revising the regulation to reflect the new organization’s relationship to  other entities within the accessions enterprise. AMRG expects the Army  to publish the updated regulation in 2020.

Updating and documenting its assessment process. AAA found that  only three of AMRG’s 23 national events during fiscal year 2016 provided  the best value for their intended purpose and recommended that AMRG  discontinue efforts that were not cost-effective in comparison to other  options and assess the cost-effectiveness of current marketing and  advertising efforts. As of September 2019, AAA reported that this  recommendation was implemented but not closed. According to AMRG  officials, AMRG discontinued all of the events that were deemed to not be  cost-effective in AAA’s report.

In its report, AAA also found that AMRG’s assessment process did not  include USAREC’s and USACC’s marketing and advertising efforts and  that AMRG did not formally document that process; AAA recommended  that AMRG establish and formally document a process with roles and  responsibilities to assess the effectiveness and efficiency of all Army  marketing and advertising efforts. As of September 2019, AAA reported  that this recommendation was implemented but not closed.

We found that AMRG formally documented how it assesses the  effectiveness of the marketing and advertising program, consistent with  the commercial best practice of seeking to develop an understanding of  how outcomes can be attributed to advertising. In January 2019, AMRG  issued guidance that outlines its assessment process, including the types  of information that are reviewed in each assessment. The guidance we  reviewed outlines three levels of assessments: Level I is a review by  program managers of their individual programs; level II, to be conducted  on a quarterly basis, is a review at the operational level in which the  directors of research and marketing review the results of marketing and  advertising efforts across multiple marketing channels; and level III, also  to be conducted on a quarterly basis, is a review at the strategic level in  which the AMRG Director reviews the Army’s progress in meeting its  strategic marketing goals. Further, in January 2019, AMRG reviewed a  summary of the number, cost, and performance of USAREC’s and  USACC’s local marketing and advertising activities as part of the level II  assessment process.

In addition, to facilitate comparison of options, AMRG developed a tool for  comparing different programs within a given marketing channel. The tool,  which AMRG calls a decision support matrix, allows officials to  comparatively rank different programs based on weighting different  factors. For example, based on our review of the decision support matrix,  we found that AMRG assigned a higher weight to a program’s  effectiveness than its timing. The tool also incorporates qualitative  feedback based on how program managers, USAREC, and USACC rank  the programs, and quantitative analysis on the cost per lead, impression,  or engagement, depending on the type of program.

Improving the reliability and capabilities of data systems. AAA  identified discrepancies between information in the Enterprise Marketing  Management (EMM) system and supporting documentation and  recommended that AMRG establish and formally document a process to  ensure that all Army marketing and advertising performance and cost  data were regularly recorded in an official marketing system of record on  a regular basis. As of September 2019, AAA reported that this  recommendation had not been fully implemented. The Army has taken  steps to improve the reliability of the data in EMM since AAA’s report. In  August 2019, the Army issued a task order on its primary marketing and  advertising contract covering EMM system support to include overseeing  and improving the quality of data in EMM. According to the performance  work statement, the Army’s new advertising agency is responsible for,  among other things, accurately documenting current data, systems, and  business processes, as well as analyzing EMM reports and  documentation for completeness and accuracy. Further, the advertising  agency is responsible for identifying and documenting business problems  and recommending areas for improvement and technology solutions.

During the focus groups we held with AMRG personnel, participants told  us that AMRG leadership was focused on demonstrating the  effectiveness of the Army’s marketing and advertising through reliable  and readily-available data. For example, the Army implemented an  electronic business reply card, which is a digital form to capture a  potential recruit’s eligibility and contact information and a means of  identifying the event where the recruit learned about the Army.  USAREC and USACC officials told us that prior to the electronic business  reply card, recruiters collected prospects’ information at events by using a  paper card. Although that card reflected the event where the potential  recruit heard about the Army, it would often take several days before the  potential recruit’s information appeared in the recruiting system,  according to these officials. As a result, recruiters would sometimes not  send in the paper card and would instead enter the prospect’s information  directly into the system. In these cases, the marketing and advertising  event would not receive credit for generating the lead. USAREC and  USACC officials stated that the electronic business reply card’s quicker  turnaround time for leads showing up in the system should improve data  reliability by ensuring that recruiters and recruiting operations officers  consistently enter an individual’s demographic data into the system, along  with the marketing and advertising activity they interacted with. These  steps to better identify the number of leads generated by marketing and  advertising activities are consistent with the commercial best practice of  conducting ongoing analysis of performance using industry standard  measures appropriate for the purpose of the advertising activity.

In addition, in August 2019, the Army issued a task order on its new  marketing and advertising contract for the maintenance and optimization  of its system that tracks analytics on the Army’s marketing and  advertising activities, which AMRG refers to as the Intelligence Hub. The  advertising agency is responsible for monitoring this system and  producing reports that track the effectiveness of marketing and  advertising activities based on key performance indicators. The  advertising agency is also responsible for upgrading the system to track  the multiple marketing and advertising resources that a potential recruit  interacts with. AMRG officials told us that the upgrade of this system is  intended to equip the Army with more complete data to demonstrate the  effectiveness of the Army’s marketing and advertising activities— consistent with the commercial best practice of seeking to develop an  understanding of how outcomes can be attributed to advertising.

Integrating national and local marketing and advertising efforts. AAA  found that AMRG did not integrate and leverage both national and local  marketing and advertising efforts to support the Army’s accessions  mission and recommended that AMRG revise the Army’s marketing  performance framework to include marketing and advertising efforts at  both the national and local levels. As of September 2019, AAA reported  that this recommendation was implemented but not closed. We found that  the Army has created programs and instituted procedures designed to  increase coordination of national and local marketing and advertising  efforts. For example, AMRG reported that it included other Army  components, including USAREC and USACC, in developing the fiscal  year 2019 marketing goals and planned to include those organizations in  its fiscal year 2020 process.

In addition, TRADOC established the Army Accessions Resource Fusion  Board, which brings together organizations from across the accessions  enterprise for quarterly meetings at which they make operational resource  sharing plans for marketing and recruiting assets. For example,  according to a March 2019 briefing for an Army Accessions Resource  Fusion Board meeting, representatives from USAREC brigades  discussed their planned marketing and advertising activities for the first  quarter of fiscal year 2020, including any requests they had for support  from other Army stakeholders for those planned activities. Further,  according to its charter, the Army Accessions Resource Fusion Board is  responsible for assessing the effectiveness of local marketing and  advertising efforts in the previous quarter.

In fiscal year 2018, the Army also created a pilot program designed to  improve how the Army’s marketing and advertising program coordinates  with its recruiting components and to produce marketing and advertising  messages that resonate more effectively with target populations. The  Army began implementing the program in Chicago in fiscal year 2019 and  as of April 2019 was planning to expand the program to Boston and four  other cities. As of July 2019, AMRG had observed positive results from  the program in Chicago. For example, AMRG reported an increase of 11  percent in the number of leads and an increase of about 7 percent in the  number of recruits who signed contracts with the Army when compared to  the prior year in that region.

Officials from TRADOC, USAREC, and USACC told us that coordination  with AMRG on marketing and advertising efforts has improved since the  time AAA conducted its audits. For example, TRADOC officials stated  that AMRG senior leaders have supported the accessions enterprise by  providing analytic support to USAREC and USACC. Further, USAREC  officials stated that in fiscal year 2018 AMRG started to provide funding  for local marketing and advertising activities near USAREC’s requested  levels, and that this change had been carried forward into fiscal year  2019.

Obtaining new tools to determine required marketing and  advertising resources. AAA found that AMRG did not use a resource  requirements projection model that supported and linked to planned  marketing efforts and recommended that AMRG develop such a model.  As of September 2019, AAA reported that this recommendation had not  been fully implemented. AMRG has contracted with the RAND  Corporation and a consulting firm to develop tools to determine the  resources AMRG needs to conduct its marketing and advertising  activities. The RAND tools include three planned models, one of which is  the recruiting resource model recommended by AAA. The recruiting  resource model has been partially completed and is being updated with  additional data with full completion scheduled for September 2020.  According to the Army’s fiscal year 2020 budget request, the Army used  the RAND report that developed this model as a justification for  increasing its advertising budget for fiscal year 2020.

Further, in consultation with a consulting firm, AMRG developed a  channel allocation simulator that allows AMRG officials to test different  funding levels for its marketing and advertising channels to see potential  outcomes. For example, based on our review of the simulator, AMRG can  enter a specific amount of funding for events to estimate how many leads  it can expect to obtain from that level of funding. AMRG officials stated  that they can use this tool to help them plan for their required level of  resources for the upcoming fiscal year. The development of this simulator  is consistent with the commercial best practice of using sophisticated  marketing mix modeling to determine an appropriate spending strategy.

The Army Has Taken Steps to Improve Its Workforce Practices and to Reorganize the Organizational Structure of Its Marketing and Advertising Program ASA(ALT) and OPM Conducted Reviews of AMRG’s Workforce and Recommended Areas for Improvement

ASA(ALT) and OPM conducted reviews of AMRG’s workforce and made  recommendations to improve the workforce practices within the marketing  and advertising program. From January to May 2018, ASA(ALT)  conducted a review of AMRG’s business processes and found high-risk  issues that contributed to organizational inefficiencies within five areas:  (1) internal communications, (2) business performance, (3) training, (4)  program performance and accountability, and (5) personnel. For example,  ASA(ALT) found that AMRG personnel were unclear about AMRG’s core  mission, objectives, and program priorities. Further, ASA(ALT) found that  AMRG’s personnel, skills, training, and physical locations were not  aligned to support AMRG’s mission. In addition, OPM conducted an  assessment from March to September 2018 to identify organizational  inefficiencies and propose solutions intended to transform AMRG into a  high-performing organization and improve its workforce morale. Similar to  ASA(ALT), OPM identified issues with a lack of mission clarity and  insufficient communication and collaboration throughout AMRG’s  workforce and with its stakeholders. In addition, OPM identified a number  of organizational design issues within AMRG, including workforce  acquisition, management, and optimization of its operational components  and staff.

ASA(ALT) and OPM made multiple recommendations to address these  issues within AMRG. For example, ASA(ALT) recommended that AMRG  establish and disseminate standard operating procedures and process  charts; clarify roles and responsibilities of the various organizational  components; and clearly communicate to staff the final annual marketing  strategy. OPM recommended a multiphased approach to implementing  its overall recommendations, identifying key actions to take in each  phase. For example, within the first phase, OPM recommended that  AMRG determine the new functional structure for AMRG because it would  improve management and accountability, collaboration, and stakeholder  satisfaction. In addition, within the second phase, OPM recommended  that AMRG develop a human capital management plan and review and  update its position descriptions regularly to ensure they align with  changing goals, staffing needs, and the organizational structure of  AMRG.

AMRG took some steps to address ASA(ALT)’s and OPM’s  recommendations to improve its workforce practices. Within its report,  ASA(ALT) noted that AMRG had started to take actions to implement  several recommendations, such as disseminating AMRG’s mission  statement, priority objectives, strategic goals, and fiscal year 2019 annual  marketing plan guidance to all AMRG personnel. Similarly, OPM noted  that AMRG had established task groups to coordinate with stakeholders,  participated in meetings with Congress and stakeholders, and was  developing a new vision for AMRG. However, as of April 2019, senior  AMRG officials stated that they had not taken steps to address all of the  reports’ recommendations because the Army was considering broader  organizational changes to the placement of AMRG within the Army. While  the recommendations in ASA(ALT) and the OPM reports were generally  specific to AMRG’s organization and workforce at that time, senior AMRG  officials stated that the Army would take additional steps to incorporate  ASA(ALT)’s and OPM’s recommendations, as appropriate, after senior  Army leadership made decisions about those changes.

In our review, AMRG personnel we met with continued to identify poor  internal communications and morale as key challenges within AMRG,  consistent with the findings from ASA(ALT) and OPM. During our focus  groups with AMRG personnel, participants repeatedly stated that senior  AMRG leadership did not communicate key information to staff. For  example, participants told us that senior AMRG leadership did not  communicate information about AMRG’s mission, strategic priorities, or  pending organizational changes. As described below, subsequent to our  focus groups, the Army began taking steps to fundamentally change the  organizational structure, workforce, and leadership of its marketing and  advertising program. In light of the timing of these substantial changes,  we did not comprehensively assess the extent to which communication  issues have been resolved in the reorganization of the marketing and  advertising program. It will be important for the new leadership to focus  on communication at the outset of this organizational change to establish  positive morale within the workforce.

The Army Reorganized Its Marketing and Advertising Program to Improve Its Structure, and Early Steps Are Consistent with Some Key Reform Practices

The Army recently reorganized its marketing and advertising program to  improve its organizational structure; the Army’s early steps to implement  the reorganization are consistent with some key practices for agency  reform efforts we identified in our prior work, as described below. In May  2019, the Secretary of the Army reassigned, redesignated, and stated  that the Army planned to relocate AMRG. The Secretary of the Army  redesignated AMRG as AEMO and reassigned the office to the Deputy  Chief of Staff of the Army, G-1. The effective date of this reassignment  was August 1, 2019. AEMO’s mission is to coordinate the Army’s national  marketing and advertising strategy; develop and maintain relationships  with the marketing and advertising industry; and develop marketing  expertise and talent within the Army to support the Army, Army National  Guard, and Army Reserve accessions. The offices will be moved from  Arlington, Virginia, and Fort Knox, Kentucky, to Chicago, Illinois.  Consistent with the key practice to designate a leader to be responsible  for the implementation of the proposed reforms, the Secretary of the Army  designated the Assistant Secretary of the Army for Manpower and  Reserve Affairs as responsible for establishing AEMO and overseeing the  transition. The Assistant Secretary stated that he expected AEMO to be  fully operational by early 2020.

The Army identified several reasons for transitioning from AMRG to  AEMO and reassigning the office to the Deputy Chief of Staff of the Army,  G-1, consistent with the key practice to define and articulate a succinct  and compelling reason for the reforms. According to the execution order  establishing AEMO, the Army needed an organization strategically  positioned to: support Army senior leadership in advertising, marketing,  and analysis; coordinate with the Army’s primary advertising agency; be  talent diverse; provide effective marketing analysis; and be able to  provide consistency of message and brand across the Army accessions  enterprise. The Assistant Secretary of the Army for Manpower and  Reserve Affairs also stated that AEMO is being assigned to the Deputy  Chief of Staff of the Army, G-1, in part, because of the continuity in  leadership that having a military officer lead the organization will provide.  Previously, AMRG was assigned to the Office of the Assistant Secretary  of the Army for Manpower and Reserve Affairs, whose leader is politically  appointed and had been vacant for two years until January 2019. In  addition, the Assistant Secretary stated that AEMO will be located in  Chicago to increase its coordination with the new advertising agency,  which is also headquartered in Chicago. The Assistant Secretary also told  us that the Army hoped to recruit civilian staff and to leverage the  marketing and advertising expertise at academic and other marketing and  advertising institutions in the region.

The Army has taken some initial steps to establish AEMO and its  operations. The Army has outlined a three-phased plan with specific tasks  and associated dates within each of these phases, which is consistent  with the key practice to establish implementation goals and a timeline to  build momentum and show progress for the reforms. Phase 1, which was  to be completed by August 1, 2019, prioritized tasks to initially establish  AEMO, such as publishing the Army directive establishing AEMO, issuing  the execution order outlining roles and responsibilities for the transition  from AMRG to AEMO, and identifying office space in Chicago. Phase 2,  which is to be completed between August 1, 2019, and February 1, 2020,  includes tasks to transition AEMO to being fully operational, such as  establishing new position descriptions and equipping the permanent office  space. Lastly, phase 3 identifies those tasks to be implemented after  February 1, 2020, when AEMO is fully operational and conducting daily  operations, such as updating roles and responsibilities in the Army’s  regulation for its marketing and advertising program and developing  policies to direct commission military personnel to the office.

The plan also identifies offices and officials who are accountable for  implementing specific tasks during the transition. The Assistant Secretary  of the Army for Manpower and Reserve Affairs told us that the Army  established an operational planning team to execute the transition from  AMRG to AEMO. The execution order also identifies key stakeholders,  including officials from TRADOC, Office of the Chief, Army Reserve, and  National Guard Bureau, who are to participate in weekly working group  meetings led by the Assistant Secretary of the Army for Manpower and  Reserve Affairs. Looking forward, by January 2020 the Army plans to  develop metrics to assess the effectiveness of the new AEMO  organization, including the purpose, expectations, and desired outcomes,  which is consistent with the key practice that calls for clear outcome- oriented goals and performance measures.

In addition, the Army has taken some initial steps to establish AEMO’s  workforce. As of September 2019, the Army had authorized 53  positions—31 military and 22 civilian—for AEMO and identified a  Brigadier General from the Army Reserve with marketing and advertising  experience as its leader. Senior Army leadership stated that they  expected to fill almost all of the positions with new military and civilian  personnel, in part, because the civilian position classifications in AMRG  do not generally align with those in AEMO. As of June 2019, the Assistant  Secretary of the Army for Manpower and Reserve Affairs told us that the  Army was identifying Active Duty, Reserve, and National Guard officers  with marketing and advertising education or experience to fill the military  positions. In addition, the Assistant Secretary stated that they were  working with OPM to develop position descriptions for the AEMO civilian  personnel and to identify the skills and expertise needed within AEMO to  fulfill its mission, consistent with the key practice to determine if the  agency will have the needed resources and capacity, including the skills  and competencies, in place for the reorganization. The Army has also  established a working group led by the U.S. Army Office of Economic and  Manpower Analysis at West Point to develop a new marketing career  path that senior Army leadership stated is intended to create a pool of  military personnel who could serve in AEMO and other Army accessions  organizations in the future.

As the Army carries out its steps to fully establish AEMO and reorganize  the marketing and advertising program, it will continue to be important for  the Army to consider and use the key practices for agency reform efforts  to guide the transition. Doing so will help ensure the success of the new  marketing and advertising organization.

We provided a draft of this report to DOD for review and comment. The  Army provided technical comments, which we incorporated as  appropriate.

We are sending copies of this report to the appropriate congressional  committees, the Secretary of Defense, and the Secretary of the Army. In  addition, the report is available at no charge on the GAO website at  http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-2775 or fielde1@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who made key contributions to this report  are listed in appendix III.

Appendix I: Army Actions to Improve Its Marketing and Advertising Program as Stated in the Army’s Report to Congress

This appendix summarizes an Army report to Congress on actions taken  to improve its marketing and advertising program. Section 599 of the  John S. McCain National Defense Authorization Act for Fiscal Year 2019  required the Army to submit a report to Congress that addressed several  elements, such as the mitigation and oversight measures implemented to  assure improved program return and contract management, and the  establishment of a review process to regularly evaluate the effectiveness  and efficiency of marketing efforts. The Army submitted the report on  February 5, 2019. Table 4 identifies the required elements of the report  and the actions that the Army has reported taking to address these  elements.

Appendix II: Army Marketing and Research Group’s Reported Actions to Address Army Audit Agency’s Recommendations

This appendix provides a summary of actions that the Army has reported  taking to address the recommendations in the April 2018 U.S. Army Audit  Agency (AAA) reports about the Army’s marketing and advertising  program. One report focused on contract oversight, and the other report  focused on what AAA termed “return on investment.” Each report  contained seven recommendations, with which the Army Marketing and  Research Group (AMRG) concurred. AMRG and AAA officials stated that  they have communicated about AMRG’s actions to implement the  recommendations and that AAA has provided feedback, as appropriate,  on actions taken. AAA officials stated that AAA may conduct a follow-up  audit in fiscal year 2020 to determine if the actions have led to  improvements in the marketing and advertising program. Table 5  summarizes the recommendations from the AAA report on contract  oversight, the actions AMRG has taken to implement them, and the  status—as of September 2019—of their implementation as reported by  AAA.

Table 6 identifies the recommendations from the AAA report on return on  investment, the actions AMRG has taken to implement them, and the  status—as of September 2019—of their implementation as reported by  AAA.

Appendix III: GAO Contact and Staff Acknowledgments

In addition to the contact named above, Margaret Best (Assistant  Director), Kendall Banks, Timothy J. DiNapoli, Jacob Fender, Alexandra  Gonzalez, Amie Lesser, Kristen Kociolek, Steven Lozano, Jonathan  Meyer, Eve Nealon, Julia Kennon, Carol Petersen, Richard Powelson,  Jerome Sandau, Jared Sippel, and Andrew Stavisky made key  contributions to this report.

The Army requested nearly $335 million for fiscal year 2020 for marketing and advertising activities intended to help recruitment.

In two reports in 2018, the Army Audit Agency found that the Army’s national-level marketing and advertising organization did not fully 1) evaluate the performance of the advertising agency it had hired, or 2) track the effectiveness of its marketing and advertising activities.

We found that the Army is taking or has taken steps to address these issues, including recently reorganizing its entire marketing and advertising program.