Workforce Automation: Better Data Needed to Assess and Plan for Effects of Advanced Technologies on Jobs

Advanced technologies—including artificial intelligence and robotics—are continually changing and emerging. While robots have existed for decades, modern robots may be equipped with learning capabilities that enable them to perform an expansive array of tasks. Advanced technologies are likely to affect the U.S. workforce by enabling firms to automate certain work tasks. Questions exist about how prepared federal agencies are to monitor workforce changes, promote economic growth, and support workers who may be negatively affected by automation.

GAO was asked to examine workforce issues related to the adoption of advanced technologies. This report examines (1) what is known about how the adoption of advanced technologies affects the U.S. workforce ; (2) federal efforts to track these effects; (3) considerations that led selected firms to adopt advanced technologies and the risks they faced; and (4) ways technology adoption has affected the workforce at selected firms.

GAO identified available federal workforce data, analyzed the extent to which those data could identify and measure workforce effects due to advanced technologies, reviewed selected research, and analyzed federal data on occupations susceptible to automation. GAO used data from the American Community Survey (2010-2016), the Current Population Survey's Displaced Worker Supplement (2016), and the Occupational Employment Statistics (2017).

GAO met with 16 firms that are using advanced technologies in their operations and seven firms that develop advanced technologies, and interviewed managers and workers, and observed firms' use of technologies. The selected firms varied in size, industry sector, types of technologies used, and geographic location. Findings from discussions with the fims are not generalizable, but provide illustrative examples about the adoption of advanced technologies. GAO interviewed officials from federal agencies, including Commerce and DOL, academic researchers, economists, labor union officials, industry association officials, officials from state economic development associations, and other knowledgeable individuals. GAO also reviewed relevant academic work.

Although existing federal data provide useful information on the U.S. workforce, they do not identify the causes of shifts in employment. As a result, it is difficult to determine whether changes are due to firms adopting advanced technologies, such as artificial intelligence and robots (see photo), or other unrelated factors. In lieu of such data, GAO analyzed employment trends and characteristics of jobs that selected researchers identified as susceptible to automation, and found that:

industries with a greater proportion of jobs susceptible to automation were more likely to have experienced growth in tech jobs (i.e., computing, engineering, and mathematics) from 2010 to 2016—possibly an indicator of industries preparing to adopt advanced technologies;

occupations susceptible to automation and industries with a greater share of these jobs did not experience meaningfully higher job loss rates in this period, though it could be too soon to observe these effects; and

certain groups, such as workers with no college education and Hispanic workers, tended to hold jobs susceptible to automation in 2016, and thus could be disproportionately affected by changes if they occur.

The Department of Labor (DOL) has a role in tracking changes in the U.S. workforce, but the data it collects related to the workforce effects of advanced technologies are limited. DOL's Bureau of Labor Statistics (BLS) identifies occupations projected to experience staffing pattern changes and the most significant causes, such as use of robotics, but its efforts are not designed to capture all instances of changes due to advanced technologies. DOL's Occupational Information Network program also collects data on tasks and technologies in occupations, such as robotics, but it was not designed to track changes over time. According to BLS, these efforts and other data they collect provide some, but not all, of the information required to identify and systematically track the impact of automation on the workforce. Without comprehensive data that link technological changes to shifts in the workforce, DOL lacks a valuable tool for ensuring that programs it funds to support workers are aligned with local labor market realities, and employers and job seekers need to rely on other sources of information to decide what training to offer or seek.

The Department of Commerce's Census Bureau (Census) has started tracking technology adoption and resulting workforce effects in the new Annual Business Survey, which was administered for the first time in June 2018 with significant support from the National Science Foundation. This first survey asked firms about their use of advanced technologies and initial results will be available in late 2019. When the survey is next administered in summer 2019, Census plans to ask additional questions about firms' motivations for adopting technologies and effects the technologies might have on workers. This survey could provide information about the prevalence of technology adoption and workforce changes (e.g., declines in production workers or increases in supervisory workers), but it is not intended to provide information on the magnitude of workforce changes. Also, it remains unclear what limitations, if any, the survey data may have.

According to officials from the 16 firms GAO interviewed, cost savings and other considerations led them to adopt advanced technologies, despite facing certain risks with the new technologies. Officials from these firms typically identified cost savings and improving job or product quality as primary motivations for adopting advanced technologies. For example, an automotive parts manufacturer said the firm adopted robots to reduce costs by using fewer workers. A door manufacturer said the firm installed two robots to lift heavy doors onto a paint line to reduce the number of worker injuries. A rubber stamp manufacturer said acquiring a robot (pictured above) allowed it to purchase and process raw materials instead of buying precut materials. Firm officials also identified risks related to adopting advanced technologies that could affect their return on investment, such as risks related to the reliability of technology and working with new tech developers.

Among the firms GAO met with, officials described various ways technology adoption has affected their workforces. On one hand, officials at many firms said they needed fewer workers in certain positions after adopting technologies. The firms generally redeployed workers to other tasks, and in some cases, reduced the size of their workforces, typically through attrition. For example, a medical center GAO visited adopted autonomous mobile robots to transport linens and waste, among other things, which officials said eliminated 17 positions and shifted workers to other positions. On the other hand, officials at some firms said advanced technologies helped them increase competitiveness and add positions. An appliance manufacturer used advanced technologies to produce more of its own parts instead of relying on suppliers and, as a result, increased the number of production jobs, according to officials. Firm officials also noted that workers' tasks and skills have been changing due to advanced technologies (see figure). Workers who can adapt to new roles may experience positive effects, such as work that is safer, while those who cannot adapt may be negatively affected.

GAO recommends that DOL develop ways to use existing or new data collection efforts to identify and systematically track the workforce effects of advanced technologies. DOL agreed with GAO's recommendation, and plans to identify and recommend data collection options to fill gaps in existing information about how the workplace is affected by new technologies, automation, and artificial intelligence. DOL also stated that it will continue coordinating with the Census Bureau on research activities in this area.

Advanced technologies and artificial intelligence, such as robotics,  machine learning, and machine vision, are likely to have significant  effects on the U.S. workforce. As firms adopt advanced technologies, the  automation of work tasks may change employment and productivity levels  and the skills needed in the workforce. Questions exist about how well- equipped federal agencies are to monitor workforce changes in the  coming years and respond in ways that encourage economic growth  while also supporting workers who may be negatively affected.

You asked us to study the effects of advanced technologies on the U.S.  workforce. This report examines 1) what is known about how the adoption  of advanced technologies affects the U.S. workforce; 2) selected federal  agency efforts to track and monitor the adoption and workforce effects of  advanced technologies; 3) considerations that led selected firms to adopt  advanced technologies and the risks they faced; and 4) ways technology  adoption has affected the workforce at selected firms.

We use “advanced technologies” as a broad term to describe  technological drivers of workforce changes, including but not limited to  those identified in a recent study by the National Academies of Sciences,  Engineering, and Medicine (National Academies): artificial intelligence;  machine learning; robotics; autonomous transport; 3D printing; advanced  manufacturing; advanced materials; computing power; and internet and  cloud technology. The technologies we observed at selected firms could  generally be categorized as applications of robotics, machine learning  (e.g., machine vision or autonomous navigation), or both. However, not all  technologies that may affect the U.S. workforce in the future—through automation or in other substantial ways—fall into these categories. Our  use of the broad term “advanced technologies” leaves open the possibility  that new technologies and other areas of focus are likely to emerge.

To analyze what is known about how the adoption of advanced  technologies affects the U.S. workforce, we explored the extent to which  available federal data could identify and measure these effects. After  considering the limitations of available data to link employment trends to  technology adoption in a comprehensive way, we used a study by  researchers Frey and Osborne that identified occupations susceptible to  automation based on the tasks associated with them. We analyzed these  occupations to glean insight about the workforce effects of advanced  technology adoption. While different studies attempt to predict what jobs  might be automated in the future, we used this study because it is widely  cited and its results are structured in a way that allowed us to identify a  broadly inclusive group of occupations susceptible to automation. The  results of our analyses could be affected by using other studies to the  extent they identify different occupations as susceptible to automation.  We analyzed the occupations Frey and Osborne identified as susceptible  to automation using employment data from the Census Bureau (Census)  and the Bureau of Labor Statistics (BLS). Specifically, we used data from  the American Community Survey (ACS), 2010-2016; the Current  Population Survey’s (CPS) Displaced Worker Supplement, 2016; and the  Occupational Employment Statistics (OES) survey, 2017. We analyzed  whether the concentration of these occupations in industries is correlated  with growth in tech jobs (i.e., jobs in the fields of computing, engineering,  and mathematics) or employment declines in those industries, whether  job displacements are more common in these occupations than in others,  the characteristics of workers who hold jobs in these occupations, and the  geographic concentration of jobs in these occupations. For more detail on  our data analysis methods, see appendix I.

We also reviewed examples of recent and ongoing studies that attempt to  measure workforce effects directly attributable to technology adoption.  We identified studies through interviews with knowledgeable individuals  and from those included in a recent review of the state of empirical work.

Our review of studies was not meant to be comprehensive of the research  in this area.

To identify selected federal agencies’ current and planned efforts to  collect data on and monitor the prevalence and effects of advanced  technologies in the economy, we met with the Departments of Labor  (DOL) and Commerce (Commerce) as the principal federal agencies  responsible for collecting data on the U.S. economy and workforce; the  White House Office of Science and Technology Policy (OSTP), which  leads interagency science and technology policy-coordination efforts  across federal agencies; and the National Science Foundation (NSF),  which was involved in the development of the Annual Business Survey.  We interviewed officials and reviewed data and information collected by  these agencies. We also reviewed Commerce’s new Annual Business  Survey questionnaire to consider the potential uses of data being  collected by the survey, and analyzed data from DOL’s Employment  Projections program and Occupational Information Network (O*NET)  database to identify information related to the adoption and workforce  effects of advanced technologies.

To understand the adoption of advanced technologies by firms and the  resulting workforce effects, we met with officials representing 16 firms that  are using advanced technologies and a systems integrator who spoke for  several of his customer firms. To identify firms to meet with, we consulted  a variety of sources, such as researchers, technology developer firms,  state economic development associations, and our own research. We  limited our focus to firms that had adopted advanced technologies and  had experienced workforce effects. Our findings from our discussions with  the selected firms are not generalizable, but do provide illustrative  examples of how various advanced technologies are being used and how  workers have been affected. We selected firms to provide a range of size,  industry sector, types of advanced technologies used, and geographic  location. For example, of the 16 firms we met with, 10 are manufacturers  and 6 are non-manufacturing firms of various types (e.g., a university- affiliated medical center and a warehouse for a regional grocery store  chain). When possible, we met with multiple managers and workers, and  observed advanced technologies in operation. Topics of discussion with  representatives of the firms included motivations for adopting advanced  technologies, the integration process, and any resulting workforce effects,  such as positions lost and gained as well as changes to workers’ tasks.

To obtain varying perspectives to supplement these profiles of firms that  use advanced technologies, we also interviewed officials at seven firms  that develop advanced technologies (hereafter referred to as developers),  two robotics integrator firms that assist client firms with adopting  advanced technologies, three industry-based organizations, two unions  representing manufacturing workers, and two worker training centers.

For all of the datasets used in our study, we reviewed documentation,  interviewed or obtained information from officials responsible for the data,  and tested the data for inaccuracies. We determined that the data were  sufficiently reliable for the purposes of this report. In addition, we  reviewed relevant federal laws and regulations related to all of the  objectives of this study. See appendix I for more detailed information  about our scope and methodology.

We conducted this performance audit from October 2017 to March 2019  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

Automation, Artificial Intelligence, and Advanced Technologies

Throughout history, new technologies have transformed societies. Many  technological advances, ranging from the steam engine to electricity and  personal computers, have enhanced productivity and improved societal  standards of living. At the same time, many technological advancements  have led to increases in automation—modifying processes to become  more automatic by reducing human involvement—and corresponding  changes in the workforce. For example, researchers have noted that  automation has replaced tasks performed by workers and also increased  production, creating a greater demand for other types of workers.

Although automation has historically been a labor disrupter in  manufacturing and physical work, various researchers have observed that  recent progress in fields such as artificial intelligence (AI) and robotics are  enabling machines to perform cognitive tasks currently performed by  humans. Artificial intelligence refers to machines and computers that  attempt to mimic various aspects of human intelligence, as we have  reported. The field of AI can be traced back to the 1950s. Early AI often  consisted of expert systems programmed by humans to perform  predefined tasks. This form of AI resulted in some degree of productivity  gains and remains an active area of development. However, numerous  factors, primarily the trends underlying big data (i.e., increased data  availability, storage, and processing power), have contributed to rapid  innovation and accomplishments in AI in recent years. Present-day AI  innovation centers more on machine learning, including deep neural  network architectures, in which systems are trained against observational  or simulated outcomes—applications include language translation and  machine vision (i.e., systems that use cameras, radar, or lasers to  observe their surroundings or recognize content). Industrial robots and  robotic machinery are often more comparable to expert systems that are  programmed to perform predefined tasks, but they can also incorporate  machine learning, such as having machine vision capabilities (e.g., object  recognition). Below are some examples of expert system and machine  learning applications of artificial intelligence.

Examples of expert system applications of AI:  software programs that prepare tax filings or schedule logistics; and  industrial robots that perform predefined or routine tasks, such as  lifting, placing, and welding pieces of metal together.

Examples of machine learning applications of AI:  software that uses a training dataset to “learn” how to read information  from a form filled out by a person;  collaborative robots that can sense when they touch a physical  obstruction and shut down to safely work alongside humans;  industrial robots with machine vision incorporated to identify and pick  up specific parts from a collection of randomly strewn pieces; and automated guided vehicles that transport materials around a  production plant and use cameras and radar to navigate  independently and re-route around obstacles.

Advanced technologies, including AI and other technological drivers of  workforce changes, are continually progressing and new developments  emerge regularly. For example, automated vehicles have varying levels of  autonomy. Similarly, while robots have existed for decades, today’s  generation of robots may be equipped with machine vision and learning  capabilities that enable them to perform a more expansive array of tasks.  How, when, or whether technologies progress from development to  commercialization (i.e., readiness for adoption), and how, when, or  whether firms adopt the technologies is generally dependent on context- specific considerations, which are difficult to predict. To better understand  these developments and how they affect the economy, the National  Academies report recommended developing three indexes (technology  progress index; AI progress index; and organizational change and  technology diffusion index) to measure technology progress and the  extent of adoption. The study suggested that indexes could be valuable  for identifying what fields are advancing rapidly and what benchmarks  might indicate the imminence of significant economic impact, as well as  tracking and predicting the types of human tasks that can be automated  and the impacts of technology adoption by industry. Stanford University’s  AI Index project is another initiative that aims to track, collate, and  visualize data related to artificial intelligence. The data collected by the AI  index measure, among other things, volume of AI activity (e.g., published  papers, course enrollment, AI-related startups, job openings) and  technical performance (e.g., object detection and speech recognition).  However, the potential uses and limitations of the data being compiled  are yet to be seen, as this initiative is still in its early stages.

Projected Workforce Effects of Advanced Technologies

While national employment data measure jobs and workers by occupation  and industry, the adoption of advanced technologies generally affects  specific work tasks, and can materialize in a variety of ways. As shown in  figure 1, industries are made up of various occupations, which in turn are  formed by a group of jobs. Underlying all, jobs are comprised of a  collection of varied work tasks.

By analyzing tasks within jobs or occupations to determine their  susceptibility to automation, a number of studies have developed models  to estimate the future workforce effects of advanced technology adoption.  The three example studies below each developed similar models, though  differences in methods and data sources produced varying conclusions  about the number of jobs that may be automated in the future.

In a 2016 article, researchers Frey and Osborne estimate that 47  percent of total U.S. employment is in occupations that are at high risk  of automation over the next decade or two (i.e., by 2030). For  example, the authors observe both that industrial robots will be able to  perform a wider scope of non-routine manual tasks and that a  substantial share of employment in services, sales, and construction  occupations exhibit high probabilities of automation.

A 2017 report by the McKinsey Global Institute estimates that 23  percent of total U.S. work hours could be automated by 2030 or as  high as 44 percent under other assumptions. The report predicts  that while labor demand will enable some re-employment of displaced  workers, up to one-third of the workforce may need to change  occupational categories.

In a 2016 paper, researchers Arntz, Gregory, and Zierahn estimate  that 9 percent of all U.S. workers hold jobs that are at high risk of  automation. The authors observe that susceptibility to automation is  lower for jobs that require cooperating or influencing others.

Studies by Autor and others also develop theoretical models exploring the  effects of automation. For example, they noted that while automation can  substitute for some tasks, it can also complement others. This can lead to  increasing value for tasks that require other attributes like creativity and  intuitive judgement. These models hypothesize that automation may have  a net positive effect on employment, or at least on employment in certain  sectors, which is consistent with historical employment trends. However,  researchers have also noted that machine learning may affect different  tasks than earlier forms of automation and may be less likely to automate  low-wage jobs—though low-wage workers may be affected in other  ways.

Workforce Effects of Advanced Technologies in Broader Context

Although the models discussed above represent ways of identifying jobs  that may be affected by the adoption of advanced technologies, they do  not provide a model for tracking the current or to-date workforce effects of  technology adoption. As the recent National Academies report states,  “making forecasts about social phenomena is perilous…  doing so  with respect to the fast-changing and dynamic area of technology is even  more challenging.” According to a different project by some of these  same experts, several factors unrelated to whether a task or job could be  automated contribute to these challenges. For example, technologies  may substitute for human labor in some tasks, but: may also complement human labor in other tasks—increasing the  demand for, or value of, human labor (e.g., the automation of  calculation tasks leading to increased demand for human  programmers); prices and demand for products may counteract this human labor  substitution (e.g., technology reducing the price of air travel, and thus  leading to increased demand for flights, and thus increased  employment in the aviation industry); and  firms may redesign operations in response to the substitution in ways  that lead to employment increases or decreases that are greater than  the direct substitution.

As discussed in the National Academies report and elsewhere,  researchers have tried to disentangle workforce effects in various ways,  such as analyzing productivity data to examine workforce trends in the  context of other economic factors, such as globalization.

As the National Academies report observes, “Predictions that new  technologies will make workers largely or almost entirely redundant are  as old as technological change itself…. However, predictions of  widespread, technologically induced unemployment have not come to  pass, at least so far.” Since recovering from the recession of 2007-2009,  the economy has recently experienced low unemployment rates—4.0  percent in January 2019—despite continued strides in advanced  technologies. However, other indicators have not recovered. For  example, the labor force participation rate—the percentage of the  population that is either employed or seeking work—declined significantly  through the recession and has generally remained at this lower level.  This may indicate that the post-recession decline in the unemployment  rate may over-represent the health of the labor market, according to BLS.  Advanced technologies and automation may also affect workers in other  ways, beyond potential changes in the workplace, such as by reducing  production costs and thus lowering the prices of consumer goods.

No Comprehensive Data Exist to Link Employment Trends to Advanced Technology Adoption, but Analyses Suggest Relationships

There are currently no comprehensive data on firms’ adoption and use of  advanced technologies. As a result, researchers have difficulty  determining whether changes in the U.S. workforce observed in existing  employment data are related to advanced technologies. The National  Academies report states that federal household and employer surveys,  such as the CPS, ACS, and OES, provide useful information about  changes to the occupational mix of the U.S. workforce over time.  However, these data cannot identify the causes of employment shifts. For  example, these data do not identify whether an employment decline in  one occupation is due to jobs being replaced as a result of automation, or  to other factors unrelated to automation. Other federal data, such as the  Job Openings and Labor Turnover Survey, provide useful information on  employment turnover and opportunities. However, although these data  are available by industry sector and firm size, the data do not capture  reasons for layoffs and discharges, and thus cannot be linked to  advanced technologies.

Employment Trends and Characteristics of Workers in Jobs Susceptible to Automation

In the absence of comprehensive data that definitively link employment  trends to technology adoption, we analyzed occupations that researchers  Frey and Osborne identified as being susceptible to automation (see  sidebar) to determine whether changes due to advanced technologies are  appearing in employment data. By exploring concentrations of these  occupations in industries, job displacements in these occupations, and   the characteristics of workers in these occupations, we found minor  indications that advanced technologies are changing the workforce and  could affect some worker populations. However, the conclusions that can  be drawn from these analyses are limited by the unpredictability of when,  if, or how automation materializes—e.g., whether worker positions are  eliminated or shifted to other non-automated tasks.

Industries with higher concentrations of jobs susceptible to automation  were more likely than others to have experienced significant growth in  their concentration of tech jobs from 2010 to 2016, according to our  analysis of employment data from the American Community Survey. For  example, as shown in figure 2, the plastics product manufacturing  industry has a relatively high concentration of jobs susceptible to  automation. Many of these jobs are in production occupations. From 2010  through 2016, this industry experienced about 11 percent annual growth  in tech jobs (i.e., jobs in the fields of computing, engineering, and  mathematics). More than half of this growth was the result of increases in  industrial engineers, engineering technicians, and miscellaneous  engineers. As we observed at some firms we visited, some of these  engineers may have been hired to program or maintain newly installed  robots. However, the data do not provide this level of information about  job tasks. Similar dynamics could also be occurring in other industries.  Across all 69 industries that had statistically significant changes in the  concentration of tech jobs, we found a positive, though weak, correlation  with the concentration of jobs susceptible to automation (see fig. 2).  This suggests that growth in tech jobs may be an indicator of industries’  preparation for, or adoption of advanced technologies. However, given  the complex causes of employment changes, there could be other  reasons for tech job growth in these industries that are unrelated to firms’  adoption of advanced technologies.

The growth in tech jobs in certain industries suggests firms in these  industries may be using more advanced technologies, which could also  signal that jobs susceptible to automation are being replaced. However,  our analysis of ACS data showed no correlation between an industry  having a higher concentration of jobs susceptible to automation and  employment changes in that industry (i.e., total employment increases or  decreases). We also found no meaningful differences in job losses,  according to our analysis of employment data from the Current Population  Survey’s Displaced Worker Supplement. Specifically, the relative rate at  which workers in occupations susceptible to automation lost a job  because their position or shift was abolished or there was insufficient  work for them to do was not meaningfully different than workers in other  occupations. There could be a number of reasons we did not find a  relationship between susceptibility to automation and employment  changes in both of these analyses, including: a relationship does not exist;  such a relationship is too complex to measure in this way (e.g.,  automation may lead to decreases in employment in some industries,  while also leading to increases in employment in other industries due  to improved competitiveness, productivity, and profitability);  it is too soon to observe the employment effects of automation (e.g.,  growth in tech jobs in an industry may be a leading indicator of  employment disruption); or our analysis covered a period of overall economic growth, which could  obscure or overwhelm other employment trends.

Existing data cannot predict with certainty when or if automation will  materialize in the workforce, as suggested by our analyses. However, the  tendency of particular worker groups to hold jobs susceptible to  automation suggests that some communities may be disproportionately  affected by changes if they occur. For example, according to our analysis  of 2016 ACS data, workers with lower levels of education are more likely  than those with higher levels to hold jobs in occupations that the Frey and  Osborne study identify as susceptible to automation. Specifically, 60.7  percent of workers with a high school degree or less hold these types of  jobs, as compared to 46.7 percent of workers with some college, 26.9  percent of workers with a bachelor’s degree, and 11.3 percent of workers  with a graduate degree. In addition, 54.1 percent of Hispanic workers hold  jobs in occupations susceptible to automation, as compared to 46.4  percent of Black workers, 40.0 percent of White workers, and 35.9  percent of Asian workers.

Certain geographic areas also rely more heavily than others on  occupations identified as susceptible to automation, according to OES  data. We identified areas where the proportion of jobs susceptible to  automation is at least 5 percentage points greater than the national  average (see fig. 3). These occupations are comprised of a diverse set  of jobs that may experience automation in different ways and at different  times, if at all. However, if employment disruptions are regionally  concentrated, groups of workers with similar skills in the same labor  market may need to adapt to changes simultaneously, which could strain  the availability of local job opportunities and support resources.

Workers in occupations that the Frey and Osborne study identify as  susceptible to automation earn less on average than other workers. For  example, the median hourly wage for workers in occupations susceptible  to automation is $14.26, compared to $22.06 for other workers, according  to our analysis of 2016 ACS data. After controlling for factors that may  affect wages, such as age, education, and industry, we found that  workers in jobs susceptible to automation earn about 17.2 percent less,  on average, than similar workers in other occupations. These results  show that, on average, workers in jobs susceptible to automation are  already in more vulnerable economic circumstances than other workers.  When or if changes brought on by automation materialize, these workers  may face additional hardships in adapting to changing workforce  demands.

Examples of Other Researchers’ Analyses that Attempt to Measure Workforce Effects Due to Advanced Technology Adoption

In the absence of comprehensive data, researchers have taken differing  approaches to exploring the relationships between technology adoption  and workforce trends. We identified some examples of recent and  ongoing work that attempt to measure workforce effects directly  attributable to technology adoption. These examples illustrate types of  data that may be useful for better understanding and measuring the use  of specific technologies (e.g., robot sales), the spread of technologies  generally (e.g., automation patents), and how specific work tasks are  changed by technology use (e.g., firm-level operations data).

Some researchers have used data on industrial robot sales collected by  the International Federation of Robotics (IFR) to approximate robotics  adoption worldwide and in the United States and to model its direct  effects on employment. Analysis by Furman and Seamans (2018)  shows that annual sales of industrial robots in the United States  increased substantially between 2010 and 2016. The analysis attributes  this growth to a combination of factors, including lower robot prices,  improved robot functionality, and greater awareness of the benefits of  robots. They also observe that the automotive sector was the largest  customer for industrial robot sales in the United States from 2004 through  2016, though robot sales to the consumer electronics sector grew the  most over that period.

Studies by Acemoglu and Restrepo (2017) and by Graetz and Michaels  (2017) both use IFR data through 2007 to model the workforce effects of  robot adoption in the United States, though their methods, results, and  conclusions differ.

Acemoglu and Restrepo estimate that each additional robot used in a  geographic area reduces employment by about six workers in that  area. They observe that their estimated employment effects are  greatest in manufacturing and other industries most exposed to  robots, in routine manual work-related occupations, and for workers  with less than a college education. They do not find corresponding  employment gains in any other occupation or education groups. They  also estimate that one more robot used per thousand workers reduces  wages by about 0.5 percent. They conclude by noting that, so far,  relatively few robots have been used in the U.S. economy and thus  the effect on jobs has been limited; however, they state that if robot  usage continues to grow as researchers expect, these effects could  be more substantial.

Graetz and Michaels estimate that increased robot use did not  significantly affect total hours worked across the 17 developed  countries in their analysis, but that work shifted from low-skilled  workers to middle-skilled and high-skilled workers. They also estimate  that increased robot use increases productivity and average wages.  While their analysis covers 17 developed countries, they note that  robot use in the United States was marginally lower than the average  across all countries. They also observe that while their results differ  from Acemoglu and Restrepo, it is possible that the effects of robot  usage are different in the United States than across the 17 countries  they analyze.

Other researchers have used U.S. patent data as an alternative way to  approximate the spread of advanced technologies and to examine the  resulting workforce effects. Mann and Püttman (2017) use machine  learning algorithms to identify patents related to automation technology.  They find that automation patents grew substantially from 1976 through  2014. After linking the patents to industries where they may be used,  they estimate that automation causes manufacturing employment to fall,  though it increases employment in the service sector, as well as overall  employment. They observe that their results depict a more positive  picture of the employment effects of new technology use than the studies  that used industrial robot sales data (discussed above). Lee Branstetter,  a researcher at Carnegie Mellon University, and his colleagues have a  similar ongoing project that uses a machine learning algorithm to identify  patents related to AI technologies. According to these researchers, their  initial results suggest a rapid rise in AI patents over the past decade and  also that AI patents are emerging in a variety of application areas. They  are also in the early stages of work linking AI patents to industries to  explore how new technology use affects the workforce.

Researchers have also identified how important micro-level data could be  for understanding the workforce effects of advanced technology adoption.

For example, reports by the National Academies and others highlight the  potential for firm-level information to augment traditional survey data to  enable analyses of the conditions under which advanced technologies  complement or substitute for workers, and what types of firms invest in  advanced technologies. Other researchers have emphasized the  importance of focusing on work tasks to analyze the effects of  technological change at workplaces. Erica Fuchs, a researcher at  Carnegie Mellon University, and her colleagues Christophe Combemale,  Katie Whitefoot, and Laurence Ales use a combined firm-level, task- based approach by collecting and analyzing production floor data from  four semiconductor firms with different levels of process automation and  parts consolidation. They map out detailed versions of firms’ production  processes and then use existing data and technical knowledge to  simulate each step to analyze the effects of technology changes. Their  preliminary results estimate that automation replaces some routine tasks,  leading to estimated declines in the number of production floor jobs  requiring medium skill levels. According to the authors, this firm-level,  task-based approach may be applicable to other manufacturing industries  and could provide insight on how the adoption of different technologies  may produce different labor outcomes. However, they note that the  approach requires detailed production process data, which may be  difficult to collect for many firms or industries.

Commerce and DOL Have Some Efforts to Track Adoption and Workforce Effects of Advanced Technologies Commerce Has Started Tracking Technology Adoption and Resulting Workforce Effects, but Data Will Not Be Available until Late 2019

Commerce’s Census Bureau has begun administering surveys with  questions that focus specifically on firms’ adoption of advanced  technologies and resulting workforce changes. According to Census, this  data collection is part of a long-standing, coordinated effort to measure  the impact of technology. In addition, consistent with Commerce’s  strategic plan, these represent new efforts to provide a timely, in-depth,  and accurate picture of the economy amidst the economic shifts and  technological advances of the 21st century. However, none of the  survey results will be available until late 2019 and later.

The new Annual Business Survey (ABS) is a joint effort by Commerce  and the National Science Foundation that has the potential to provide  insight on the spread of advanced technologies in the economy and could  be used to examine the workforce effects of technology adoption, but the  first ABS results are not expected until late 2019. Census administered  the 2017 ABS in June 2018 to collect information on firms’ use of  advanced technologies, such as automated guided vehicles, machine  learning, machine vision, and robotics, among other things (see example  in sidebar). The survey asks whether firms are testing a given technology  or using it for either less than 5 percent, 5 to 25 percent, or more than 25  percent of their production or service. Census officials said this question  should provide information about the extent of technology adoption  nationwide, including whether there are any industry concentrations of  advanced technologies.

Census plans to add questions on the workforce effects of advanced  technologies when it administers the 2018 ABS during July through  December 2019, pending final approval by the Office of Management and  Budget. Census plans to release these survey results in December  2020. Specifically, Census plans to include new questions that ask firms  about: (1) their use of advanced technologies such as AI, cloud  computing, robotics, and specialized software and equipment; (2) their  motivation for adopting and using artificial intelligence and advanced  technologies; (3) the impact these technologies might have on the  number and skill level of workers; and (4) the factors that could adversely  affect the adoption or production of these technologies. The new  questions also ask about changes in the number of production workers,  non-production workers, supervisors, and non-supervisors. These new  questions could be used to characterize the prevalence of workforce  changes in the economy caused by advanced technology adoption (e.g.,  declines in production workers, or increases in supervisory workers) and  whether this differs by industry sector. However, these planned questions  are not intended to provide information to quantify the magnitude of  workforce changes, in part to minimize respondent burden and potential  survey error, according to Census. In addition, until the ABS data are  available and evaluated, it remains unclear what limitations, if any, the  data may have.

Census also plans to expand other surveys to track the spread of  advanced technologies in the economy, including its Annual Survey of  Manufactures (ASM) and Annual Capital Expenditures Survey (ACES).

Census plans to administer the 2018 ASM in May 2019, pending final  approval by the Office of Management and Budget. The survey will  collect capital expenditures data for industrial robotics at  approximately 50,000 manufacturing plants, as well as the number of  industrial robots purchased by and in use at these plants. Census  officials stated these two measures might be useful in understanding  the impact that industrial robots could have on productivity as well as  the impact robots could have on the manufacturing labor force once  the survey results are available in the spring of 2020.

Census plans to administer the 2018 ACES during March through  May 2019 and to have the survey results available in February  2020.The survey will include questions on robotics expenditures,  similar to those in the 2018 ASM. However, the ACES collects  expenditure data from 50,000 employer firms across all non-farm  sectors of the economy—instead of just manufacturers—and will also  ask about firms’ use of both industrial and service robots.

Some Commerce offices also track issues related to the adoption and  workforce effects of advanced technologies on a limited or intermittent  basis. For example, National Institute of Standards and Technology  officials stated that the Hollings Manufacturing Extension Partnership  collects limited information about the number of jobs gained and retained  by small and medium businesses adopting new technologies. National  Telecommunications and Information Administration officials said they  monitor developments in AI on an intermittent basis and also direct a  project that examines new applications of small and large internet  devices.

DOL’s Current Efforts Provide Limited Information for Tracking the Workforce Effects of Advanced Technologies

DOL has a role in collecting data that track changes occurring in the U.S.  economy and workforce, including developing new ways to track  emerging economic trends, though as we previously discussed, currently  available federal data do not link shifts in the workforce to technological  changes. BLS is the principal federal statistical agency responsible for  measuring labor market activity. According to DOL’s strategic plan, BLS is  to support public and private decision-making and meet the needs of its  many stakeholders, including the general public, educational institutions,  and the public workforce system. This includes regularly identifying  structural shifts in the economy and developing new data products that  reflect economic changes. In addition, DOL’s Employment and Training  Administration (ETA) is to assist workers’ entry and reentry into in- demand industries and occupations. This assistance includes providing  job seekers with accurate labor market data and guidance about  opportunities, aligning training services to industry needs, and helping  connect businesses with properly skilled workers. Internal control  standards state that agencies should use quality information to identify,  analyze, and respond to significant changes, including external conditions  such as economic and technological changes that may affect an agency’s  ability to achieve its objectives. DOL collects workforce data through  various surveys, including the Current Population Survey’s Displaced  Worker Supplement, and produces other data products such as the  occupational employment projections and Occupational Information  Network database that include information related to advanced  technologies. However, these data are limited, and according to BLS,  provide some, but not all, of the information required to assess the impact  of automation on the workforce.

BLS’s Employment Projections program identifies and provides limited  information about occupations expected to experience declines in their  share of employment in an industry or group of industries as a result of  the adoption of advanced technologies. On a biennial basis, this program  analyzes changes in the economy to project how employment by  occupation may change over 10 years, including which occupations may  be affected by advanced technologies. Factors that can affect  occupational employment include but are not limited to technological  innovation; changes in business practices or production methods;  organizational restructuring of work; changes to the size of business  establishments; and offshore and domestic outsourcing, according to  BLS. As part of this program, BLS develops a table of occupations that  are projected to have direct employment changes due to some identified  reason. This table identifies projected staffing pattern changes and  BLS’s qualitative judgment of the most significant factor or factors  projected to affect the occupation. The table also indicates whether an  occupation’s share of employment is expected to change within a single  industry or within multiple or all industries. For example, the table includes  the following selected entries:

Librarians: Employment share is projected to decline in the  information services industry as internet-based research continues to  displace library-based research.

Stock clerks and order fillers: Employment share is projected to  decline in two industries (the warehousing and storage industry and  the grocery and merchant wholesalers industry) as firms increasingly  adopt automated storage-and-retrieval systems.

Aircraft structure and systems assemblers: Employment share is  projected to decline in all industries as collaborative robotics increase  efficiency, producing more output with the same amount of labor.

We identified 100 occupations in BLS’s table that are projected to  experience declines in their shares of employment in an industry or group  of industries as a result of the adoption of advanced technologies.  Similar to the examples above, reasons could be related to automation,  the increased use of robots or artificial intelligence, advances in machine  or software technologies, or other changes resulting from the adoption of  advanced technologies. As shown in figure 4, most of these occupations  are production occupations (40 of 100) or office and administrative  support occupations (30 of 100). BLS officials told us they do not currently  track groups of occupations projected to experience employment share  declines due to specific reasons, such as advanced technology adoption.  Officials also said they do not aggregate total projected employment  effects stemming from similar causes because they are unable to identify  ripple effects in all occupations—e.g., automation in one occupation  affecting employment in a different occupation.

Information contained in ETA’s Occupational Information Network  (O*NET) database includes, among other things, information about work  activities, tools and technologies used, and required skills associated with  over 1,000 occupations. According to ETA officials, the primary purpose  of O*NET is to assist job seekers in making employment decisions.  However, the O*NET database can be used to identify occupations that  use certain types of advanced technologies. For example, we identified  15 occupations in which workers monitor, install, develop, troubleshoot,  debug, or perform other tasks with robots as part of their daily work  activities and 63 occupations in which workers use robots as a tool or  technology in their daily work activities (see table 1). In addition, states,  federal officials including at BLS, and academic researchers use these  data to inform, among other things, worker support programs. DOL  officials told us they do not use O*NET data to analyze changes in  occupations over time, such as robots being used in additional  occupations, because the methodology is not currently structured to  capture these kinds of changes systematically. For example, data are  collected from a selection of occupations at varying frequencies, rather  than at the same time, which could make it challenging to track changes  in certain occupations over time.

Without comprehensive data linking employment shifts and technological  changes, policymakers and DOL may not be prepared to design and  implement programs that both encourage economic growth and provide  support for workers affected by changes. DOL-funded programs rely on  accurate information to guide job seekers to employment opportunities  and to help align training services with local industry needs. For example,  the O*NET database identifies high-growth, high-demand occupations for  job seekers based largely on BLS employment projections data. While  these employment projections provide valuable information, they are not  designed to identify the full extent of occupational shifts due to advanced  technology adoption. Similarly, other workforce surveys, such as the  Current Population Survey’s Displaced Worker Supplement and the Job  Openings and Labor Turnover Survey, do not collect information about  the causes of job losses and gains. This information could be a valuable  tool for designing programmatic or policy supports for workers. For  example, data on whether advanced technologies have resulted in worker  displacements, work hour reductions, or substantial adjustments to work  tasks could better position BLS to meet stakeholder needs.

Congress has expressed concern that there continues to be insufficient  data on the effects advanced technologies are having on the U.S.  workforce. On January 2, 2019, BLS reported to Congress that it plans  to work with a contractor during fiscal year 2019 to study the interaction  between labor and capital in the workplace and how it is affected by new  technologies; identify ways to supplement BLS data with additional  information on automation; and produce a report that recommends data  collection options to fill those gaps. In fiscal year 2020, BLS also plans to  identify pilot projects to test the feasibility of new data collection based on  the recommendations in its final report, resources permitting. However,  these plans are still in their early stages, according to BLS officials.

Commerce and DOL Face Challenges Tracking the Workforce Effects of Advanced Technologies

Officials at Commerce and DOL stated that collecting data on the  adoption and workforce effects of advanced technologies is challenging  because it is difficult to identify which new and emerging technologies to  track; employment trends generally occur at the occupation and industry  levels but the effects of advanced technologies typically occur at the task  or job level; and employment trends have a complex and diverse set of  causes. Specifically:  Identifying which new and emerging technologies to track.  Census officials said there is uncertainty about how an emerging  technology might affect the economy and thus whether it should be  tracked systematically. For example, self-service technology  appeared at grocery stores in 1916, other self-service technology  appeared at gas stations later, and more recently self-service  technologies are being adopted by some restaurants, according to  researchers. Periodically, Census has included questions in its firm  surveys about the use of these technologies. Past surveys asked  questions about the use of self-service at gas stations until the  technology became ubiquitous and was dropped from the survey. As  self-service technologies have expanded to other areas of the  economy such as restaurants, Census has again added questions  about self-service to recent surveys because information is lacking on  the growth of this phenomenon.

Trends and effects appear at different levels. BLS officials said  employment changes due to technology typically occur at the  individual task or job level and employment trend data are at the  industry and occupation levels. Officials also said that identifying  technology-related effects in occupations, such as changes related to  uses of machine learning algorithms, is difficult because some  workers within an occupation might be affected by the technology  while others might not. For example, some computer scientists and  engineers might be involved in the development or application of  machine learning algorithms while others are not.

Causes of trends are complex and diverse. BLS officials said that  employment trends’ complex and diverse causes make it difficult to  identify occupations that are changing because of advanced  technologies. Changes in one occupation may have ripple effects in  other occupations. Partly as a result of this complexity, BLS’s  Employment Projections program identifies examples of technology- impacted occupations, but it does not attempt to identify all instances  where technology impacts occupations nor does it attempt to quantify  an overall projected employment effect of advanced technologies.

White House Office of Science and Technology Policy Coordinates Policy and Research Activities Related to Advanced Technologies

The White House Office of Science and Technology Policy (OSTP) is  responsible for coordinating AI related policy across government  agencies and for overseeing the National Science and Technology  Council’s subcommittees and their ongoing activities. For example, the  Subcommittee on Machine Learning and Artificial Intelligence was  originally chartered in 2016 to monitor machine learning and artificial  intelligence and to watch for the arrival of important technology  milestones in the development of AI, among other things. OSTP officials  told us that the Subcommittee has been re-chartered, now receives  direction from OSTP’s Select Committee on Artificial Intelligence, and is  presently focused on federal resources related to AI research and  development.

Cost Savings and Other Considerations Motivated Selected Firms to Adopt Advanced Technologies, Despite Facing Risks Such As the Reliability of Technologies

Selected firms generally adopted advanced technologies through a  phased process of innovation and technology adoption (see fig. 5). We  met with officials representing 16 firms that are using advanced  technologies and a systems integrator who spoke for a number of his  customer firms. Many firm officials described the path to integrating  technology into operations as lengthy, complex, and iterative. For  example, some firms we visited have had to build and test different  mechanical “grippers” attached to robot arms to pick up and handle  particular objects; one firm had high school participants at a local training  center develop a gripper solution for one of the firm’s robots. Some of the  large firms we visited had their own internal teams that identified, tested,  and integrated advanced technologies. Other firms we visited used third- party integrator companies to help with incorporating technologies into  their operations. We spoke with firm officials about their motivations for  adopting advanced technologies, as well as challenges they faced  throughout the process, and they identified a number of similar issues.

Selected Firms Identified Cost Savings and Job Quality Among Key Motivations for Adopting Advanced Technologies

Most selected firms cited cost savings as a primary consideration for  adopting advanced technologies. Firm officials discussed cost-related  motivations in various forms, such as remaining competitive in a global  economy, increasing productivity (i.e., lower cost per unit), decreasing  labor costs, and saving on physical space.

Firms said they adopted advanced technologies as a way of reducing  operational costs—including labor costs—to increase competitiveness  and profitability. Some officials also specifically identified the pressure of  large low-cost competitors, both in the United States and globally, as a  major motivation to reduce costs and product prices.

Officials at a medium-sized door manufacturer told us that increased  use of advanced technologies, such as robots, enabled the firm to  increase efficiency, reduce labor costs, and re-focus its product line  on custom doors to survive the entry of manufacturers in China that  could sell mass-produced doors for lower prices.

The original motivation for adopting robots at a medium-sized  automotive parts manufacturer was a customer’s price demand that  the firm could not meet and still remain profitable, according to  officials. Integrating more robots enabled the firm to reduce production  costs by using fewer workers.

At a large manufacturing corporation of household and personal care  goods, officials told us the company had a goal of reducing its  workforce size by 1,500 full-time positions per year for 5 years (across  its subsidiaries), and specifically using robotic automation to  accomplish 40 percent of its reduction goal.

The constant pressure to keep costs low in the health care sector  motivated a university-affiliated medical center we visited to explore  adopting more advanced technologies, such as autonomous mobile  robots that could decrease expenses by reducing the number of  positions in some departments.

Firm officials also told us about other, non-labor-related cost savings  considerations that led to the adoption of advanced technologies.

Officials at a large automotive manufacturer told us they recently  upgraded a laser welding system to use fewer, more advanced robots  to save production line space—which is a valuable commodity in  manufacturing. They also pursued this change to increase overall  production capacity because the physical space they saved could be  used to install more robots for other production steps.

The integration of autonomous mobile robots to deliver prescription  drugs to patient wards at a university-affiliated medical center was  intended, in part, to save costs related to medicines that go missing  when delivered and processed manually, according to officials.

According to officials at selected firms, the desire to improve jobs led  firms to adopt advanced technologies. The firms wanted to automate  tasks that are dangerous, difficult, dull, or dirty in large part to improve  worker safety, and to optimize the value added by workers. For example:

Dangerous work: Two robots were installed to pick up doors  weighing between 90 and 300 pounds, and place them on a paint line  at a medium-sized door manufacturer we visited. Prior to the robots,  workers who performed this dangerous task experienced work related  injuries, and the firm paid large amounts of money in workers’  compensation claims, according to officials. Once the robots were  installed, the firm experienced a decrease in the number of worker  compensation claims.

Dull work: A small automotive parts manufacturer we visited installed  an industrial robot to perform a machine-to-machine transfer of a  heavy part. Prior to the robot, the firm had three workers performing  this task—even though the task only required two—because workers  would eventually quit due to the tedium of the job and new workers  would require time to be trained, according to officials.

Value-added work: Some officials told us they adopted advanced  technologies because they wanted to maximize human labor that  provided value to the firm and reduce labor that did not. Officials at a  warehouse for a regional grocery store chain and a university- affiliated medical center said they wanted to minimize time workers  spent traveling between tasks (as opposed to performing tasks).  Warehouse officials said their workers spend up to 60 percent of their  time traveling back and forth between shelves and products, which is  time that could be spent selecting and sorting items. Thus, at the time  of our visit, the warehouse was in the early stages of adopting  automated guided vehicles to eliminate the need for workers to travel  between points. Similarly, officials at a university-affiliated medical  center that adopted autonomous mobile robots to transport, among  other things, prescription drugs, said nurses and pharmacy  technicians used to walk back and forth between the patient ward and  the pharmacy to pick up and deliver these drugs, which diverted them  from performing other tasks. They said that the medical center wanted  them to have more time to provide valuable work, especially for  employees who are highly-paid.

Officials at many firms said that adopting advanced technologies can help  them deal with the challenges of recruiting and retaining skilled workers.  They explained that worker shortages and high turnover can result from  skill gaps in the local or national workforce, low unemployment, and  certain work being viewed as unappealing, among other reasons. For  example, officials at a warehouse for a regional grocery store chain we  visited told us they struggle with high worker turnover and the constant  need to hire new workers. In addition, low unemployment can make it  difficult to retain workers with the right skills to operate machinery  according to officials at a small automotive parts manufacturer. Similarly,  at the university-affiliated medical center, an official said that positions for  pharmacy and other types of medical technicians can be difficult to fill. By  using autonomous mobile robots to automate some tasks, the medical  center can streamline its operations to more efficiently use the  technicians it already has.

Recruitment in Manufacturing  Officials at some manufacturing firms we visited said they have had trouble attracting  new workers into the sector, and officials at two firms said that adopting advanced  technologies is one way they have sought to make manufacturing more attractive and to  appeal to more and younger workers. One younger worker at a small automotive parts  manufacturer talked about how appealing his workplace was due to the firm’s use of  advanced technologies, specifically robots. Officials at a large automotive manufacturer  viewed their tech development facility, which includes spaces to tinker with virtual reality,  augmented reality (i.e., technology that superimposes images on a user’s view of the  real world; for example, by wearing augmented reality glasses), and other emerging  technologies, as an asset to recruit young talent.

Improving product quality, expanding product offerings, and supply chain  reliability were primary motivations for adopting advanced technologies,  according to officials at some firms.

Product quality: Quality is paramount in the automotive industry,  where mistakes are costly and can have implications for a firm’s  reputation, according to officials at a medium-sized automotive parts  manufacturer we visited. For this reason, they decided to use robots  rather than workers for welding in order to standardize the processes,  reduce errors, and improve product consistency and quality. Officials  at a large automotive manufacturer similarly said that the firm has  pursued machine learning technologies to ensure fewer defects and  problems in vehicles. Engineers at the firm are developing a smart  watch for workers who connect wires that will provide feedback to  these workers if a proper connection is not made, based on the sound  of the connection. The firm is already using machine vision technology  that inspects vehicles as they pass through a section of the production  line to ensure the correct pieces have been used for each vehicle  model.

Expanding product offerings: At a medium-sized fruit processing  plant, an official said that integrating robots, an advanced conveyer  system, and machine vision inspection technologies, among other  advanced technologies, enabled the firm to begin producing  applesauce in a highly automated and safe way. Had manual  production been the only option, officials said they would not have  considered producing applesauce due, in part, to safety issues.

Supply chain reliability: One small manufacturer of rubber stamps  and embossing seals (hereafter referred to as a small stamp  manufacturer) used to rely on a single supplier for pre-cut materials,  which was not always reliable. The firm adopted a collaborative robot,  in part, so it could purchase raw materials directly and then have the  robot cut the materials as part of the production process (see fig. 6).

Selected Firms Cited Various Risks with Adopting Advanced Technologies, Such as the Reliability of Technology, and Working with New Tech Developers

In addition to the capital cost of advanced technologies, which some firms  told us can be substantial, firms face a number of risks that can affect  their return on investment, such as the reliability of technology and  working with new tech developers. While the firms we met with had  already adopted advanced technologies, officials had to consider and  overcome various risks during the adoption process. Some of these firms  decided against adopting other advanced technologies upon evaluating  these risks.

Being an early adopter of a technology is risky because the new  technology may not yet be sufficiently reliable for firms’ operations.  Officials at a large appliance manufacturer we visited showed us  technology that was supposed to use machine vision to autonomously  inspect the wire connections for clothes dryers. They told us that the  vision technology had been ineffective, so they took it off the production  line for engineers to continue working with it in the lab; they planned to  bring the technology back onto the line a few weeks after our visit.  Officials at this firm said that the vision technology was still relatively  immature, as it had a limited field of vision and yielded numerous false  readings. Similarly, a warehouse we visited that invested in automated  guided vehicles used them to move pallets for a short time, but then put  them into storage because these vehicles did not have mature enough  machine learning and vision capabilities for the firm’s purposes.  Eventually, officials from this warehouse began working closely with the  developer firm to improve the vehicle technology, which advanced  enough that it could be used. For instance, officials from the warehouse  suggested adding turn signals to the vehicles to alert nearby workers of  intended movements and improving the vehicles’ ability to travel over  spills without triggering the system’s sensors to shut down.

Firm Size Might Affect Risk Tolerance  An official at one small manufacturing firm stated that larger firms may be more willing to  be early adopters of technology, as they may be able to absorb the high risks of  experimenting with expensive technologies, while smaller firms tend to wait until a  technology has been optimized before deciding to adopt it. Accordingly, his firm only  purchases industrial robots from an established manufacturer, although it would like to  experiment with newer technologies in the future, such as augmented reality.  Officials at a large manufacturing firm told us they have purchased a number of  advanced technologies to experiment with, even though they do not know yet how the  technologies may ultimately be used in their production process. This firm also has  teams of technicians and engineers who can adapt the technology for operations. During  our visit, we met with engineers who demonstrated different potential applications of  technologies that are still being tested, including using virtual reality to test new part  design and augmented reality glasses to provide interactive training to workers.

Officials at some firms explained that installing advanced technologies at  times necessitated building manual redundancies into their operations  due to reliability concerns. Officials at a construction consulting company  and a municipal township that adopted a machine learning technology to  inspect roads said the technology would miscategorize road quality at  times, such as identifying tree branch shadows on the road as pavement  cracks. While working with the developer to improve the technology,  officials said they continued to conduct redundant manual inspections to  ensure they were making road repair decisions based on accurate  information. During our visit to a large appliance manufacturer, we saw  multiple collaborative robots that were not working properly. As a result,  workers were performing these tasks manually while the robots were  down; officials told us that each of the firm’s automated processes has  workers trained to perform the tasks in case a technology was not  working properly.

Technologies Viewed Differently by Firms  Some firms find a technology to be useful while others find little practical application for  that technology, as illustrated by the various opinions firm officials had about  collaborative robots.  Officials at one small manufacturer we visited said that a collaborative robot was well  suited for the firm’s production process and environment because, among other reasons:  (1) the firm produces small durable goods that require dexterity rather than speed, which  the collaborative robot could provide; (2) the collaborative robot would be safe around  workers and could be trained by non-technical staff, so the firm’s small workforce could  adapt to its use; and (3) the collaborative robot could fit in the firm’s limited floor space,  as it would not require a cage.  On the other hand, officials at other manufacturing firms we visited told us that  collaborative robots were less useful in their settings because they have significant  weight and speed limitations in order to be safe enough to operate outside of a cage,  limiting their usefulness for their firms.

Some firm officials told us it could be risky to work with tech developers  with limited experience. Officials at a large appliance manufacturer said  that newer developers may go out of business or be bought out by a  larger firm, which could render the technology acquired from them  obsolete (especially in terms of future servicing of parts and software  updates). The officials stated that emerging technologies, both hardware  and software, tend to not be standardized, so investing in a developer  likely means investing in a type of technology that may not be supported  by other developers if issues arise. We heard from some firms that they  purchased technology from developers who already had established  reputations and longevity. For example, a small manufacturer of durable  goods selected a robotics company because of the founder’s reputation  and track record, among other reasons.

Operational slowdowns: The time period between initial adoption and  optimization of a technology varies widely and can sometimes be a  lengthy and ongoing process, according to officials.

One small stamp manufacturer experienced a lengthy and iterative  implementation process for an off-the-shelf collaborative robot they  purchased. For example, they had to construct a customized  environment for the robot to function in, make parts by hand,  purchase a 3-D printer to develop tools for the robot, and build  additional parts to take care of increased byproducts like sawdust.

Officials at a large automotive manufacturer told us that new  technology, such as machine vision technology used for automated  inspections, is often integrated on the weekends or during off-shifts.  Then, on the first day of production after the new technology is  integrated, the production line starts slowly and speeds up as worker  comfort and experience increases.

Outside of manufacturing, a consultant that helps facilitate the  adoption of advanced technologies at firms said that firms’ existing, or  legacy, computer infrastructure can be a barrier to integrating  machine learning technology, increasing complexity and causing an  extended implementation process as his firm integrates the new  technology platform with the legacy infrastructure.

Worker concerns: Officials at some manufacturing firms said they have  encountered worker concerns with advanced technologies, and have  employed various tactics to mitigate this, such as introducing workers to  the technology in offsite demonstrations and involving them during the  decision-making and planning before the technology was integrated. In  one case, workers were able to ask questions about a collaborative robot  as it was being installed and were provided with orientation training. The  robot was then phased into operations—used initially for short periods of  time so workers would become accustomed to its physical presence and  proximity to their workstations.

Deciding Not to Adopt Advanced Technologies  Officials at the firms we visited identified instances in which they chose not to adopt  certain advanced technologies, or not to use advanced technologies that were working  well in other processes. Reasons we heard included:  a product line had too much variation to benefit from advanced technologies (i.e.,  that some advanced technologies work better for standardized products and  processes);  a certain manufacturing process was too low-volume to invest time and resources  into automation; and  human dexterity is difficult to replicate.

Officials from a large appliance manufacturer showed us an instance where using  automation would not make sense. We observed a worker performing a simple, single  task: grabbing a metal heat shield and plastic dishwasher spinner from separate bins  and clipping one on to the other. Because of the shape of the pieces and because they  were lying unorganized in boxes, the task requires human dexterity, making the process  difficult to automate, according to officials.

Adopting Advanced Technologies Has Had Varied Effects on the Workforces of Selected Firms, Including Declines in Some Types of Work and Gains in Others Officials Said Advanced Technologies Have Replaced Positions at Some Selected Firms, and Most Firms Relied on Redeployment of Workers and Attrition Rather than Direct Layoffs

Officials at many of the firms we visited said they needed fewer workers  in certain positions after adopting advanced technologies to perform tasks  previously done by workers. Officials at these firms generally told us  they adjusted by redeploying workers to other responsibilities and, in  certain instances, reducing the firm’s workforce size through attrition. We  also heard examples of direct layoffs due to the adoption of technologies.  There may also be other types of adjustments firms can make that we did  not observe or discuss with these officials. The complexity of these  workforce adjustments makes it difficult to determine or measure the  effects of technology adoption on workers. For example, although  workers may not have lost their jobs due to an adopted technology taking  over specified work tasks—either because of redeployments or attrition— fewer job opportunities might be available in the future for workers with  similar skills. In addition, the iterative and sometimes lengthy process of  incorporating advanced technologies can delay workforce effects. Thus,  the absence of short-term effects of technology adoption does not  necessarily preclude long-term implications, such as reductions or slower  growth rates in workforce size over time (see text box below). As  discussed in the prior section, one reason firm officials are motivated to  use advanced technologies is to decrease labor costs.

Slower Workforce Growth than Revenue Growth  An official from a small automotive parts manufacturer told us that advanced  technologies and automation resulted in revenue increasing by more than 400 percent  over the last 12 years while the workforce increased about 15 percent. Production  workers now make up a smaller percentage of the overall firm workforce than prior to  automation, and sales and support staff now make up a greater percentage. The firm  official described this change as an increase in higher-skilled jobs and a decrease in  lower-skilled jobs.  Similarly, according to firm officials at a different medium-sized automotive parts  manufacturer, revenue has grown six times in the past 15 years while the workforce has  grown four times, largely as a result of adopting robotics technology.

Redeployments without job loss: When advanced technologies  replaced positions, firms we visited often shifted, or redeployed, workers  to different responsibilities. For example, officials at a medium-sized  automotive parts manufacturer we visited told us they had nine workers  who smoothed sharp edges and removed burrs on hydraulic cylinders  prior to installing two robots to perform these tasks. Now, with the robots  in these positions, three workers load the robots and then inspect and de- burr any parts of the cylinders the robots missed. The other six workers  were redeployed to other tasks, according to a firm official. At a large  appliance manufacturer we visited, officials told us that two workers used  to move large parts from one line to another line to be painted. Now, as  we observed, a collaborative robot performs this function alone; a worker  monitors the operation to ensure it is running smoothly, and the original  workers were moved to different tasks on the production line, according to  officials. Although the size of these firms’ workforces did not decrease as  a result of the technology adoption, the numbers of certain positions were  adjusted—for example, production positions decreased while monitoring  positions increased. Differences in skills required for these positions may  also affect the ability of current workers to transition and could have  implications for individual workers even though the number of jobs at the  firm does not change. These sorts of changes may or may not appear in  firms’ reported employment data, depending on whether redeployed  workers change occupations or what other workforce changes may be  occurring simultaneously (e.g., if other production workers are hired for  reasons unrelated to the technology adoption).

Redeployments with job loss through attrition: Officials at some of the  selected firms that redeployed workers said they also reduced their  overall workforce size through attrition, as a result of adopting advanced  technologies.

Autonomous mobile robots independently transported biohazardous  waste, linens, meals, and prescription drugs throughout the university- affiliated medical center we visited. Officials told us they eliminated 17  positions after they deployed the robots. No workers were laid off;  instead, they relied on high staff turnover rates and moved workers to  vacant positions elsewhere.

At a medium-sized fruit processing plant, firm officials told us they  replaced 150 to 200 jobs with various advanced technologies over the  past 3 to 4 years. However, they relied on attrition rather than layoffs.  For example, the plant adopted a robot to pack food into boxes. Prior  to using the robot, officials told us there were 26 workers per shift  performing this job; as of our visit, there were 13 workers per shift.

A medium-sized door manufacturer reduced its workforce from 650  employees to less than 500 over approximately the last 20 years due  to, among other things, their adoption of robots, according to firm  officials. For example, we observed industrial robots that load steel  sheets into a cutting machine, reading a barcode on each sheet that  tells them what size sheet is being lifted and how it should be placed  in the cutting machine. This process only requires a single worker to  monitor the robots during each of two shifts, where previously three  workers per shift were on this production step (i.e., a change from six  to two workers total).

How quickly workforce reductions materialize for firms using attrition can  vary greatly. We visited firms with low employee turnover rates and firms  with high turnover rates. High worker turnover rates allowed some firms to  more quickly adjust their workforces when deploying advanced  technologies and may be a reason we were told about job loss through  attrition rather than layoffs at these firms.

Job loss through layoffs: An official from a systems integrator firm  (“integrator”) provided examples of significant layoffs as a direct result of  advanced technologies. This integrator provides machine learning  technology and other similar products to automate office and  administrative processes, among other things. One of the integrator’s  customers—a U.S. automotive parts firm facing competition from online  retailers—adopted machine learning technology to take over its accounts  payable and distribution system. As a result, according to the integrator’s  official, this firm reduced the number of employees in one of its U.S.  offices from 500 to 200. Another of this same integrator’s customers—a  firm that sells telecommunication circuits—adopted machine learning  technology to automate product returns processing. As a result, the firm  experienced a 30 percent reduction in customer care calls, and replaced  about 150 jobs in a U.S. call center with 110 jobs at a call center in a  different country (i.e., about 150 U.S. jobs lost; and an overall workforce  reduction), according to the integrator’s official.

Advanced Technologies Helped Increase Competitiveness and Enabled Employment Growth Despite Positions Being Replaced, According to Officials at Some Selected Firms

According to officials at some selected firms, greater competitiveness and  productivity due to the adoption of advanced technologies (see sidebar)  has helped firms grow their workforces. For example, some hired  additional production workers due to increased production (despite some  production tasks being taken on by the adopted technologies), or new  types of workers, such as technicians to maintain the technologies. Some  officials also said that although they may not have grown their workforces,  adopting advanced technologies helped them stay in business by  allowing them to compete effectively, and thus to preserve jobs and retain  workers. For example, officials at a medium-sized door manufacturer,   where we observed numerous robots in the production facility, told us that  their firm “could not survive” global competition without the use of  advanced technologies.

Productivity and Efficiency Gains  Adopting advanced technologies has helped  some firms improve their product quality and  increase their production efficiency.  For example, according to officials at a  medium-sized fruit processing plant, after the  firm began using an automated fruit grading  technology, the process took significantly less  time and resulted in far fewer complaints from  farmers about the grading. Farmers thought  the automated grading technology was fairer  and more accurate than having workers  manually and subjectively grade the fruit.  A large appliance manufacturer that began  using a collaborative robot to apply sealant to  an appliance door observed improved  consistency, which led to fewer service calls  from retailers and customers about  excessive, insufficient, or incorrect seals.  One medium-sized door manufacturer said  that automation technologies enabled them to  produce and ship doors in 3 days, as  opposed to 4 to 6 weeks.   An official from a warehouse for a regional  chain of grocery stores said that using  automated guided vehicles allowed the firm to  save time moving pallets from one end of the  warehouse to the other, and also save worker  hours. The warehouse saves just over $2 per  pallet moved by an automated guided vehicle  rather than a worker, and up to $3,500 a day  based on volume, according to the official.

Advanced technologies enabled some selected firms to increase  production or produce a larger range of goods, and thus to hire additional  production workers. This also led to workforce increases for suppliers and  other firms, according to officials.

One large appliance manufacturer increased its use of robots and  other advanced technologies to produce more of its own component  parts internally instead of relying on suppliers. As a result, the firm  was also able to increase the number of production jobs, according to  firm officials.

Due to advanced technologies, a small automotive parts manufacturer  was able to bid on a contract to produce a new and more intricate part  for a major automotive manufacturer. An official described how the  part was so intricate that it could not have been produced manually  with the required level of consistency and speed. Although the firm  adopted six robots to produce this part, winning the contract also  created nine new jobs. While the robots are completing much of the  production, the volume of parts demanded and the existence of some  tasks that only workers can complete has led to this job growth.

A developer of autonomous mobile robots said that, as a result of  increased business, his firm has created jobs among its eight local  suppliers where he buys parts, such as motherboards for the robots.

Growth of Developer and Integrator Firms  Selected developer firms we met with said they grew their technical and non-technical  staff as a result of increasing demand for their technologies.

A firm that develops and produces robots had tripled its workforce size, to about 130  employees, in the last year alone, according to officials.  An official at another developer firm that makes inspection robots said they had  grown from three workers to about 20 and envisions expanding to 100 in the near  future. The official said that the firm’s first years were spent on technology  development, but that once the technology was deployable to customers, the firm  grew its workforce size.

Integrator firms that help companies adopt advanced technologies have also grown in  size, and new types have emerged, according to integrators we visited. For example,  with the development of smarter robots, one integrator firm we visited entered the  industry to recondition and sell old robots; the firm also adds newer technology to these  robots if requested. This integrator has grown from 35 to 45 employees in the last 10  years, according to officials, with the new positions being primarily robot technician jobs.

As a result of technology adoption, some firms hired more workers with  technical skills, and in other instances lower-skilled workers, according to  firm officials.

An official from a warehouse for a regional chain of grocery stores  said that adopting an advanced automation system created a need for  three additional workers to provide preventive maintenance on the  machines. These additional positions pay about 25 percent more than  the standard warehouse positions, according to officials.

At a large automotive manufacturer, officials told us the firm increased  its number of lower-skilled cleaning jobs when robots began  producing large amounts of byproduct.

Officials Said Workers’ Roles, Tasks, and Skills Have Been Changing Due to Advanced Technologies at Selected Firms

At the firms we visited, workers changed roles and tasks as a result of  advanced technology adoption, such as focusing more on interactive,  cognitive, higher-skilled, and monitoring tasks, and in other cases  focusing more on lower-skilled tasks. Workers who can adapt and be  flexible to task changes may experience positive effects, including work  that is less physically taxing, safer, more ergonomic, less monotonous, or  higher paying. On the other hand, workers who are unable to adjust to  changing tasks may be negatively affected. Officials at some of the firms  told us that their firms provided internal training or leveraged external  resources to develop workers’ skills to help them move into new  positions. During our visits to selected firms, we saw a variety of ways in  which tasks for workers are changing.

Interactive work: The use of autonomous mobile robots to deliver  prescription drugs for patients enabled nurses at the university-affiliated  medical center we visited to focus more of their time on patient  interaction, according to officials. The small stamp manufacturer we  visited would like to continue to automate its ordering process and focus  more on providing customer service. Officials there said for future hires,  they plan to recruit for data and people skills, rather than production skills.

Cognitive work: A federal statistical agency adopted machine learning  technology to automatically interpret text narratives on forms and assign  codes to the data. As a result, staff who previously entered this  information manually are able to spend more time on analytical tasks  such as reviewing the accuracy of the auto-coding, correcting issues,  obtaining clarifications about information submitted on the forms, and  following up with non-respondents, according to officials.

Higher-skilled work: At a large automotive manufacturer, due to  increased use of advanced technologies, workers who are hired today  need to have greater technical proficiencies than workers hired in the  past. For example, to adapt to their changing roles working with robotic  equipment, non-technical production staff need machine maintenance  and technical skills, rather than only manual dexterity skills. Officials at a  large appliance manufacturer that adopted an automated machine to  stamp metal said that the resulting process required a single worker to  monitor the machine and provide basic maintenance. This worker needed  technical skills and at least 6 months of training to effectively perform  these duties. In contrast, at another one of this firm’s global plants, four  separate pressers are used and each requires workers to load and  unload metal.

Monitoring work: Officials at the large appliance manufacturer  mentioned above showed us a step in their production process in which  two small pieces of plastic and metal need to be attached. Three workers  used to perform this task by hand, which caused ergonomic challenges,  and inconsistencies in both quality and production cycle times. Now, the  firm uses three robots to perform this work and a single worker loads the  pieces for all three robots and monitors their performance. At a small  automotive parts manufacturer, production operators who work in cells  with robots monitor multiple machines and sometimes also monitor  multiple work cells, so a greater aptitude level is needed. As a result,  these operators earn $3 per hour more than operators in work cells  without robots, according to a firm official.

Less physically taxing work: Staff at some firms also told us how  advanced technologies have made worker tasks less physically  demanding. For example, we talked with one warehouse worker who  used to lift heavy boxes, but who now operates a forklift after his old task  was automated with a conveyer belt and sorting system. He described his  new position as having ergonomic benefits, including experiencing less  back pain. At a large automotive manufacturer, officials said the firm  installed six robots to paint vehicle interiors. This production step was a  major ergonomic hazard and workers who did this painting had a  relatively high injury rate, according to officials. Officials told us that  adopting the robots lowered the injury rate among these workers and  resulted in faster vehicle painting.

Simplified work: At a small stamp manufacturer that adopted a  collaborative robot, officials told us that as the firm continues to redesign  and optimize operations, the robot will take on more complex tasks. As a  result, the remaining production work performed by the firm’s production  worker will be simpler (see fig. 7). Officials said that in the future, after the  firm’s current production worker retires, the firm may rely on contingent  workers to perform any needed production work not completed by the  robot because the tasks will be simpler and easier to train a new,  temporary worker to complete. Officials said the firm may also hire a  worker with a different and more varied skillset who can perform the few  remaining production tasks along with other types of tasks.

Lower-skilled work: Officials at a medium-sized door manufacturer  installed a robot to facilitate the firm’s redesigned door sealant system  and production process. The original process of manually applying door  sealant was physically-intensive, ergonomically challenging, and required  significant skill and experience to precisely apply the sealant. With the  new design, a robot applies the sealant autonomously. As a result,  workers perform lower-skilled tasks in this process, including placing a  piece on a platform, visually inspecting the robot’s work, cleaning and  setting up the robot’s work station, and confirming the correct program is  entered in the computer.

Adaptability to changing daily work demands: Officials from selected  firms told us that due to advanced technology adoption, workers need to  change tasks depending on the day and circumstances. For example, at  a large appliance manufacturer some workers serve in different capacities  depending if the robots are functioning properly and depending on the  production needs of that day. On the day we visited the plant, several of  the robots were malfunctioning and workers were performing the robots’  tasks. Firm officials said that some of their workers serve in swing roles  and move around to different production processes and assist as needed.

Training Centers for Advanced Tech Skills  We met with officials at a training center that  re-trains adults and teaches high school  students to work with advanced technologies  used in manufacturing. We visited two firms  in the area that told us that this training center  helps fill a local shortage in maintenance  technician skills, and that they have hired  workers who graduated from the center.  Officials at the training center said that there  is a high demand in the area for maintenance  technicians. For example, they said that a  large automotive manufacturer in the area is  planning to hire 800 maintenance technicians  over the next 3 years, and that the firm is  worried about how it will fill these positions.  Officials at the training center also said that  some firms have such a high demand for  maintenance technicians that they hire high  school students who complete the training  program before they graduate high school.  The training center is piloting its adult training  program. The program recruits adults who  are underemployed and have some  mechanical aptitude, then trains them in  advanced technologies used in  manufacturing. Most of the students who  participated in an early pilot obtained higher  paying jobs than those they held before the  program, according to officials at the training  center.

Many firms we visited offered training for workers to adapt to their  changing roles and tasks, particularly when the tasks or roles became   more technical. Some firms used internal training resources and some  leveraged local training centers (see sidebar). Some technology  developers also offered training to firms that adopted their technologies.  Officials at some firms told us that training current workers for more  technical positions was easier than finding workers with the appropriate  skills. For example, officials at one medium-sized door manufacturer said  they needed highly specialized engineers, but could not find any in the  region. As a result, this firm offered tuition reimbursement for workers  who were willing to go back to school to become engineers. They also  partnered with local community colleges to train students to become  future maintenance technicians. Officials at a large automotive  manufacturer said that due to increases in the firm’s use of advanced  technologies, the plant has needed to hire more technicians. As a result,  this firm added programs to its on-site training center to train workers for  these roles.

The complex job changes we observed at the selected firms we visited  are not currently captured in federal data, though they may have  significant implications for broader employment shifts. As the primary  agencies responsible for monitoring the U.S. economy and workforce, the  Departments of Commerce and Labor are aware of the importance of  advanced technologies as major drivers of changes. For example,  Census’ newly administered Annual Business Survey may provide  valuable information in the future about the adoption and use of advanced  technologies nationwide and the prevalence of resulting workforce  effects. However, comprehensive data on firms’ adoption and use of  advanced technologies do not currently exist, which prevents federal  agencies and others from fully monitoring the spread of advanced  technologies throughout the economy and linking their use to changes in  employment levels or structural shifts in the tasks and skills associated  with jobs.

Observations from our visits to selected firms illustrate the complex and  varied workforce effects that result from firms’ adoption of advanced  technologies. In some circumstances, technology adoption will lead to  increases in different types of jobs and in other cases technology  adoption will lead to workforce reductions—either over time or  immediately. Regardless of the firm-level workforce effects, worker roles  and responsibilities are likely to change as advanced technologies take  over tasks that workers previously performed. These changes could  positively affect some workers, but could also have negative  consequences for other workers, especially those who are unable to  adapt to changes. For example, workers whose previous work tasks are  automated and who are unable to perform new tasks required of them  may need to seek new employment. If these changes occur occupation- wide, across many firms, workers may need to re-train or seek new  employment in entirely different occupations or industries. To the extent  that these changes are concentrated among occupations susceptible to  automation, certain groups of workers (e.g., those with lower education  levels) may be disproportionately affected and may lack the opportunity to  develop skills needed to enter growing occupations. These workers will  be in greater need of programmatic or policy supports, and federal  workforce programs will need to be aligned with in-demand skills for the  changing economy.

Without comprehensive data that can measure the magnitude and variety  of these firm-level changes, the workforce effects of the adoption of  advanced technologies will remain unclear, job seekers may not be fully  informed about their best future career prospects, and federally funded  programs to support workers may be misaligned with labor market  realities. DOL’s ability to collect information regularly on jobs and workers  may enable the agency to fill these information gaps. Specifically, better  data could be used by policymakers and DOL to proactively design and  fund worker training programs that meet the job needs of the future.

The Secretary of Labor should direct the Bureau of Labor Statistics (BLS)  and the Employment and Training Administration (ETA) to develop ways  to use existing or new data collection efforts to identify and systematically  track the workforce effects of advanced technologies. For example, the  Secretary could select any of the following possibilities, or could identify  others.

BLS could expand existing worker or firm surveys to ask respondents  whether advanced technologies have resulted in worker  displacements, work hour reductions, or substantial adjustments to  work tasks.

BLS could expand its employment projections work to regularly  identify occupations projected to change over time due to advanced  technologies.

ETA could expand the O*NET data system to identify changes to  skills, tasks, and tools associated with occupations, as the information  is updated on its rotational basis, and consider how this could be used  to track the spread of advanced technologies.  (Recommendation 1)

We provided a draft of this report to DOL, Commerce, NSF, and OSTP for  review and comment. We received written comments from DOL that are  reprinted in appendix II and summarized below. DOL and Commerce  provided technical comments, which we incorporated as appropriate. NSF  and OSTP told us that they had no comments on the draft report.

DOL agreed with our recommendation to develop ways to identify and  track the workforce effects of advanced technologies. DOL stated that it  will continue coordinating with the Census Bureau on research activities  in this area, and that it plans to identify and recommend data collection  options to fill gaps in existing information about how the workplace is  affected by new technologies, automation, and AI. DOL also stated that it  plans to release employment projections annually instead of every 2  years, beginning in 2019.

As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to the appropriate  congressional committees, the Secretary of Labor, the Secretary of  Commerce, the Director of the National Science Foundation, the Director  of the White House Office of Science and Technology Policy, and other  interested parties. In addition, the report is available at no charge on the  GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-7215 or brownbarnesc@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

The objectives of this review were to examine (1) what is known about  how the adoption of advanced technologies affects the U.S. workforce;  (2) selected federal agency efforts to track and monitor the adoption and  workforce effects of advanced technologies; (3) considerations that led  selected firms to adopt advanced technologies and the risks they faced;  and (4) ways technology adoption has affected the workforce at selected  firms.

Throughout the report, we use “advanced technologies” as a broad term  to describe technological drivers of workforce changes, including but not  limited to those identified in the National Academies study: artificial  intelligence; machine learning; robotics; autonomous transport; advanced  manufacturing; 3D printing; advanced materials; computing power; and  internet and cloud technology. The technologies we observed at work  sites could generally be categorized as applications of robotics, machine  learning (e.g., machine vision or autonomous navigation), or both.  However, not all technologies that may affect the U.S. workforce in the  future—through automation or in other substantial ways—fall into these  categories. Our use of the broad term “advanced technologies” leaves  open the possibility that new technologies and other areas of focus are  likely to emerge.

To examine what is known about how the adoption of advanced  technologies affects the U.S. workforce, we explored the extent to which  available federal data could identify and measure these effects, and we  identified limitations with available data. Because there was no  comprehensive data that link employment trends to technology adoption,  we used a study by Frey and Osborne to identify a group of occupations  susceptible to automation. We then analyzed whether the concentration  of these occupations in industries is correlated with growth in tech jobs or  employment declines in those industries, whether job displacements are  more common in these occupations than in others, the characteristics of  workers who hold jobs in these occupations, and the geographic  concentration of jobs in these occupations. We analyzed employment  data from the Census Bureau (Census) and the Bureau of Labor  Statistics (BLS); specifically, the American Community Survey (ACS), the  Current Population Survey’s (CPS) Displaced Worker Supplement, and  the Occupational Employment Statistics (OES) survey. For more  information, see detailed discussions of our data analyses in sections 1-3  below.

Identifying occupations susceptible to automation: Using a model  that evaluates tasks within an occupation, Frey and Osborne estimate a  probability of automation for 702 occupations. They identify occupations  with a probability greater than 0.7 as being at high risk of automation. In  our analyses, we thus consider this collection of occupations as those  susceptible to automation. While there are different studies that attempt to  predict what occupations or jobs may be automated in the future, we use  the work by Frey and Osborne because it is widely cited and because its  results are structured to allow us to identify a broadly inclusive collection  of occupations susceptible to automation. The results of our analyses  could be affected by using other studies to the extent that they identify  different occupations as susceptible to automation. The accuracy of any  collection of occupations is limited by the unpredictability of when or if  jobs are automated, as well as the fact that occupations are comprised of  a variety of jobs, which may experience automation to varying degrees or  in different ways.

We also reviewed examples of recent and ongoing studies that attempt to  measure workforce effects directly attributable to technology adoption.  We identified examples of research through interviews with  knowledgeable individuals and from among those included in a recent  review of the state of empirical work. Our review of studies was not  meant to be comprehensive of the research in this area.

To identify selected federal agencies’ current and planned efforts to  collect data on, and monitor the prevalence and effects of advanced  technologies in the economy, we met with the Departments of Labor  (DOL) and Commerce (Commerce), as the principal federal agencies  responsible for collecting data on the U.S. economy and workforce; the  White House Office of Science and Technology Policy (OSTP), which  leads interagency science and technology policy-coordination efforts across federal agencies; and the National Science Foundation (NSF),  which was involved in the development of the Annual Business Survey.  We interviewed officials and reviewed data and information collected by  these agencies. We also reviewed the Annual Business Survey’s  questionnaire to consider the potential uses of data being collected by the  survey, and analyzed data from DOL’s Employment Projections program  and Occupational Information Network (O*NET) database to identify  information related to the adoption and workforce effects of advanced  technologies.

Annual Business Survey: The Annual Business Survey was  administered for the first time in summer 2018, and collects  information from firms about various topics, including innovation and  technology use. The survey is a joint effort by the Census Bureau and  the National Center for Science and Engineering Statistics within the  National Science Foundation and Census plans to administer the  survey annually for 5 years. The Annual Business Survey replaces the  5-year Survey of Business Owners, the Annual Survey of  Entrepreneurs, the Business R&D and Innovation for Microbusinesses  survey, and the innovation section of the Business R&D and  Innovation Survey.

Employment Projections program: BLS’s Employment Projections  program analyzes changes in the economy, among other things, to  project how employment by occupation may change over the next 10  years, including which occupations may be affected by advanced  technologies. BLS’s projections are for the most part structured  around the Occupational Employment Statistics, which produces  employment and wage estimates for over 800 occupations. As part of  this program, BLS develops a table of occupations that are projected  to have direct employment changes due to some identified reason.  According to BLS officials, the specific reason listed for each  occupation is based on BLS’s judgment of the most significant factor  or factors affecting the occupation (i.e., based on a qualitative  assessment). We examined the reasons listed in this table and  identified those related to the adoption of advanced technologies in an  occupation, such as through automation, the increased use of robots  or artificial intelligence, advances in machine or software  technologies, or other similar changes. We then counted the number  of unique occupations projected to experience declines in their shares  of employment in an industry or group of industries due to one of  these reasons. We also counted these occupations according to their  major occupation group. BLS projected that some of these  occupations would experience employment share declines in all  industries and some would experience employment share declines in  a single industry only. We counted unique occupations regardless of  what industries or how many were noted (e.g., all industries or only  one). We chose to do this to capture an inclusive list of occupations  projected to be affected by advanced technologies, and because we  are not using the list to quantify total projected employment changes.  Of the 247 unique occupations BLS includes in its table as projected  to have direct employment changes due to some identified reason,  BLS projects that 163 will experience employment share declines— 100 of those occupations are projected to change broadly as a result  of the adoption of advanced technologies. An employment share  decline indicates that employment in an occupation will decline  relative to others in a given industry or group of industries, not that the  occupation will necessarily experience a decrease in employment in  absolute terms.

Occupational Information Network (O*NET) database: The O*NET  database contains information about the skills, tasks, and tools (i.e.,  use of technology) associated with specific occupations. We  downloaded two components of the database that (1) list the various  work tasks associated with each occupation, and (2) list the various  tools and technologies used by each occupation. In each database  component, we searched for and identified tasks, tools, and  technologies that involved robots in some way—e.g., tasks such as  working with robots, robotic systems, or robotic applications, and tools  such as welding robots, loading robots, or robot automation tools. We  then counted the number of unique occupations that (1) had an  associated work task related to robots, or (2) used a robot-related tool  in the occupation.

To understand firms’ adoption of advanced technologies and any  resulting workforce effects, we met with officials representing 16 different  firms that are using advanced technologies in their operations, as well as  a systems integrator who provided detailed information about how several  customer firms are using advanced technologies. Most of the meetings  with firms were in-person site visits; three of the meetings with firms and  the meeting with the systems integrator were by phone. Throughout this  report, we use the term “firm” for simplicity, although the “firms” we met  with included production plants of large manufacturers, single-location  firms, public sector agencies, and other entities (see below). We also  identify the manufacturing firms we visited as falling into one of three  different size groups to describe their relative size differences from each  other. The manufacturing firms we visited ranged from eight employees to  thousands, according to firm officials. For the purposes of our study, we  define small as fewer than 200 employees; medium as 200 employees to  1,000; and large as over 1,000 employees.

Among the 16 firms we met with that are using advanced technologies,  10 are manufacturing firms: a small manufacturer of rubber stamps and embossing seals (also  referred to as a small stamp manufacturer);  two medium-sized door manufacturers; a small automotive parts manufacturer; a medium-sized automotive parts manufacturer;  two large appliance manufacturers; a large automotive manufacturer; a large manufacturing corporation of household and personal care a medium-sized fruit processing plant.

Six are non-manufacturing firms of various types: a construction consulting company; a federal statistical agency; a food retail corporation; a municipal township; a university-affiliated medical center; and a warehouse for a regional grocery store chain.

The firms about which we received information from the systems  integrator were business, administrative, and customer relations offices of  various firm types.

To identify firms to meet with, we consulted and sought referrals from a  variety of knowledgeable sources, including academic researchers,  technology developer firms, technology integrator firms, state economic  development associations, and our own research. We selected firms that  varied in size, industry sector, types of advanced technology used, and  geography. We limited our focus to firms that had adopted advanced  technologies and had experienced workforce effects. Our selection of  firms is not a generalizable sample, but does provide illustrative examples  of the adoption and workforce effects of advanced technologies.

During our site visits at firms, we met with one or more management  officials and, at times, with workers. We were also able to view the  advanced technologies being used in operations. Our discussions with  officials included topics such as motivations for adopting advanced  technologies, the integration process, and any workforce effects that  resulted from the technologies, including positions lost or gained and how  workers’ tasks and skills may have changed. Our site visits and interviews  with firm officials ranged from hour-long conversations to full-day visits, so  some site visits yielded more detailed information than others.

In addition to the firms that use advanced technologies, we interviewed  seven technology developer firms and two robotics integrator firms (in  addition to the systems integrator mentioned above). We met with these  firms to learn more about some of the technologies being used and the  adoption process, as well as about workforce effects at these firms. We  identified these developer and integrator firms from various sources,  including our conversations with academic researchers and our own  research.

We conducted additional interviews to obtain background and context for  our work. We met with individuals knowledgeable about issues related to  the adoption and workforce effects of advanced technologies, such as  academic researchers and economists, officials from two unions  representing manufacturing workers, officials at three industry-based  organizations, officials from two state economic development  associations, and officials at two worker training centers. For all  objectives, we also reviewed relevant federal laws and regulations.

The remainder of this appendix provides detailed information about the  data and quantitative analysis methods we used to examine what is  known about the workforce effects of automation and the adoption of  advanced technologies (objective 1), as follows:

Section 1: Analyses using data from the ACS

Section 2: Analyses using data from the CPS’s Displaced Worker

Section 3: Analyses using data from the OES survey  For each of the datasets described below, we conducted a data reliability  assessment of variables included in our analyses. We reviewed technical  documentation and related publications and websites with information  about the data. We spoke with BLS and Census officials who maintain the  datasets to gain an understanding of and provide context for the various  data that we analyzed, as well as to resolve any questions about the data  and to identify any known limitations. We also tested the data, as  applicable, to check for logical consistency, missing data, and  consistency with data reported in technical documentation. We  determined that the variables we used from the data we reviewed were  sufficiently reliable for the purposes of this report.

Section 1: Analyses Using Data from the American Community Survey

This section describes the quantitative analysis methods we used to  examine employment trend correlations and the characteristics and  earnings of workers in occupations susceptible to automation (as  identified by Frey and Osborne; see above). We used ACS data for these  analyses.

The ACS is administered by the Census Bureau and is an ongoing  national survey that uses a series of monthly samples to produce  annually updated estimates for the same areas surveyed via the  decennial census. The ACS collects a range of information about  individuals from a large sample of households—over 2.2 million  respondent households in 2016—including employment information such  as occupation, industry, and earnings, and demographic information such  as age, gender, race, ethnicity, and educational attainment. We limited  our analysis to workers who were classified as current employees, and  who had earned positive wage and salary income in the prior 12 months.  In 2016, this resulted in observations representing 136 million workers,  close to the number reported by BLS for that same period using a  different survey. This report primarily used ACS data from 2010 through  2016—specifically, we relied on the Census Bureau’s Public Use  Microdata Sample of the ACS for the single years 2010, 2011, 2012,  2013, 2014, 2015, and 2016.

Analyses of Employment Trend Correlations

To test whether industries with higher concentrations of individuals in  occupations susceptible to automation (as identified by Frey and  Osborne) have experienced employment changes, we examined their  correlation with changes in tech job concentration and changes in overall  employment from 2010 through 2016. We limited the analysis to this  period both because the ACS occupation codes changed in 2010 and  because it allowed our results to post-date the economic recession of  2007-2009. We used industry definitions set by the ACS data, which  groups some industries together—e.g., residential and nonresidential  construction industries are combined in a single construction industry  grouping. We defined tech jobs as those in computing, engineering, and  mathematics occupations, consistent with previous GAO work on the tech  field. We also examined an alternative definition of tech jobs in which  we included those with “computer” in the occupation title. For both  definitions, we estimated the number of tech jobs in each industry in each  year, 2010-2016. We then calculated the growth rate in the number of  tech jobs in each industry, and correlated that growth rate with the  percentage of workers in that industry in occupations susceptible to  automation (as identified by Frey and Osborne). We also estimated the  number of workers overall in each industry in each year (2010-2016) and  correlated the trend in total employment with the percentage of workers in  that industry in occupations susceptible to automation (as identified by  Frey and Osborne). We restricted our correlation analyses to those  industries where the tech job growth rate or the overall employment trend  was statistically significant.

We performed two correlation tests. The Spearman test measures  correlation between the rank of the two sets of values. The Pearson test  measures correlation between the values themselves. As shown in table  2, we found a positive but weak correlation between industries with higher  concentrations of jobs susceptible to automation and their concentration  of tech jobs, based on both correlation tests and both definitions of tech  jobs, and we found no meaningful correlation with change in overall  employment in either test.

To explore an example industry—the plastics product manufacturing  industry—in further detail, we identified the number of jobs susceptible to  automation within that industry, by occupation and groups of occupations.  We also examined the growth in tech jobs within the industry, by tech  occupation. We approximated each occupation’s contribution to the  overall growth of tech jobs in the industry by multiplying their individual  growth rates over the period 2010-2016 by their employment in 2010. The  growth rates for the three engineering occupations, which when  combined, account for more than half of the industry’s growth in tech jobs,  were each significant at the 85 percent confidence level.

Analyses of Worker Characteristics and Earnings

To analyze the characteristics of workers in occupations susceptible to  automation (as identified by Frey and Osborne), as well as the  characteristics of workers with tech jobs, we used 2016 ACS data. We  examined data on the workers’ gender, level of education, age, race and  ethnicity, and hourly wage, and compared distributions of workers in  occupations susceptible to automation and workers in all other  occupations (see table 3). For race and ethnicity categories, we included  only non-Hispanic members of White, Black, Asian, and Other categories,  and the Hispanic category included Hispanics of all races. The “Other”  category included American Indian or Alaskan Native, Native Hawaiian or  Pacific Islander, two or more races, and other race. To analyze education  level, we combined all attainment levels from a high school degree or  less. To estimate the hourly wage of workers, we divided the wage and  salary earnings of the worker by their usual hours worked and weeks  worked. To test the reliability of this measure, we compared our results to  average hourly wages reported by other BLS surveys; we found that the  average values were sufficiently close to determine that this method was  sufficiently reliable for our purposes.

To investigate whether differences in hourly wage might be due to other  factors, we estimated multiple regression models that enabled us to  control for additional variables. Specifically, we estimated wage  differences between workers in occupations susceptible to automation  and workers in other occupations—i.e., whether a worker was in an  occupation susceptible to automation (as identified by Frey and Osborne)  was our primary independent variable (a binary, yes/no variable).  Because we used the natural log of the hourly wage as the dependent  variable, the standard interpretation of the regression coefficient of this  variable is that it represents the average log point difference in hourly  wages between occupations susceptible to automation and all other  occupations. This coefficient can be made to more closely approximate a  percentage difference in hourly wages or an earnings gap by taking the  exponent and subtracting 1. As noted previously, we limited our analysis  to workers who earned positive wage and salary income in the prior 12  months. We also removed observations with outlier values for wages  (e.g., wage rates above $140 per hour); this represented about 1 percent  of the sample in 2016.

We ran five regression models with different sets of independent variable  controls.

Regression (1) estimates the earnings gap without any controls (the  uncorrected earnings gap).

Regression (2) estimates the earnings gap with a set of independent  variables that control for characteristics of the individual; these  variables included age, race and ethnicity, gender, marital status,  state of residence, and education level.

Regression (3) estimates the earnings gap with independent dummy  variables for 2-digit industry codes added; this corrects for any  differences between industries at the 2-digit level.

Regression (4) estimates the earnings gap with independent dummy  variables for 2-digit occupation codes added; this corrects for any  differences between occupations at the 2-digit level.

Regression (5) includes both 2-digit industry and 2-digit occupation  code dummy variables.

As table 4 shows, we found a significant difference in hourly wages  between workers in occupations susceptible to automation compared to  workers in other occupations, even after independent variables to control  for worker characteristics, industry, and occupation codes were included.  Including the additional independent variables caused the earnings gap to  fall from just over -34 percent to just over -10 percent. Regression model  3, which estimated an earnings gap of about -17.2 percent, is our  preferred model, as it controls for individual worker characteristics and for  any differences between industries at the 2- digit level, but does not  include occupation as an independent variable. Including occupation  variables controls for any differences between occupations at the 2- digit  level. However, because we identify workers in jobs susceptible to  automation based on their occupations, these occupation control  variables are likely highly predictive of Frey and Osborne’s estimated  probability of automation, which is used to categorize workers in jobs  susceptible to automation. We also ran these regression models for other  years from 2010 to 2016 and we found substantively similar results.

Section 2: Analyses Using Data from the Current Population Survey’s Displaced Worker Supplement

This section discusses the quantitative analysis methods we used to  compare relative job displacement rates between workers in occupations  susceptible to automation (as identified by Frey and Osborne; see above)  and workers in other occupations. We used data from the CPS’s  Displaced Worker Supplement for these analyses.

The CPS is sponsored jointly by Census and BLS and is the source of  official government statistics on employment and unemployment in the  United States. The basic monthly survey is used to collect information on  employment, such as employment status, occupation, and industry, as  well as demographic information, among other things. The survey is  based on a sample of the civilian, non-institutionalized population of the  United States. Using a multistage stratified sample design, about 56,000  households are interviewed monthly based on area of residence to  represent the country as a whole and individual states; the total sample  also includes additional households, some of which are not interviewed in  a given month for various reasons, such as not being reachable. The  CPS Displaced Worker Supplement has been administered every other  year since 1984, and provides supplemental data on persons age 20  years or older who lost a job involuntarily in the prior 3 years, including  data on reasons for job displacement, as well as industry and occupation  of the former job. This report used data from the January 2016 Displaced  Worker Supplement.

Analyses of Relative Job Displacement Rates

To analyze whether workers in occupations susceptible to automation (as  identified by Frey and Osborne) experience job displacement at differing  rates than workers in other occupations, we used data from the CPS’s  January 2016 Displaced Worker Supplement. We identified workers who  lost or left a job involuntarily during the 3 calendar years prior to the  survey (i.e., January 2013 through December 2015) because their  position or shift was abolished or because there was insufficient work for  them to do. We focused on these reasons for displacement as those that  most closely approximate how advanced technologies could replace  workers at a given firm. We also limited our analysis to those workers  who did not expect to be recalled to their jobs within the next 6 months.  We categorized these displaced workers according to the occupations  from which they were displaced (e.g., workers displaced from occupations  susceptible to automation and workers displaced from all other  occupations).

We calculated relative job displacement rates as the number of  displacements over the period 2013-2015 reported by a given population  (e.g., workers in occupations susceptible to automation), over that  population’s total current employment in January 2016. Although this  measure does not represent the total number of jobs that existed annually  that could have resulted in displacements, it allows us to control for  population size and to approximate a relative displacement rate. We  examined various populations, including occupations identified as  susceptible to automation by Frey and Osborne, occupations BLS  projects will experience declines in their share of employment due to  advanced technologies (see above), and production occupations. To  categorize occupations, Frey and Osborne and BLS use Standard  Occupational Classification (SOC) codes, whereas the Displaced Worker  Supplement uses Census occupation codes. We used a crosswalk  provided by Census to match these occupation classifications. SOC  codes have a hierarchical structure—e.g., a “broad” occupation group  contains a subset of “detailed” occupations. For example, SOC code 13- 1031 is the detailed occupation “claims adjusters, examiners, and  investigators” within the broad group SOC 13-1030 (“claims adjusters,  appraisers, examiners, and investigators”). When a direct crosswalk  between SOC and Census occupation codes was not available at the  detailed level, we used the associated broad SOC group to identify a  Census occupation code. There were some respondents in the Displaced  Worker Supplement who did not report the occupation from which they  were displaced, and these were dropped from our analysis.

To estimate the sampling errors for each estimate, we used strata defined  by state because the Displaced Worker Supplement data did not provide  replicate weights or the sampling strata necessary to obtain standard  errors. When estimating the number of job displacements over the period  2013-2015 reported by a given population (e.g., workers in occupations  susceptible to automation), we used the supplement weight for  respondents. When estimating the population’s total current employment  in January 2016, we used the CPS 2016 weight for respondents. We  used a Taylor series linearization to estimate the sampling error of the  ratio of estimated number of job displacements over the period 2013- 2015 to the estimated number of current employment in 2016.

While our primary analysis examined relative displacement rates for  workers in occupations susceptible to automation, we also conducted  sensitivity analyses by considering other groups of occupations.  Specifically, we examined the relative displacement rates of the following  groups:  Jobs susceptible to automation had a relative displacement rate of 3.4  percent +/- 0.3, and all other jobs combined had a relative  displacement rate of 2.9 percent, +/- 0.2.

Jobs in occupations BLS projects will experience relative declines in  employment due to advanced technologies (see above) had a relative  displacement rate of 3.7 percent, +/- 0.5, and all other jobs combined  had a relative displacement rate of 3.6 percent, +/- 0.2.

Jobs in production occupations had a relative displacement rate of 3.7  percent +/- 0.8, and all other jobs combined had a relative  displacement rate of 3.1 percent, +/- 0.2.

Section 3: Analyses Using Data from the Occupational Employment Statistics survey

This section discusses the quantitative analysis methods we used to  analyze geographic reliance on occupations susceptible to automation  (as identified by Frey and Osborne; see above). We used OES data for  these analyses.

The OES survey is a federal-state cooperative effort between BLS and  state workforce agencies, which collects information on occupational  employment and wage rates for wage and salary workers in nonfarm  establishments. The survey is based on a sample drawn from about 7.6  million in-scope nonfarm establishments in the United States that file  unemployment insurance reports to the state workforce agencies. Using a  stratified sample design, about 200,000 establishments are surveyed  semiannually and employment estimates are based on six panels of data  collected over a 3-year cycle. The final in-scope sample size when six  panels are combined is approximately 1.2 million establishments. The  OES survey includes all full- and part-time wage and salary workers in  nonfarm industries, but excludes self-employed workers, owners and  partners in unincorporated firms, household workers, and unpaid family  workers. OES data provide occupational employment estimates by  industry for the country as a whole, for individual states, and for more  local geographic areas (e.g., metropolitan and nonmetropolitan areas).  This report used data from the May 2017 Occupational Employment  Statistics.

Analyses of Geographic Reliance on Occupations Susceptible to Automation

To analyze what U.S. geographic areas rely more heavily on employment  in occupations susceptible to automation, we used data from the May  2017 OES. For each local geographic area, we estimated how many jobs  were in occupations identified as susceptible to automation by Frey and  Osborne (see above) and how many jobs were in all other occupations.  We also estimated how many jobs were in each group of occupations  nationwide (using national-level data). We then calculated a location  quotient for each local geographic area, which measures the proportion of  each area’s jobs that were in occupations susceptible to automation  compared to the national proportion of employment in these occupations.  This measure depicts the extent to which a local geographic area relies  on certain jobs for the employment of its population, relative to other  areas.

Based on their location quotients, we categorized and mapped 589 local  geographic areas in the following three groups:

Relatively High Concentration: Areas where the proportion of jobs  susceptible to automation is at least 5 percentage points greater than  the national average, and the difference is statistically significant at  the 95 percent confidence level. This translates to an estimated  location quotient of at least 1.1.

Average or Relatively Low Concentration: Areas where the proportion  of jobs susceptible to automation is within 5 percentage points above  the national average or lower.

Undetermined Reliance: Areas where the proportion of jobs  susceptible to automation is undetermined. We classify an area’s  proportion as “undetermined” if the estimated margin of error at the 95  percent confidence level is larger than 5 percentage points.

We conducted one sided z-tests at the 95 percent confidence level to  analyze each area’s estimated location quotient. The null hypothesis is  that the area location quotient is less than or equal to 1.1 (i.e., the  proportion of employment in the group of occupations in an area is 1.1  times the national proportion). The alternative hypothesis is that the area  location quotient is greater than 1.1. Because estimated area employment  proportions are based on a sample, we also restricted our tests to those  areas that were reliable for our purposes by requiring that areas had  sampling errors of no greater than 5 percentage points for a 95 percent  confidence interval.

According to BLS, employment estimates for individual occupations in  individual local geographic areas may not be available in the public data  for a variety of reasons, including for example, failure to meet BLS quality  standards or to ensure the confidentiality of survey respondents. Because  we aggregate data across multiple occupations, our methodology treats  these cases as if employment in the given occupation in the given area  was zero, which is not the case and which introduces imprecision into our  analysis and the resulting location quotients. However, because ensuring  confidentiality is a primary concern, we assume that most of these cases  where data are suppressed would have relatively small numbers of jobs,  and thus have minimal effects on our results. To test this assumption and  to ensure the appropriateness of our methods, we compared the total  number of jobs we analyzed across all local geographic areas to the total  number of jobs reported at the national level (which do not have data  suppressed). The total number of jobs analyzed across our local  geographic areas was 5.5 percent lower than the total number of jobs  reported at the national level, which we concluded was within an  acceptable threshold to determine that the data were sufficiently reliable  for our purposes and our analysis. In addition, according to BLS,  because occupational employment estimates are rounded to the nearest  10 before publication, estimates of location quotients calculated from the  public data will be subject to some rounding error, compared with location  quotients calculated from the unrounded pre-publication data.

Appendix II: Comments from the Department of Labor

Appendix III: GAO Contact and Staff Acknowledgments

In addition to the contact named above, Blake Ainsworth (Assistant  Director), Michael Kniss (Analyst-in-Charge), Shilpa Grover, and John  Lack made key contributions to this report. Also contributing to this report  were James Bennett, Benjamin Bolitzer, Melinda Cordero, Holly Dye,  Jonathan Felbinger, Sheila R. McCoy, Jean McSween, James Rebbe,  Krishana Routt-Jackson, Benjamin Sinoff, Almeta Spencer, and Sonya  Vartivarian.

Robots, artificial intelligence, and other advanced technologies are changing the workplace. We visited companies to observe the effects on workers. Effects varied, with some companies reducing their workforces, many moving workers to different roles, and some hiring workers due to increased production or new skill needs.

Workforce data doesn't identify the causes of employment shifts, making it difficult to assess technology's effects. Additional information could help agencies design programs to prepare workers for jobs of the future.

We recommended that the Department of Labor develop ways to better track workforce effects of technologies.