A deep reinforcement learning model for predictive maintenance planning of road assets: Integrating LCA and LCCA

Road maintenance planning is an integral part of road asset management. One of the main challenges in Maintenance and Rehabilitation (M&R) practices is to determine maintenance type and timing. This research proposes a framework using Reinforcement Learning (RL) based on the Long Term Pavement Performance (LTPP) database to determine the type and timing of M&R practices. A predictive DNN model is first developed in the proposed algorithm, which serves as the Environment for the RL algorithm. For the Policy estimation of the RL model, both DQN and PPO models are developed. However, PPO has been selected in the end due to better convergence and higher sample efficiency. Indicators used in this study are International Roughness Index (IRI) and Rutting Depth (RD). Initially, we considered Cracking Metric (CM) as the third indicator, but it was then excluded due to the much fewer data compared to other indicators, which resulted in lower accuracy of the results. Furthermore, in cost-effectiveness calculation (reward), we considered both the economic and environmental impacts of M&R treatments. Costs and environmental impacts have been evaluated with paLATE 2.0 software. Our method is tested on a hypothetical case study of a six-lane highway with 23 kilometers length located in Texas, which has a warm and wet climate. The results propose a 20-year M&R plan in which road condition remains in an excellent condition range. Because the early state of the road is at a good level of service, there is no need for heavy maintenance practices in the first years. Later, after heavy M&R actions, there are several 1-2 years of no need for treatments. All of these show that the proposed plan has a logical result. Decision-makers and transportation agencies can use this scheme to conduct better maintenance practices that can prevent budget waste and, at the same time, minimize the environmental impacts.


Introduction:
The performance of all national economies is dependent on road infrastructure, which delivers a wide range of economic and social advantages. At the same time, it has a significant environmental impact throughout its construction, maintenance, and use (Santero and Horvath 2009). The necessity of sustainable development of infrastructure systems, such as transportation networks, is now widely acknowledged, especially after the UN's International panel on climate change (IPCC) reported for "code red" in its latest report (Cartwright 2021). Maintaining and repairing the continually deteriorating pavement networks sustainably with minimal financial expenditure, greenhouse gas emissions, and non-renewable materials consumption has steadily attracted attention (Wang and Gangaram 2014).
Pellicer et al. (Pellicer, Sierra et al. 2016) found that conventional evaluations of pavement maintenance alternatives concentrated on economic and technical parameters, ignoring the relevance of environmental impacts. Air pollution and climate change are two of the most pressing environmental issues facing the globe today. If temperature levels continue to rise owing to anthropogenic greenhouse gas (GHG) emissions into the atmosphere, global warming might have severe environmental, social, and economic consequences throughout the world (Santos, Torres-Machi et al. 2020). With that being said, special attention must be paid to the transportation sector since it is responsible for spewing considerable amounts of pollutants up into the air (Watts, Amann et al. 2018). Moreover, in the presence of global warming, it is anticipated to have more Maintenance and Rehabilitation (M&R) practices because of increased natural disasters, and thus more pollutants will be released into the atmosphere (Santos, Torres-Machi et al. 2020).
Developing a practical long-term maintenance framework will not only help policymakers to achieve an acceptable level of service but enable them to avoid significant capital waste since there are limitations on the budget (France-Mensah and O'Brien 2018). There is a need for prediction models for developing an effective maintenance plan. Many budget allocation models have been proposed to address this issue for various purposes, both on the project and network level (Yu, Gu et al. 2015). There are lots of studies that have been conducted in the field of pavement performance forecast in recent years using the Long-Term Pavement Performance (LTPP) data (Piryonesi andEl-Diraby 2020, Yamany, Saeed et al. 2020) or Local data sets (Yao, Dong et al. 2019). The cost-effectiveness of methods, whether in data collection (Piryonesi and El-Diraby 2020) or maintenance framework, is essential because, in the case of data gathering, it should be practical for small firms to be cost-effective and in case of maintenance framework, it can prevent budget waste.
Engineers have been attempting to determine which maintenance techniques yield the best result. Many researchers used the notion of "Cost-effectiveness" to evaluate the efficiency of their methods. Torres-Machi et al (Torres-Machi, Pellicer et al. 2017) and Osorio et al (Osorio-Lird, Chamorro et al. 2018) employed cost-effectiveness or cost-benefit techniques extensively to examine both performance and economy. In order to make better decisions that are cost-effective and more environmentally friendly, pavement repair must be planned over a lengthy period (Alam, Hossain et al. 2020, Huang, Dong et al. 2021. Combining treatments and developing more 2 predictable schedules would considerably increase pavement service life and save a significant amount of capital (Irfan, Khurshid et al. 2012).
Another important parameter that must be considered is treatment timing. Determining a maintenance schedule aims to keep the pavement in good working order while saving money on maintenance. Much research has focused on estimating maintenance timing based on long-term pavement performance deterioration (Donev and Hoffmann 2020, Li, Feng et al. 2021, Qiao, Du et al. 2021, Renard, Corbett et al. 2021.
Numerous research has attempted to combine machine learning and civil engineering to tackle various engineering challenges, motivated by machine learning's incredible capacity to employ models built on data in order to replicate human intelligence activities (Neves, González et al. 2017, Chopra, Sharma et al. 2018, Deka 2019, Naranjo-Pérez, Infantes et al. 2020. Deep learning is the most prevalent among them and is commonly used in damage recognition, crack detection, and prediction , Athanasiou, Ebrahimkhanlou et al. 2020, Azimi and Pekcan 2020, Jiang and Zhang 2020, Piryonesi and El-Diraby 2020 . A Recent approach for finding the best treatment timing is through maximizing cost-effectiveness with reinforcement learning (Yao, Dong et al. 2020, Renard, Corbett et al. 2021. In table below, a summary of previous research on AI application in infrastructure management is presented in table 1. A machine learning approach known as reinforcement learning (RL) might be used to develop better maintenance techniques to solve the problems, that is, to deal with the long-term maintenance decision-making (Yao, Dong et al. 2020). Pavement authorities are now focusing more on the long-term cost-effectiveness of maintenance treatments, making maintenance planning a sequential decision-making issue. However, whether employing either professional expertise or mathematical programming approaches, long-term maintenance planning is problematic due to the complex process of pavement deterioration, multiple alternative treatments, and various pavement performance indicators. RL is an excellent tool for simulating sequential decision-making situations on a big scale. It discovers the best approach by maximizing the agent's cumulative rewards from the environment. RL-based pavement management decision-making techniques can include sophisticated pavement performance prediction, more specific treatment techniques and materials comparison, and numerous pavement treatment applications in a longterm optimization through agent-action environment interactions, as opposed to traditional pavement management decision-making techniques (Liu, Chen et al. 2020, Han, Ma et al. 2021, Renard, Corbett et al. 2021 RL has been widely employed in civil engineering to solve transportation issues. Its use in traffic signal control to reduce vehicle delays is a good example (Zolfpour-Arokhlo, Selamat et al. 2014, Zhang, Mao et al. 2015, Mannion, Duggan et al. 2016, Aslani, Seipel et al. 2018, Müller, Rangras et al. 2021. Moreover, RL has been used in route planning, traffic congestion alleviation and the traffic flow optimization problem (Walraven, Spaan et al. 2016, Peng, Xi et al. 2021, Zhu, Lu et al. 2021. To balance the distribution of passenger demand along a metro line, Jiang et al executed a coordinated passenger inflow management during peak hours. However, there has been less research that has applied RL in pavement management decision-making. This study's significant contribution may be summarized as follows: (a) for the first time, this paper is developing an RL framework that considers costs and environmental effects at the same time, and (b) developing an RL framework based on a public database (LTPP) is the first one in its kind. We used the LTPP database, as it is the most comprehensive database for road maintenance practices by now. Although there are many deficiencies in data, like missing data and irrational changes, we used some methods to tackle the problems, which will be discussed later in the data preparation section.
Following this introduction, section 2 introduces the approach we took in this paper. Section 3 discusses the proposed approach's result, including future maintenance planning, costs, and behavior of the approach under different conditions. Section 4 concludes this study and guides further research.

Deep reinforcement learning
In this paper, we aim to find a cost and environmentally-effective sequence of maintenance actions to provide the most budget-efficient set of maintenance actions and contribute to a more sustainable maintenance management system.
Initially, some basic principles of Deep reinforcement learning are briefly introduced. Reinforcement learning is a branch of machine learning algorithms specialized in solving sequential decision-making problems to allow software-defined agents to learn the optimum behaviors in a virtual environment to achieve their objectives (Andrew Barto 2014).
Reinforcement Learning differs from other categories in Machine Learning, considering the kind of feedback (reward) that the agent (algorithm) receives after making a decision. In supervised learning, the agent will simply understand how good it was in predicting (doing an action) by immediately comparing its result with the label (Cunningham, Cord et al. 2008), and the agent uses this method to improve its next prediction (action). For unsupervised learning, there is no feedback as the data is unlabeled (Ghahramani 2003). Reinforcement learning is something in between, which works with the term delayed reward, and agents are learning through a trial-anderror procedure of interacting with the environment. It starts with an initial state, and every time the agent makes a decision, it receives a reward and changes the state to another one. The goal is for the agent to find a Policy that leads to the maximum reward. As RL works on the Markov Decision Process (MDP) principles, we introduce our sustainable road maintenance decisionmaking system based on the MDP system (Van Otterlo and Wiering 2012). is a scalar discount factor. In a model-free prediction, T is always unknown and is thus omitted afterwards in this paper.
The best (optimal) policy for RL is one that maximizes the cumulative reward. At each step in policy π the agent at one specific state t s takes an action t a to observe the next resulting state 1 t s + and obtains a reward 1 t r + . The final cumulative reward in step t is defined by t G and can be defined as follow: n is the number of steps. As mentioned in Defenition2.1,  is called discount factor, and it plays a key role in "delayed reward" phenomenon. The more  is closed to 0, the less important are future rewards, and the agent focuses on immediate rewards (Sutton 1992). Achieving an optimal policy *  means that our agent has succeeded in finding the optimal sequence of actions in each time step which maximizes t G .

5
Before introducing different ways of finding *  , we first define our sustainable maintenance planning system based on MDP, thus defining Environment, states, actions and rewards in our proposed system ( Figure 2).

Environment
The agent's Environment is the world it moves and interacts with. The Environment receives the agent's current state and action as inputs and delivers the reward and the agent's future state as output.
In our sustainable maintenance planning system, the Environment should be a place that simulates the initial road condition and then determines the following condition of the road based on the chosen action. In a perfect world, the agent can perform its action on an actual road segment and quickly evaluate the post-action condition of the road to evaluate the effectiveness of the action. However, these real-world solutions and even their laboratory simulations in civil engineering situations are not practical; they are complex and time-consuming.
To overcome this problem, Machine Learning models are used to simulate the Environment and estimate future road conditions. This paper uses a Deep Artificial Neural Network (DANN) to estimate the following condition of a road section based on some initial (input) features. This ANN model is used further in RL to play the role of the Environment.
There are 40 factors used as the inputs for the ANN model, which are related to road structure features (24 factors), traffic conditions (6 factors), weather conditions (4 factors), initial performance indicator (1 factor), the interval between two indicator evaluations (1 factor), 6 maintenance type (3 factors) and maintenance date (1 factor). As we have considered two indicators in our Sustainable Maintenance Planning Model, two ANNs are used to predict each indicator separately. The Flowchart of these ANN models and their performance are illustrated in Figure 3 and 4, respectively.

Data
Long-term pavement performance (LTPP) data were employed in this paper. The following indicators for flexible pavements are presented in the dataset: 1) International Roughness Index (IRI), 2) Rutting Depth (RD), 3) Cracking Metric (CM) and 4) Faulting (F). However, several studies developed methods to create indicators that are not initially in the database (Piryonesi and El-Diraby 2020). Although the LTPP database is probably the most comprehensive dataset of road assets, there are deficiencies, including 1) missing data and 2) conflicting data. In order to deal with these shortcomings, several approaches have been employed to improve the quality of the data set, which will be discussed in the data preparation part. In addition, Based on the research conducted by (Lupia and Alter 2014), the transparency of the data is important; thus, it is worth 7 mentioning that the research in this paper is conducted based on an open-source dataset (as opposed to previous similar work), that may enhance the validity of the proposed methods.

Indicator Selection
Selecting a single indicator will result in inaccuracies since it cannot offer a comprehensive picture of road conditions. As a result, a collection of indicators is required to depict the whole situation accurately. As we previously noted, choosing the right indicator has a significant impact on the outcomes and may turn a credible, extensive, and complete maintenance system into a meaningless one. Different research has been considered in the process of selection (Domitrović, Dragovan et al. 2018, Marcelino, Lurdes Antunes et al. 2018, Visintine, Simpson et al. 2018, Fakhri and Dezfoulian 2019, Nabipour, Karballaeezadeh et al. 2019, Li, Liu et al. 2020, Cruz, Mendoza et al. 2021, Fakhri, Dezfoulian et al. 2021, Song, Thatcher et al. 2021. Based on previous research, the performance indicators should have the following characteristics:

Appropriateness
Is suitable as a measure at the national level (including accuracy, timeliness, consistency, and precision) Based on these considerations, three indicators had been selected initially, International Roughness Index (IRI), Cracking Metric (CM), Rutting Depth (RD). Later in the modeling for RL, the authors of this paper realized that incorporating the Cracking Metric as an indicator, due to much fewer data, will result in irrational final results (maintenance and rehabilitation practices). Thus, it has been omitted from consideration for the RL model.

Data Preparation
A major and time-consuming task for academic research papers dealing with big databases is the data preparation part (García, Luengo et al. 2015). As the required dataset for training the nextcondition predictor model described in the above section is based on some separate data sources (LTPP database and climate module MERRA provided by NASA), these data need to be first merged into a single one. Data preprocessing is also a must-do for our neural network prediction model (Environment). Without preprocessing of raw data, the neural network will not produce accurate forecasts.

8
The main tasks of the preprocessing part are as follows: First, the categorical data (e.g., road function class, Surface type, etc.) are converted into numerical ones, using the one-hot encoding method. Second, a Min-Max Normalizer is applied to the data for faster convergence and higher proficiency (Patro and Sahu 2015). Furthermore, considering traffic and weather data, there are some missing data, which can be handled by interpolation, fillna methods and deletion. In addition, because of some illogical cases in the dataset, IRI and RD might have decreased over time without any actions taken. To overcome this problem, a calibration method has been applied on those data points that have decreased between two action-performed dates (if two maintenance actions have been done on a single road segment, its indicator performance must increase gradually in this interval. Otherwise, it is erroneous and must be calibrated). The calibration process is illustrated in Figure 5, in which The blue squares are actual data. The red vertical lines show the time when a maintenance action has been done. In a rational situation, the performance indicator like RD decreases right after a maintenance action but then increases. Thus, those data points that decrease without performing maintenance action are wrong and must be calibrated. In this figure, the calibrated data points are indicated with green cross-shapes.

State (S)
A state is a situation that the agent knows about itself in the environment. In this paper, 19 features combined with the performance indicators build up the state space. These features are presented in Table 3. This set of features should include those that will change during the maintenance horizon. They are separated into three categories: 1) Structure, 2) Traffic, and 3) Climate.

Action
All motions that the agent can take are known as Action (A). Although an action is relatively selfexplanatory, agents typically pick from a list of discrete, viable acts. Our actions are the maintenance and rehabilitation that are practiced on the road and are also used as the inputs in the neural network prediction model. In order to make the developed models more realistic, we chose the actions based on their frequency in the data set (these sets of actions represent more than 80% of the actions practiced in the data set). "Do-nothing" is also considered to be an action. Different actions have been made through combinations of types, thicknesses and materials. The reason behind selecting these features to make different combinations is to reach the highest accuracy with the least computation. Table 4 represents actions that the agent can take. With the combination of different types, thicknesses and materials, the Action space would be built from which the agent can perform its action.

Reward
A reward is feedback that we use to determine if an agent's activities in a particular state were successful or unsuccessful. The purpose of incentives is to offer feedback on an RL model's actions' performance. As a result, it is critical to define the reward accurately to direct the learning process properly. We proposed the reward in this paper based on the idea in . However, our reward function differs from the previous work in several terms, including the environmental cost. Figure  6 represents the calculation of actions' performance on an indicator, in which the initial point is the first year of planning. The purple curve shows the indicator performance after taking an action. The blue dotted area is the area that is used in the reward calculation formula, and the black curve is a hypothetical curve of not doing an action at all during the planning horizon. When the planning starts, the indicator is in an initial state. However, each year it decreases if no maintenance action is performed on the road, and it follows the black curve. The first time an action is performed (purple curve), it increases and separates from the Non_action curve. Generally, the performance will decrease between two maintenance actions (time t and 1 t + ). It is worth noting that for the sake of simplicity, we illustrate a decreasing performance indicator in Figure 6, while the proposed ones (IRI and RD) are inherently increasing.

Figure 6. The mechanism of calculating the effectiveness of actions for an indicator
The proposed reward function is as follows: In each time step, firstly, the cumulative area bounded with Non-action curve, Action curve, current time t and the next time 1 t + is calculated for each indicator (eq 1). In eq1, i indicates the indicator. In addition, the total cumulative cost is calculated as in eq 2. In this paper, as we have considered our maintenance system to be even environmentally effective as opposed to similar research, the total cost is the sum of both an action's cost and its environmental cost. In eq2, 1.04 is for 4% discount rate as it has been largely used in the previous research. Furthermore, the environmental cost of each action is calculated based on eq 3, in which In the next step, the effective cost is calculated based on eq 4, which is the effectiveness of applying an action (including non-action) so far in the system.
The effective cost for each indicator is then combined into a single term, as shown in eq 5.  (Mnih, Kavukcuoglu et al. 2013) is an off-policy method that learns the value of the optimal policy independently of the agent's actions. It uses an ε-greedy approach to choose an action in every time step. In DQN, samples are selected randomly from a replay memory as the inputs of a DNN to produce less correlated results than selected directly from consecutive samples. According to the loss function, in each step, the q-value (the output of the proposed action) together with the obtained reward will be combined with the max q of the same network ( * q ) to form the loss as mentioned earlier, and then update the weights through a back-propagation and gradientdescent approach.
One problem that arises by having only one DNN to approximate q values is that both q -values and target * q will update in the same direction, and the correlation between the q-values and target q -values can be high. To tackle this issue, a more efficient version of DQN, Double DQN uses two separate neural networks, one as policy network with parameter  , and another as target network with parameter  . The separated network uses the same architecture as the original one and updates the weights with regard to it every t-timesteps (  θ). Thus, the loss function is modified to be:

Proximal Policy Optimization (PPO)
Despite DQN, PPO is an on-policy method that uses policy gradient approaches to estimate the optimum policy. PPO algorithm makes the use of importance sampling similar to TRPO (Schulman, Wolski et al. 2017) method and KL divergence constraints on the basis of traditional policy gradient method. In PPO, the network tends to increase the probability of sampled actions through a gradient ascent approach. The gradient estimator in TRPO is as follows: is the updated policy that derives from the old policy old   and β is some coefficient.
However, due to the complexity of TRPO in policy estimation, PPO has been introduced, which is compatible with SGD and simplifies the algorithm by removing the KL penalty and needs to make adaptive updates with its clipped surrogate objective. It also ensures that the deviation from the previous policy is relatively small. The objective is then as follows:

PPO VS DQN
It can be observed that PPO provides a better convergence and performance rate than other techniques but is sensitive to changes. On the other hand, DQN alone is unstable and gives poor convergence, hence requiring several add-ons.
In our sustainable road maintenance decision-making system, we experimented with both DQN and PPO and compared the results in order to choose the best one. Both methods were implemented, and their parameters were tuned with several trials. However, our results showed a more stable convergence of PPO compared to DQN, and PPO gained a better cumulative reward with fewer samples. Thus, the final implementation is obtained using the PPO algorithm.
The obtained hyperparameters are shown in Table 5.

Cost Calculations: Integrating LCA and LCCA
To calculate the cost-effectiveness of the actions, first, we need to calculate the cost and environmental impact of treatments. The environmental damage cost (EDC) has generally been overlooked when calculating pavement costs. To consider EDC alongside economic cost, there should be an integration between LCA and LCCA. A model provided by Tol (Tol 2005) has been utilized for the integration. After this step, we integrated the costs and environmental impacts (as marginal costs) of the M&R actions. In the following paragraphs, a discussion has been made about our approaches.

LCA
For the LCA, a tool developed by a consortium on green design and manufacturing in the University of California, Berkeley (paLATE 2.0) (Figure 7) has been used with some modifications. The tool takes user input for the design, initial construction, maintenance, equipment use, and costs for a roadway, and provides outputs for the life-cycle environmental effects and costs. In this research, only production of material, transportation and construction have been considered. Other parts are simply beyond the scope of this research. Environmental effects investigated include CO2, CH4, N2O, PM2.5, SO2, CO and NOx. All of these pollutants have been considered in LCA as an equal amount of CO2, based on the global warming potential (GWP) ( Table 6) (Jing, Bai et al. 2012). In the following parts, more details will be discussed:

LCCA
The LCCA method has been established as a standard method for assessing the monetary competitiveness of various designs. A median value for road project item costs in Texas has been considered for cost calculations.

Case study
Our case study is a hypothetical 6 lane (3 lanes in each direction) highway located in Texas. The highway is set to be in a warm and wet climate. Daily traffic conditions (AADT, ESAL, different axle counts, Truck ration) are compatible with the road structure. The highway is composed of 46 sections and a total length of 23 kilometers For ease of calculation, all road segments are considered to be 500 meters in length, and the width of all lanes is 3.66 meters (46 segments in total). The reason for selecting a hypothetical case study is to implement a comprehensive set of characteristics in the case study to compare the performance of the model under different conditions. Different conditions for the case study are defined within the limits of similar highways in the same region. Road sections were classified as "wet" because the average site precipitation was greater than 20 inches/yr. Moreover, they are classified as "freeze" since the average site freeze index was greater than 46 °Cdegree days. Figure 8 denotes the learning process for the agent, which takes thousands of iterations. Each iteration or round is a set of 20 years of performing action on a road segment. As it can be seen the model converges after iteration 7000.  Figure 9 shows the projected maintenance and rehabilitation costs of all segments between 2022 and 2041. It is important to note that more budget is needed in some years. This is because the road is in an excellent overall condition in the early years, and thus, there is no need for heavy M&R practices. After several years that the road has been subjected to heavy traffics, the need for heavy M&R practices increased. Since after the heavy M&R, the condition of the road improves, there is a sharp decline in budget allocation. In Figure 10, which is only for a single road segment, it is obvious that in some years, there is no M&R, which happens after heavy rehabilitation practices. This means that the model can decide that there is no need for more M&R practices after recent rehabilitation.  Table 7 shows the percent of actions for each year. Since the road is in good condition at the start of 2022 (the beginning of the 20-year period), 60 percent of the first year's actions are non-actions. Moreover, the other 40 percent is a slight maintenance practice (aggregate seal coat). After the first year, the road will be subjected to heavy traffic, and the need for heavy M&R practices is necessary.

Result
In order to see if the model responds logically to different conditions, a sensitivity analysis has been conducted. Figure 11 shows that as the traffic volume increases (by 50, 100, 150, 200, and 250 percent), the need for maintenance and rehabilitation increases, which results in a drop in the cost-effectiveness of the model. It can be concluded that the developed model responds logically.

CONCLUSION
This study introduced a new framework for long-term rehabilitation and road maintenance strategies. A PPO model has been utilized to maximize the cost-effectiveness of actions. This tool can provide a basis to help decision-makers. The result shows that the proposed framework can successfully produce promising results. We also tested a DQN model, but PPO results were more accurate and logical. After each intervention, all lanes must reach the same elevation. Since the real-time practice for measuring the impact of traffic and M&R practices is not achievable, we used an ANN model to simulate the real-world condition. The environment comprises 40 parameters divided into four subcategories of 1) Structure, 2) Traffic, 3) PIs and, 4) Climate. The action space is made up of 32 actions (including do-nothing) which only 16 of them are taken by the agent. The main features of the study are as follows: 1) The proposed framework solves the problem of conflicting data by correcting the false data. 2) A neural network model specifically created to simulate the environment. 3) A sensitivity analysis was conducted to evaluate the proposed model's behavior. 4) The model integrates the environmental and economic impact of M&R practices to produce a more comprehensive basis for decision-makers. 5) the outcome of the research is a decision support tool for organizations to go towards M&R strategies that align with sustainable development goals.
Future studies could address the impact of climate change and evaluate how different climate change scenarios can affect the M&R strategies. Another direction can be pushing models toward a network of roads instead of considering one segment of road. Incorporating budget restrictions into the model will lead to more realistic results since there are always funding restrictions on the side of organizations responsible for M&R practices in the real condition. Finally, implementing the performance indicators in the reward calculation, so that different service levels have different values in the reward will lead to a more logical behavior of the model.