In order to clarify the purpose of AutoRL, assume Traveling Salesman Problems (TSP) as an example. Recently, DRL has been popular for learning heuristics and solving different variants of TSP. The first step in solving TSP with DRL is to model the problem as a sequential decision making and determine MDP components. Vanilla TSP is defined as finding a tour with minimum length in a graph where all the nodes are visited exactly once. In a constructive solution approach, an agent starts from the source and walks through nodes until building a tour. At each timestep, the graph, current node, visited nodes and remained nodes are the observations of an environment. There are several ways like graph neural networks [1]} and structure2vec [2]} to convert the observations to state representation and AutoRL helps to find the best approach among the possible methods for this conversion. The second step is to define a RL algorithm for updating the policy network. Popular RL algorithms such as A2C [3]}, PPO [4]}, ACER [5]} and DQN [6]} are available to train the policy network. Each of them is useful for particular problems and it is not easy to determine the best algorithm for a problem. AutoRL aims to provide the opportunity of searching among these possible algorithms to find the suitable one. We focus on model-free RL algorithms where the transition probabilities between states are unknown. The third step is to set the hyper-parameters such as network architecture, learning rate, discount factor, etc. AutoRL employs hyper-parameter optimization methods in its framework to derive the optimal hyper-parameters for an algorithm and a set of problem instances.
