With the rapid development of artificial intelligence, some researchers have adopted reinforcement learning technique for solving the TSP. One category is to combine reinforcement learning or deep reinforcement learning with existing (meta) heuristics. Ant-Q ([1]} [1]}) and Q-ACS ([3]} [3]}) replace the pheromone in the ant colony algorithm with the Q-table in the Q-learning algorithm. However, the effect of Q-table is similar to that of the pheromone. Liu and Zeng ([5]}) employ reinforcement learning to construct mutation individuals in the previous version of EAX-GA ([6]} [6]}) and report results on instances with up to 2,400 cities, but their proposed algorithm RMGA is inefficient when compared with EAX-GA and LKH. The improvements of these reinforced algorithms to the original heuristics are not obvious. Another category is to apply DRL to directly solve the TSP. Bello et al. ([8]}) address TSP by using the actor-critic method to train a pointer network ([9]} [9]}). The S2V-DQN ([11]} [11]}) apply reinforcement learning to train graph neural networks so as to solve several combinatorial optimization problems, including minimum vertex cover, maximum cut and TSP. Xing et al. ([13]}) use deep neural network combined with Monte Carlo tree search to solve the TSP. It is generally hard for them to scale to large TSP instances with thousand cities, indicating that current DRL methods still have a gap to the competitive heuristic algorithms in solving the TSP.
