Machine learning has been considered as a promising technique to tackle challenges in traditional optimization methods [1]}, [2]}. Supervised learning and reinforcement learning (RL) are two widely used machine learning techniques in wireless communications. Supervised learning heavily relies on labelled data [3]}, i.e., the optimal solutions, which are not easy to obtain. Different from supervised learning, RL aims to learn a better policy on decision-making through exploration and exploitation [4]}. By combining RL with DNN, deep RL (DRL) is designed in [5]}, which extends RL to deal with the large state-action space. DRL has demonstrated its strong ability in solving a more complex Markov decision process (MDP) with a high-dimensional state-action space in wireless communications applications [6]}, especially for decision-making problems in V2X communications [7]}, [8]}. DRL has been widely adopted to address the more complex resource management problems in V2X communications scenarios [9]}, [10]}, [11]}, [12]}, [13]}, [14]}. In order to improve the performance of the next generation vehicular networks, the authors proposed a deep Q-network (DQN)-based algorithm to control the complex resources via an integrated framework [9]}. The DRL technique was applied to address the resource allocation in the Internet of Vehicle (IoV) case, which adopts the concept of Internet of Things (IoT) to vehicle communications scenarios to improve the road safety and satisfy the ubiquitous connectivity requirements [10]}, [11]}. In [10]}, the safety and QoS issues in the IoV case with the battery-powered vehicles were solved by exploiting DQN to learn an optimal mapping from the current characteristics of the underlying model to the scheduling policy. A DQN-based algorithm was designed to overcome the dynamic topology and time-varying spectrum states by learning the optimal scheduling policy in a cognitive radio-based vehicular network [11]}. The resource allocation problem, which considers the latency of V2V links and the sum rate of V2I links, was investigated in [12]} and [13]} by using the single-agent and multi-agent approaches, respectively. A centralized learning and distributed implementation are adopted to select the resources based on the proposed multi-agent RL algorithm [13]}. The decentralized DQN-based algorithm was proposed to find the optimal sub-band and power level selection in the unicast and broadcast V2X scenario [12]}. In order to enhance the reliability of safety-critical messages delivery in V2V links, a two-timescale federated DRL-based semi-decentralized algorithm was proposed to optimize the selection of transmission mode and resources in V2X communications [14]}.
