In this part, we describe how to use DQN to make the decision on the sub-band selection. Deep Q-learning (DQL), which is implemented by combining DNN and Q-learning, has been considered as one of the useful value-based off-policy DRL techniques on handling the problems with large state spaces and discrete actions [1]}. The DNN used in deep Q-learning is called DQN, which is used to estimate the action-value function. The DQL technique can be directly applied to solve the sub-band assignment problem due to the discrete feature of the sub-band. The DQN unit is shown in the left part of Fig. REF , which includes one \(Q\)  network and one target \(Q\)  network. The use of the target network is to improve the stability of DQL. Both networks have the same DNN architecture. At the beginning of the time slot \(t\) , the \(k\) -th V2V agent chooses an action \(a_k^s(t)\)  from \(\mathbf {a}_k^s(t)\)  based on the \(\epsilon \) -greedy policy and the state \(\mathbf {s}_k(t)\) . The \(\epsilon \) -greedy policy is adopted to balance the exploration of new actions and the exploitation of known actions. It indicates that the \(k\) -th agent randomly selects \(a_k^s(t)\)  with probability \(\epsilon \in (0,1)\) , or selects the action \(a_k^s(t)\)  according to the following equation with probability \(1-\epsilon \) :
\(a_k^s(t) = \mathrm {arg~max}_{a_k^s(t)\in \mathbf {\mathcal {A}}_k}[Q(\mathbf {s}_k(t),a_k^s(t),\theta )],\forall k,\) 
