We employ DQN as the optimization policy \(\pi (s_t, a_t)\)  to update all the five Q-values (corresponding to the five candidate strategies in \(a_t\) ) at each step.
The structure of each tree \(s_t\)  is stored as a matrix: the raw data represents each node of the tree, and the column data represents elements of each node.
We follow the standard assumption that delayed rewards are discounted by a factor \(\gamma \)  at each time step. Action-value function \(Q(s_t, a_t)\)  is defined as the expected return based on model structure \(s_t\)  and updating strategy \(a_t\) . The optimal action-value function \(Q^{\prime }(s_t, a_t)\)  has the maximum expected return by the optimal policy.
However, the state space in our framework is enormous because tree structures are diverse with different elements of nodes. Thus, we use the Bellman equation [1]} to estimate \(Q^{\prime }\)  for each state-action pair.
We utilize deep neural networks with parameters \(\theta \)  as function approximator, i.e., \(Q(s_t,a_t;\theta ) \approx Q^{\prime }(s_t,a_t)\)  . The Q-network is trained by minimizing a loss function:
\(L(\theta ) = {{\rm E}_{{s_t},{a_t},{r_t},{s_{t + 1}}}}[{(l - Q(s_t,a_t;\theta ))^2}],\) 
