
PSRL-CFR: Posterior sampling for reinforcement learning with counterfactual regret minimization (PSRL-CFR) [1]} proposes a framework that combines PSRL and CFR. The environment is transformed into known by Thompson sampling, and the CFR algorithm is used to calculate the approximate Nash equilibrium. In order to update data in the environment, the special strategies \(\hat{\sigma }_{1}\)  and \(\hat{\sigma }_{2}\)  are regarded as interaction strategies.

FSP: Fictitious self-play [2]}, [3]} is a popular algorithm to find Nash equilibrium (NE) in uncertain setting. FSP needs to make use of Fitted-Q iteration (see PSRL-CFR) with initial hyperparameters to learn the best response to the average strategy of each player's opponent when the environment is unknown.

MCCFR-OS: Monte Carlo counterfactual regret minimization based on outcome sampling [4]} is a popular variant of CFR that avoids traversing the entire game-tree by sampling only a single playing of the game on each iteration. The \(\epsilon \) -greedy is the exploration strategy where \(\epsilon =0.1\) .

Variants of VCFR: In order to better measure the validity of our experimental methods, we use three additional variants of VCFR: 1) Naive: The reward without information gain is inputted into CFR procedure,and the average strategy by calculating is used as exploration policy to interact with the unknown environment; 2) Naive-DCFR: To prove the generalization of our algorithm framework, we use other variant algorithm Discounted CFR. DCFR [5]} is a variant algorithm of CFR, and it has three parameters \(\alpha \) , \(\beta \) , and \(\gamma \)  as discount factors to improve speed of solution. In every round t, the effects of three parameters are multiplying cumulative regrets \(R\)  by \(t^{\alpha }/ (t^{\alpha }+1)\)  when \(R>0\) , by \(t^{\beta }/(t^{\beta }+1)\)  when \(R<0\) , and contributions to average policy \(\bar{\sigma }\)  by \(\left(t/(t+1)\right)^{\gamma }\) . The basic process is the same as Navie-CFR, only replacing CFR with DCFR. The previous experiment has an excellent performance when setting \(\alpha =\frac{3}{2}\) , \(\beta = 0\) , and \(\gamma =2\) ; 3) Random: The players take random actions in each round.

