We qualitatively compare the policies learned with our mutual information constraint to an \((\epsilon , \delta )-\) differentially private Q-learning policy obtained via the DPQL algorithm described in [1]}using their implementationFound at https://github.com/wangbx66/differentially-private-q-learning.
The provided implementation only supports a one-dimensional continuous state, so we consider a reparameterization of the VPN environment described in section REF  with \(n=4\)  mirrors, mapping the eight possible states (\(u=\lbrace 0, 1, 2, 3\rbrace \) , VPN=\(\lbrace 0, 1\rbrace \) ) to eight equal sub-regions of the interval \([0, 1]\) . The inputs to DPQL are a differential privacy budget \(\epsilon \)  and a noise level \(\sigma \) . As long as \(\sigma \ge \theta \) , the mechanism is then \((\epsilon , \delta )\) -DP (in the sense described in section ) where \(\theta , \delta \)  are depend on the batch size, learning rate, Lipschitz constant of the value approximator and other parameters. We compare several values of \(\sigma \)  with \(\epsilon = 0.05\)  against our method.
