A well-known reward shaping approach is potential-based reward shaping [1]}. Let \(r(s,a,s^{\prime })\)  be the instant reward of taking action \(a\)  in state \(s\)  and going to state \(s^{\prime }\) . Potential-based reward shaping employs a function \(F(s,a,s^{\prime })=\gamma \Phi (s^{\prime })-\Phi (s)\)  where, \(\gamma \)  is discount factor, and \(\Phi (s)\)  and \(\Phi (s^{\prime })\)  are potential functions of states \(s\)  and \(s^{\prime }\) , respectively. The new reward function is \(r(s,a,s^{\prime })+F(s,a,s^{\prime })\)  which is the sum of original reward and the potential-based reward shaping. In [2]}, potential-based reward shaping is analyzed in episodic RL. The authors show that temporal credit assignment or reward value could speed up training deep neural networks if a potential based reward function is added to the original reward. An important requirement of this work is that the policy derived by shaped reward should be equivalent to the original policy. Potential-based Reward shaping preserves the optimal policy when the goal states are predefined terminal states and shaping reward is zero in the goal states [1]}. It is shown in [2]} that policy invariance is violated for finite horizon domains with multiple terminal states and they propose to set the potential value of terminal states as zero to solve this issue.
