Q:

Any example code of REINFORCE algorithm proposed by Williams?

Does any one know example of an Algorithm Williams proposed in Paper "A class of gradient-estimating algorithms for reinforcement learning in neural networks" http://incompleteideas.net/sutton/williams-92.pdf

A:

From David Silver's RL lecture on Policy Gradient methods, slide 21 here is pseudo-code for the episodic Reinforce algorithm, which basically is a gradient-based method where the expected return is sampled directly from the episode (as opposed to estimating it with some learned function). In this case the expected return is actually the total episodic reward onward that step, $G_t$.
initialise $\theta$
for each episode {$s_1, a_1, r_2 ... s_{T-1}, a_{T-1}, r_T$} sampled from policy $\pi_\theta$ do
    for t = 1 to T - 1 do
        $\theta \leftarrow \theta + \alpha \nabla_\theta \log \pi_\theta(s_t,a_t) G_t$
    end for
end for
This algorithm suffers from high variance because the sampled rewards can be very different from one episode to another therefore this algorithm is usually used with a baseline substracted from the policy. Here is a more detailed explanation complete with code samples.

