The idea of using human feedback in the RL framework to accelerate its learning speed has been explored before. There are mainly two ways of influencing the agent's learning. One modifies (`shapes') reward signals, while the other modifies the agent's policy directly.
Most methods to incorporate human feedback modify a reward signal from the environment [1]}, [2]}, [3]}. For example, Knox et al. proposed the TAMER framework, which assumes that an agent has a table for each state-action pair \(\hat{H}(s,a)\)  to accumulate the human feedback[1]}, and use it to generate the pseudo-rewards. However, it is in general difficult to come up with a reward value based on human feedback, such as “yes, it is correct” or “no, it is wrong”.
Also, recent papers have observed that a more effective use of human feedback is to directly inform on concrete policies [1]}, [6]}.
Within this remit, Griffith et al. proposed Advise[7]}, which estimates the Bayes optimum feedback policy and directly applies it to an agent's policy.
The authors show that it outperforms existing approaches and it is robust against infrequent and inconsistent feedback. Outside RL, the problem of using feedback (labels) coming from multiple trainers (or annotators in this case), has been addressed under the learning from crowds paradigm for both non-sequential [8]} and sequential data [9]}. In these works, the idea is both to obtain accurate aggregate feedback as well as estimating the reliability of the trainers.
