Recent studies have shown the vulnerability of reinforcement learning (RL) models in noisy settings. The sources of noises differ across scenarios. For instance, in practice, the observed reward channel is often subject to noise (e.g., when observed rewards are collected through sensors), and thus observed rewards may not be credible as a result. Also, in applications such as robotics, a deep reinforcement learning (DRL) algorithm can be manipulated to produce arbitrary errors. In this paper, we consider noisy RL problems where observed rewards by RL agents are generated with a reward confusion matrix. We call such observed rewards as perturbed rewards. We develop an unbiased reward estimator aided robust RL framework that enables RL agents to learn in noisy environments while observing only perturbed rewards. Our framework draws upon approaches for supervised learning with noisy data. The core ideas of our solution include estimating a reward confusion matrix and defining a set of unbiased surrogate rewards. We prove the convergence and sample complexity of our approach. Extensive experiments on different DRL platforms show that policies based on our estimated surrogate reward can achieve higher expected rewards, and converge faster than existing baselines. For instance, the state-of-the-art PPO algorithm is able to obtain 67.5% and 46.7% improvements in average on five Atari games, when the error rates are 10% and 30% respectively. Designing a suitable reward function plays a critical role in building reinforcement learning models for real-world applications. Ideally, one would want to customize reward functions to achieve application-specific goals (Hadfield-Menell et al., 2017) . In practice, however, it is difficult to design a function that produces credible rewards in the presence of noise. This is because the output from any reward function is subject to multiple kinds of randomness:• Inherent Noise. For instance, sensors on a robot will be affected by physical conditions such as temperature and lighting, and therefore will report back noisy observed rewards.• Application-Specific Noise. In machine teaching tasks BID13 Loftin et al., 2014) , when an RL agent receives feedback/instructions from people, different human instructors might provide drastically different feedback due to their personal styles and capabilities. This way the RL agent (machine) will obtain reward with bias.• Adversarial Noise. Adversarial perturbation has been widely explored in different learning tasks and shows strong attack power against different machine learning models. For instance, Huang et al. (2017) has shown that by adding adversarial perturbation to each frame of the game, they can mislead RL policies arbitrarily.Assuming an arbitrary noise model makes solving this noisy RL problem extremely challenging. Instead, we focus on a specific noisy reward model which we call perturbed rewards, where the observed rewards by RL agents are generated according to a reward confusion matrix. This is not a very restrictive setting to start with, even considering that the noise could be adversarial: Given that arbitrary pixel value manipulation attack in RL is not very practical, adversaries in the real-world have high incentives to inject adversarial perturbation to the reward value by slightly modifying it. For instance, adversaries can manipulate sensors via reversing the reward value.In this paper, we develop an unbiased reward estimator aided robust framework that enables an RL agent to learn in a noisy environment with observing only perturbed rewards. Our solution framework builds on existing reinforcement learning algorithms, including the recently developed DRL ones (Q-Learning BID19 BID18 , Cross-Entropy Method (CEM) BID11 , Deep SARSA BID10 , Deep Q-Network (DQN) (Mnih et al., 2013; BID6 , Dueling DQN (DDQN) BID17 , Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al., 2015) , Continuous DQN (NAF) (Gu et al., 2016) and Proximal Policy Optimization (PPO) BID4 Only an underwhelming amount of reinforcement learning studies have focused on the settings with perturbed and noisy rewards, despite the fact that such noises are common when exploring a realworld scenario, that faces sensor errors or adversarial examples. We adapt the ideas from supervised Er |r (r) = Pr |r (r =r − )r − + Pr |r (r =r + )r + .When r = r + , from the definition in Lemma 1:Pr |r (r =r − ) = e + , Pr |r (r =r + ) = 1 − e + . Taking the definition of surrogate rewards Eqn. FORMULA2 DISPLAYFORM0 Similarly, when r = r − , it also verifies Er |r [r(s t , a t , s t+1 )] = r(s t , a t , s t+1 ).Proof of Lemma 2. The idea of constructing unbiased estimator is easily adapted to multi-outcome reward settings via writing out the conditions for the unbiasedness property (s.t. Er |r [r] = r.). For simplicity , we shorthandr(r = R i ) asR i in the following proofs. Similar to Lemma 1, we need to solve the following set of functions to obtainr: DISPLAYFORM1 whereR i denotes the value of the surrogate reward when the observed reward is R i . Define R := [R 0 ; R 1 ; · · · ; R M −1 ], andR := [R 0 ,R 1 , ...,R M −1 ], then the above equations are equivalent to: R = C ·R. If the confusion matrix C is invertible, we obtain the surrogate reward: DISPLAYFORM2 According to above definition, for any true reward level R i , i = 0, 1, · · · , M − 1, we have DISPLAYFORM3 Furthermore, the probabilities for observing surrogate rewards can be written as follows: DISPLAYFORM4 wherep i = j p j c j,i , andp i , p i represent the probabilities of occurrence for surrogate rewardR i and true reward R i respectively. Corollary 1. Letp i and p i denote the probabilities of occurrence for surrogate rewardr(r = R i ) and true reward R i . Then the surrogate reward satisfies, DISPLAYFORM5 Proof of Corollary 1. From Lemma 2, we have, DISPLAYFORM6 Consequently, DISPLAYFORM7 To establish Theorem 1, we need an auxiliary result (Lemma 3) from stochastic process approximation, which is widely adopted for the convergence proof for Q-Learning (Jaakkola et al., 1993; BID14 . Lemma 3. The random process {∆ t } taking values in R n and defined as DISPLAYFORM8 converges to zero w.p.1 under the following assumptions: DISPLAYFORM9 Here F t = {∆ t , ∆ t−1 , · · · , F t−1 · · · , α t , · · · } stands for the past at step t, α t (x) is allowed to depend on the past insofar as the above conditions remain valid. The notation || · || W refers to some weighted maximum norm.Proof of Lemma 3. See previous literature (Jaakkola et al., 1993 ; BID14 .Proof of Theorem 1. For simplicity, we abbreviate s t , s t+1 , Q t , Q t+1 , r t ,r t and α t as s, s , Q, Q , r,r, and α, respectively.Subtracting from both sides the quantity Q * (s, a) in Eqn. (3): DISPLAYFORM10 In consequence, DISPLAYFORM11 Finally, DISPLAYFORM12 Becauser is bounded, it can be clearly verified that DISPLAYFORM13 for some constant C. Then, due to the Lemma 3, ∆ t converges to zero w.p.1, i.e., Q (s, a) converges to Q * (s, a).The procedure of Phased Q-Learning is described as Algorithm 2: DISPLAYFORM14 DISPLAYFORM15 Note thatP here is the estimated transition probability, which is different from P in Eqn. FORMULA22 .To obtain the sample complexity results , the range of our surrogate reward needs to be known. Assuming reward r is bounded in [0, R max ], Lemma 4 below states that the surrogate reward is also bounded, when the confusion matrices are invertible:Lemma 4. Let r ∈ [0, R max ] be bounded, where R max is a constant ; suppose C M ×M , the confusion matrix, is invertible with its determinant denoting as det(C). Then the surrogate reward satisfies DISPLAYFORM16 Proof of Lemma 4. From Eqn. FORMULA4 , we have, DISPLAYFORM17 where adj(C ) is the adjugate matrix of C; det(C) is the determinant of C. It is known from linear algebra that, DISPLAYFORM18 where M ji is the determinant of the (M − 1) × (M − 1) matrix that results from deleting row j and column i of C. Therefore, M ji is also bounded: DISPLAYFORM19 where the sum is computed over all permutations σ of the set {0, 1, · · · , M − 2}; c is the element of M ji ; sgn(σ) returns a value that is +1 whenever the reordering given by σ can be achieved by successively interchanging two entries an even number of times, and −1 whenever it can not.Consequently, DISPLAYFORM20 Proof of Theorem 2. From Hoeffding's inequality, we obtain: DISPLAYFORM21 In the same way,r t is bounded by M det(C) · R max from Lemma 4. We then have, DISPLAYFORM22 Further, due to the unbiasedness of surrogate rewards, we have st+1∈S P a (s t , s t+1 )r t = st+1∈S;rt∈R P a (s t , s t+1 ,r t )r t .As a result, DISPLAYFORM23 In the same way, DISPLAYFORM24 Recursing the two equations in two directions (0 → T ), we get DISPLAYFORM25 Combining these two inequalities above we have: DISPLAYFORM26 For arbitrarily small , by choosing m appropriately, there always exists 1 = 2 =(1−γ) 2(1+γ) such that the policy error is bounded within . That is to say, the Phased Q-Learning algorithm can converge to the near optimal policy within finite steps using our proposed surrogate rewards.Finally, there are |S||A|T transitions under which these conditions must hold, where | · | represent the number of elements in a specific set. Using a union bound, the probability of failure in any condition is smaller than DISPLAYFORM27 We set the error rate less than δ, and m should satisfy that DISPLAYFORM28 In consequence, after m|S||A|T calls, which is, O DISPLAYFORM29 , the value function converges to the optimal one for every state s, with probability greater than 1 − δ.The above bound is for discounted MDP setting with 0 ≤ γ < 1. For undiscounted setting γ = 1, since the total error (for entire trajectory of T time-steps) has to be bounded by , therefore, the error for each time step has to be bounded by T . Repeating our anayslis, we obtain the following upper bound: DISPLAYFORM30 Proof of Theorem 3. DISPLAYFORM31 Using the CauchySchwarz inequality, DISPLAYFORM32 So we get, Var(r ) − Var(r) ≥ 0. In addition, DISPLAYFORM33
A new approach for learning with noisy rewards in reinforcement learning
{"src": "scitldr-AIC_train", "paper_id": "BkMWx309FX"}