Problem 3.1 (Causal Bandit Problem, [1]})  Consider an episodic MDP with \(K\)  actions (arms) and a single state. At the beginning of each episode, a context \(c \in [K]\)  is chosen uniformly at random and represents the correct arm for that episode. Pulling an arm leads to binary feedback: \(+\)  if the arm was correct and \(-\)  otherwise. This feedback is flipped with probability \(\epsilon _{obs} \in [0, 1]\) . The learner observes expert demonstrations where at each timestep, the expert plays the correct arm with probability \(1 - \epsilon _{exp} \in [0, 1]\)  and another arm uniformly at random otherwise. The reward function is 1 for pulling the correct arm and 0 otherwise. We emphasize that the learner does not observe the rewards, just noisy binary feedback as an observation after each pull.

