Context Word: problems. 
Question: PersonX's problems don't really seem to affect PersonY that much although _ is pretty annoying. 
Answer: PersonX
PersonX's problems don't really seem to affect PersonY that much because _ is pretty calm.