He is dismayed to learn from Jacelyn Bywater that many of the City Watch and the common folk consider Tyrion the problem in the city, and would revolt given the chance.
Bronn remarks, while Varys is in the room, that things would be a lot easier if Tommen were king ...