Renly had the iron throne in his grasp until a shadow assassin killed him in the night. There were many rumours, yes, but Brienne of Tarth's claim of a shadow wearing the face of Stannis slaying her former liege made Varys easily connect the dots. Only a shadowbinder could spawn such a foul fiend, and Stannis famously had a red priestess from Asshai in his service, which was the cradle of the vilest of sorcerers. A witch that completely matched the description of the woman in front of him. Oh, Varys hated magic with a passion ever since he was unmanned, and he went to great lengths to know how to know, possibly identify, and avoid them. He had even bitterly dived into the sparse records and rumours of their unnatural capabilities to know what to watch out for should he meet another sorcerer.