"Well?"
"If the company of fools irritates, as you say, the society of clever men leaves its own peculiar pain also. Where the goodness or talent of your friend is beyond and above all doubt, your own worthiness to be his associate often becomes a matter of question."

 the society of clever men brings its own peculiar pain also. You wonder whether you are worthy to be his associate."
