If Avery, of all people, is the person who knows Albus well enough to track him down, that gives him countless opportunities to contaminate their bond, and the moment Albus breaks the Curse, he will realise that they have become estranged.