He has made sure that anyone who wants to communicate with him comes to him, never vice versa. The office at the Manor serves as his private sanctum; he can summon others at will. No one, except perhaps the Dark Lord, can summon Draco Malfoy anywhere.