Sirius frowns, "She never mentioned it; just said he was with family. Dumbledore said his guardians were living in the muggle world, to keep Harry safe and away from all the fame." The fame—and Sirius’ own fame/infamy—had been a reason Dumbledore gave for Sirius approaching the Harry situation with caution.