Sirius took ample advantage of this, taking Harry out at any opportune moment. James was always happy to tag along, but Regulus rarely left the house. And though Harry enjoyed all of the trips, he was always eager to return home. His childhood was not spent in... isolation per se. But it was spent with very select people. He didn't much enjoy the crowds of the outside world.