Morgana became someone that he missed when she went away for long periods of time. She left him with Remus for the most part. He wasn’t a babysitter though. Remus just watched the projects that happened out of the farmhouse. Where Tom was expected to be if he was not enjoying his freedom. If anything happened he was to apparate straight there or use the portkey that was enchanted for him.