He and Riddle were enemies, and that’s why they had to spend so much time together: to make sure they didn’t break the terms of their deal, which was the only thing keeping the peace between them. Harry was keeping an eye on Riddle, for the good of the Wizarding world, and Riddle was exploiting Harry’s knowledge of the future.