It doesn't take them long to settle on an explanation that's close to the truth: that the Grangers" visit earlier in the day had been taxing and Harry would rather be well-rested for the first visit with Sirius. He promises to make it soon, as soon as he feels settled in, and challenges Sirius to a Seeker's Game when they see each other next. It feels so dishonest, though, because he knows that he'll put off this visit for as long as he can. At the very least until he can honestly tell Sirius that he's settled in and things are good, which Harry isn't so certain is true right now.