Tom knew he was a ghost the entire time, which is why, despite seemingly falling for Kate as she was for him, he kept her at an arm’s distance. All the things that Tom did with Kate, such as ice skating and walking around London, Kate was seemingly doing “alone” — only she could see Tom. This is why no one else directly interacted with Tom, why Tom always wore the same outfit, and why he seemingly avoided bumping into anyone on the street.