Ever since the Battle of Hogwarts he hadn’t returned to Hogwarts. Harry wasn’t there when they fixed Dumbledore’s grave, wasn’t there when they installed his portrait. Although he hadn’t flat out refused to set foot, it was pretty clear he was reluctant to.