If he had a guest inside, Hermione didn't want to push her way in. After seeing him in the flesh with Ginny, she never wanted to come face to face with another one of his illicit affairs. The next time it happened she was afraid she wouldn't be able to calm her rage again. She still itched to slice Ginny's throat open and crucio her husband every time she recalled that day.