Hermione would have been happy if their relationship had come to light. She said she loved him and gossip didn't matter to her, but Draco insisted. He didn't want people to start talking about him again at every corner of the school, judging him for his past as well as his present. He wanted to be with her without any kind of pressure from the outside.