Not reacting was easy, something Draco was used to. When he was put in a position where other wizards were taunting him, he just thought about Hermione. She was his happy place and one of the reasons why he was eager to finally give up his job at the Ministry. He only stayed for another year so it would not look suspicious when he finally decided to play husband full time.