He was angry at Hermione for this. She was the one who brought the Malfoys into their home. She was the one who wrote to Ginny and made her reply back such cruel words. She made Harry choose between him and Malfoy.