Hermione hadn't shown up, which Draco saw as a good thing. He couldn't handle confronting her as well. He let Ron vent his rage in the garden. When he ran out of voice and was forced to leave with his brothers, hours had already passed.