Because of his foolishness he could have caused the death of at least one of those under his care. He couldn't let it happen again, Stopping Tom was important but he had already crossed too many lines, broken many of his own rules. Leaving Harry with the Dursleys was one of those yet no matter how much he hated himself for it, he couldn't risk the boy.