It simply would not do for him to legitimately attack her. His war crimes had technically been forgiven, but he would not put it past the Ministry to withdraw their allegiance should he harm the Golden Girl. Plus, the fact that he was not thrown in Azkaban did not mean he was respected and trusted by all. There were still many who would not even look at him, and he did not want to make that any worse.