Everything would've been different if he desired to be the next Dark Lord. There were those who would follow him to the very top of the regime if he asked. Obsessed with power and privilege, if their former brothers and sisters from the Inner Circle believed they would be rewarded for their loyalty when the ascension was complete, they wouldn't hesitate. Considering he changed his plans to instead be nothing more than a concerned father and husband who wanted nothing to do with ruling, he had fewer friends. Where was the benefit in following a man who didn't seek or desire power? They could all end up dead for nothing. Their own families would be left unprotected to be picked over by the one who was strong enough, ambitious enough to claw their way to the top. That person, to Hermione's great relief, would not be her husband.