Before invading, make sure you have your passive stacked so you can win the trade. If they invade you, be ready to fight, but only do so if your passive is piled up and you can win, if not just step back.