Oddly enough, you do have to say one thing: "I am invoking my right to remain silent." or some variant thereof. It's not enough to simply remain silent. You have to acknowledge the right and that you're taking advantage of it.