No, the Twins could remain. Rickon could smash it with laughable ease now if he wanted to, but it would expose his presence here for all to see, and his travel as a nameless hedge knight would be cut short before it even began. The Crossing could remain standing for another seven moons.