That's a good way to injure it; you should always pick up the clam and the piece together, and be very careful when handling the two.
You should never try to pull a crocea off anything that it's attached to because you can rip the byssal organ out of it.