Don't treat it like a trade secret, but like a commodity to trade with.
Your contract should have stated that you are only creating a design (or print job, website, whatever) for the client, and that you explicitly are not surrendering source files.