One such drawback is that the encryption usually involves a cryptographic key, and management and record keeping with respect to all of the different keys is not a simple task.
Some sophisticated mathematical algorithms exist which can decipher the cryptographic information, even without advance knowledge of the cryptographic key.