Deniable encryption

In cryptography and steganography, plausibly deniable encryption describes encryption techniques where the existence of an encrypted file or message is deniable in the sense that an adversary cannot prove that the plaintext data exists.
The users may convincingly deny that a given piece of data is encrypted, or that they are able to decrypt a given piece of encrypted data, or that some specific encrypted data exists. Such denials may or may not be genuine. For example, it may be impossible to prove that the data is encrypted without the cooperation of the users. If the data is encrypted, the users genuinely may not be able to decrypt it. Deniable encryption serves to undermine an attacker's confidence either that data is encrypted, or that the person in possession of it can decrypt it and provide the associated plaintext.
Function.
Deniable encryption makes it impossible to prove the existence of the plaintext message without the proper decryption key. This may be done by allowing an encrypted message to be decrypted to different sensible plaintexts, depending on the key used. This allows the sender to have plausible deniability if compelled to give up their encryption key.
The notion of "deniable encryption" was used by Julian Assange and Ralf Weinmann in the Rubberhose filesystem and explored in detail in a paper by Ran Canetti, Cynthia Dwork, Moni Naor, and Rafail Ostrovsky in 1996.
Scenario.
Deniable encryption allows the sender of an encrypted message to deny sending that message. This requires a trusted third party. A possible scenario works like this:
Another possible scenario involves Alice sending the same ciphertext (some secret instructions) to Bob and Carl, to whom she has handed different keys. Bob and Carl are to receive different instructions and must not be able to read each other's instructions. Bob will receive the message first and then forward it to Carl.
Forms of deniable encryption.
Normally, ciphertexts decrypt to a single plaintext that is intended to be kept secret. However, one form of deniable encryption allows its users to decrypt the ciphertext to produce a different (innocuous but plausible) plaintext and plausibly claim that it is what they encrypted. The holder of the ciphertext will not be able to differentiate between the true plaintext, and the bogus-claim plaintext. In general, decrypting one ciphertext to multiple plaintexts is not possible unless the key is as large as the plaintext, so this is not practical for most purposes. However, some schemes allow decryption to decoy plaintexts that are close to the original in some metric (such as edit distance).
Modern deniable encryption techniques exploit the fact that without the key, it is infeasible to distinguish between ciphertext from block ciphers and data generated by a cryptographically secure pseudorandom number generator (the cipher's pseudorandom permutation properties).
This is used in combination with some decoy data that the user would plausibly want to keep confidential that will be revealed to the attacker, claiming that this is all there is. This is a form of steganography.
If the user does not supply the correct key for the truly secret data, decrypting it will result in apparently random data, indistinguishable from not having stored any particular data there.
One example of deniable encryption is a cryptographic filesystem that employs a concept of abstract "layers", where each layer can be decrypted with a different encryption key. Additionally, special "chaff layers" are filled with random data in order to have plausible deniability of the existence of real layers and their encryption keys. The user can store decoy files on one or more layers while denying the existence of others, claiming that the rest of space is taken up by chaff layers. Physically, these types of filesystems are typically stored in a single directory consisting of equal-length files with filenames that are either randomized (in case they belong to chaff layers), or cryptographic hashes of strings identifying the blocks. The timestamps of these files are always randomized. Examples of this approach include Rubberhose filesystem and PhoneBookFS. 
Another approach used by some conventional disk encryption software suites is creating a second encrypted volume within a container volume. The container volume is first formatted by filling it with encrypted random data, and then initializing a filesystem on it. The user then fills some of the filesystem with legitimate, but plausible-looking decoy files that the user would seem to have an incentive to hide. Next, a new encrypted volume (the hidden volume) is allocated within the free space of the container filesystem which will be used for data the user actually wants to hide. Since an adversary cannot differentiate between encrypted data and the random data used to initialize the outer volume, this inner volume is now undetectable.
LibreCrypt and BestCrypt can have many hidden volumes in a container; TrueCrypt is limited to one hidden volume.
Detection.
The existence of hidden encrypted data may be revealed by flaws in the implementation. It may also be revealed by a so-called 'watermarking attack' if an inappropriate cipher mode is used.
The existence of the data may be revealed by it 'leaking' into non-encrypted disk space where it can be detected by forensic tools.
Doubts have been raised about the level of plausible deniability in 'hidden volumes' – the contents of the "outer" container filesystem have to be 'frozen' in its initial state to prevent the user from corrupting the hidden volume (this can be detected from the access and modification timestamps), which could raise suspicion. This problem can be eliminated by instructing the system not to protect the hidden volume, although this could result in lost data.
Drawbacks.
Deniable encryption has been criticized because it does not defend users from revealing keys under coercion or torture. Possession of deniable encryption tools could lead attackers to continue torturing a user even after the user has revealed all their keys, because the attackers could not know whether the user had revealed their last key or not.
Deniable authentication.
Some in-transit encrypted messaging suites, such as Off-the-Record Messaging, offer deniable authentication which gives the participants plausible deniability of their conversations. While deniable authentication is not technically "deniable encryption" in that the encryption of the messages is not denied, its deniability refers to the inability of an adversary to prove that the participants had a conversation or said anything in particular.
This is achieved by the fact that all information necessary to forge messages is appended to the encrypted messages – if an adversary is able to create digitally authentic messages in a conversation (see hash-based message authentication code (HMAC)), he is also able to forge messages in the conversation. This is used in conjunction with perfect forward secrecy to assure that the compromise of encryption keys of individual messages does not compromise additional conversations or messages.