<commit_before># Saltpack Binary Signcryption Format

The regular encryption format provides privacy and authenticity using long-term
asymmetric keys, but sometimes that's not possible or practical:

- We might want to encrypt for a key that's shared by many recipients. Doing
  that retains privacy, but it means that the encryption step no longer
  provides authenticity. (Any of the recipients could forge the message,
  because the MAC is no longer pairwise.)
- We might want to encrypt for future recipients who haven't published any keys
  yet, using a secret that we'll distribute later.

For these cases, we need to combine encryption for privacy with signing for
authenticity. This is what the saltpack signcryption mode provides. Note that
this is [not as
simple](http://world.std.com/~dtd/sign_encrypt/sign_encrypt7.html) as layering
the encryption mode the signing mode. Note also that this gives up the
repudiability property of the encryption mode.

## Design

At a high-level, the message is encrypted once using a symmetric key, which is
shared with all recipients. The recipients may be either public Curve25519
encryption keys, or long-term 32-byte symmetric secrets. It is then signed with
a long-term signing key belonging to the sender.

The message is chunked into 1MB chunks. A sequential nonce used for the
encryption and MAC's ensures that the 1MB chunks cannot be reordered. The end
of the message is marked with an authenticated flag to prevent truncation
attacks.

As in the encryption mode, each encrypted copy of the message encryption key is
given a recipient identifier. Unlike the encryption mode, signcryption doesn't
use the literal public key bytes as the identifier (in the case of an
asymmetric recipient), but instead an opaque identifier derived from the
recipient's key and the ephemeral sender key. This preserves recipient
anonymity by default. Applications lose the ability to make suggestions like
"your phone can't read this message, but your laptop can" based on the
recipients list, but the intention is that distributing long-term shared
symmetric keys to all of a user's devices can avoid that situation entirely.
The recipient identifiers for symmetric recipients are entirely up to the
application.

The signcryption scheme allows an anonymous sender by specifying a signing key
that is all zeroes. In this case the signature is filled with zeroes also, and
the verification step is skipped. This would be similar to using the encryption
mode with an anonymous sender and a shared asymmetric recipient key, though the
encryption mode wasn't designed with shared recipient keys in mind. (In
particular it would be very wrong to share recipient keys in the encryption
mode, if the sender was authenticated).

## Implementation

An signcrypted message is a series of concatenated [MessagePack
objects](https://github.com/msgpack/msgpack/blob/master/spec.md). The first is
a header packet, followed by one or more payload packets, the last of which is
indicated with a final packet flag.

### Header Packet
The header packet is a MessagePack list with these contents:

```
[
    format name,
    version,
    mode,
    ephemeral public key,
    sender secretbox,
    recipients list,
]
```

- The **format name** is the string "saltpack".
- The **version** is a list of the major and minor versions, currently
  `[2, 0]`. Note that saltpack version 1 did not include a signcryption mode.
- The **mode** is the number 3, for signcryption. (1 and 2 are attached and
  detached signing, and 3 is signcryption.)
- The **ephemeral public key** is a NaCl public encryption key, 32 bytes. The
  ephemeral keypair is generated at random by the sender and only used for one
  message.
- The **sender secretbox** is a
  [`crypto_secretbox`](http://nacl.cr.yp.to/secretbox.html) containing the
  sender's long-term public _signing_ key, encrypted with the **payload key**
  from below.
- The **recipients list** contains a recipient pair for each recipient key,
  including an encrypted copy of the **payload key**. See below.

A recipient pair is a two-element list:

```
[
    recipient identifier,
    payload key box,
]
```

- The **recipient identifier** depends on the type of recipient key. For an
  asymmetric recipient key, the identifier is derived from shared DH output.
  For a symmetric key, the identifier is left to the application.
- The **payload key box** is a [`crypto_box`](http://nacl.cr.yp.to/box.html)
  containing a copy of the **payload key**, encrypted with the recipient's
  public key, the ephemeral private key, and a counter nonce.

#### Generating a Header Packet

When composing a message, the sender follows these steps to generate the
header:

1. Generate a random 32-byte **payload key**.
2. Generate a random ephemeral keypair, using
   [`crypto_box_keypair`](http://nacl.cr.yp.to/box.html).
3. Encrypt the sender's long-term public key _signing_ key using
   [`crypto_secretbox`](http://nacl.cr.yp.to/secretbox.html) with the **payload
   key** and the nonce `saltpack_sender_key_sbox`, to create the **sender
   secretbox**.
4. Encrypt a copy of the **payload key** for each recipient, and create an
   identifier for each resulting secretbox. The procedure here is different for
   the two different types of recipients:

   For Curve25519 recipient public keys, first derive a shared symmetric key by
   boxing 32 zero bytes with the recipient public key, the ephemeral private
   key, and the nonce `saltpack_derived_sboxkey`, and taking the last 32 bytes
   of the resulting box. Seal the **payload key** using this derived symmetric
   key, with the nonce `saltpack_recipsbXXXXXXXX`, where `XXXXXXXX` is the
   8-byte big-endian unsigned recipient index. To compute the recipient
   identifier, concatenate the derived symmetric key and the
   `saltpack_recipsbXXXXXXXX` nonce together, and HMAC-SHA512 them under the
   key `saltpack signcryption box key identifier`. The identifier is the first
   32 bytes of that HMAC.

   For recipient symmetric keys, first derive a shared symmetric key.
   Concatenate the ephemeral public Curve25519 key and the recipient symmetric
   key, and HMAC-SHA512 them under the key `saltpack signcryption derived
   symmetric key`. The derived key is the first 32 bytes of that HMAC. Seal the
   **payload key** using this derived symmetric key, with the nonce
   `saltpack_recipsbXXXXXXXX`, where `XXXXXXXX` is the 8-byte big-endian
   unsigned recipient index. The recipient identifier in this case is up to the
   application.

5. Collect the **format name**, **version**, and **mode** into a list, followed
   by the **ephemeral public key**, the **sender secretbox**, and the nested
   **recipients list**.
6. Serialize the list from #5 into a MessagePack `array` object.
7. Take the [`crypto_hash`](http://nacl.cr.yp.to/hash.html) (SHA512) of the
   bytes from #6. This is the **header hash**.
8. Serialize the bytes from #6 *again* into a MessagePack `bin` object. These
   twice-encoded bytes are the header packet.

Encrypting the sender's long-term public key in step #3 allows Alice to stay
anonymous to eavesdroppers. If Alice wants to be anonymous to recipients as
well, she can supply an all-zero signing public key in step #3. In this case,
recipients should skip the signature verification step and indicate that the
message is from an anonymous sender.<commit_msg>avoid using the word 'seal', since it's kind of ambiguous<commit_after># Saltpack Binary Signcryption Format

The regular encryption format provides privacy and authenticity using long-term
asymmetric keys, but sometimes that's not possible or practical:

- We might want to encrypt for a key that's shared by many recipients. Doing
  that retains privacy, but it means that the encryption step no longer
  provides authenticity. (Any of the recipients could forge the message,
  because the MAC is no longer pairwise.)
- We might want to encrypt for future recipients who haven't published any keys
  yet, using a secret that we'll distribute later.

For these cases, we need to combine encryption for privacy with signing for
authenticity. This is what the saltpack signcryption mode provides. Note that
this is [not as
simple](http://world.std.com/~dtd/sign_encrypt/sign_encrypt7.html) as layering
the encryption mode the signing mode. Note also that this gives up the
repudiability property of the encryption mode.

## Design

At a high-level, the message is encrypted once using a symmetric key, which is
shared with all recipients. The recipients may be either public Curve25519
encryption keys, or long-term 32-byte symmetric secrets. It is then signed with
a long-term signing key belonging to the sender.

The message is chunked into 1MB chunks. A sequential nonce used for the
encryption and MAC's ensures that the 1MB chunks cannot be reordered. The end
of the message is marked with an authenticated flag to prevent truncation
attacks.

As in the encryption mode, each encrypted copy of the message encryption key is
given a recipient identifier. Unlike the encryption mode, signcryption doesn't
use the literal public key bytes as the identifier (in the case of an
asymmetric recipient), but instead an opaque identifier derived from the
recipient's key and the ephemeral sender key. This preserves recipient
anonymity by default. Applications lose the ability to make suggestions like
"your phone can't read this message, but your laptop can" based on the
recipients list, but the intention is that distributing long-term shared
symmetric keys to all of a user's devices can avoid that situation entirely.
The recipient identifiers for symmetric recipients are entirely up to the
application.

The signcryption scheme allows an anonymous sender by specifying a signing key
that is all zeroes. In this case the signature is filled with zeroes also, and
the verification step is skipped. This would be similar to using the encryption
mode with an anonymous sender and a shared asymmetric recipient key, though the
encryption mode wasn't designed with shared recipient keys in mind. (In
particular it would be very wrong to share recipient keys in the encryption
mode, if the sender was authenticated).

## Implementation

An signcrypted message is a series of concatenated [MessagePack
objects](https://github.com/msgpack/msgpack/blob/master/spec.md). The first is
a header packet, followed by one or more payload packets, the last of which is
indicated with a final packet flag.

### Header Packet
The header packet is a MessagePack list with these contents:

```
[
    format name,
    version,
    mode,
    ephemeral public key,
    sender secretbox,
    recipients list,
]
```

- The **format name** is the string "saltpack".
- The **version** is a list of the major and minor versions, currently
  `[2, 0]`. Note that saltpack version 1 did not include a signcryption mode.
- The **mode** is the number 3, for signcryption. (1 and 2 are attached and
  detached signing, and 3 is signcryption.)
- The **ephemeral public key** is a NaCl public encryption key, 32 bytes. The
  ephemeral keypair is generated at random by the sender and only used for one
  message.
- The **sender secretbox** is a
  [`crypto_secretbox`](http://nacl.cr.yp.to/secretbox.html) containing the
  sender's long-term public _signing_ key, encrypted with the **payload key**
  from below.
- The **recipients list** contains a recipient pair for each recipient key,
  including an encrypted copy of the **payload key**. See below.

A recipient pair is a two-element list:

```
[
    recipient identifier,
    payload key box,
]
```

- The **recipient identifier** depends on the type of recipient key. For an
  asymmetric recipient key, the identifier is derived from shared DH output.
  For a symmetric key, the identifier is left to the application.
- The **payload key box** is a [`crypto_box`](http://nacl.cr.yp.to/box.html)
  containing a copy of the **payload key**, encrypted with the recipient's
  public key, the ephemeral private key, and a counter nonce.

#### Generating a Header Packet

When composing a message, the sender follows these steps to generate the
header:

1. Generate a random 32-byte **payload key**.
2. Generate a random ephemeral keypair, using
   [`crypto_box_keypair`](http://nacl.cr.yp.to/box.html).
3. Encrypt the sender's long-term public key _signing_ key using
   [`crypto_secretbox`](http://nacl.cr.yp.to/secretbox.html) with the **payload
   key** and the nonce `saltpack_sender_key_sbox`, to create the **sender
   secretbox**.
4. Encrypt a copy of the **payload key** for each recipient, and create an
   identifier for each resulting secretbox. The procedure here is different for
   the two different types of recipients:

   For Curve25519 recipient public keys, first derive a shared symmetric key by
   boxing 32 zero bytes with the recipient public key, the ephemeral private
   key, and the nonce `saltpack_derived_sboxkey`, and taking the last 32 bytes
   of the resulting box. Secretbox the **payload key** using this derived
   symmetric key, with the nonce `saltpack_recipsbXXXXXXXX`, where `XXXXXXXX`
   is the 8-byte big-endian unsigned recipient index. To compute the recipient
   identifier, concatenate the derived symmetric key and the
   `saltpack_recipsbXXXXXXXX` nonce together, and HMAC-SHA512 them under the
   key `saltpack signcryption box key identifier`. The identifier is the first
   32 bytes of that HMAC.

   For recipient symmetric keys, first derive a shared symmetric key.
   Concatenate the ephemeral public Curve25519 key and the recipient symmetric
   key, and HMAC-SHA512 them under the key `saltpack signcryption derived
   symmetric key`. The derived key is the first 32 bytes of that HMAC.
   Secretbox the **payload key** using this derived symmetric key, with the
   nonce `saltpack_recipsbXXXXXXXX`, where `XXXXXXXX` is the 8-byte big-endian
   unsigned recipient index. The recipient identifier in this case is up to the
   application.

5. Collect the **format name**, **version**, and **mode** into a list, followed
   by the **ephemeral public key**, the **sender secretbox**, and the nested
   **recipients list**.
6. Serialize the list from #5 into a MessagePack `array` object.
7. Take the [`crypto_hash`](http://nacl.cr.yp.to/hash.html) (SHA512) of the
   bytes from #6. This is the **header hash**.
8. Serialize the bytes from #6 *again* into a MessagePack `bin` object. These
   twice-encoded bytes are the header packet.

Encrypting the sender's long-term public key in step #3 allows Alice to stay
anonymous to eavesdroppers. If Alice wants to be anonymous to recipients as
well, she can supply an all-zero signing public key in step #3. In this case,
recipients should skip the signature verification step and indicate that the
message is from an anonymous sender.
{'fraction_non_alphanumeric': 0.06090002694691458, 'fraction_numerical': 0.008488278092158448, 'mean_word_length': 4.26418439716312, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 10, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '226685', 'n_tokens_mistral': 4045, 'n_tokens_neox': 3784, 'n_words': 2136}