<commit_before>
DFU File Format - Metadata Store Proposal
=========================================

Introduction
------------

The DFU specification version 1.1 defines some target-specific data that can
optionally be included in the DFU file to ease firmware deployment.
These include items such as the runtime vendor, product and device release,
but nothing else and with no provision for extra metadata.

The DFU file specification does not specify any additional data structures
allowing vendors to include additional metadata, although does provide the
chance to include future additional data fields in the header in a backwards and
forwards compatible way by increasing the `header_len` value.

All software reading and writing DFU-format files should already be reading the
footer length from the file, rather than assuming a fixed footer length of 0x10
bytes.
This ensures that only the raw device firmware is sent to the device and not any
additional data fields added in future versions of the DFU specification.

There are valid reasons why we would want to add additional metadata into the
distributed DFU file. Reasons are listed as follows:

 * Legal compliance, to tag a file with copyright and licensing information
 * Business-specific metadata, for instance the SHA-1 git commit for the source
 * Cryptographic information such as a SHA-256 hash or a detached GPG signature

Although the original authors of the specification allowed for future additions
to the specification, they only allowed us to extend the footer by 239 bytes as
the `header_len` value is specified as just one byte, and 16 bytes are already
specified by the specification.

This would explain why some vendors are using vendor-specific file prefix data
segments, for instance the DfuSe prefix specification from ST.
This specification is not aiming to expand or standardize the various
incompatible vendor-specific prefix specifications, but tries to squeeze the
additional metadata into the existing DFU footer space which is compatible with
all existing DFU-compliant software.

Specification
-------------

An additional structure would be present after the binary firmware data, and
notionally contained within the DFU footer itself, although specified as a
seporate object.

The representation in memory and on disk would be as follows:

    uint16      signature='MD'
    uint8       number_of_keys
    uint8       key(n)_length
    ...         key(n) (no NUL)
    uint8       value(n)_length
    ...         value(n) (no NUL)
    <existing DFU footer>

If there are no metadata keys being set, it is expected that the metadata table
signature is not be written to the file, and that the footer should be again
0x10 bytes in length.

The signature of `MD` should also be checked before attempting to parse the
metadata store structure to ensure other vendor-specific extensions are not
already in use.

Conclusions
-----------

The metadata store proposal allows us to store a small amount of metadata
inside the DFU file footer.
If the original specification had included just one more byte for the footer
length (for instance a `uint16`, rather than a `uint8`) type then I would have
proposed a key type allowing integers, IEEE floating point, and strings, and
also made the number of keys and the length of keys much larger.
Working with what we've been given, we can support a useful easy-to-parse
extension that allows us to solve some of the real-world problems vendors are
facing when trying to distribute firmware files for devices that support
in-the-field device firmware upgrading.<commit_msg>trivial: Add actual example in the DFU metadata spec document<commit_after>
DFU File Format - Metadata Store Proposal
=========================================

Introduction
------------

The DFU specification version 1.1 defines some target-specific data that can
optionally be included in the DFU file to ease firmware deployment.
These include items such as the runtime vendor, product and device release,
but nothing else and with no provision for extra metadata.

The DFU file specification does not specify any additional data structures
allowing vendors to include additional metadata, although does provide the
chance to include future additional data fields in the header in a backwards and
forwards compatible way by increasing the `header_len` value.

All software reading and writing DFU-format files should already be reading the
footer length from the file, rather than assuming a fixed footer length of 0x10
bytes.
This ensures that only the raw device firmware is sent to the device and not any
additional data fields added in future versions of the DFU specification.

There are valid reasons why we would want to add additional metadata into the
distributed DFU file. Reasons are listed as follows:

 * Legal compliance, to tag a file with copyright and licensing information
 * Business-specific metadata, for instance the SHA-1 git commit for the source
 * Cryptographic information such as a SHA-256 hash or a detached GPG signature

Although the original authors of the specification allowed for future additions
to the specification, they only allowed us to extend the footer by 239 bytes as
the `header_len` value is specified as just one byte, and 16 bytes are already
specified by the specification.

This would explain why some vendors are using vendor-specific file prefix data
segments, for instance the DfuSe prefix specification from ST.
This specification is not aiming to expand or standardize the various
incompatible vendor-specific prefix specifications, but tries to squeeze the
additional metadata into the existing DFU footer space which is compatible with
all existing DFU-compliant software.

Specification
-------------

An additional structure would be present after the binary firmware data, and
notionally contained within the DFU footer itself, although specified as a
seporate object.

The representation in memory and on disk would be as follows:

    uint16      signature='MD'
    uint8       number_of_keys
    uint8       key(n)_length
    ...         key(n) (no NUL)
    uint8       value(n)_length
    ...         value(n) (no NUL)
    <existing DFU footer>

If there are no metadata keys being set, it is expected that the metadata table
signature is not be written to the file, and that the footer should be again
0x10 bytes in length.

The signature of `MD` should also be checked before attempting to parse the
metadata store structure to ensure other vendor-specific extensions are not
already in use.

The key and value fields should be parsed as UTF-8, although in the pursuit of
space minimisation ASCII values are preferred where possible.

Example
-------

The following table shows an example firmware file with the payload 'DATA' set
with vendor 0x1234, product 0xABCD and no metadata table.

    neg.
    offset  description     byte
    ----------------------------
    13      firmware'D'     0x44
    12      firmware'A'     0x41
    11      firmware'T'     0x54
    10      firmware'A'     0x44
    0f      bcdDevice       0xFF
    0e      bcdDevice       0xFF
    0d      idProduct       0xCD
    0c      idProduct       0xAB
    0b      idVendor        0x34
    0a      idVendor        0x12
    09      bcdDFU          0x00
    08      bcdDFU          0x01
    07      ucDfuSig'U'     0x55
    06      ucDfuSig'F'     0x46
    05      ucDfuSig'D'     0x44
    04      bLength         0x10
    03      dwCRC           0x52
    02      dwCRC           0xB4
    01      dwCRC           0xE5
    00      dwCRC           0xCE

The following table shows a second firmware file with the same payload but
with the addition of a metadata table with a single metadata pair of `test=val`:

    neg.
    offset  description     byte
    ----------------------------
    1f      firmware'D'     0x44
    1e      firmware'A'     0x41
    1d      firmware'T'     0x54
    1c      firmware'A'     0x44
    1b      ucMdSig'M'      0x4D
    1a      ucMdSig'D'      0x44
    19      bMdLength       0x01
    18      bKeyLen         0x04
    17      KeyData't'      0x74
    16      KeyData'e'      0x65
    15      KeyData's'      0x73
    14      KeyData't'      0x74
    13      bValueLen       0x03
    12      ValueData'v'    0x76
    11      ValueData'a'    0x61
    10      ValueData'l'    0x6c
    0f      bcdDevice       0xFF
    0e      bcdDevice       0xFF
    0d      idProduct       0xCD
    0c      idProduct       0xAB
    0b      idVendor        0x34
    0a      idVendor        0x12
    09      bcdDFU          0x00
    08      bcdDFU          0x01
    07      ucDfuSig'U'     0x55
    06      ucDfuSig'F'     0x46
    05      ucDfuSig'D'     0x44
    04      bLength         0x1C
    03      dwCRC           0x1B
    02      dwCRC           0x25
    01      dwCRC           0x6D
    00      dwCRC           0xF5

Conclusions
-----------

The metadata store proposal allows us to store a small amount of metadata
inside the DFU file footer.
If the original specification had included just one more byte for the footer
length (for instance a `uint16`, rather than a `uint8`) type then I would have
proposed a key type allowing integers, IEEE floating point, and strings, and
also made the number of keys and the length of keys much larger.
Working with what we've been given, we can support a useful easy-to-parse
extension that allows us to solve some of the real-world problems vendors are
facing when trying to distribute firmware files for devices that support
in-the-field device firmware upgrading.
{'fraction_non_alphanumeric': 0.04855889724310777, 'fraction_numerical': 0.02850877192982456, 'mean_word_length': 2.76010993325481, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6170036', 'n_tokens_mistral': 2619, 'n_tokens_neox': 2367, 'n_words': 1307}