<commit_before>### TMTP Protocol

"Trusted Messaging Transfer Protocol" defines a simple client/server scheme 
for reliable store-and-forward message delivery. 
It needs no other protocol in the way that POP & IMAP need SMTP. 
TMTP may be conveyed by any secure, reliable means, e.g. TCP+TLS. 
TMTP sessions are typically long-duration, and may idle for extended periods. 
After the client completes a login or register request, either side may contact the other.
A client may simultaneously contact multiple TMTP servers via separate connections. 

The rationale underlying this work is outlined in 
[_Why TMTP?_](https://mnmnotmail.org/rationale.html)
The project homepage is [mnmnotmail.org](https://mnmnotmail.org).

__Status:__ Version 1 is a work in progress, and aspects need to be revised. 
All of the elements defined herein are implemented in the 
[mnm server](https://github.com/networkimprov/mnm).

__Todo:__ Collect relevant items from the 
[todo list](https://github.com/networkimprov/mnm/blob/master/Todo.txt) into a _Future_ section.
Define types for msgid, uid, datetime, etc.

__Conventions:__ Header definitions herein use these conventions:
```
  "nameT": type,           // field gives a certain type
  "nameV": value,          // field gives a certain value
  "nameW": value | value,  // field gives one of many certain values
 ["nameA": t/v, ...        // alternative fieldsets
 |"nameB": t/v, ... ],     //   nth alternative
 <"nameO": t/v, ... >,     // optional fieldset
 <[ ... | ... ]>,          // optional alternative fieldsets
  (ref),                   // previously defined fieldset
```

### Definitions

Each message starts with a header, wherein four hex digits give the size of a JSON metadata object, 
which may be followed by arbitrary 8-bit data if `.datalen` is non-zero.
The data may begin with a secondary JSON metadata object if `.datahead` is non-zero.
```
hhhh{ ... <"datalen":uint, <"datahead":uint>> }datahead octets, datalen - datahead octets
```

Messages are not interleaved/multiplexed within a link; that should be considered for a later revision.

Link errors & timeouts cause the server to close the connection without notice. 
Protocol errors or login failure by TMTP clients cause the server to close the connection 
after emitting a quit response:
```
{ "op":    "quit",
  "error": string}   // reason the connection was closed
```

Ack responses `(ack)` from the server have the following format:
```
{ "op":     "ack",
  "id":     string,  // matches id on the message being ack'd
 <"msgid":  string,  // id assigned to the message being ack'd
  "posted": string   // datetime assigned to the message being ack'd
 |"error":  string>} // reason the message was not delivered
```

Messages from the server typically have the following required headers `(std)`:
```
  "from":    string, // uid of the sender
  "id":      string, // id assigned to the message
  "posted":  string, // datetime assigned to the message
  "headsum": uint    // checksum for the message header
```

Messages carrying arbitrary data have the following headers `(data)`:
```
  "datalen":  uint,  // count of octets following the header
 <"datahead": uint>, // count <= datalen of octets which comprise a secondary header
 <"datasum":  uint>, // checksum for datalen octets
```

Id strings generated by the server (`(ack) .msgid` and `(std) .id`) must be sequential, 
but need not be contiguous. An easy way to generate them is to contact an NTP 
(network time protocol) service on startup for the current time, calculate the number of 
nanoseconds since the epoch, and increment that value for each id string generated. 
This supports generation of up to one billion ids per second, and doesn't require a persistent 
record of the last id generated before shutdown.

### Client messages and resulting server messages

After the client completes a Login or Register sequence, either side may contact the other.

0. __TmtpRev__ gives the latest recognized protocol version; it must be the first message.
   ```
   { "op": 0,
     "id": "1"}       // protocol version string
   ```
   Response:
   ```
   { "op": "tmtprev",
     "id": "1"}       // protocol version string
   ```

0. __Register__ creates a user account with a single node.  
_todo: accept credentials for third party authentication services_  
   ```
   { "op":       1,
     "newnode":  string,     // user label for a client device
    <"newalias": string>}    // user alias, must be 8+ printable characters
   ```
   Response: same as _Login_  
   To sender's node:
   ```
   { "op":     "registered",
     "uid":    string,       // permanent id for new user
     "nodeid": string,       // password for first node
    <"error":  string>}      // reason alias was not allowed
   ```

0. __Login__ connects a client to a user node.  
_todo: response provides connection timeout period_  
   ```
   { "op":   2,
     "uid":  string,        // permanent user id
     "node": string}        // password for this node
   ```
   Response:
   ```
   { "op":   "info",
     "info": "login ok",    // todo: drop this
     "ohi":  [string, ...]} // list of uids now online
   ```
   To sender's nodes:  
   _todo: replace with messages on ohi channel?_
   ```
   { "op":   "login",
     (std),
     "node": "tbd"}
   ```

0. __UserEdit__ updates a user account to add an alias or node.  
_todo: disallow node label reuse?_  
_todo: dropnode and dropalias; prevent account hijacking from stolen client/nodeid_  
_todo: cork pauses delivery to user's nodes, params date & for-list_  
   ```
   { "op":       3,
     "id":       string,  // referenced by (ack) response
    ["newnode":  string   // user label for a client device
    |"newalias": string]} // user alias, must be 8+ printable characters
   ```
   Response: `(ack) // without .msgid or .posted`  
   To sender's nodes:
   ```
   { "op":       "user",  // these have higher priority than normal messages
     (std),
    ["newnode":  string,  // from client request
     "nodeid":   string   // password for new node
    |"newalias": string]} // from client request
   ```

0. __OhiEdit__ notifies selected contacts of a user's presence.
   ```
   { "op":   4,
     "id":   string,                  // referenced by (ack) response
     "for":  [{"id": string}, ...],   // list of uids
     "type": "add" | "drop"
           | "init"}                  // no "ohiedit" message is sent to sender's nodes
   ```
   Response: `(ack) // without .msgid or .posted`  
   To sender's nodes:
   ```
   { "op":   "ohiedit",
     (std),
     "for":  [{"id": string}, ...],   // from client request
     "type": "add" | "drop"}          // from client request
   ```
   To recipients:
   ```
   { "op":     "ohi",
     "from":   string,                // uid
     "status": 1 | 2}                 // 1 online, 2 offline
   ```

0. __GroupInvite__ performs a _Ping_, and authorizes the recipient to join a group, 
creating it if necessary.  
_todo: closed groups; invitations by moderators_  
   ```
   { "op":   5,
     "id":   string,       // referenced by (ack) response
     "gid":  string,       // group name, must be 8+ printable characters
     "from": string,       // sender alias
     "to":   string,       // invitee alias
     (data)}               // .datalen max is limited by server; data must be valid UTF-8
   ```
   Response: `(ack)`  
   To recipient:
   ```
   { "op":    "invite",
     (std),
     "gid":   string,      // from client request
     "alias": string,      // from client request .from
     "to":    string,      // from client request
     (data)}               // from client request
   ```
   To group members:
   ```
   { "op":       "member",
     (std),
     "act":      "invite",
     "gid":      string,   // from client request
     "alias":    string}   // from client request .to
   ```

0. __GroupEdit__ updates a group, allowing an invitee to join, or a member to be dropped, 
or a member to update their recorded alias.
   ```
   { "op":        6,
     "id":        string,  // referenced by (ack) response
     "act":       "join"
                | "alias"
                | "drop",
     "gid":       string,  // group name
    [<"newalias": string>  // user alias, for "join"
    |"newalias":  string   // user alias, for "alias"
    |"to":        string]} // user alias, for "drop"
   ```
   Response: `(ack) // without .msgid or .posted`  
   To group members:
   ```
   { "op":       "member",
     (std),
     "act":      string,   // from client request
     "gid":      string,   // from client request
     "alias":    string,   // last reported value
    <"newalias": string>}  // from client request
   ```

0. __Post__ sends a message to users and/or groups.
   ```
   { "op":  7,
     "id":  string,               // referenced by (ack) response
     "for": [{"id":   string,     // uid or group name
              "type": 1 | 2 | 3}, // 1 uid, 2 gid (include self), 3 gid (exclude self)
             ... ]
          | [],                   // only sender's nodes
     (data)}
                                  // .datahead segment
   { "threadid":      string,     // empty for new thread
     "alias":         string,     // user alias
     "subject":       string,     // optional if .threadid set
    <"confirmid":     string,     // id of confirmed forwarded message, see PostNotify
     "confirmposted": string>,    // posted date of confirmed forwarded message
    <"cc":                        // subscribers, omit unless .threadid empty
      [{ "who":       string,     // user alias
         "whouid":    string,     // uid
         "by":        string,     // user alias that included .who
         "byuid":     string,     // uid
         "date":      string,     // datetime when .who was included
         "note":      string,     // comment from .by
         "subscribe": bool},      // .who receives new thread messages
       ... ]>,
    <"attach":                    // file contents appear in the data segment following the message body
      [{ "name":      string,     // unique in .attach, prefix u:/f:/r:
         "size":      uint,       // count of octets
        <"ffn":       string>},   // filled-form name, omit for .name u:*
       ... ]> }
                                  // todo: filled-form attachment

   { "nodesync": true}            // .datahead segment for node sync
                                  // todo: data segment for node sync
   ```
   Response: `(ack)`  
   To recipients:
   ```
   { "op":  "delivery",
     (std),
     (data)}                      // from client request
   ```

0. __PostNotify__ sends a message to the `.for` list and a separate notification 
to the `.for` and `.notefor` lists.  
_todo: PostNotify synchronization; see Todo.txt_

   This enables a decentralized subscriber list per-thread, and forwarding a thread to new subscribers. 
The `(data)` message delivers a copy of the thread, without attachments, to new subscribers.
Recipients flag its messages as unconfirmed.
The notification delivers the list of new subscribers to current & new subscribers.
The current subscribers then Post a copy of each thread message authored by them, with attachments, 
to the new subscribers, including the `.confirmid` & `.confirmposted` fields of `.datahead`.
   ```
   { "op":         8,
     "id":         string,  // referenced by (ack) response
     "for":        [ ... ], // list of uids and/or group names, see Post
    <"fornotself": true>,   // exclude sender's nodes (normally implicit in .for list)
     "notefor":    [ ... ], // list of uids and/or group names, see Post
     "notelen":    uint,    // count of octets following the header for notification
    <"notehead":   uint>,   // analagous to .datahead for notification
    <"notesum":    uint>,   // analagous to .datasum for notification
     (data)}                // .datahead & .datasum pertain to octets following notification

                            // .notehead segment
   { "threadid": string,    // id of forwarded thread
     "cc":       [ ... ]}   // new subscribers, see Post

                            // .datahead segment
   { "threadid": string}    // id of forwarded thread
                            // todo: data segment for forwarded thread
   ```
   Response: `(ack)`  
   To recipients:
   ```
   { "op":     "delivery",
     (std),
     "notify": uint,        // count of items in .for & .notefor, including self
     (data)}                // from client request (.datalen - .notelen), .datahead, .datasum
   ```
   To notification recipients:
   ```
   { "op":     "notify",
     (std),
     "postid": string,      // (ack) .msgid
     (data)}                // from client request .notelen, .notehead, .notesum
   ```

0. __Ping__ sends a short text message via a user's alias.
On receipt, the recipient may thereafter contact the sender by uid, given in `(std) .from`.
A server may limit the number of pings and consecutive failed pings per 24h.
   ```
   { "op":   9,
     "id":   string,  // referenced by (ack) response
     "from": string,  // sender alias
     "to":   string,  // invitee alias
     (data)}          // .datalen max is limited by server; data must be valid UTF-8
   ```
   Response: `(ack)`  
   To recipient:
   ```
   { "op":    "ping",
     (std),
     "alias": string, // from client request .from
     "to":    string, // from client request
     (data)}          // from client request
   ```

0. __Ack__ acknowledges receipt of a message.
   ```
   { "op":   10,
     "id":   string, // from .id of message being ack'd
     "type": string} // unused
   ```

0. __Pulse__ resets the connection timeout.
   ```
   { "op": 11}
   ```

0. __Quit__ performs logout.
   ```
   { "op": 12}
   ```


### License

Copyright 2020 <NAME>  
Published at https://github.com/networkimprov/mnm

This Source Code Form is subject to the terms of the Mozilla Public
License, v. 2.0. If a copy of the MPL was not distributed with this
file, You can obtain one at http://mozilla.org/MPL/2.0/

<commit_msg>protocol: Add human-readable text section<commit_after>### TMTP Protocol

"Trusted Messaging Transfer Protocol" defines a simple client/server scheme 
for reliable store-and-forward message delivery. 
It needs no other protocol in the way that POP & IMAP need SMTP. 
TMTP may be conveyed by any secure, reliable means, e.g. TCP+TLS. 
TMTP sessions are typically long-duration, and may idle for extended periods. 
After the client completes a login or register request, either side may contact the other.
A client may simultaneously contact multiple TMTP servers via separate connections. 

The rationale underlying this work is outlined in 
[_Why TMTP?_](https://mnmnotmail.org/rationale.html)
The project homepage is [mnmnotmail.org](https://mnmnotmail.org).

__Status:__ Version 1 is a work in progress, and aspects need to be revised. 
All of the elements defined herein are implemented in the 
[mnm server](https://github.com/networkimprov/mnm).

__Todo:__ Collect relevant items from the 
[todo list](https://github.com/networkimprov/mnm/blob/master/Todo.txt) into a _Future_ section.
Define types for msgid, uid, datetime, etc.

__Conventions:__ Header definitions herein use these conventions:
```
  "nameT": type,           // field gives a certain type
  "nameV": value,          // field gives a certain value
  "nameW": value | value,  // field gives one of many certain values
 ["nameA": t/v, ...        // alternative fieldsets
 |"nameB": t/v, ... ],     //   nth alternative
 <"nameO": t/v, ... >,     // optional fieldset
 <[ ... | ... ]>,          // optional alternative fieldsets
  (ref),                   // previously defined fieldset
```

### Definitions

Each message starts with a header, wherein four hex digits give the size of a JSON metadata object, 
which may be followed by arbitrary 8-bit data if `.datalen` is non-zero.
The data may begin with a secondary JSON metadata object if `.datahead` is non-zero.
```
hhhh{ ... <"datalen":uint, <"datahead":uint>> }datahead octets, datalen - datahead octets
```

Messages are not interleaved/multiplexed within a link; that should be considered for a later revision.

Link errors & timeouts cause the server to close the connection without notice. 
Protocol errors or login failure by TMTP clients cause the server to close the connection 
after emitting a quit response:
```
{ "op":    "quit",
  "error": string}   // reason the connection was closed
```

Ack responses `(ack)` from the server have the following format:
```
{ "op":     "ack",
  "id":     string,  // matches id on the message being ack'd
 <"msgid":  string,  // id assigned to the message being ack'd
  "posted": string   // datetime assigned to the message being ack'd
 |"error":  string>} // reason the message was not delivered
```

Messages from the server typically have the following required headers `(std)`:
```
  "from":    string, // uid of the sender
  "id":      string, // id assigned to the message
  "posted":  string, // datetime assigned to the message
  "headsum": uint    // checksum for the message header
```

Messages carrying arbitrary data have the following headers `(data)`:
```
  "datalen":  uint,  // count of octets following the header
 <"datahead": uint>, // count <= datalen of octets which comprise a secondary header
 <"datasum":  uint>, // checksum for datalen octets
```

Id strings generated by the server (`(ack) .msgid` and `(std) .id`) must be sequential, 
but need not be contiguous. An easy way to generate them is to contact an NTP 
(network time protocol) service on startup for the current time, calculate the number of 
nanoseconds since the epoch, and increment that value for each id string generated. 
This supports generation of up to one billion ids per second, and doesn't require a persistent 
record of the last id generated before shutdown.

Human-readable text must be UTF-8. 
For _Ping_ and _GroupInvite_ messages, the text must be displayed verbatim. 
For _Post_ and _PostNotify_ messages, the text should be displayed per the 
[CommonMark](https://commonmark.org) specification, with extensions for strikethrough, 
[tables](https://help.github.com/articles/organizing-information-with-tables/), 
slide decks, charts, forms, and hyperlinks to attachments & messages. \
_todo: document mnm CommonMark extensions_

### Client messages and resulting server messages

After the client completes a Login or Register sequence, either side may contact the other.

0. __TmtpRev__ gives the latest recognized protocol version; it must be the first message.
   ```
   { "op": 0,
     "id": "1"}       // protocol version string
   ```
   Response:
   ```
   { "op": "tmtprev",
     "id": "1"}       // protocol version string
   ```

0. __Register__ creates a user account with a single node.  
_todo: accept credentials for third party authentication services_  
   ```
   { "op":       1,
     "newnode":  string,     // user label for a client device
    <"newalias": string>}    // user alias, must be 8+ printable characters
   ```
   Response: same as _Login_  
   To sender's node:
   ```
   { "op":     "registered",
     "uid":    string,       // permanent id for new user
     "nodeid": string,       // password for first node
    <"error":  string>}      // reason alias was not allowed
   ```

0. __Login__ connects a client to a user node.  
_todo: response provides connection timeout period_  
   ```
   { "op":   2,
     "uid":  string,        // permanent user id
     "node": string}        // password for this node
   ```
   Response:
   ```
   { "op":   "info",
     "info": "login ok",    // todo: drop this
     "ohi":  [string, ...]} // list of uids now online
   ```
   To sender's nodes:  
   _todo: replace with messages on ohi channel?_
   ```
   { "op":   "login",
     (std),
     "node": "tbd"}
   ```

0. __UserEdit__ updates a user account to add an alias or node.  
_todo: disallow node label reuse?_  
_todo: dropnode and dropalias; prevent account hijacking from stolen client/nodeid_  
_todo: cork pauses delivery to user's nodes, params date & for-list_  
   ```
   { "op":       3,
     "id":       string,  // referenced by (ack) response
    ["newnode":  string   // user label for a client device
    |"newalias": string]} // user alias, must be 8+ printable characters
   ```
   Response: `(ack) // without .msgid or .posted`  
   To sender's nodes:
   ```
   { "op":       "user",  // these have higher priority than normal messages
     (std),
    ["newnode":  string,  // from client request
     "nodeid":   string   // password for new node
    |"newalias": string]} // from client request
   ```

0. __OhiEdit__ notifies selected contacts of a user's presence.
   ```
   { "op":   4,
     "id":   string,                  // referenced by (ack) response
     "for":  [{"id": string}, ...],   // list of uids
     "type": "add" | "drop"
           | "init"}                  // no "ohiedit" message is sent to sender's nodes
   ```
   Response: `(ack) // without .msgid or .posted`  
   To sender's nodes:
   ```
   { "op":   "ohiedit",
     (std),
     "for":  [{"id": string}, ...],   // from client request
     "type": "add" | "drop"}          // from client request
   ```
   To recipients:
   ```
   { "op":     "ohi",
     "from":   string,                // uid
     "status": 1 | 2}                 // 1 online, 2 offline
   ```

0. __GroupInvite__ performs a _Ping_, and authorizes the recipient to join a group, 
creating it if necessary.  
_todo: closed groups; invitations by moderators_  
   ```
   { "op":   5,
     "id":   string,       // referenced by (ack) response
     "gid":  string,       // group name, must be 8+ printable characters
     "from": string,       // sender alias
     "to":   string,       // invitee alias
     (data)}               // .datalen max is limited by server; data must be valid UTF-8
   ```
   Response: `(ack)`  
   To recipient:
   ```
   { "op":    "invite",
     (std),
     "gid":   string,      // from client request
     "alias": string,      // from client request .from
     "to":    string,      // from client request
     (data)}               // from client request
   ```
   To group members:
   ```
   { "op":       "member",
     (std),
     "act":      "invite",
     "gid":      string,   // from client request
     "alias":    string}   // from client request .to
   ```

0. __GroupEdit__ updates a group, allowing an invitee to join, or a member to be dropped, 
or a member to update their recorded alias.
   ```
   { "op":        6,
     "id":        string,  // referenced by (ack) response
     "act":       "join"
                | "alias"
                | "drop",
     "gid":       string,  // group name
    [<"newalias": string>  // user alias, for "join"
    |"newalias":  string   // user alias, for "alias"
    |"to":        string]} // user alias, for "drop"
   ```
   Response: `(ack) // without .msgid or .posted`  
   To group members:
   ```
   { "op":       "member",
     (std),
     "act":      string,   // from client request
     "gid":      string,   // from client request
     "alias":    string,   // last reported value
    <"newalias": string>}  // from client request
   ```

0. __Post__ sends a message to users and/or groups.
   ```
   { "op":  7,
     "id":  string,               // referenced by (ack) response
     "for": [{"id":   string,     // uid or group name
              "type": 1 | 2 | 3}, // 1 uid, 2 gid (include self), 3 gid (exclude self)
             ... ]
          | [],                   // only sender's nodes
     (data)}
                                  // .datahead segment
   { "threadid":      string,     // empty for new thread
     "alias":         string,     // user alias
     "subject":       string,     // optional if .threadid set
    <"confirmid":     string,     // id of confirmed forwarded message, see PostNotify
     "confirmposted": string>,    // posted date of confirmed forwarded message
    <"cc":                        // subscribers, omit unless .threadid empty
      [{ "who":       string,     // user alias
         "whouid":    string,     // uid
         "by":        string,     // user alias that included .who
         "byuid":     string,     // uid
         "date":      string,     // datetime when .who was included
         "note":      string,     // comment from .by
         "subscribe": bool},      // .who receives new thread messages
       ... ]>,
    <"attach":                    // file contents appear in the data segment following the message body
      [{ "name":      string,     // unique in .attach, prefix u:/f:/r:
         "size":      uint,       // count of octets
        <"ffn":       string>},   // filled-form name, omit for .name u:*
       ... ]> }
                                  // todo: filled-form attachment

   { "nodesync": true}            // .datahead segment for node sync
                                  // todo: data segment for node sync
   ```
   Response: `(ack)`  
   To recipients:
   ```
   { "op":  "delivery",
     (std),
     (data)}                      // from client request
   ```

0. __PostNotify__ sends a message to the `.for` list and a separate notification 
to the `.for` and `.notefor` lists.  
_todo: PostNotify synchronization; see Todo.txt_

   This enables a decentralized subscriber list per-thread, and forwarding a thread to new subscribers. 
The `(data)` message delivers a copy of the thread, without attachments, to new subscribers.
Recipients flag its messages as unconfirmed.
The notification delivers the list of new subscribers to current & new subscribers.
The current subscribers then Post a copy of each thread message authored by them, with attachments, 
to the new subscribers, including the `.confirmid` & `.confirmposted` fields of `.datahead`.
   ```
   { "op":         8,
     "id":         string,  // referenced by (ack) response
     "for":        [ ... ], // list of uids and/or group names, see Post
    <"fornotself": true>,   // exclude sender's nodes (normally implicit in .for list)
     "notefor":    [ ... ], // list of uids and/or group names, see Post
     "notelen":    uint,    // count of octets following the header for notification
    <"notehead":   uint>,   // analagous to .datahead for notification
    <"notesum":    uint>,   // analagous to .datasum for notification
     (data)}                // .datahead & .datasum pertain to octets following notification

                            // .notehead segment
   { "threadid": string,    // id of forwarded thread
     "cc":       [ ... ]}   // new subscribers, see Post

                            // .datahead segment
   { "threadid": string}    // id of forwarded thread
                            // todo: data segment for forwarded thread
   ```
   Response: `(ack)`  
   To recipients:
   ```
   { "op":     "delivery",
     (std),
     "notify": uint,        // count of items in .for & .notefor, including self
     (data)}                // from client request (.datalen - .notelen), .datahead, .datasum
   ```
   To notification recipients:
   ```
   { "op":     "notify",
     (std),
     "postid": string,      // (ack) .msgid
     (data)}                // from client request .notelen, .notehead, .notesum
   ```

0. __Ping__ sends a short text message via a user's alias.
On receipt, the recipient may thereafter contact the sender by uid, given in `(std) .from`.
A server may limit the number of pings and consecutive failed pings per 24h.
   ```
   { "op":   9,
     "id":   string,  // referenced by (ack) response
     "from": string,  // sender alias
     "to":   string,  // invitee alias
     (data)}          // .datalen max is limited by server; data must be valid UTF-8
   ```
   Response: `(ack)`  
   To recipient:
   ```
   { "op":    "ping",
     (std),
     "alias": string, // from client request .from
     "to":    string, // from client request
     (data)}          // from client request
   ```

0. __Ack__ acknowledges receipt of a message.
   ```
   { "op":   10,
     "id":   string, // from .id of message being ack'd
     "type": string} // unused
   ```

0. __Pulse__ resets the connection timeout.
   ```
   { "op": 11}
   ```

0. __Quit__ performs logout.
   ```
   { "op": 12}
   ```


### License

Copyright 2020 <NAME>  
Published at https://github.com/networkimprov/mnm

This Source Code Form is subject to the terms of the Mozilla Public
License, v. 2.0. If a copy of the MPL was not distributed with this
file, You can obtain one at http://mozilla.org/MPL/2.0/


{'fraction_non_alphanumeric': 0.12220904270164666, 'fraction_numerical': 0.004081775048841753, 'mean_word_length': 2.1786427145708585, 'pattern_counts': {'":': 274, '<': 43, '<?xml version=': 0, '>': 41, 'https://': 12, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7502079', 'n_tokens_mistral': 8502, 'n_tokens_neox': 8107, 'n_words': 3283}