<commit_before>
# Keys

in order to execute these script, you need to have a JSON file in
this directory, named `parsers-keys.json` containing all the keys of the 
parser you run. (Keys has to be shared with the server), this is an example:

[
  {
    "key": "<KEY>",
    "name": "feedPostHref",
    "fields": ["feedPostHref"]
  },
  {
    "key": "<KEY>",
    "name": "postType",
    "fields": ["postType"]
  },
  {
    "key": "test-2-dghy654edfghy65rfgh",
    "name": "promotedInfo",
    "fields": ["ownerName","promotedInfo","promotedMedia","promotedPage"]
  }
]
```

The key is associated to the parser name through the file name, the
library in lib/parse.js look for this explicitly and will fail.

open an issue in (facebook.tracking.exposed)[https://github.com/tracking-exposed/facebook] repository if you want develop a parser, and we'll start to manage keys distribution. You could write to claudio﹫tracking・exposed using [this PGP key](https://keybase.io/vecna).

## Environment 

some environment variables are checked by the parser library:

  * DEBUG: as usual by debug module
  * url: by default https://facebook.tracking.exposed
  * concurrency: numbers of snipped parsed in parallel (but node VM is single thread so I don't really know what happen), default 1
  * delay: milliseconds to wait after every parsing, default 200
  * since: the starting date by default is 2016-09-11
  * until: the end data of the window in analysis (to be tested?)
  * repeat: can be undefined, and by default you get only the HTMLs that are not yet been checked by the parser. can be `true`, and return all the objects in which previous parsing has been successfull, or `false` the object in which parser failed in the previous executions.


## The API and the parser...

are documented in the website via docco! (**TODO**)<commit_msg>Fix some parsers documentation typos<commit_after># Keys

In order to execute these scripts you need to have a JSON file in
this directory, named `parsers-keys.json`: the library in `lib/parse.js` looks for this explicitly and will fail if the file cannot be found.

`parsers-keys.json` should contain all the keys of the
parser you run (keys have to be shared with the server). This is an example:

```
[
  {
    "key": "<KEY>",
    "name": "feedPostHref",
    "fields": ["feedPostHref"]
  },
  {
    "key": "<KEY>",
    "name": "postType",
    "fields": ["postType"]
  },
  {
    "key": "test-2-dghy654edfghy65rfgh",
    "name": "promotedInfo",
    "fields": ["ownerName","promotedInfo","promotedMedia","promotedPage"]
  }
]
```

Open an issue in [facebook.tracking.exposed](https://github.com/tracking-exposed/facebook) repository if you want develop a parser, and we'll start to manage keys distribution. You could write to claudio﹫tracking・exposed using [this PGP key](https://keybase.io/vecna).

## Environment

Some environment variables are checked by the parser library:

  * `concurrency`: numbers of snipped parsed in parallel (but node VM is single thread so I don't really know what happen), default `1`
  * `DEBUG`: as usual by debug module
  * `delay`: milliseconds to wait after every parsing, default `200`
  * `repeat`: can be undefined, and by default you get only the HTMLs that have not yet been checked by the parser. If `true`, return all the objects in which previous parsing has been successful. If `false`, return the object in which the parser failed during the previous executions.
  * `since`: the starting date by default is `2016-09-11`
  * `until`: the end data of the window in analysis (to be tested?)
  * `url`: by default https://facebook.tracking.exposed


## The API and the parser...

are documented in the website via docco! (**TODO**)
{'fraction_non_alphanumeric': 0.10739341608202914, 'fraction_numerical': 0.009713977334052886, 'mean_word_length': 4.036684782608695, 'pattern_counts': {'":': 18, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2310245', 'n_tokens_mistral': 1110, 'n_tokens_neox': 1039, 'n_words': 484}