<commit_before>Instructions for processing input repositories:
`tac-kbp-eal` refers to your clone of the github repository or the `tac-kbp-eal` subproject.
Any docidmap is a File with, one per line, a document id, a tab character, and a path to an existing file.

1. Convert to canonical using `tac-kbp-eal/target/appassembler/bin/importForeignIDs`:  
    Params:

    ```
    input: /path/to/your/system/output
    output: /path/to/canonical/id/output
    doMultipleStores: false
    outputLayout: KBP_EAL_2016
    ```

2. Validate system output using `tac-kbp-eal/target/appassembler/bin/validateSystemOutput2016`:  
    Params:

    ```
    systemOutputStore: /path/to/canonical/id/output
    dump: false
    docIDMap: a map of docid to raw text
    validRoles: tac-kbp-eal/data/2016.types.txt
    linkableTypes: tac-kbp-eal/data/2016.linkable.txt
    ```

    Note that you may filter out invalid event types/roles using `tac-kbp-eal/target/appassembler/bin/filterOutInvalidArguments`, although this will silently ignore any errors in your system.
    Note that you may filter out any Generic responses from the linking store and corresponding entries from the CorpusLinking (if any) using `tac-kbp-eal/target/appassembler/bin/filterLinkingStore`.


3. Minimize system outputs to just the best it produced using `tac-kbp-eal/target/appassembler/bin/keepOnlyBestResponses`:  
    Params:

    ```
    inputStore: /path/to/canonical/id/output
    outputStore: keepBestOutput
    outputLayout: KBP_EAL_2016
    keepInferenceCases: false
    ```
    
4. Extract query responses using `tac-kbp-eal/target/appassembler/bin/queryResponseFromERE`:  
    Params (these may change come the evaluation and feedback from the LDC):
    ```
    com.bbn.tac.eal.storeDir: /output/dir/of/keep/Best
    com.bbn.tac.eal.storesToProcess: systemNamesInTheAboveDirectory
    com.bbn.tac.eal.outputFile: outputFile
    com.bbn.tac.eal.eremap: ere docid to filepath map.<commit_msg>Add quote filtering params<commit_after>Instructions for processing input repositories:
`tac-kbp-eal` refers to your clone of the github repository or the `tac-kbp-eal` subproject.
Any docidmap is a File with, one per line, a document id, a tab character, and a path to an existing file.

1. Convert to canonical using `tac-kbp-eal/target/appassembler/bin/importForeignIDs`:  
    Params:

    ```
    input: /path/to/your/system/output
    output: /path/to/canonical/id/output
    doMultipleStores: false
    outputLayout: KBP_EAL_2016
    ```

2. Validate system output using `tac-kbp-eal/target/appassembler/bin/validateSystemOutput2016`:  
    Params:

    ```
    systemOutputStore: /path/to/canonical/id/output
    dump: false
    docIDMap: a map of docid to raw text
    validRoles: tac-kbp-eal/data/2016.types.txt
    linkableTypes: tac-kbp-eal/data/2016.linkable.txt
    ```

    Note that you may filter out invalid event types/roles using `tac-kbp-eal/target/appassembler/bin/filterOutInvalidArguments`, although this will silently ignore any errors in your system.
    Note that you may filter out any Generic responses from the linking store and corresponding entries from the CorpusLinking (if any) using `tac-kbp-eal/target/appassembler/bin/filterLinkingStore`.


3. Filter out responses that appear in quotes:
    Build the quote filter using `tac-kbp-eal/target/appassembler/bin/buildQuoteFilter`:
    Params:
    ```
    quoteFilter: /path/to/output/file
    docIdToFileMap: a map or docid to raw text
    ```

    Filter the responses: using `tac-kbp-eal/target/appassembler/bin/applyQuoteFilter`:
    Params:

    ```
    inputStore:
    outputStore:
    quoteFilter:
    ```

4. Minimize system outputs to just the best it produced using `tac-kbp-eal/target/appassembler/bin/keepOnlyBestResponses`:  
    Params:

    ```
    inputStore: /path/to/canonical/id/output
    outputStore: keepBestOutput
    outputLayout: KBP_EAL_2016
    keepInferenceCases: false
    ```
    
5. Extract query responses using `tac-kbp-eal/target/appassembler/bin/queryResponseFromERE`:  
    Params (these may change come the evaluation and feedback from the LDC):
    ```
    com.bbn.tac.eal.storeDir: /output/dir/of/keep/Best
    com.bbn.tac.eal.storesToProcess: systemNamesInTheAboveDirectory
    com.bbn.tac.eal.outputFile: outputFile
    com.bbn.tac.eal.eremap: ere docid to filepath map.
{'fraction_non_alphanumeric': 0.09180628873077806, 'fraction_numerical': 0.011246270369520313, 'mean_word_length': 3.5395833333333333, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5645528', 'n_tokens_mistral': 1490, 'n_tokens_neox': 1414, 'n_words': 382}