<commit_before># Word Co-occurence

Tools to find word co-occurence in given sentences.

## Format definition

### Text input

The input text file is defining like this:

a tab seperated file(TSV):

`ID \t sentence`

### Keyword(entity) input

The keyword input can should be a json file with objects(dictionaries).

```json

{
    "keyword1": {
        "abbreviation": [
            "key1_abbr1",
            "key1_abbr2"
        ],
        "full_name": [
            "key1_form1",
            "key1_form2"
        ]
    },
    "keyword2": {
        "abbreviation": [
        ],
        "full_name": [
            "key2_form1",
        ]
    }
}

```

### ID-Entity Output

a tab seperated file(TSV):

`ID \t keyword \t key_form`

### ID-Sentence-Entities Output

a tab seperated file(TSV):

`ID \t sentence \t keyword1, keyword2, keyword3`<commit_msg>doc: add regular expression support<commit_after># Word Co-occurence

Tools to find word co-occurence in given sentences.

## Format definition

### Text input

The input text file is defining like this:

a tab seperated file(TSV):

`ID \t sentence`

### Keyword(entity) input

The keyword input can should be a json file with objects(dictionaries).

```json

{
    "keyword1": {
        "abbreviation": [
            "key1_abbr1",
            "key1_abbr2"
        ],
        "full_name": [
            "key1_form1",
            "key1_form2"
        ]
    },
    "keyword2": {
        "full_name": [
            "key2_form1",
        ],
        "regular_expression":[
            "key2_re1"
        ]
    }
}

```

### ID-Entity Output

a tab seperated file(TSV):

`ID \t keyword \t key_form`

### ID-Sentence-Entities Output

a tab seperated file(TSV):

`ID \t sentence \t keyword1, keyword2, keyword3`
{'fraction_non_alphanumeric': 0.1224373576309795, 'fraction_numerical': 0.018223234624145785, 'mean_word_length': 2.1887477313974593, 'pattern_counts': {'":': 12, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '159193', 'n_tokens_mistral': 606, 'n_tokens_neox': 599, 'n_words': 162}