<commit_before>### Syntagmatic relations / word associations

In this problem we have a collection of documents, each document represented by a single line of text.
Hence, the number of documents equals the number of lines.
```python
def read_doc():
    from sys import stdin
    return stdin.readlines()

text = read_doc()
```

The text might contain escape sequences, so we will get rid of those:
```python
def unescape(s):
    """
    http://stackoverflow.com/a/24519338/2003487
    """
    import re
    import codecs
    ESCAPE_SEQUENCE_RE = re.compile(r'''
    ( \\U........      # 8-digit hex escapes
    | \\u....          # 4-digit hex escapes
    | \\x..            # 2-digit hex escapes
    | \\[0-7]{1,3}     # Octal escapes
    | \\N\{[^}]+\}     # Unicode characters by name
    | \\[\\'"abfnrtv]  # Single-character escapes
    )''', re.UNICODE | re.VERBOSE)
    def decode_match(match):
        return codecs.decode(match.group(0), 'unicode-escape')
        pass

    return ESCAPE_SEQUENCE_RE.sub(decode_match, s)

text = [unescape(sent) for sent in text]
```
Then, we'll break each document by whitespace, also skipping empty documents
```python
from nltk.tokenize.regexp import WhitespaceTokenizer
text = [WhitespaceTokenizer().tokenize(sent) for sent in text if len(sent) > 0]
```
convert everything to lowercase
```python
text = [[token.lower() for token in sent] for sent in text]
```

and clean all words from characters which are not letters (except apostrophes)
```python
text = [[''.join(ch for ch in token if ch.isalpha() or ch == '\'') for token in sent] for sent in text]
```

remove all words which are shorter than 2 or longer than 35 characters:
```python
text = [[token for token in sent if len(token) >= 2 and len(token) <= 35] for sent in text]
```

Now, it's time to get rid of all stopwords. Conveniently, `nltk` itself provides such:
```python
from nltk.corpus import stopwords
text = [[token for token in sent if not token in set(stopwords.words('english'))] for sent in text]
```

Finally, we will apply Porter2 stemmer:
```python
from nltk.stem.snowball import SnowballStemmer
stemmer = SnowballStemmer("english")
text = [[stemmer.stem(token) for token in sent] for sent in text]
```<commit_msg>stopwords work better this way<commit_after>### Syntagmatic relations / word associations

In this problem we have a collection of documents, each document represented by a single line of text.
Hence, the number of documents equals the number of lines.
```python
def read_doc():
    from sys import stdin
    return stdin.readlines()

text = read_doc()
```

The text might contain escape sequences, so we will get rid of those:
```python
def unescape(s):
    """
    http://stackoverflow.com/a/24519338/2003487
    """
    import re
    import codecs
    ESCAPE_SEQUENCE_RE = re.compile(r'''
    ( \\U........      # 8-digit hex escapes
    | \\u....          # 4-digit hex escapes
    | \\x..            # 2-digit hex escapes
    | \\[0-7]{1,3}     # Octal escapes
    | \\N\{[^}]+\}     # Unicode characters by name
    | \\[\\'"abfnrtv]  # Single-character escapes
    )''', re.UNICODE | re.VERBOSE)
    def decode_match(match):
        return codecs.decode(match.group(0), 'unicode-escape')
        pass

    return ESCAPE_SEQUENCE_RE.sub(decode_match, s)

text = [unescape(sent) for sent in text]
```
Then, we'll break each document by whitespace, also skipping empty documents
```python
from nltk.tokenize.regexp import WhitespaceTokenizer
text = [WhitespaceTokenizer().tokenize(sent) for sent in text if len(sent) > 0]
```
convert everything to lowercase
```python
text = [[token.lower() for token in sent] for sent in text]
```

and clean all words from characters which are not letters (except apostrophes)
```python
text = [[''.join(ch for ch in token if ch.isalpha() or ch == '\'') for token in sent] for sent in text]
```

remove all words which are shorter than 2 or longer than 35 characters:
```python
text = [[token for token in sent if len(token) >= 2 and len(token) <= 35] for sent in text]
```

Now, it's time to get rid of all stopwords. Conveniently, `nltk` itself provides such:
```python
from nltk.corpus import stopwords
stopwords = set(stopwords.words('english'))
text = [[token for token in sent if not token in stopwords] for sent in text]
```

Finally, we will apply Porter2 stemmer:
```python
from nltk.stem.snowball import SnowballStemmer
stemmer = SnowballStemmer("english")
text = [[stemmer.stem(token) for token in sent] for sent in text]
```
{'fraction_non_alphanumeric': 0.12524939037907337, 'fraction_numerical': 0.013744180891154955, 'mean_word_length': 3.512, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1113035', 'n_tokens_mistral': 1500, 'n_tokens_neox': 1373, 'n_words': 536}