<jupyter_start><jupyter_text>Fichiers d'inputs et d'outputs<jupyter_code>infile = "../data/all.txt"
outfile = "../data/sents.txt"<jupyter_output><empty_output><jupyter_text>Tokenisation du corpus ligne par ligne **Important** : pour traiter tout le corpus, mettez `LIMIT = None`<jupyter_code>LIMIT =None
with open(outfile, 'w',encoding="utf-8") as output:
    with open(infile, encoding="utf-8", errors="backslashreplace") as f:
        content = f.readlines()
        content = content[:LIMIT] if LIMIT is not None else content
        n_lines = len(content)
        for i, line in enumerate(content):
            if i % 100000 == 0:
                print(f'processing line {i}/{n_lines}')
            sentences = sent_tokenize(line)
            for sent in sentences:
                output.write(sent + "\n")<jupyter_output>processing line 0/12204593
processing line 100000/12204593
processing line 200000/12204593
processing line 300000/12204593
processing line 400000/12204593
processing line 500000/12204593
processing line 600000/12204593
processing line 700000/12204593
processing line 800000/12204593
processing line 900000/12204593
processing line 1000000/12204593
processing line 1100000/12204593
processing line 1200000/12204593
processing line 1300000/12204593
processing line 1400000/12204593
processing line 1500000/12204593
processing line 1600000/12204593
processing line 1700000/12204593
processing line 1800000/12204593
processing line 1900000/12204593
processing line 2000000/12204593
processing line 2100000/12204593
processing line 2200000/12204593
processing line 2300000/12204593
processing line 2400000/12204593
processing line 2500000/12204593
processing line 2600000/12204593
processing line 2700000/12204593
processing line 2800000/12204593
processing line 2900000/12204593
processing line 3000000/1[...]
{'fraction_non_alphanumeric': 0.07751091703056769, 'fraction_numerical': 0.24672489082969432, 'mean_word_length': 4.674922600619195, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 8, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '195774', 'license': 'MIT', 'n_tokens_mistral': 875, 'n_tokens_neox': 574, 'n_words': 158, 'path': 'module4/s3_sentence_tokenizer.ipynb', 'repo_name': 'oussamagh97/tac', 'size': '1832'}