<jupyter_start><jupyter_text>About 60% of the sentences consist of 3 to 25 tokens, 30% of the sentences consist of 26 to 40 tokens, and 10% of the sentences consist of 41 to 70 tokens.<jupyter_code>wiki_corpus = load_wiki()
wiki_sent_lengths = np.array([len(sent) for sent in wiki_corpus])
print(
    (np.logical_and(wiki_sent_lengths >= 3, wiki_sent_lengths <= 25)).sum()
    / len(wiki_sent_lengths)
)
print(
    (np.logical_and(wiki_sent_lengths >= 26, wiki_sent_lengths <= 40)).sum()
    / len(wiki_sent_lengths)
)
print(
    (np.logical_and(wiki_sent_lengths >= 41, wiki_sent_lengths <= 70)).sum()
    / len(wiki_sent_lengths)
)<jupyter_output><empty_output><jupyter_text>The median and the mean of the sentence lengths are about 23 and 25 respectively<jupyter_code>print(np.median(wiki_sent_lengths))
print(np.mean(wiki_sent_lengths))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11379310344827587, 'fraction_numerical': 0.0367816091954023, 'mean_word_length': 6.08130081300813, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 12, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '446231', 'license': 'MIT', 'n_tokens_mistral': 353, 'n_tokens_neox': 314, 'n_words': 73, 'path': 'notebooks/wiki_sentence_length_statistcs.ipynb', 'repo_name': 'dschaehi/capturing-word-order', 'size': '870'}