<jupyter_start><jupyter_text>Passage reduced to slots<jupyter_code>TfP.passage('Barwar', 'A Hundred Gold Coins', 1)<jupyter_output><empty_output><jupyter_text>Passage reduced to morphemes<jupyter_code>TfP.passage('Barwar', 'A Hundred Gold Coins', 1, baseTypes={'morpheme'})<jupyter_output><empty_output><jupyter_text>Passage reduced to words<jupyter_code>TfP.passage('Barwar', 'A Hundred Gold Coins', 1, baseTypes={'word'})<jupyter_output><empty_output><jupyter_text>Passage reduced to words and then morphemes<jupyter_code>TfP.passage('Barwar', 'A Hundred Gold Coins', 1, baseTypes={'morpheme', 'word'})<jupyter_output><empty_output><jupyter_text>Passage reduced to words and then morphemes and then letters<jupyter_code>TfP.passage('Barwar', 'A Hundred Gold Coins', 1, baseTypes={'morpheme', 'word', 'letter'})<jupyter_output><empty_output><jupyter_text>Passage reduced to morphemes with feature foreign included<jupyter_code>TfP.passage('Barwar', 'Gozali and Nozali', 29, baseTypes={'morpheme'}, features='foreign')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14026717557251908, 'fraction_numerical': 0.006679389312977099, 'mean_word_length': 12.802631578947368, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '481170', 'license': 'MIT', 'n_tokens_mistral': 409, 'n_tokens_neox': 395, 'n_words': 70, 'path': 'test/provider/nena.ipynb', 'repo_name': 'ancient-data/text-fabric', 'size': '1048'}