<jupyter_start><jupyter_text>The Tidy Data<jupyter_code>doc1.sample(20)<jupyter_output><empty_output><jupyter_text>Show me all highlights<jupyter_code>doc1[doc1['hl-id'].notnull()].groupby("hl-id")['word'].apply(lambda x: " ".join(x))<jupyter_output><empty_output><jupyter_text>Show me all my annotations<jupyter_code>doc1[doc1['is-annotation']].groupby("hl-id")['word'].apply(lambda x: " ".join(x))<jupyter_output><empty_output><jupyter_text>Show me tagged sections<jupyter_code>sids = set(doc1[doc1.tag]['section-id'])
doc1[doc1['section-id'].isin(sids)].groupby("section-id")['word'].apply(lambda x: " ".join(x))<jupyter_output><empty_output><jupyter_text>Plot of the most frequent words for each section<jupyter_code>doc1.groupby("section")['word'].apply(lambda x: nltk.FreqDist(x))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.19117647058823528, 'fraction_numerical': 0.014705882352941176, 'mean_word_length': 13.086206896551724, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '404041', 'license': 'MIT', 'n_tokens_mistral': 330, 'n_tokens_neox': 322, 'n_words': 31, 'path': 'analysis.ipynb', 'repo_name': 'RaoOfPhysics/zotero-insights', 'size': '816'}