<jupyter_start><jupyter_text>Exercise 06: store annotated text as JSON files Much of the preceding NLP code has been worked into a small library, and we'll call functions from that library to help keep these notebooks more readable. Take a look at the source code in `pynlp.py`, and an example usage:<jupyter_code>import pynlp

html_file = "html/article1.html"
json_file = "a1.json"

pynlp.full_parse(html_file, json_file)<jupyter_output><empty_output><jupyter_text>That extracts text from HTML in the first article, then stores the parsed and annotated text as JSON, one line per sentence. Let's look at the first two sentences:<jupyter_code>%sx more a1.json
# representation of paragraphs, sentences, words, and sentence annotation<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06684141546526867, 'fraction_numerical': 0.00655307994757536, 'mean_word_length': 6.2075471698113205, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '564757', 'license': 'MIT', 'n_tokens_mistral': 239, 'n_tokens_neox': 224, 'n_words': 93, 'path': '1j_NLP_Python/ex06.ipynb', 'repo_name': 'yevheniyc/Python', 'size': '763'}