<jupyter_start><jupyter_text>Jollity Manual Jollity is a small library of Python functions that process Jupyter notebooks.Jollity does _not_ convert notebooks from/to other formats,like Markdown, PDF and HTML.There are plenty of tools for that, including [pandoc](https://pandoc.org),[nbconvert](https://nbconvert.readthedocs.io), [Jupytext](https://jupytext.readthedocs.io),[nbsphinx](https://nbsphinx.readthedocs.io) and [Jupyter Book](https://jupyterbook.org).  Example Let's first see an example of how Jollity is used in practice.I'm authoring in Markdown a textbook for M269,the algorithms and data structures module at The Open University, UK.I wrote a Python script that:1. processes the Markdown files2. converts the Markdown files to Jupyter notebooks3. processes the notebooks4. executes the notebooks with nbconvert5. converts the notebooks to PDF and HTML with nbsphinx6. processes the notebooks7. zips all files into an archive for uploading to the M269 website.Markdown cells at the end of stage 2 may look like this: ```The best-case complexity is ø(1) and the worst-case complexity is ø(2^n).Exponential functions were introduced in [MU123](mu123).**Exercise:** Explain why the worst-case is exponential.In practice the worst-case may only occur very rarely.**Exercise**: Edit the next cell to complete the sentence.The average-case complexity is ...``` This example shows the four kinds of special comments used in M269.(You can define your own.)- The `ANSWER` comment becomes a separate Markdown cell with text  '_Write your answer here._' in the final notebook that goes to students,  but nothing appears in the PDF or HTML.- The `INFO` and `NOTE` comments are replaced with HTML code that puts the text  in a coloured box (different colours for info and note boxes).- The text within `EDIT` comments is put in a separate Markdown cell, so that  students don't edit by mistake all other surrounding text.The example also shows the use of special characters or character combinationsto make typing the text faster. The script uses Jollity to replace alloccurrences of ø with Θ and of ^ followed by n with ⁿ.The ø letter is quick to type on my keyboard (Alt-o), but there's noshortcut for uppercase theta.The script also replaces occurrences of `mu123` within a linkwith the corresponding URL. Keeping a mapping of abbreviations to URLsavoids repeatedly writing and updating the same URL in several notebooks.Code cells often include `%timeit` commands to measure the run-time of code.This slows down the execution of the code cells in stage 4, so the scriptcan be run in a 'draft' mode that comments out the `%timeit` commands\before the notebooks are executed.Stage 3 uses Jollity to:- Put the four special comments into their own Markdown cells.- Add HTML code at the start and end of the info and note cells  to generate the boxes.- Strip spaces and newlines from the start and end of each cell.- Do text replacements to obtain special characters like superscripts.- Replace `_text_` with `*text*` in certain contexts to avoid a Jupyter bug when  rendering italicised text.- Replace one or more spaces between certain words and digits with a single  non-breaking space, for example `step 1.2 takes 45 µs` becomes `step&nbsp;1.2 takes 45&nbsp;µs`.- Expand abbreviated URLs and check all URLs lead to existing pages.- Comment out all timing commands in code cells, if in draft mode.- Report code lines longer than 69 characters: they wrap around in the PDF.- Report invisible line breaks in Markdown.- Report when a heading level is skipped.- Remove all code between ` skip` and ` \skip`,  as illustrated in the [README](../README.md).- If the notebook is the start of a chapter, add at the end a reminder to  check the M269 website for news and errata before working through the chapter.Stage 6 (after execution and conversion to PDF and HTML) uses Jollity to:- Convert the Markdown syntax within the info and note boxes to HTML.  Nbsphinx correctly converts the boxes' Markdown content to PDF and HTML, but  the Jupyter interface can't render Markdown within HTML.- Insert the boilerplate text in the empty `ANSWER` cells.- If a notebook has code, extract it to a separate file, with a copyright  notice before the code. Comment out all IPython commands starting with `%`.File `m269.py` has the code that calls Jollity to do all the above.The Jollity functions are explained below.  Using Jollity Jollity uses the `NotebookNode` class of the `nbformat` module torepresent a notebook in memory.Each function takes an instance of that class and modifies it.To process your notebooks, you will need to write code like this: ```pyimport globimport jollityimport nbformat go through all notebooks in `folder` or a subfolder of itfor file in glob.iglob('folder/**/*.ipynb', recursive=True):    notebook = nbformat.read(file, 4)    4 is the notebook format version    if 'introduction' in file:           pre-process chapter introductions        jollity.one_function(notebook)    jollity.another_function(notebook)   same processing for all notebooks    nbformat.write(notebook, file)       overwrites the original file``` In most authoring workflows you will wish to preserve the originaland write the processed notebook to a different file or folder.For an alternative way of going through files in a folder,see script `generate_doc.py`.It reads the source Markdown file of this manual in folder `md`and writes the notebook to folder `doc`.The script uses [Jupytext](https://jupytext.readthedocs.io) to converta Markdown file to a Jupyter notebook. Jollity requires Python 3.8 or later.  Logging The Jollity functions log any warnings and errors as they process notebooks.By default, the warning and error messages are printed on the screen,but you can collect them in a file.If you're using a bash-like command line, you can type`python your_script.py 2> log_file` to redirect the messages,for example `python generate_doc.py 2> log.txt`.This will overwrite the log file every time you run your script.Alternatively, add the following to your script:<jupyter_code>import logging

logging.basicConfig(filename='log.txt')<jupyter_output><empty_output><jupyter_text>This will append the messages to the file, if it exists.In this way you can preserve the log of previous runs of your script.If you want the log file to start afresh every time you run the script, write<jupyter_code>logging.basicConfig(filename='log.txt', filemode='w')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.05864197530864197, 'fraction_numerical': 0.008179012345679013, 'mean_word_length': 4.735398230088496, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 11, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '448422', 'license': 'MIT', 'n_tokens_mistral': 1734, 'n_tokens_neox': 1640, 'n_words': 924, 'path': 'doc/manual.ipynb', 'repo_name': 'mwermelinger/jollity', 'size': '6480'}