<commit_before>- upload new docs along with the new version of the library (or do not publish the docs at all)

- in the downloading code, handle 404 errors more gently. Place information that given file was not available due to broken link.
- add more unit tests for different scenarios of algorithm and tree state e.g.:
	- tree with the root only
	- a large tree
	- tree with no missing nodes and without errors
	- root's children are leafs
- in the Unit tests, when the mechanize.Browser opens the "/root/2011-07-02" webpage, the "HTTP Error 404: File not found" error _sometimes_ shows up. The entry in the logs looks as follows: `WARNING thread=="Thread-95", exception in node "/root/2011-07-12": "Moving to child failed". Detailed message: "HTTP Error 404: File not found".`. 
	- probably a fault of the HTTP file server?
- ? change the order of nodes saved in the `*.xml` file: order by names, not by state of the node?
- add command-line option to try to visit once again the ERROR nodes of a tree read from a file (it might be helpful because in case of a web crawler, the pages which were not available before might be available now)

Documentation
-------------
? generate source docs with sphinx instead of epydoc?
- In the `docs/tree_exploration_algorithm/text.tex` document,
	- make a distinction between tree as a data structure shared between the threads and explored tree
	- describe more precisely the example of the upward update of the state (the last figure).<commit_msg>Small modification of the TODO file<commit_after>- in the downloading code, handle 404 errors more gently. Place information that given file was not available due to broken link.
- add more unit tests for different scenarios of algorithm and tree state e.g.:
	- tree with the root only
	- a large tree
	- tree with no missing nodes and without errors
	- root's children are leafs
- in the Unit tests, when the mechanize.Browser opens the "/root/2011-07-02" webpage, the "HTTP Error 404: File not found" error _sometimes_ shows up. The entry in the logs looks as follows: `WARNING thread=="Thread-95", exception in node "/root/2011-07-12": "Moving to child failed". Detailed message: "HTTP Error 404: File not found".`. 
	- probably a fault of the HTTP file server?
- ? change the order of nodes saved in the `*.xml` file: order by names, not by state of the node?
- add command-line option to try to visit once again the ERROR nodes of a tree read from a file (it might be helpful because in case of a web crawler, the pages which were not available before might be available now)

Documentation
-------------
? generate source docs with sphinx instead of epydoc?
- In the `docs/algorithm/index.tex` document,
	- make a distinction between tree as a data structure shared between the threads and explored tree
	- describe more precisely the example of the upward update of the state (the last figure).
{'fraction_non_alphanumeric': 0.06878672658140339, 'fraction_numerical': 0.018665744901486345, 'mean_word_length': 3.7913907284768213, 'pattern_counts': {'":': 2, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3531544', 'n_tokens_mistral': 775, 'n_tokens_neox': 712, 'n_words': 453}