<jupyter_start><jupyter_text>Using books for query purposes can be regarded as a continual learning process of the bot. As the bot is positioned as an expert, what better way to further improve his knowledge base?We will use the Book processing results in terms of paragraphs that are keeping the original chronology and dependence on the book/chapters/subchapters. This process is similar with training databases for Pirkin 2 Model- SPCA, except the labeled answers which will be in scope of Labyrinth model and described more in NOG from books.We will also use an encoder similar to BERT and use book corpuses to fine-tune the language model of the back-up models.<jupyter_code># IN: Books processing/ NIU
# OUT: trained books for NOG<jupyter_output><empty_output><jupyter_text>Objectives:•	Training books inputs;•	Using Books processing output;•	Making summarization using SPCA for all books output database;Dependencies: NOG from books/NIU/ Books processing;Database/ Vocabularies/External needed: Books Input Database, Books Output Database<jupyter_code># To dos:
# Inputs:
 # The Book Input Database will go through NIU – Machine Learning pipelines except: Backup, Secondary, Books processing, Auto Complete, Triggering, Domain Validation, SPCA memory – keep in mind that in Book processing we’ve also used Machine Education;
 # The resulting SPCAs will be linked with immediate results from the Book Output Database, to establish the chapter/subchapter that will be queried. These results will be linked with the Labyrinth model and the principles from the deep conversational answers.
 # The chapter will be blocked in the user queries states in order not to exist without considering the exit terms stipulated in the user queries pipeline.  
# NOG5 Summarization
 # The Book Outputs Database will go through the same process as Database training for Pirkin Model.
 # The resulted SPCA will be kept for Labyrinth model – NOG5 Summarization; the immediate answer will apply,

# use codes from NIU<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.03984259714707329, 'fraction_numerical': 0.0014756517461878996, 'mean_word_length': 5.277777777777778, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '326067', 'license': 'Apache-2.0', 'n_tokens_mistral': 512, 'n_tokens_neox': 483, 'n_words': 288, 'path': 'NOG-NLG dummy codes/36_Books_training_dummy_code.ipynb', 'repo_name': 'Cezanne-ai/project-2021', 'size': '2033'}