<jupyter_start><jupyter_text>This is a complex part in the model because we are dealing with 4 different types of database training (some types of training are not, however, similar with the AI concept of training):•	Pirkin 2 model database training by using experts’ answers (core inputs). Experts in different fields (programmers, AI/NLP engineers, doctors, lawyers, HR specialists, investment advisors, restaurants evaluators…) will provide answers to different possible questions or to advisory/consultancy sessions (which can be addressed by the user in a million possible ways, in short inputs or complex phrases). The bot must identify if the user’s input, summarized in a SPCA sentence (received by the bot in 2 vectors: k-word + embedded vector for each word in the SPCA) has a correspondent in the expert’s answers. For this scope we will need a database with questions/answers or situations/information that can be obtained from a specialized site, from an interview with the expert or even specialized books. The bot cannot interfere in the answer and generate a different/adapted answer (in SPCA answer, slots included in NER and DER will be treated as slots types). The labeling can be done with very simplistic utterances as a pro for the Pirkin 2 model. This will provide a very smooth training process without an important focus on testing that anyway will be in scope of the Cezanne-ai project when we put side by side the results from the 2 models’ implementation (SPCA vs back-up).•	Back up answer for core inputs is generated by one of the three E2E models. We made the setup in NIU, so there is no need for further clarifications.•	Pirkin 2 model database training of chit chats (in a similar way, we will use databases for reply to feed up the Reaction analysis from CPL). This process of training will be close to the first one, but NER and DER are not an issue.•	Secondary flow training (chit chats and replies) is an intent-based model that comes first in the NOG flow.All utterances in the database will be trained and we are going to allocate an incremented number (notation) to each correspondent answer to check the duplications in the conversations. It’s important to state that the training process will be ongoing and will be automatically updated - see Auto-training pipeline.<jupyter_code># IN: all databases
# OUT: k-vectors, embedding vectors, E2E training<jupyter_output><empty_output><jupyter_text>Objectives:•	Training the database for Pirkin Model;•	Training the database for Chitchats, Replies and back up answer;Dependencies: NOG from databases/NIU;Database/ Vocabularies/External needed: all,<jupyter_code># To dos:
# Pirkin model
 # The database with questions will go through NIU pipeline (the same as an input) except: CVM, IVM, AVM, Backup, Secondary, Books processing, Auto Complete, Triggering, Domain Validation, SPCA memory;
 # Each resulting SPCA (that will have k-vector’s and an embedding vector’s) will be linked with the answer from the databases and chronological indices will be applied.
# E2E – DL model
  # It will be a standard training included into an E2E model. Also the answers will be marked not to duplicate.<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.043668122270742356, 'fraction_numerical': 0.0031191515907673115, 'mean_word_length': 4.983208955223881, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '333005', 'license': 'Apache-2.0', 'n_tokens_mistral': 810, 'n_tokens_neox': 769, 'n_words': 492, 'path': 'NOG-NLG dummy codes/35_Database_training_dummy_code.ipynb', 'repo_name': 'Cezanne-ai/project-2021', 'size': '3206'}