<jupyter_start><jupyter_text>Now, we need to transmit data to the bot for computation and understanding. We already transmitted important info through matrices and entities and we must transmit the core input (which is at this moment the remaining corpus of the input). As we don’t have a huge database and we created this detailed/unique architecture, it’s only natural to make our own word embedded vector & rules. Let’s say we want to describe the pointer finger functions. A solution is to mask the finger and then reconstruct it based on similarities with what we have in the datasets (this is pretty much the current state of the art solution). Another solution is to show the finger, independently, or in a relation with the previous finger, like RNN's. We are going to refer to <NAME>'s work to better understand how humans do it. A word doesn’t have only intrinsic or extrinsic representation, and it’s meaning can be extracted due to the relations with other words (the same is with the pointer finger). This solution limits the size of corpuses needed to build an accurate embedding if used in a sentence-intent model that already extracted the important meaning in terms of words and their syntactic functions.A word has 4 dimensions in terms of understanding:•	Its intrinsic meaning; •	Its meaning for other words; •	Its meaning inside the sentence/phrase/description;•	Its meaning for the persons/bot to whom you conversate.Consequently, in our model, a word will be vectorized through a special word vector  that covers the relations between the words in the entire input as fundamental for the real meaning of the word. The same vectorization process will be applied on the database and books. Doing this process will allow us to apply simple similarity functions between embedded word vectors from the input and the ones from the databases in order to extract/generate the answers.<jupyter_code># IN: clean-up input/ core input + databases
# OUT: embedding vector + k-word vector for every word in the input and databases<jupyter_output><empty_output><jupyter_text>Objectives:•	Analyzing words that are used to empower other words;•	Finding the most familiar word (for example a synonym to the word from the user that is more known) - we include here also words that are coming from Splitting sentences and composed words;•	Making a word embedding vector to cover all 4 dimensions of the word;Language specificities: yes for words connectors (ex: ''at", “in”) and specific forms of the verbs (ex: “will be", “would”); Dependencies: Splitting sentences/Composed words/ Input processing ll/Database processing/Books processing;Database/ Vocabularies needed: vocabularies for synonyms, regional words, neologisms: vocabulary with connectors;<jupyter_code># To dos:
# 1.	Assess the syntactic functions of words using connectors (ex : "at”, “in”) + other types (like “will be”) + negations; then eliminate these words.
# 2.	Built a matrix (we will name it k-word) that has on the first column the following:
# •	Index no. of the word in the Lexicon/specific vocabularies/additional index for the composed words;
# •	Domain identification; 
# •	NER identification – will be regarded as slot types in training;
# •	The POS of the word;
# •	Frequency of the word in the databases. NER will not have frequencies.
# 3.	Include in the k-word additional columns: synonyms/ neologisms, regional words and words from splitting sentences.
# 4.	The column with the highest frequency or with a slot type will become the k-word vector of the word.
# 5.	Build the embedding vector of the word with the following data range between -3 and 3: frequency in the databases, frequency of the word in the input (if the words repeat in the same input), position of the word inside the input, position of the word vs the verbs/nouns/adjectives, category of the adjectives (from composed words).<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.048811653462816254, 'fraction_numerical': 0.002300025555839509, 'mean_word_length': 4.824404761904762, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 10, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '207617', 'license': 'Apache-2.0', 'n_tokens_mistral': 963, 'n_tokens_neox': 917, 'n_words': 603, 'path': 'NIU-NLU main dummy codes/19_Embeddings_dummy_code.ipynb', 'repo_name': 'Cezanne-ai/project-2021', 'size': '3922'}