<jupyter_start><jupyter_text>Using NLP question-answering solution to conversational AI can be tricky if we are seeing the conversation for what it really means . The same is with intent-based solutions. Let’s take some examples:“I am going home” vs “Am I going home?”“I do not intend to go home and watch tv” vs “I do intend to go home and not watch TV”In question-answering or chatbots the hypothesis is that the user is addressing a question which is generally built in a positive sentence. In this way the similarity scores will be high with all the available functions and probably will not have a huge impact if you forget to put the question mark. In a conversation, the two pairs of examples have completely different meanings and, even if you use dense layers for embeddings and a very accurate similarity score, you can fail to understand what the user is saying (especially if you don’t have a large corpus). For this reason, we will build a matrix (CVM) to constantly double check the context and the user’s intention. Another way to do it is to check your pipelines and better optimize the tokenizer and sub-word function that you are using. This method can have some effect for huge corpuses and languages like English, however this is not in scope of our paper. Not to mention that if you are using a pre-trained model you need to stick with the model tokenizer and most of the time is out of your hands. This matrix is crucial for layer 2: Conversational Policies Learning (CPL) that must use this matrix (together with other matrices) to decide the policies and the states of the dialogue/conversation. The following information needs to be retrieved and kept in the matrix:•	If the input is an affirmation, a negation or a question (or a request).•	If the user is talking about the present, the future or the past.•	If the user is speaking in first, second or third person.Principles: •	Each sentence will have a CVM.•	The priorities are as follows: negation, question, affirmation.•	The principal CVM will be chosen in the SPCA  algorithm.•	CVM will be updated in the memory update algorithm.•	For deep conversational inputs without subject, CVM will complete SPCA with the subject.The dependencies with other validation matrices are in scope of layer 2.<jupyter_code># IN: inputs from the pipelines: Grammar & Splitting sentences
# OUT: data for CPL<jupyter_output><empty_output><jupyter_text>Objectives:Populating a 3*3 matrix with 1 or 0 for the following:•	Question/Request or not, Negation or not, Affirmation or not;•	Future, present or past;•	First, second or third person of the verb.Each sentence will have a CVM, if the sum of row 1>1 (priority: negation & question), if sum=0 than affirmation=1. Language Specificities: yes (specific forms of negations, retrieving questions even if “?” doesn't exist);Dependencies: Grammar-Semantics/Splitting sentences/CPLDatabase/ Vocabularies needed: Vocabularies for negations& questions + involve linguists<jupyter_code># To dos:
# 1.	Retrieve info from Grammar-Semantics and populate the 2nd and 3rd row of the CVM.
# 2.	Search each sentence for key words for negation 
# 3.	Searching for wording/punctuation for questions. For every language there are different ways of assessing negations.<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.04527499240352476, 'fraction_numerical': 0.004557885141294439, 'mean_word_length': 4.857651245551601, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 10, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '125101', 'license': 'Apache-2.0', 'n_tokens_mistral': 848, 'n_tokens_neox': 811, 'n_words': 511, 'path': 'NIU-NLU dummy codes/9_CVM_dummy_code.ipynb', 'repo_name': 'Cezanne-ai/project-2021', 'size': '3291'}