<jupyter_start><jupyter_text>Spoken Language Understanding (SLU) Semantic decoding system is responsible for decoding the meaning of utterances sent to the system. Often dialogue systems are deployed in a noisy setting, where background noise and diverse user populations can result in high speech recognition error rates. Thus, SLU systems operate on the top $k$ hypotheses output by the speech recogniser and this creates additional challenges for effective transformation of spoken utterances to dialogue acts. Let's look at the illustrative procedure:<jupyter_code>Dialogue turn: Is there maybe a cheap place in the north of town please?
Dialogue act type: inform
Semantic slots: price, area
Semantic values: cheap, north
Dialogue act: inform(price=cheap, area=north)<jupyter_output><empty_output><jupyter_text>As we can see, during the semantic decoding process we extract the information about dialogue act type,which encodes the system or the user intention in a dialogue turn, and semantic slots and values that furtherdescribe entities from the ontology that a dialogue turn refers to.We can tackle this problem from two perspectives - first one involves creation of list of grammar rules that parse user utterances. Second approach relies on statistically trained models where we train classifiers to directly label utterances based on training data.  Content:* [Rule-based models](first-bullet)* [Support Vector Machines](6-bullet)* [PyDial/semi](0-bullet)  Rule-based model: A natural solution to the problem is to create a set of rules that transforms the utterance to the dialogue act basedon the domain ontology. In the case of the general communication with the system, they might look like this:<jupyter_code>self.rHELLO = "(\b|^|\ )(hi|hello)\s"
self.rNEG =  "(\b|^|\ )(no\b|wrong|incorrect|error)|not\ (true|correct|right)\s" 
self.rAFFIRM = "(yes|ok\b|OK\b|okay|sure|(that('?s| is) )?right)" 
self.rBYE = "(\b|^|\ )(bye|goodbye|that'*s*\ (is\ )*all)(\s|$|\ )"
self.GREAT = "(great|good|awesome)"
self.HELPFUL = "(that((\')?s|\ (is|was))\ (very\ )?helpful)"<jupyter_output><empty_output><jupyter_text>More specific expressions depend completely on the domain that we are operating on.  A general framework for implementation rule-based system can be derived from `semi/RegexSemI.py`.You can find specific scripts with regular expression for every domain in folder `semi/RegexSemI_DOMAIN`. In order to test this model you can just run:<jupyter_code>pydial chat config/texthub.cfg<jupyter_output><empty_output><jupyter_text>To use it in your specific config file you have to change in `semi` section the variable `semitype` to `RegexSemI`, for example:<jupyter_code>[semi_CamRestaurants]
semitype = RegexSemI<jupyter_output><empty_output><jupyter_text>Support Vector Machines  As it was mentioned previously, we can look at semantic decoding as a classification task in whichwe need to make predictions about the set of semantic concepts. However, this requires corpora data with labelled semantic concepts. Such an approach can easily handle $N$-best list of automatic speech recogniser hypotheses as in real conversational systems error rate of the top hypothesis is typically about 20-30% and thus robust system is needed.Classification model that is implemented in PyDial is support vector machines which map input into a high-dimensional feature space where data is linearly separable. It was shown that such statistical approach can substantially  improve  performance  both  in  terms  of  accuracy  and  overall  dialogue  reward. To use this approach with your specific config file you have to change in `semi` section the variable `semitype` to `SVMSemI`, for example:<jupyter_code>[semi_CamRestaurants]
semitype = SVMSemI<jupyter_output><empty_output><jupyter_text>You can test it with just chatting with the system through pydial:<jupyter_code>pydial chat config/texthub_svm.cfg<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07112014224028448, 'fraction_numerical': 0.001524003048006096, 'mean_word_length': 5.884615384615385, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '189450', 'license': 'Apache-2.0', 'n_tokens_mistral': 1090, 'n_tokens_neox': 1046, 'n_words': 485, 'path': 'Tutorials/Spoken Language Understanding.ipynb', 'repo_name': 'helloric/pydial3', 'size': '3937'}