as the cosine vector similarity using BERT embeddings, denoting the "semantic similarity",
 of \(T\)  and \(Q_{NL}\)  with both \(A_V\)  and \(A_T\) . We record the mean \(\mu \)  and standard deviation \(\sigma \)  for all the comparisons over the randomly sampled data.
<FIGURE>Experimental Results:
The results of our experiment are depicted in Table: REF . On the basis of Levenshtein distance, we observe that the syntactic similarity between the verbalized RDF triple \(A_T\)  and the RDF triple \(T\)  is more than the similarity between \(T\)  and the verbalized answer \(A_V\) , i.e., \(Sim(A_T, T) > Sim(A_V, T)\) . In contrary, the syntactic similarity between \(A_V\)  and the user question \(Q_{NL}\)  is more than the similarity \(Q_{NL}\)  and the verbalized RDF triple \(A_T\) , i.e., \(Sim(A_V, Q_{NL}) > Sim(A_T, Q_{NL})\) . We also observe the same in the case of the cosine similarity measure.
Thus, empirically proving hypothesis H1.
<TABLE>Additionally, Figure: REF  illustrates the results of the experiment calculating the Levenshtein distances. In the first plot, we compare the Levenshtein distance between (\(T\) , \(A_T\) ) and (\(T\) , \(A_V\) ) over the 500 randomly sampled questions. The plot clearly depicts that the purple line indicating the distance between (\(T\) , \(A_T\) ) is higher than the green line indicating the distance between (\(T\) , \(A_V\) ) at most data point, thus demonstrating that the pair (\(T\) , \(A_V\) ) is syntactically less similar than the (\(T\) , \(A_T\) ). Furthermore in the second plot, we compare the Levenshtein distance between (\(Q_{NL}\) , \(A_T\) ) and (\(Q_{NL}\) , \(A_V\) ), which clearly depicts the red line indicating the distance between (\(Q_{NL}\) , \(A_V\) ) is higher than the blue line indicating the distance between (\(Q_{NL}\) , \(A_T\) ). Thus, verifying that \(A_V\)  is syntactically more similar to \(Q_{NL}\)  than \(A_T\) . However, in both case there are a few points of exception where the similarity measures are the same. This is due to questions in the dataset which contain same surface labels as in the RDF triples.

Experiment: 2

Objective:
The objective of this experiment is to record the performance of our baseline models on VANiLLa dataset based on certain evaluation criteria.


Baseline models:
This section presents an overview of our baseline models used for evaluating and assessing the quality of the data. We decided to use some conventional sequence-to-sequence models following the underlying Encoder-Decoder pipeline [1]}, [2]}.
The question \(Q_{NL}\)  and the retrieved answer \(A_T\)  are separated with the separation "\(<\)sep\(>\) " tag to form the input to the encoder and the decoder produces the verbalized answer in natural language \(A_V\)  as the output. The inputs to both the encoder and the decoder end with the end-of-sequence "\(<\)eos\(>\) " tag and the output from the decoder is appended with the start-of-sequence "\(<\)sos\(>\) " tag before the start of prediction. The four baseline models are as follows:
Seq2Seq model with attention mechanism:
Sequence-to-Sequence model with attention mechanism [3]} proposed by Bahdanau simultaneously learns to align and translate by employing a bi-directional RNN as the encoder and using a weighted sum of the vector representation generated at the encoder level as the context in the decoder. These sequence-to-sequence models have been widely used in the NLP community for tasks such as Machine Translation, Question Answering, etc.

Convolution based Encoder-Decoder model:
Facebook AI introduced the use of CNN in sequence learning [4]}. The concept is to implement a fully convolutional architecture, by incorporating CNN at both encoder and decoder level. These models implement a position embedding, which provides an understanding about the portion of input or output sequence currently in consideration, along with a multi-step attention.

Transformer:
Transformers are novel architectures that solves sequence-to-sequence tasks quite nicely because of its ability to handle long-term dependencies. Recently, Transformers achieved state-of-the art in wide range of NLP tasks such as Natural Language Generation, Machine Translation. The architecture of transformer was first proposed by Vaswani et al. [5]}. Followed by that many pre-trained transformer models are proposed such as BERT [6]}, GPT [7]}, RoBERTa [8]}, which are trained on a large number of data. For simplicity, we use a simple transformer model for the evaluation of our dataset. Multi-headed attention and positional-encoding are used in this transformer model. Multi-headed attention helps to capture global dependencies between input and output and positional-encoding encodes information about the relative or absolute position of each token in the sequence.



Dataset:
We use our dataset VANiLLa with a 80-20 split to generate the training and test sets. The training set consists of 85732 examples whereas the test set consists of 21434 examples. We also implement a \(k\) -cross validation with \(k=5\)  on training set for generating the validation set in both cases.


Experimental Settings:
We use standard dropout value of 0.5 for the recurrent layers, 0.25 for convolution layers and 0.1 for multi-head attention mechanism. The hidden layers dimensions for transformers is set to 200 with embedding dimension of 200, while for the other baselines are set the hidden dimension to 512 and embedding dimension to 300. In the case of Convolution based Encoder Decoder model, we utilize 10 layers of convolutions with kernel size 3. For the multi-headed attention mechanism in Transformer model, we set 8 as the number of head with 4 layers. For the sake of consistency, we train all the models for
7 epochs on the our dataset, with Adam optimizer and cross entropy loss function.


Evaluation Criteria:
We implement the following criteria for the evaluation and comparison of our baseline models:
Criteria#1
Perplexity: Perplexity defines the quality of predicting real sentences for a probability model. A good model will have a lower perplexity indicating a high probability of prediction. It is the average branching factor for predicting the next word. It is calculated with the help of cross-entropy as follows:
\(PPL = 2^{H(p,q)}\) 
where, \(H(p,q)\)  is the cross-entropy of the model which tries to learn the probability distribution \(q\)  close to the unknown probability distribution of the data \(p\) .
Criteria#2
Modified Precision: Modified Precision is the sum of the clipped n-gram counts of the generated sequence or hypothesis divided by the total number of n-grams in the hypothesis. The range of the modified precision is between 100 and 0 with 100 being the best score. This scores depicts two aspects:
 [(i)]

adequacy (generated hypothesis sentence contains similar words as that in the reference sentence), and

fluency (the n-gram matches between the hypothesis and reference).
Criteria#3
BLEU: The BLEU score, Bilingual Evaluation Understudy, is a modified precision score [9]} combined with a brevity penalty for penalizing shorter generation. Hence, the BLEU score checks for matches in length, order and word choice between the hypothesis and the references. The range of BLEU score is between 100 and 0 with 100 being the best score.

<FIGURE>
<TABLE>

Experimental Results:
The perplexity value on the validation set after every epoch for all our baselines are plotted in Figure: REF  and the test results of the three baseline models trained on the two sizes of our dataset, VANiLLa has been reported in table: REF . The results clearly indicate that the transformer model performs the best among the three baselines with perplexity value of 12.10 and BLEU score of 30.80.
The transformer model puts attention on the whole sentence instead of attention on word by word, hence capturing the context better. Additionally, transformer handles the relationship between words in a sentence by positional encoding and multi-headed attention mechanism [5]}. These characteristics helps transformer model achieve better result than other baseline models.

Experiment: 3
Objective:
Empirically it is evident that the transformer model is the best amongst the baseline models. Hence, in this experiment we examine the improvement of the evaluation metrics for the transformer model trained on the combination of VANiLLa and VQuAnDa [11]} datasets (5000 question-answer pairs).
<TABLE>

Experimental Settings:
In this experiment we combine the training data of VANiLLa with the training data of VQuAnDa dataset and feed it into the transformer model. Although, we combine the training data for the training phase but we test the model on both the VANiLLa and VQuAnDa separately. Note that the majority of the combined data is from VANiLLa as the number of data in VANiLLa is 20 times more than VQuAnDa.

Experimental Results:
The combined data used for training improves the precision score (80.45) of VQuAnDa test data by a fair margin. Additionally, transformer model achieves best BLEU score on VANiLLa test set. So, it is evident that this combined data training approach helps in creating a strong baseline for future research. Furthermore, our baseline transformer model with the suggested parameters (described in the experimental settings) achieve SOTA in BLEU score on VQuAnDa data which is 26.25 where in VQuAnDa paper BLEU is reported 18.18.

Conclusion and Future Work
We introduce the VANiLLa dataset with the intent to support the verbalization of answers in the KGQA task. The dataset consisting of over 100k examples was generated semi-automatically by adapting questions from other datasets such as CSQA and SimpleQuestionsWikidata and then verbalizing the answers by crowd-sourcing. We empirically show that the verbalizations in our dataset are syntactically and semantically closer to the question than to the KG triple labels. Additionally, we evaluate multiple baseline models based on generic sequence to sequence models already in use in the NLP community. These baseline models present a reference point for future research. The experimental results of these baselines indicate that there still exists substantial room for enhancement for solving the task of verbalizing answers.
One major use case of answer verbalizations are dialogue systems. We believe that our dataset along with our design framework presents a valuable contribution in this field. In the future, we plan to continuously aid researchers with more variations and expansions of our dataset.
