For the IC task, evaluation is performed on the utterance level. The typical evaluation metric for IC is accuracy, calculated as the number of the correct predictions made by the model divided by the total number of predictions. As for SF, the evaluation is performed on the entity level. The common metrics used is the metric introduced in CoNLL-2003 shared task [1]} to evaluate Named Entity Recognition (NER) by computing the F-1 score. The F1-score, is the harmonic mean score between precision and recall. Precision is the percentage of slot predictions from the model which are correct, while recall is the percentage of slots in the corpus that are found by the model. A slot prediction is considered correct when an exact match is found [1]}. As the slot is annotated in BIO format to mark the boundary of the slot (see Table REF ), a correct prediction is only counted when the model can predict the correct slot label on the correct token offset. Consequently, the exact match metrics does not reward cases when the model predict correct slot label but get the incorrect slot boundary (partial match).
