We compare DILOG with two baselines: DAMD [1]} is the state-of-the-art model with dialog action prediction, and MLP is a naive baseline. We calculate the following metrics to evaluate the quality of the models: Inform is a per diaog metric that measures whether the system finally provides a correct entity, Success measures whether all the requested information are provided, and Action F1 is a per turn metric that checks if the predicted dialog action (both intent and entities) matches the ground truth.
We use a template-based NLG to convert dialog actions predicted by DILOG and MLP to compute inform and success on generated utterances. There are three slots (postcode, phone, address) that only appear in the system actions but not dialog states. We manually add an option to the template-based NLG to inform those slots whenever an entity is informed, and denote that variant as -AM, which significantly improves the success rate. We do not compare the -AM variant against other methods.
