@inproceedings{kwan-etal-2024-jotr,
    title = "{J}o{TR}: A Joint Transformer and Reinforcement Learning Framework for Dialogue Policy Learning",
    author = "Kwan, Wai-Chung  and
      Wang, Huimin  and
      Wang, Hongru  and
      Wang, Zezhong  and
      Liang, Bin  and
      Wu, Xian  and
      Zheng, Yefeng  and
      Wong, Kam-Fai",
    editor = "Calzolari, Nicoletta  and
      Kan, Min-Yen  and
      Hoste, Veronique  and
      Lenci, Alessandro  and
      Sakti, Sakriani  and
      Xue, Nianwen",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.lrec-main.837",
    pages = "9578--9588",
    abstract = "Dialogue policy learning (DPL) aims to determine an abstract representation (also known as action) to guide what the response should be. Typically, DPL is cast as a sequential decision problem across a series of predefined action candidates. However, such static and narrow actions can limit response diversity and impede the dialogue agent{'}s adaptability to new scenarios and edge cases. To overcome these challenges, we introduce a novel \textbf{Jo}int \textbf{T}ransformer \textbf{R}einforcement Learning framework, coined as \textbf{JoTR}, where a text-to-text Transformer-based model is employed to directly generate dialogue actions. More concretely, JoTR formulates a token-grained policy, facilitating more dynamic and adaptable dialogue action generation without the need for predefined action candidates. This method not only enhances the diversity of responses but also significantly improves the system{'}s capability to manage unfamiliar scenarios. Furthermore, JoTR utilizes Reinforcement Learning with a reward-shaping mechanism to efficiently fine-tune the token-grained policy. This allows the model to evolve through interactions, thereby enhancing its performance over time. Our extensive evaluation demonstrates that JoTR surpasses previous state-of-the-art models, showing improvements of 9{\%} and 13{\%} in success rate, and 34{\%} and 37{\%} in the diversity of dialogue actions across two benchmark dialogue modeling tasks respectively. These results have been validated by both user simulators and human evaluators. Code and data are available at ://github.com/KwanWaiChung/JoTR.",
}

Dialogue policy learning (DPL) aims to determine an abstract representation (also known as action) to guide what the response should be. Typically, DPL is cast as a sequential decision problem across a series of predefined action candidates. However, such static and narrow actions can limit response diversity and impede the dialogue agent{'}s adaptability to new scenarios and edge cases. To overcome these challenges, we introduce a novel \textbf{Jo}int \textbf{T}ransformer \textbf{R}einforcement Learning framework, coined as \textbf{JoTR}, where a text-to-text Transformer-based model is employed to directly generate dialogue actions. More concretely, JoTR formulates a token-grained policy, facilitating more dynamic and adaptable dialogue action generation without the need for predefined action candidates. This method not only enhances the diversity of responses but also significantly improves the system{'}s capability to manage unfamiliar scenarios. Furthermore, JoTR utilizes Reinforcement Learning with a reward-shaping mechanism to efficiently fine-tune the token-grained policy. This allows the model to evolve through interactions, thereby enhancing its performance over time. Our extensive evaluation demonstrates that JoTR surpasses previous state-of-the-art models, showing improvements of 9{\%} and 13{\%} in success rate, and 34{\%} and 37{\%} in the diversity of dialogue actions across two benchmark dialogue modeling tasks respectively. These results have been validated by both user simulators and human evaluators. Code and data are available at ://github.com/KwanWaiChung/JoTR.
Kwan, Wai-Chung Wang, Huimin Wang, Hongru Wang, Zezhong Liang, Bin Wu, Xian Zheng, Yefeng Wong, Kam-Fai
JoTR: A Joint Transformer and Reinforcement Learning Framework for Dialogue Policy Learning