Multimodal models for various tasks with both image and text input have been widely studied.  [1]} and  [2]} achieve good performance in SIMMC 1.0 by using visual information as a text description of the object. However, these approaches have limitations in that object descriptions must be given, and it is difficult to regard them as a multimodal model using image information. MTN [3]} achieved the best performance in DSTC 7 (Video Scene-aware Dialogue) [4]} and is a baseline model of SIMMC 2.0. MTN extracts each feature for each modal and generates an appropriate response through a transformer [5]}.  [6]} improves performance by pretraining VilBERT [7]} for Conceptual Captions and Visual Question Answering and finetuning for VisDial. CLIP [8]} is trained to match the transformer between image and text description, so it has the advantage of text-image matching for the unseen object class.
