In this paper, we explore learning from two training sets: of images and layouts. Specifically, we use panoptic segmentations [1]}, which specify for every segment \(s_i\)  its instance identity \(p_i\)  and semantic category \(m_i\) , but other types of segmentations can likewise be used. The images and layouts can either be paired or unpaired, and our training scheme accommodates both options: either by pre-training each stage independently first and then fine-tune them jointly in the paired case, or training them together from scratch in the unpaired one (See section  for further details).
