@inproceedings{
lin2024icondm,
title={Icon{DM}: Text-Guided Icon Set Expansion Using Diffusion Models},
author={Jiawei Lin and Zhaoyun Jiang and Jiaqi Guo and Shizhao Sun and Ting Liu and Zijiang James Yang and Jian-Guang Lou and Dongmei Zhang},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=SsVVrDheMH}
}
Icons are ubiquitous visual elements in graphic design. However, their creation is non-trivial and time-consuming. To this end, we draw inspiration from the booming text-to-image field and propose Text-Guided Icon Set Expansion, a task that allows users to create novel and style-preserving icons using textual descriptions and a few handmade icons as style reference. Despite its usefulness, this task poses two unique challenges. (i) Abstract Concept Visualization. Abstract concepts like technology and health are frequently encountered in icon creation, but their visualization requires a mental grounding process that connects them to physical and easy-to-draw concepts. (ii) Fine-grained Style Transfer. Unlike ordinary images, icons exhibit far richer fine-grained stylistic elements, including tones, line widths, shapes, shadow effects, etc, setting a higher demand on capturing and preserving them during generation.

To address the challenges, we propose IconDM, a method based on pre-trained text-to-image (T2I) diffusion models. It involves a one-shot domain adaptation process and an online style transfer process. The domain adaptation aims to improve the pre-trained T2I model in understanding abstract concepts by finetuning on high-quality icon-text pairs. To achieve so, we construct IconBank, a large-scale dataset of 2.3 million icon-text pairs, where the texts are generated by the state-of-the-art vision-language model from icons. In style transfer, we introduce a Style Enhancement Mod- ule into the T2I model. It explicitly extracts the fine-grained style features from the given reference icons, and is jointly optimized with the T2I model during DreamBooth tuning. To assess IconDM, we present IconBench, a structured suite with 30 icon sets and 100 concepts (including 50 abstract concepts) for generation. Quantitative results, qualitative analysis, and extensive ablation studies demonstrate the effectiveness of IconDM.
IconDM: Text-Guided Icon Set Expansion Using Diffusion Models
Jiawei Lin Zhaoyun Jiang Jiaqi Guo Shizhao Sun Ting Liu Zijiang James Yang Jian-Guang Lou Dongmei Zhang