@inproceedings{
jiang2023on,
title={On Transferring Expert Knowledge from Tabular Data to Images},
author={Jun-Peng Jiang and Han-Jia Ye and Leye Wang and Yang Yang and Yuan Jiang and De-Chuan Zhan},
booktitle={UniReps:  the First Workshop on Unifying Representations in Neural Models},
year={2023},
url={https://openreview.net/forum?id=qNZ3Tyrq9C}
}
Transferring knowledge across modalities has garnered significant attention in the field of machine learning as it enables the utilization of expert knowledge from diverse domains. In particular, the representation of expert knowledge in tabular form, commonly found in fields such as medicine, can greatly enhance the comprehensiveness and accuracy of image-based learning.
However, the transfer of knowledge from tabular to image data presents unique challenges due to the distinct characteristics of these data types, making it challenging to determine "how to reuse" and "which subset to reuse". To address this, we propose a novel method called CHannel tAbulaR alignment with optiMal tranSport (CHARMS) that automatically and effectively transfers relevant tabular knowledge. Specifically, by maximizing the mutual information between a group of channels and tabular features, our method modifies the visual embedding and captures the semantics of tabular knowledge. The alignment between channels and attributes helps select the subset of tabular data which contains knowledge to images. Experimental results demonstrate that {\sc Charms} effectively reuses tabular knowledge to improve the performance and interpretability of visual classifiers.
On Transferring Expert Knowledge from Tabular Data to Images
Jun-Peng Jiang Han-Jia Ye Leye Wang Yang Yang Yuan Jiang De-Chuan Zhan