@inproceedings{
shi2024lcaontheline,
title={{LCA}-on-the-Line: Benchmarking Out of Distribution Generalization with Class Taxonomies},
author={Jia Shi and Gautam Rajendrakumar Gare and Jinjin Tian and Siqi Chai and Zhiqiu Lin and Arun Balajee Vasudevan and Di Feng and Francesco Ferroni and Shu Kong and Deva Ramanan},
booktitle={NeurIPS 2023 Workshop on Distribution Shifts: New Frontiers with Foundation Models},
year={2024},
url={https://openreview.net/forum?id=7CUutNeDDg}
}
We introduce `Least Common Ancestor (LCA)-on-the-line' as a method for predicting models' Out-of-Distribution (OOD) performance using in-distribution measurements, without the need for OOD data. We revisit the LCA distance, a concept from the pre-deep-learning era, which calculates the hierarchical distance between labels and predictions in a predefined class hierarchy tree, such as WordNet. Our evaluation of 75 models across five significantly shifted ImageNet-OOD datasets demonstrates the robustness of LCA-on-the-line. It reveals a strong linear correlation between in-domain ImageNet LCA distance and OOD Top-1 accuracy across various datasets, including ImageNet-S/R/A/ObjectNet. Compared to previous methods such as Accuracy-on-the-line and Agreement-on-the-line, LCA-on-the-line shows superior generalization across a wide range of models. This includes models trained with different supervision types, such as class labels for vision models (VMs) and textual captions for vision-language models (VLMs). Our method offers a compelling alternative perspective on why vision-language models tend to generalize better to OOD data compared to vision models, even those with similar or lower in-domain (ID) performance. In addition to presenting an OOD performance indicator, we also demonstrate that aligning model predictions more closely with the class hierarchy and integrating a training loss objective with soft-labels can enhance model OOD performance.
LCA-on-the-Line: Benchmarking Out of Distribution Generalization with Class Taxonomies
Jia Shi Gautam Rajendrakumar Gare Jinjin Tian Siqi Chai Zhiqiu Lin Arun Balajee Vasudevan Di Feng Francesco Ferroni Shu Kong Deva Ramanan