Automatic Concept Extraction for Concept Bottleneck-based Video Classification
Recent efforts in interpretable deep learning models have shown that
concept-based explanation methods achieve competitive accuracy with standard
end-to-end models and enable reasoning and intervention about extracted
high-level visual concepts from images, e.g., identifying the wing color and
beak length for bird-species classification. However, these concept bottleneck
models rely on a necessary and sufficient set of predefined concepts-which is
intractable for complex tasks such as video classification. For complex tasks,
the labels and the relationship between visual elements span many frames, e.g.,
identifying a bird flying or catching prey-necessitating concepts with various
levels of abstraction. To this end, we present CoDEx, an automatic Concept
Discovery and Extraction module that rigorously composes a necessary and
sufficient set of concept abstractions for concept-based video classification.
CoDEx identifies a rich set of complex concept abstractions from natural
language explanations of videos-obviating the need to predefine the amorphous
set of concepts. To demonstrate our method's viability, we construct two new
public datasets that combine existing complex video classification datasets
with short, crowd-sourced natural language explanations for their labels. Our
method elicits inherent complex concept abstractions in natural language to
generalize concept-bottleneck methods to complex tasks.