Answering visual queries is a complex task that requires both visual processing and reasoning.
End-to-end models, the dominant approach for this task, do not explicitly differentiate between the two, limiting interpretability and generalization.
Learning modular programs presents a promising alternative, but has proven challenging due to the difficulty of learning both the programs and modules simultaneously.
We introduceViperGPT, a framework that leverages code-generation models to compose vision-and-language models into subroutines to produce a result for any query.ViperGPTutilizes a provided API to access the available modules, and
composes them by generating Python code that is later executed.
This simple approach requires no further training, and achieves state-of-the-art results across various complex visual tasks.