Learning Behavior Representations Through Multi-Timescale Bootstrapping
Natural behavior consists of dynamics that are both unpredictable, can switch
suddenly, and unfold over many different timescales. While some success has
been found in building representations of behavior under constrained or
simplified task-based conditions, many of these models cannot be applied to
free and naturalistic settings due to the fact that they assume a single scale
of temporal dynamics. In this work, we introduce Bootstrap Across Multiple
Scales (BAMS), a multi-scale representation learning model for behavior: we
combine a pooling module that aggregates features extracted over encoders with
different temporal receptive fields, and design a set of latent objectives to
bootstrap the representations in each respective space to encourage
disentanglement across different timescales. We first apply our method on a
dataset of quadrupeds navigating in different terrain types, and show that our
model captures the temporal complexity of behavior. We then apply our method to
the MABe 2022 Multi-agent behavior challenge, where our model ranks 3rd overall
and 1st on two subtasks, and show the importance of incorporating
multi-timescales when analyzing behavior.