In recent years, dynamic vision sensors (DVS), also known as event-based cameras or neuromorphic sensors, have seen increased use due to various advantages over traditional frame-based cameras. Its high temporal resolution overcomes motion blurring, its high dynamic range overcomes extreme illumination conditions and its low power consumption makes it suitable for platforms such as drones and self-driving cars. While frame-based computer vision is mature due to the large amounts of data and ground truth available, event-based computer vision is still a work in progress as data sets are not as aplenty and ground truth is scarce for tasks such as object detection. In this paper, we suggest a way to overcome the lack of labeled ground truth by introducing a simple method to generate pseudo-labels for dynamic vision sensor data, assuming that the corresponding frame-based data is also available. These pseudo-labels can be treated as ground truth when training on supervised learning algorithms, and we show, for the first time, event-based car detection under ego-motion in a realistic environment at 100 frames per second with an average precision of 41.7 relative to pseudo-ground truth.
We learn rich natural sound representations by capitalizing on large amounts of unlabeled sound data collected in the wild. We leverage the natural synchronization between vision and sound to learn an acoustic representation using two-million unlabeled videos. Unlabeled video has the advantage that it can be economically acquired at massive scales, yet contains useful signals about natural sound. We propose a student-teacher training procedure which transfers discriminative visual knowledge from well established visual recognition models into the sound modality using unlabeled video as a bridge. Our sound representation yields significant performance improvements over the state-of-the-art results on standard benchmarks for acoustic scene object classification. Visualizations suggest some high-level semantics automatically emerge in the sound network, even though it is trained without ground truth labels. In this work we propose a technique that transfers supervision between images from different modalities. We use learned representations from a large labeled modality as a supervisory signal for training representations for a new unlabeled paired modality. Our method enables learning of rich representations for unlabeled modalities and can be used as a pre-training procedure for new modalities with limited labeled data. We show experimental results where we transfer supervision from labeled RGB images to unlabeled depth and optical flow images and demonstrate large improvements for both these cross modal supervision transfers. Code, data and pre-trained models are available at this https URL