@inproceedings{
cai2023jolt,
title={Jo{LT}: Jointly Learned Representations of Language and Time-Series},
author={Yifu Cai and Mononito Goswami and Arjun Choudhry and Arvind Srinivasan and Artur Dubrawski},
booktitle={Deep Generative Models for Health Workshop NeurIPS 2023},
year={2023},
url={https://openreview.net/forum?id=UVF1AMBj9u}
}
Time-series and text data is prevalent in healthcare and frequently exist in tandem, for e.g., in electrocardiogram (ECG) interpretation reports. Yet, these modalities are typically modeled independently. Even studies that jointly model time-series and text do so by converting time-series to images or graphs. We hypothesize that explicitly modeling time-series jointly with text can improve tasks such as summarization and question answering for time-series data, which have received little attention so far. To address this gap, we introduce JoLT to jointly learn desired representations from pre-trained time-series and text models. JoLT utilizes a Querying Transformer (Q-Former) to align the time-series and text representations. Our experiments on a large real-world electrocardiography dataset for medical time-series summarization show that JoLT outperforms state-of-the-art image captioning and medical question-answering approaches, and that the decoder architecture, size, and pre-training data can vary the performance on said tasks.
JoLT: Jointly Learned Representations of Language and Time-Series
Yifu Cai Mononito Goswami Arjun Choudhry Arvind Srinivasan Artur Dubrawski