@inproceedings{jain-etal-2024-saliency,
    title = "Saliency-Aware Interpolative Augmentation for Multimodal Financial Prediction",
    author = "Jain, Samyak  and
      Chhabra, Parth  and
      Neerkaje, Atula Tejaswi  and
      Mathur, Puneet  and
      Sawhney, Ramit  and
      Agarwal, Shivam  and
      Nakov, Preslav  and
      Chava, Sudheer  and
      Manocha, Dinesh",
    editor = "Calzolari, Nicoletta  and
      Kan, Min-Yen  and
      Hoste, Veronique  and
      Lenci, Alessandro  and
      Sakti, Sakriani  and
      Xue, Nianwen",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.lrec-main.1244",
    pages = "14285--14297",
    abstract = "Predicting price variations of financial instruments for risk modeling and stock trading is challenging due to the stochastic nature of the stock market. While recent advancements in the Financial AI realm have expanded the scope of data and methods they use, such as textual and audio cues from financial earnings calls, limitations exist. Most datasets are small, and show domain distribution shifts due to the nature of their source, suggesting the exploration for data augmentation for robust augmentation strategies such as Mixup. To tackle such challenges in the financial domain, we propose SH-Mix: Saliency-guided Hierarchical Mixup augmentation technique for multimodal financial prediction tasks. SH-Mix combines multi-level embedding mixup strategies based on the contribution of each modality and context subsequences. Through extensive quantitative and qualitative experiments on financial earnings and conference call datasets consisting of text and speech, we show that SH-Mix outperforms state-of-the-art methods by 3-7{\%}. Additionally, we show that SH-Mix is generalizable across different modalities and models.",
}

Predicting price variations of financial instruments for risk modeling and stock trading is challenging due to the stochastic nature of the stock market. While recent advancements in the Financial AI realm have expanded the scope of data and methods they use, such as textual and audio cues from financial earnings calls, limitations exist. Most datasets are small, and show domain distribution shifts due to the nature of their source, suggesting the exploration for data augmentation for robust augmentation strategies such as Mixup. To tackle such challenges in the financial domain, we propose SH-Mix: Saliency-guided Hierarchical Mixup augmentation technique for multimodal financial prediction tasks. SH-Mix combines multi-level embedding mixup strategies based on the contribution of each modality and context subsequences. Through extensive quantitative and qualitative experiments on financial earnings and conference call datasets consisting of text and speech, we show that SH-Mix outperforms state-of-the-art methods by 3-7{\%}. Additionally, we show that SH-Mix is generalizable across different modalities and models.
Jain, Samyak Chhabra, Parth Neerkaje, Atula Tejaswi Mathur, Puneet Sawhney, Ramit Agarwal, Shivam Nakov, Preslav Chava, Sudheer Manocha, Dinesh
Saliency-Aware Interpolative Augmentation for Multimodal Financial Prediction