metadata

datasets:
  - mhan/Shot2Story-20K
  - mhan/shot2story
language:
  - en
metrics:
  - bleu
pipeline_tag: visual-question-answering

Shot2Story: A New Benchmark for Comprehensive Understanding of Multi-shot Videos

Repository: Shot2Story
Paper: 2312.10300
Point of Contact: mailto:Mingfei Han

Training Dataset

Please download the multi-shot videos here.

We are excited to release a new video-text benchmark for multi-shot video understanding. This release contains a 134k version of our dataset. It includes detailed long summaries (human annotated + GPTV generated) for 134k videos and shot captions (human annotated) for 188k video shots. Please check the dataset here.

Models

We are releasing the checkpoints trained with our Shot2Story-20K and Shot2Story-134K.

{20k,134k}-version/sum_shot_best_epoch.pth: Model tuned on our multi-shot summary data. Used in the config files ckpt.
{20k,134k}-version/shot_av_best_epoch.pth: Model trained on our single-shot caption data. Used in the config files ckpt.
transnetv2-pytorch-weights.pth: Checkpoint used for automatic shot detection method, which is used in the Bot demo. Please following the original license of the TransNetv2.
BLIP.cache.tar: Cached checkpoints for training, testing and offline demos. This is only to ease the usage case that servers can't access huggingface. Please be restriected the original license to the different models.

License

Our text annotations are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0) License. They are available strictly for non-commercial research.

Users must refer to HD-VILA-100M for original video access. By downloading our annotations, you agree to these terms. Respect for video copyright holders is paramount. Ensure your use of the videos aligns with the original source's terms.

Citation

If you find our work useful for your research, please consider citing the paper

@misc{han2023shot2story20k,
    title={Shot2Story20K: A New Benchmark for Comprehensive Understanding of Multi-shot Videos}, 
    author={Mingfei Han and Linjie Yang and Xiaojun Chang and Heng Wang},
    year={2023},
    eprint={2312.10300},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}