File size: 1,432 Bytes
49419dc fd15a67 f63bcb5 9c13556 bac9f3a |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 |
---
license: mit
pipeline_tag: image-to-text
datasets:
- antoniorv6/camera_grandstaff
tags:
- omr
- camera_grandstaff
arxiv: 2402.07596
---
# Sheet Music Transformer (base model, fine-tuned on the Camera Grandstaff dataset)
The SMT model fine-tuned on the _Camera_ GrandStaff dataset for pianoform transcription.
The code of the model is hosted in [this repository](https://github.com/antoniorv6/SMT).
## Model description
The SMT model consists of a vision encoder (ConvNext) and a text decoder (classic Transformer).
Given an image of a music system, the encoder first encodes the image into a tensor of embeddings (of shape batch_size, seq_len, hidden_size), after which the decoder autoregressively generates text, conditioned on the encoding of the encoder.
<img src="https://github.com/antoniorv6/SMT/raw/master/graphics/SMT.jpg" alt="drawing" width="720"/>
## Intended uses & limitations
This model is fine-tuned on the _Camera_ GrandStaff dataset, its use is limited to transcribe pianoform images only.
### BibTeX entry and citation info
```bibtex
@misc{RiosVila2024,
title={Sheet Music Transformer: End-To-End Optical Music Recognition Beyond Monophonic Transcription},
author={Antonio Ríos-Vila and Jorge Calvo-Zaragoza and Thierry Paquet},
year={2024},
eprint={2402.07596},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2402.07596},
}
``` |