|
--- |
|
license: mit |
|
datasets: |
|
- allenai/MADLAD-400 |
|
language: |
|
- en |
|
- ru |
|
- bg |
|
- uk |
|
- kk |
|
base_model: |
|
- mistralai/Mistral-7B-v0.1 |
|
--- |
|
VocADT is a solution for vocabulary adaptation using adapter modules that are trained to learn the optimal linear combination of existing embeddings while keeping the model’s weights fixed. |
|
VocADT offers a flexible and scalable solution without requiring external resources or language constraints. |
|
|
|
|
|
## New Vocabulary Adapted Models |
|
Only the input/output embeddings are replaced, while all other original weights of base model remain fixed. |
|
These are the merged version: after training the adapters, we merge the original embeddings with the adapter to generate the new embeddings. |
|
| Name | Adapted Model | Base Model | New Vocab Size | Focused Languages | |
|
|---|---|---|---|---| |
|
| VocADT-Latin | [h-j-han/Mistral-7B-VocADT-50k-Latin](https://huggingface.co/h-j-han/Mistral-7B-VocADT-50k-Latin) | [Mistral](https://huggingface.co/mistralai/Mistral-7B-v0.1) | 50k | Swahili (sw), Indonesian (id), Estonian (et), Haitian Creole (ht), English (en)| |
|
| VocADT-Mixed | [h-j-han/Mistral-7B-VocADT-50k-Mixed](https://huggingface.co/h-j-han/Mistral-7B-VocADT-50k-Mixed) | [Mistral](https://huggingface.co/mistralai/Mistral-7B-v0.1) | 50k | Korean (ko), Greek (el), Russian (ru), Bulgarian (bg), English (en) | |
|
| VocADT-Cyrillic | [h-j-han/Mistral-7B-VocADT-50k-Cyrillic](https://huggingface.co/h-j-han/Mistral-7B-VocADT-50k-Cyrillic) | [Mistral](https://huggingface.co/mistralai/Mistral-7B-v0.1) | 50k | Russian (ru), Bulgarian (bg), Ukrainian (uk), Kazakh (kk), English (en) | |
|
|
|
|
|
## Quick Start |
|
```python |
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
|
# model_name = "mistralai/Mistral-7B-v0.1 # Base Model |
|
model_name = "h-j-han/Mistral-7B-VocADT-50k-Cyrillic" # Vocabulary Adapted Model |
|
tokenizer = AutoTokenizer.from_pretrained(model_name) |
|
model = AutoModelForCausalLM.from_pretrained(model_name) |
|
|
|
prefix = "\nEnglish: Hello!\nUkrainian: Добрий день!\nEnglish: How are you?\nUkrainian: Як справи?\nEnglish: " |
|
line = "Do you speak English?" |
|
suffix = f"\nUkrainian:" |
|
prompt = prefix + line + suffix |
|
|
|
inputs = tokenizer(prompt, return_tensors="pt") |
|
outputs = model.generate(**inputs, max_new_tokens=7) |
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
|
|
|
# Base Model Output: "Чи мовляєш англі" # Wrong output and need more tokens to complete |
|
# VocADT Output: "Чи розмовляєш ти англійською?" # Complete and good output within 7 tokens |
|
``` |
|
|
|
## Reference |
|
We provide code in Github repo: https://github.com/h-j-han/VocADT |
|
Also, please find details in this paper: |
|
``` |
|
@misc{han2024vocadt, |
|
title={Adapters for Altering LLM Vocabularies: What Languages Benefit the Most?}, |
|
author={HyoJung Han and Akiko Eriguchi and Haoran Xu and Hieu Hoang and Marine Carpuat and Huda Khayrallah}, |
|
year={2024}, |
|
eprint={2410.09644}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.CL}, |
|
url={https://arxiv.org/abs/2410.09644}, |
|
} |
|
``` |