Safetensors
mistral
File size: 3,043 Bytes
1b581ab
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
---
license: mit
datasets:
- allenai/MADLAD-400
language:
- en
- ko
- el
- ru
- bg
base_model:
- mistralai/Mistral-7B-v0.1
---
VocADT is a solution for vocabulary adaptation using adapter modules that are trained to learn the optimal linear combination of existing embeddings while keeping the model’s weights fixed.
VocADT offers a flexible and scalable solution without requiring external resources or language constraints.


## New Vocabulary Adapted Models
Only the input/output embeddings are replaced, while all other original weights of base model remain fixed.
These are the merged version: after training the adapters, we merge the original embeddings with the adapter to generate the new embeddings.
| Name | Adapted Model | Base Model | New Vocab Size | Focused Languages |
|---|---|---|---|---|
| VocADT-Latin | [h-j-han/Mistral-7B-VocADT-50k-Latin](https://huggingface.co/h-j-han/Mistral-7B-VocADT-50k-Latin) | [Mistral](https://huggingface.co/mistralai/Mistral-7B-v0.1) | 50k | Swahili (sw), Indonesian (id), Estonian (et), Haitian Creole (ht), English (en)|
| VocADT-Mixed | [h-j-han/Mistral-7B-VocADT-50k-Mixed](https://huggingface.co/h-j-han/Mistral-7B-VocADT-50k-Mixed) | [Mistral](https://huggingface.co/mistralai/Mistral-7B-v0.1) | 50k | Korean (ko), Greek (el), Russian (ru), Bulgarian (bg), English (en) |
| VocADT-Cyrillic | [h-j-han/Mistral-7B-VocADT-50k-Cyrillic](https://huggingface.co/h-j-han/Mistral-7B-VocADT-50k-Cyrillic) | [Mistral](https://huggingface.co/mistralai/Mistral-7B-v0.1) | 50k | Russian (ru), Bulgarian (bg), Ukrainian (uk), Kazakh (kk), English (en) |


## Quick Start
```python
from transformers import AutoModelForCausalLM, AutoTokenizer

# model_name = "mistralai/Mistral-7B-v0.1 # Base Model
model_name = "h-j-han/Mistral-7B-VocADT-50k-Mixed" # Vocabulary Adapted Model
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

prefix = "\nEnglish: Hello \nKorean: μ•ˆλ…•ν•˜μ„Έμš” \nEnglish: Thank you\nKorean: κ³ λ§™μŠ΅λ‹ˆλ‹€\nEnglish: "
line = "I lived in Korea for seven years"
suffix = f"\nKorean:"
prompt = prefix + line + suffix

inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=10)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

# Base Model Output: "ν•œκ΅­μ— 7λ…„ μ‚΄" # This short incomplete phrase in Korean is 10 tokens for the base model.
# VocADT Output: "μ €λŠ” ν•œκ΅­μ— 7λ…„ λ™μ•ˆ μ‚΄μ•˜μŠ΅λ‹ˆλ‹€." # Complete and good output within 10 tokens
```

## Reference
We provide code in Github repo : https://github.com/h-j-han/VocADT  
Also, please find details in this paper :
```
@misc{han2024vocadt,
      title={Adapters for Altering LLM Vocabularies: What Languages Benefit the Most?}, 
      author={HyoJung Han and Akiko Eriguchi and Haoran Xu and Hieu Hoang and Marine Carpuat and Huda Khayrallah},
      year={2024},
      eprint={2410.09644},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2410.09644}, 
}
```