File size: 9,018 Bytes
96ed42b d07be75 826de2e 96ed42b c58a89d 2e03180 96ed42b 0004e73 96ed42b 17ecd35 ae50909 6ca0c05 17ecd35 ae50909 17ecd35 ae50909 6ca0c05 17ecd35 ae50909 17ecd35 4aff21d 96ed42b da41283 96ed42b 5988290 59c4c44 6138e69 684d079 6138e69 684d079 6138e69 b72a5ef 6138e69 684d079 6138e69 684d079 6138e69 b72a5ef 6138e69 59c4c44 96ed42b 17ecd35 96ed42b 252f3f9 96ed42b 17ecd35 96ed42b 4aff21d 96ed42b 5ffe8ae 96ed42b 17ecd35 96ed42b 4aff21d 96ed42b 4fd96f0 96ed42b fd84ee3 7ee6d92 fd84ee3 5da6e9e ce7071f d07be75 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 |
---
language:
- en
- ko
license: llama3
library_name: transformers
base_model:
- meta-llama/Meta-Llama-3-8B
---
<a href="https://github.com/MLP-Lab/Bllossom">
<img src="https://github.com/teddysum/bllossom/blob/main//bllossom_icon.png?raw=true" width="40%" height="50%">
</a>
# Bllossom | [Demo]() | [Homepage](https://www.bllossom.ai/) | [Github](https://github.com/MLP-Lab/Bllossom) | [Colab-tutorial](https://colab.research.google.com/drive/1fBOzUVZ6NRKk_ugeoTbAOokWKqSN47IG?usp=sharing) |
```bash
์ ํฌ ์์ธ๊ณผ๊ธฐ๋ MLP์ฐ๊ตฌ์ค์์ ํ๊ตญ์ด-์์ด ์ด์ค ์ธ์ด๋ชจ๋ธ์ธ Bllossom์ ๊ณต๊ฐํ์ต๋๋ค!
์์ธ๊ณผ๊ธฐ๋ ์ํผ์ปดํจํ
์ผํฐ์ ์ง์์ผ๋ก 100GB๊ฐ๋๋ ํ๊ตญ์ด๋ก ๋ชจ๋ธ์ ์ฒด๋ฅผ ํํ๋ํ ํ๊ตญ์ด ๊ฐํ ์ด์ค์ธ์ด ๋ชจ๋ธ์
๋๋ค!
ํ๊ตญ์ด ์ํ๋ ๋ชจ๋ธ ์ฐพ๊ณ ์์ง ์์ผ์
จ๋์?
- ํ๊ตญ์ด ์ต์ด! ๋ฌด๋ ค 3๋ง๊ฐ๊ฐ ๋๋ ํ๊ตญ์ด ์ดํํ์ฅ
- Llama3๋๋น ๋๋ต 25% ๋ ๊ธด ๊ธธ์ด์ ํ๊ตญ์ด Context ์ฒ๋ฆฌ๊ฐ๋ฅ
- ํ๊ตญ์ด-์์ด Pararell Corpus๋ฅผ ํ์ฉํ ํ๊ตญ์ด-์์ด ์ง์์ฐ๊ฒฐ (์ฌ์ ํ์ต)
- ํ๊ตญ์ด ๋ฌธํ, ์ธ์ด๋ฅผ ๊ณ ๋ คํด ์ธ์ดํ์๊ฐ ์ ์ํ ๋ฐ์ดํฐ๋ฅผ ํ์ฉํ ๋ฏธ์ธ์กฐ์
- ๊ฐํํ์ต
์ด ๋ชจ๋ ๊ฒ ํ๊บผ๋ฒ์ ์ ์ฉ๋๊ณ ์์
์ ์ด์ฉ์ด ๊ฐ๋ฅํ Bllossom์ ์ด์ฉํด ์ฌ๋ฌ๋ถ ๋ง์ ๋ชจ๋ธ์ ๋ง๋ค์ด๋ณด์ธ์ฅ!
๋ฌด๋ ค Colab ๋ฌด๋ฃ GPU๋ก ํ์ต์ด ๊ฐ๋ฅํฉ๋๋ค. ํน์ ์์ํ ๋ชจ๋ธ๋ก 4GB GPU์์ฌ๋ ค๋ณด์ธ์ [์์ํ๋ชจ๋ธ](https://huggingface.co/MLP-KTLim/llama-3-Korean-Bllossom-8B-4bit)
1. Bllossom-8B๋ ์์ธ๊ณผ๊ธฐ๋, ํ
๋์ธ, ์ฐ์ธ๋ ์ธ์ด์์ ์ฐ๊ตฌ์ค์ ์ธ์ดํ์์ ํ์
ํด ๋ง๋ ์ค์ฉ์ฃผ์๊ธฐ๋ฐ ์ธ์ด๋ชจ๋ธ์
๋๋ค! ์์ผ๋ก ์ง์์ ์ธ ์
๋ฐ์ดํธ๋ฅผ ํตํด ๊ด๋ฆฌํ๊ฒ ์ต๋๋ค ๋ง์ด ํ์ฉํด์ฃผ์ธ์ ๐
2. ์ด ๊ฐ๋ ฅํ Advanced-Bllossom 8B, 70B๋ชจ๋ธ, ์๊ฐ-์ธ์ด๋ชจ๋ธ์ ๋ณด์ ํ๊ณ ์์ต๋๋ค! (๊ถ๊ธํ์ ๋ถ์ ๊ฐ๋ณ ์ฐ๋ฝ์ฃผ์ธ์!!)
3. Bllossom์ NAACL2024, LREC-COLING2024 (๊ตฌ๋) ๋ฐํ๋ก ์ฑํ๋์์ต๋๋ค.
4. ์ข์ ์ธ์ด๋ชจ๋ธ ๊ณ์ ์
๋ฐ์ดํธ ํ๊ฒ ์ต๋๋ค!! ํ๊ตญ์ด ๊ฐํ๋ฅผ์ํด ๊ณต๋ ์ฐ๊ตฌํ์ค๋ถ(ํนํ๋
ผ๋ฌธ) ์ธ์ ๋ ํ์ํฉ๋๋ค!!
ํนํ ์๋์ GPU๋ผ๋ ๋์ฌ ๊ฐ๋ฅํํ์ ์ธ์ ๋ ์ฐ๋ฝ์ฃผ์ธ์! ๋ง๋ค๊ณ ์ถ์๊ฑฐ ๋์๋๋ ค์.
```
The Bllossom language model is a Korean-English bilingual language model based on the open-source LLama3. It enhances the connection of knowledge between Korean and English. It has the following features:
* **Knowledge Linking**: Linking Korean and English knowledge through additional training
* **Vocabulary Expansion**: Expansion of Korean vocabulary to enhance Korean expressiveness.
* **Instruction Tuning**: Tuning using custom-made instruction following data specialized for Korean language and Korean culture
* **Human Feedback**: DPO has been applied
* **Vision-Language Alignment**: Aligning the vision transformer with this language model
**This model developed by [MLPLab at Seoultech](http://mlp.seoultech.ac.kr), [Teddysum](http://teddysum.ai/) and [Yonsei Univ](https://sites.google.com/view/hansaemkim/hansaem-kim)**
## Demo Video
<div style="display: flex; justify-content: space-between;">
<!-- ์ฒซ ๋ฒ์งธ ์ปฌ๋ผ -->
<div style="width: 49%;">
<a>
<img src="https://github.com/lhsstn/lhsstn/blob/main/x-llava_dem.gif?raw=true" style="width: 100%; height: auto;">
</a>
<p style="text-align: center;">Bllossom-V Demo</p>
</div>
<!-- ๋ ๋ฒ์งธ ์ปฌ๋ผ (ํ์ํ๋ค๋ฉด) -->
<div style="width: 49%;">
<a>
<img src="https://github.com/lhsstn/lhsstn/blob/main/bllossom_demo_kakao.gif?raw=true" style="width: 70%; height: auto;">
</a>
<p style="text-align: center;">Bllossom Demo(Kakao)ใ
คใ
คใ
คใ
คใ
คใ
คใ
คใ
ค</p>
</div>
</div>
## NEWS
* [2024.05.08] Vocab Expansion Model Update
* [2024.04.25] We released Bllossom v2.0, based on llama-3
* [2023/12] We released Bllossom-Vision v1.0, based on Bllossom
* [2023/08] We released Bllossom v1.0, based on llama-2.
* [2023/07] We released Bllossom v0.7, based on polyglot-ko.
## Example code
### Colab Tutorial
- [Inference-Code-Link](https://colab.research.google.com/drive/1fBOzUVZ6NRKk_ugeoTbAOokWKqSN47IG?usp=sharing)
### Install Dependencies
```bash
pip install torch transformers==4.40.0 accelerate
```
### Python code with Pipeline
```python
import transformers
import torch
model_id = "MLP-KTLim/llama-3-Korean-Bllossom-8B"
pipeline = transformers.pipeline(
"text-generation",
model=model_id,
model_kwargs={"torch_dtype": torch.bfloat16},
device_map="auto",
)
pipeline.model.eval()
PROMPT = '''๋น์ ์ ์ ์ฉํ AI ์ด์์คํดํธ์
๋๋ค. ์ฌ์ฉ์์ ์ง์์ ๋ํด ์น์ ํ๊ณ ์ ํํ๊ฒ ๋ต๋ณํด์ผ ํฉ๋๋ค.
You are a helpful AI assistant, you'll need to answer users' queries in a friendly and accurate manner.'''
instruction = "์์ธ๊ณผํ๊ธฐ์ ๋ํ๊ต MLP์ฐ๊ตฌ์ค์ ๋ํด ์๊ฐํด์ค"
messages = [
{"role": "system", "content": f"{PROMPT}"},
{"role": "user", "content": f"{instruction}"}
]
prompt = pipeline.tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True
)
terminators = [
pipeline.tokenizer.eos_token_id,
pipeline.tokenizer.convert_tokens_to_ids("<|eot_id|>")
]
outputs = pipeline(
prompt,
max_new_tokens=2048,
eos_token_id=terminators,
do_sample=True,
temperature=0.6,
top_p=0.9,
repetition_penalty = 1.1
)
print(outputs[0]["generated_text"][len(prompt):])
# ์์ธ๊ณผํ๊ธฐ์ ๋ํ๊ต MLP์ฐ๊ตฌ์ค์ ๋ฉํฐ๋ชจ๋ฌ ์์ฐ์ด์ฒ๋ฆฌ ์ฐ๊ตฌ๋ฅผ ํ๊ณ ์์ต๋๋ค. ๊ตฌ์ฑ์์ ์๊ฒฝํ ๊ต์์ ๊น๋ฏผ์ค, ๊น์๋ฏผ, ์ต์ฐฝ์, ์์ธํธ, ์ ํ๊ฒฐ, ์ํ์, ์ก์น์ฐ, ์ก์ ํ, ์ ๋์ฌ ํ์์ด ์์ต๋๋ค.
```
### Python code with AutoModel
```python
import os
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = 'MLP-KTLim/llama-3-Korean-Bllossom-8B'
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
device_map="auto",
)
model.eval()
PROMPT = '''๋น์ ์ ์ ์ฉํ AI ์ด์์คํดํธ์
๋๋ค. ์ฌ์ฉ์์ ์ง์์ ๋ํด ์น์ ํ๊ณ ์ ํํ๊ฒ ๋ต๋ณํด์ผ ํฉ๋๋ค.
You are a helpful AI assistant, you'll need to answer users' queries in a friendly and accurate manner.'''
instruction = "์์ธ๊ณผํ๊ธฐ์ ๋ํ๊ต MLP์ฐ๊ตฌ์ค์ ๋ํด ์๊ฐํด์ค"
messages = [
{"role": "system", "content": f"{PROMPT}"},
{"role": "user", "content": f"{instruction}"}
]
input_ids = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt"
).to(model.device)
terminators = [
tokenizer.eos_token_id,
tokenizer.convert_tokens_to_ids("<|eot_id|>")
]
outputs = model.generate(
input_ids,
max_new_tokens=2048,
eos_token_id=terminators,
do_sample=True,
temperature=0.6,
top_p=0.9,
repetition_penalty = 1.1
)
print(tokenizer.decode(outputs[0][input_ids.shape[-1]:], skip_special_tokens=True))
# ์์ธ๊ณผํ๊ธฐ์ ๋ํ๊ต MLP์ฐ๊ตฌ์ค์ ๋ฉํฐ๋ชจ๋ฌ ์์ฐ์ด์ฒ๋ฆฌ ์ฐ๊ตฌ๋ฅผ ํ๊ณ ์์ต๋๋ค. ๊ตฌ์ฑ์์ ์๊ฒฝํ ๊ต์์ ๊น๋ฏผ์ค, ๊น์๋ฏผ, ์ต์ฐฝ์, ์์ธํธ, ์ ํ๊ฒฐ, ์ํ์, ์ก์น์ฐ, ์ก์ ํ, ์ ๋์ฌ ํ์์ด ์์ต๋๋ค.
```
## Citation
**Language Model**
```text
@misc{bllossom,
author = {ChangSu Choi, Yongbin Jeong, Seoyoon Park, InHo Won, HyeonSeok Lim, SangMin Kim, Yejee Kang, Chanhyuk Yoon, Jaewan Park, Yiseul Lee, HyeJin Lee, Younggyun Hahm, Hansaem Kim, KyungTae Lim},
title = {Optimizing Language Augmentation for Multilingual Large Language Models: A Case Study on Korean},
year = {2024},
journal = {LREC-COLING 2024},
paperLink = {\url{https://arxiv.org/pdf/2403.10882}},
},
}
```
**Vision-Language Model**
```text
@misc{bllossom-V,
author = {Dongjae Shin, Hyunseok Lim, Inho Won, Changsu Choi, Minjun Kim, Seungwoo Song, Hangyeol Yoo, Sangmin Kim, Kyungtae Lim},
title = {X-LLaVA: Optimizing Bilingual Large Vision-Language Alignment},
year = {2024},
publisher = {GitHub},
journal = {NAACL 2024 findings},
paperLink = {\url{https://arxiv.org/pdf/2403.11399}},
},
}
```
## Contact
- ์๊ฒฝํ(KyungTae Lim), Professor at Seoultech. `[email protected]`
- ํจ์๊ท (Younggyun Hahm), CEO of Teddysum. `[email protected]`
- ๊นํ์(Hansaem Kim), Professor at Yonsei. `[email protected]`
## Contributor
- ์ต์ฐฝ์(Chansu Choi), [email protected]
- ๊น์๋ฏผ(Sangmin Kim), [email protected]
- ์์ธํธ(Inho Won), [email protected]
- ๊น๋ฏผ์ค(Minjun Kim), [email protected]
- ์ก์น์ฐ(Seungwoo Song), [email protected]
- ์ ๋์ฌ(Dongjae Shin), [email protected]
- ์ํ์(Hyeonseok Lim), [email protected]
- ์ก์ ํ(Jeonghun Yuk), [email protected]
- ์ ํ๊ฒฐ(Hangyeol Yoo), [email protected]
- ์ก์ํ(Seohyun Song), [email protected] |