Yi-Ko-34B-Instruct / README.md
maywell's picture
Update README.md
ee2aaae verified
metadata
license: other
license_name: yi-license
license_link: LICENSE
language:
  - en
  - ko
pipeline_tag: text-generation
inference: false
base_model: beomi/Yi-Ko-34B
tags:
  - pytorch
  - Yi-Ko
  - 01-ai
  - Yi
library_name: transformers

Yi Ko 34B Instruct

Training Process

  1. Further trained with Korean corpus.
  2. SFT
  3. DPO (Dataset URL)

Model Info

Context Length Parameter Prompt Template KMMLU(5-shot)
4k(4096) 34B ChatML 49.03

Acknowledgement

The training is supported by Sionic AI.

Original Model Card by beomi

Yi-Ko series models serve as advanced iterations of 01-ai/Yi models, benefiting from an expanded vocabulary and the inclusion of Korean/English corpus in its further pretraining. Just like its predecessor, Yi-Ko series models operate within the broad range of generative text models that stretch from 6 billion to 34 billion parameters. This repository focuses on the 34B pretrained version, which is tailored to fit the Hugging Face Transformers format. For access to the other models, feel free to consult the index provided below.

Model Details

Model Developers Junbum Lee (Beomi)

Variations Yi-Ko-34B will come in a range of parameter sizes — 6B and 34B — with Ko(Korean+English).

Input Models input text only.

Output Models generate text only.

Model Architecture

Yi-Ko series models are an auto-regressive language model that uses an optimized transformer architecture based on Llama-2*.

*Yi model architecture is based on Llama2, so it can be loaded via LlamaForCausalLM class on HF.

Model Name Training Data Params Context Length GQA Trained Tokens LR Train tokens (per batch)
Yi-Ko-34B A mix of Korean + English online data 34B 4k O 40B+ 5e-5 4M

Vocab Expansion

Model Name Vocabulary Size Description
Original Yi-Series 64000 Sentencepiece BPE
Expanded Yi-Ko Series 78464 Sentencepiece BPE. Added Korean vocab and merges

Tokenizing "안녕하세요, 오늘은 날씨가 좋네요.ㅎㅎ"

Model # of tokens Tokens
Original Yi-Series 47 ['<0xEC>', '<0x95>', '<0x88>', '<0xEB>', '<0x85>', '<0x95>', '하', '<0xEC>', '<0x84>', '<0xB8>', '<0xEC>', '<0x9A>', '<0x94>', ',', '▁', '<0xEC>', '<0x98>', '<0xA4>', '<0xEB>', '<0x8A>', '<0x98>', '은', '▁', '<0xEB>', '<0x82>', '<0xA0>', '<0xEC>', '<0x94>', '<0xA8>', '가', '▁', '<0xEC>', '<0xA2>', '<0x8B>', '<0xEB>', '<0x84>', '<0xA4>', '<0xEC>', '<0x9A>', '<0x94>', '.', '<0xE3>', '<0x85>', '<0x8E>', '<0xE3>', '<0x85>', '<0x8E>']
Expanded Yi-Ko Series 10 ['▁안녕', '하세요', ',', '▁오늘은', '▁날', '씨가', '▁좋네요', '.', 'ㅎ', 'ㅎ']
*Equal Korean vocab with Llama-2-Ko Series

Tokenizing "Llama 2: Open Foundation and Fine-Tuned Chat Models"

Model # of tokens Tokens
Original Yi-Series 21 ['The', '▁Y', 'i', '▁series', '▁models', '▁are', '▁large', '▁language', '▁models', '▁trained', '▁from', '▁scratch', '▁by', '▁developers', '▁at', '▁', '0', '1', '.', 'AI', '.']
Expanded Yi-Ko Series 21 ['▁The', '▁Y', 'i', '▁series', '▁models', '▁are', '▁large', '▁language', '▁models', '▁trained', '▁from', '▁scratch', '▁by', '▁developers', '▁at', '▁', '0', '1', '.', 'AI', '.']
*Equal Korean vocab with Llama-2-Ko Series *Since Expanded Yi-Ko Series prepends _ at the beginning of the text(to ensure same tokenization for Korean sentences), it shows negilible difference for the first token on English tokenization.

Model Benchmark

LM Eval Harness - Korean Benchmarks

Tasks Version Filter n-shot Metric Value Stderr
kmmlu_direct N/A none 5 exact_match 0.5027 ± 0.1019
kobest_boolq 1 none 5 acc 0.9202 ± 0.0072
none 5 f1 0.9202 ± N/A
kobest_copa 1 none 5 acc 0.8480 ± 0.0114
none 5 f1 0.8479 ± N/A
kobest_hellaswag 1 none 5 acc 0.5320 ± 0.0223
none 5 f1 0.5281 ± N/A
none 5 acc_norm 0.6340 ± 0.0216
kobest_sentineg 1 none 5 acc 0.9874 ± 0.0056
none 5 f1 0.9874 ± N/A
haerae N/A none 5 acc 0.7965 ± 0.0116
none 5 acc_norm 0.7965 ± 0.0116
- haerae_general_knowledge 1 none 5 acc 0.5114 ± 0.0378
none 5 acc_norm 0.5114 ± 0.0378
- haerae_history 1 none 5 acc 0.8511 ± 0.0260
none 5 acc_norm 0.8511 ± 0.0260
- haerae_loan_word 1 none 5 acc 0.8402 ± 0.0283
none 5 acc_norm 0.8402 ± 0.0283
- haerae_rare_word 1 none 5 acc 0.8642 ± 0.0170
none 5 acc_norm 0.8642 ± 0.0170
- haerae_standard_nomenclature 1 none 5 acc 0.8301 ± 0.0305
none 5 acc_norm 0.8301 ± 0.0305

LICENSE

Follows Yi License

Citation

Acknowledgement

The training is supported by TPU Research Cloud program.