Hugging Face's logo

aisingapore
/
sea-lion-7b-instruct-research

Text Generation
Transformers
Safetensors
mpt
conversational
custom_code
text-generation-inference
Inference Endpoints
Model card Files Community
6
sea-lion-7b-instruct-research / README.md
tainc's picture
tainc
Update README.md
4820c08 verified
preview code
|
raw
history blame contribute delete
8.31 kB
metadata
new_version: aisingapore/gemma2-9b-cpt-sea-lionv3-instruct
base_model: aisingapore/sea-lion-7b
license: cc-by-nc-sa-4.0
language:
  - en
  - zh
  - id
  - ms
  - tl
  - my
  - vi
  - th
  - lo
  - km
  - ta

SEA-LION-7B-Instruct-Research

SEA-LION is a collection of Large Language Models (LLMs) which has been pretrained and instruct-tuned for the Southeast Asia (SEA) region. The size of the models range from 3 billion to 7 billion parameters. This is the card for the SEA-LION 7B Instruct (Non-Commercial) model.

For more details on the base model, please refer to the base model's model card.

For the commercially permissive model, please refer to the SEA-LION-7B-Instruct.

SEA-LION stands for Southeast Asian Languages In One Network.

Model Details

Model Description

The SEA-LION model is a significant leap forward in the field of Natural Language Processing, specifically trained to understand the SEA regional context.

SEA-LION is built on the robust MPT architecture and has a vocabulary size of 256K.

For tokenization, the model employs our custom SEABPETokenizer, which is specially tailored for SEA languages, ensuring optimal model performance.

The pre-training data for the base SEA-LION model encompasses 980B tokens. The model was then further instruction-tuned on Indonesian data only.

  • Developed by: Products Pillar, AI Singapore
  • Funded by: Singapore NRF
  • Model type: Decoder
  • Languages: English, Chinese, Indonesian, Malay, Thai, Vietnamese, Filipino, Tamil, Burmese, Khmer, Lao
  • License: CC BY-NC-SA 4.0 License

Benchmark Performance

SEA-LION-7B-Instruct-NC performs better than other models of comparable size when tested on tasks in the Indonesian language.

We evaluated SEA-LION-7B-Instruct-NC on the BHASA benchmark and compared it against Llama-2-7B, Mistral-7B and Falcon-7B.

We only evaluated it on the Indonesian tasks as the model was only instruction-tuned in Indonesian.

The evaluation was done zero-shot with Indonesian prompts and only a sample of 100 - 1000 instances per dataset was used as per the setting described in the BHASA paper.

The scores shown in the tables below have been adjusted to only consider answers provided in the appropriate language.

For Natural Language Understanding (NLU) tasks, we tested the model on Sentiment Analysis (Sent) using the NusaX dataset, Question Answering (QA) using the TyDiQA dataset, and Toxicity Detection (Tox) using the Indonesian Multi-Label Hate Speech Detection dataset. The metrics used are F1 score for all three tasks.

For Natural Language Generation (NLG) tasks, we tested the model on Machine Translation from English to Indonesian (MT-EN-ID) and from Indonesian to English (MT-ID-EN) using the FLORES-200 dataset, and Abstractive Summarization (AbsSum) using the XLSum dataset. The metrics used for Machine Translation are ChrF++ and COMET22, and ROUGE-L is used for Abstractive Summarization.

For Natural Language Reasoning (NLR) tasks, we tested the model on Natural Language Inference (NLI) using the IndoNLI lay dataset and on Causal Reasoning (Causal) using the XCOPA dataset. The metrics are accuracy for both tasks.

Model QA (F1) Sentiment (F1) Toxicity (F1) Eng>Indo (ChrF++) Indo>Eng (ChrF++) Summary (ROUGE-L) NLI (Acc) Causal (Acc)
SEA-LION-7B-Instruct-Research 24.86 76.13 24.45 52.50 46.82 15.44 33.20 23.80
SEA-LION-7B-Instruct 68.41 91.45 17.98 57.48 58.04 17.54 53.10 60.80
SeaLLM 7B v1 30.96 56.29 22.60 62.23 41.55 14.03 26.50 56.60
SeaLLM 7B v2 44.40 80.13 55.24 64.01 63.28 17.31 43.60 82.00
Sailor-7B (Base) 65.43 59.48 20.48 64.27 60.68 8.69 15.10 38.40
Sailor-7B-Chat 38.02 87.64 52.07 64.25 61.87 15.28 68.30 85.60
Llama 2 7B Chat 11.12 52.32 0.00 44.09 57.58 9.24 0.00 0.00
Mistral 7B Instruct v0.1 38.85 74.38 20.83 30.60 51.43 15.63 28.60 50.80
GPT-4 (gpt-4-0314) 73.60 74.14 63.96 69.38 67.53 18.71 83.20 96.00

Technical Specifications

Model Architecture and Objective

SEA-LION is a decoder model using the MPT architecture.

Parameter SEA-LION 7B
Layers 32
d_model 4096
head_dim 32
Vocabulary 256000
Sequence Length 2048

Tokenizer Details

We sample 20M lines from the training data to train the tokenizer.
The framework for training is SentencePiece.
The tokenizer type is Byte-Pair Encoding (BPE).

Example Usage 

# Please use transformers==4.34.1

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("aisingapore/sea-lion-7b-instruct-nc", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("aisingapore/sea-lion-7b-instruct-nc", trust_remote_code=True)

prompt_template = "### USER:\n{human_prompt}\n\n### RESPONSE:\n"
prompt = """Apa sentimen dari kalimat berikut ini?
Kalimat: Buku ini sangat membosankan.
Jawaban: """
full_prompt = prompt_template.format(human_prompt=prompt)

tokens = tokenizer(full_prompt, return_tensors="pt")
output = model.generate(tokens["input_ids"], max_new_tokens=20, eos_token_id=tokenizer.eos_token_id)
print(tokenizer.decode(output[0], skip_special_tokens=True))

The Team

Lam Wen Zhi Clarence
Leong Wei Qi
Li Yier
Liu Bing Jie Darius
Lovenia Holy
Montalan Jann Railey
Ng Boon Cheong Raymond
Ngui Jian Gang
Nguyen Thanh Ngan
Ong Tat-Wee David
Rengarajan Hamsawardhini
Susanto Yosephine
Tai Ngee Chia
Tan Choon Meng
Teo Jin Howe
Teo Eng Sipp Leslie
Teo Wei Yi
Tjhi William
Yeo Yeow Tong
Yong Xianbin

Acknowledgements

AI Singapore is a national programme supported by the National Research Foundation, Singapore and hosted by the National University of Singapore. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.

Contact

For more info, please contact us using this SEA-LION Inquiry Form

Link to SEA-LION's GitHub repository

Disclaimer

This the repository for the non-commercial instruction-tuned model. The model has not been aligned for safety. Developers and users should perform their own safety fine-tuning and related security measures. In no event shall the authors be held liable for any claim, damages, or other liability arising from the use of the released weights and codes.