OxxoCodes/Pula-XLMR-large-v0.1

Pula-XLMR-Large

Model Information

The largest encoder model in the BOTS-LM suite of language models, Pula-XLMR-Large is a highly capable encoder-style langauge model built for Setswana. Based on the Afro-XLMR-Large architecture and fine-tuned on a massive copora of Setswana and English webtext, instruction following, and synthetic data, Pula-XLMR-Large reaches competitive levels of performance compared to existing open models for Setswana and English.

The BOTS-LM suite of language models is fine-tuned on a custom-made massive corpora of Setswana and English web documents (Marothodi), instruction following examples and synthetic data (Medupe), multilingual data (Aya Dataset, Inkuba Instruct), and subsets of OpenHermes 2.5, MagPie Pro MT, and The Tome. The Pula-XLMR models are trained on an additional subset of MC4 and the AfriBERTa Corpus.

Model Description

• Developed by: Nathan Brown (@OxxoCodes) and Vukosi Marivate (@vukosi)
• Funded by: [More Information Needed]
• Model type: Meta Llama 3.1
• Language(s) (NLP): Setswana, English
• License: [More Information Needed]
• Finetuned from model: Afro-XLMR-Large

Model Sources

• Repository: https://github.com/OxxoCodes/BOTS-LM
• Paper: [2408.02239] BOTS-LM: Training Large Language Models for Setswana
• Demo: [More Information Needed]

Uses

Direct Use

BOTS-LM models and data are intended for commercial and research use in Setswana and English. All Pula models are trained on predominantly instruction data, and as such are primarily meant for assistant-like chat. However, they may still function for general langauge generation, although this is predominantly untested. The BOTS-LM suite also allows for the use of its outputs to improve other models, including synthetic data generation and distillation.

Downstream Use

BOTS-LM's Pula LLMs are trained on a wide variety of tasks including (but not limited to) general conversations, Setswana <--> English translation, writing, question answering, tool use/function calling, Named Entity Recognition (NER), and Part of Speech (POS) tagging.

Out-of-Scope Use

The use of any model, data, or resource in the BOTS-LM suite that violates applicable laws or regulations, is intended to cause direct or indirect harm, or is otherwise generally considered immoral or unethical, is explicitly prohibited.

While every model in the BOTS-LM suite has been trained on a small amount of data covering several langauges, use of these models is only officially supported for Setswana and English.

Bias, Risks, and Limitations

As with any language model, BOTS-LM models are susceptible to generating inaccurate or biased responses to user prompts. It is recommended the user(s) of BOTS-LM models develop additional safety safeguards or perform additional safety training prior to deployment in customer- or client-facing environments.

How to Get Started with the Model

Use the code below to get started with the model.

from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("OxxoCodes/Pula-XLMR-large")
model = AutoModelForMaskedLM.from_pretrained("OxxoCodes/Pula-XLMR-large")

# prepare input
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors="pt")

# forward pass
output = model(**encoded_input)

Training Details

Training Data

Pula-XLMR was fine-tuned on a corpus of 0.5 billion tokens of Setswana and English text for 3 epochs, totaling of 1.5 Billion tokens, as measured using the Llama 3.1 tokenizer.

To increase the occurances of high-quality instruction data, all non-OPUS data sources appear twice per epoch, while certain synthetic datasets appear five times per epoch.

We ensure 80% of training tokens are made up of SetsText, SetsText-Instruct, and Hugging Face's Everyday Conversations dataset. The other 20% consists of various high-quality and multilingual data sources.

Following these procedures, the final training data distribution is as follows:

• SetsText-Instruct: 66.43% (336M tokens)

• SetsText: 13.5% (68M tokens)

• MagPie Pro MT: 10% (50M tokens)

• OpenHermes 2.5: 7% (35M tokens)

• Aya Dataset: 2% (10M tokens)

• Inkuba Instruct: 1% (5M tokens)

• Everyday Conversations: 0.07% (0.3M tokens)

Preprocessing

All Pula models in the BOTS-LM series are trained on sequences up to 4096 tokens. For all sequences longer than 4096 tokens, such as large documents or long multi-turn instruction conversations, they are split into chunks of 4096 tokens. In addition, we utilize the Liger Kernel to reduce GPU memory requirements.

Training Hyperparameters

• Training regime: BF16 mixed precision
• Epochs: 3.0
• Learning Rate: 2e-5
• Learning Rate Scheduler: Cosine
• Warmup Ratio: 5%
• LoRA Rank: 64
• LoRA Alpha: 32
• Per Device Train Batch Size: 1
• Gradient Accumulation Steps: 8
• GPUs: 4x Nvidia H100s 80GB
• Sequence Length: 4096
• Effective Batch Size: 32 (131k tokens)

Technical Specifications

Compute Infrastructure

Compute is provided by the Clemson University Palmetto Cluster.

Hardware

Pula-XLMR-Large is trained using a single NVIDIA H100 80GB GPU with sixteen CPUs and 64GB RAM.

Software

Training is performed using Hugging Face transformers, accelerate, peft, and trl with DeepSpeed and ZeRO-3.

Citation

BibTeX:

@misc{2408.02239,
  Author = {Nathan Brown and Vukosi Marivate},
  Title = {BOTS-LM: Training Large Language Models for Setswana},
  Year = {2024},
  Eprint = {arXiv:2408.02239},
  Note = {Hugging Face repository: \url{https://huggingface.co/collections/OxxoCodes/bots-lm-66af1106ccc0fb38839f39da}}
}

APA:

Brown, N., & Marivate, V. (2024). BOTS-LM: Training Large Language Models for Setswana. arXiv. [2408.02239] BOTS-LM: Training Large Language Models for Setswana. Hugging Face repository: https://huggingface.co/collections/OxxoCodes/bots-lm-66af1106ccc0fb38839f39da