mistral-7b-english-welsh-translate

This model is a fine-tuned version of mistralai/Mistral-7B-Instruct-v0.3 on the Welsh Government Alpaca Welsh-English Instructions dataset.

This model is trained for English-Welsh translation (in any direction), with a focus on government documents, using Markdown formatting.

To ensure the highest quality translations, use the Alpaca instruction prompt format with the below structure.

    ### Instruction: {instruction}

    ### Input: {input}

    ### Response:

Your instruction should be "Translate the text from English to Welsh." (or vice versa).

The model is also available quantized as GGUF. This version be be tested in this interactive space.

Running the model

The model is intended to be run locally, ideally using Text generation web UI to ensure correct prompt structure.

Start the UI as instructed for your system.

• In the "Model" tab, download either this model or the quantized version. Once the download is complete, load the model.
• In the "Parameters" tab, under "Generation", set "auto_max_new_tokens" to maximum, and "Ban the eos_token" to True. In "Custom stopping strings", add "### Input"
• In the "Notebook" tab, make sure you are using the "Alpaca-with-input" prompt. Set the instruction as "Translate the text from Welsh to English." (or vice versa).#
• Add the text you would like to translate (replacing "Input"), and hit "generate"

Performance may start to degrade past a certain context window (especially if using the quantized models). Convert in chunks of under 1000 words to avoid these issues.

LLM Evals

Thanks to YALL - Yet Another LLM Leaderboard. Full results available here.

Model	AGIEval	TruthfulQA	Bigbench
mistral-7b-english-welsh-translate	35.31	54.5	38.4

Training procedure

See axolotl config

axolotl version: 0.4.1

base_model: mistralai/Mistral-7B-Instruct-v0.3
model_type: MistralForCausalLM
tokenizer_type: LlamaTokenizer

load_in_8bit: false
load_in_4bit: 
strict: false

# huggingface repo
datasets:
  - path: AndreasThinks/welsh-translation-instruction
    type: alpaca
val_set_size: 0.04
output_dir: ./outputs/mistral-welsh

hub_model_id: AndreasThinks/mistral-7b-english-welsh-translate


sequence_len: 8192
sample_packing: true
pad_to_sequence_len: true
eval_sample_packing: false

wandb_project: mistral-nemo-welsh
wandb_entity:
wandb_watch:
wandb_name: mistral-nemo-welsh-v1
wandb_log_model:

gradient_accumulation_steps: 4
micro_batch_size: 2
num_epochs: 4
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.000005

train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false

gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true

warmup_steps: 10
evals_per_epoch: 4
eval_table_size:
eval_max_new_tokens: 128
saves_per_epoch: 1
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-06
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 8
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 10
• num_epochs: 4

Training results

Training Loss	Epoch	Step	Validation Loss
1.5781	0.0013	1	1.5514
0.4427	0.2506	194	0.4841
0.4142	0.5011	388	0.4271
0.4001	0.7517	582	0.3996
0.4155	1.0023	776	0.3828
0.3178	1.2296	970	0.3792
0.3156	1.4801	1164	0.3732
0.3115	1.7307	1358	0.3678
0.2722	1.9813	1552	0.3633
0.2492	2.2089	1746	0.3809
0.2159	2.4595	1940	0.3828
0.2277	2.7100	2134	0.3810
0.2435	2.9606	2328	0.3809
0.158	3.1899	2522	0.3961
0.1802	3.4404	2716	0.3966
0.1679	3.6910	2910	0.3966

Framework versions

• Transformers 4.42.4
• Pytorch 2.3.1+cu121
• Datasets 2.19.1
• Tokenizers 0.19.1