Minueza-32M-Base
Summary
Minueza-32M-Base is a foundation model with 32 million parameters trained from scratch on a large corpus of text in English.
It's available in the following formats: Safetensors, GGUF, and ONNX.
And it's being released alongside some fine-tuned versions:
- Minueza-32M-UltraChat: Trained on a single conversational dataset.
- Minueza-32M-Chat: Trained on a mix of conversational datasets.
- Minueza-32Mx2-Chat: Sparse Mixture of Experts trained on interleaved conversational datasets.
- And more...
Intended Uses
This model was created with the following objectives in mind:
- Run on mobile web browsers via Transformers.js.
- Run fast on machines without GPU.
- Serve as a base for fine-tunes using ChatML format, hence the two additional special tokens (
<|im_start|>
and<|im_end|>
) with<|im_end|>
as default EOS token.- ChatML works great for both instruction and chat models, so if all fine-tunes are made following the ChatML pattern, other users might benefit from the easiness of creating merges.
Datasets
The model was trained on a subset of each of the following non-synthetic datasets:
- Skylion007/openwebtext
- c4
- wikimedia/wikipedia - 20231101.simple
- tiiuae/falcon-refinedweb
- izumi-lab/open-text-books
- togethercomputer/RedPajama-Data-V2
- databricks/databricks-dolly-15k
- euclaise/reddit-instruct-curated
- CohereForAI/aya_dataset - original english annotations
The subsets were interleaved to form the final training corpus of approximately 650 million tokens.
Model Architecture
This is a transformer model with the Mistral architecture, trained on a context window of 2048 tokens.
Configuration | Value |
---|---|
max_position_embeddings | 2048 |
hidden_size | 312 |
intermediate_size | 1092 |
num_attention_heads | 12 |
num_hidden_layers | 10 |
num_key_value_heads | 4 |
vocab_size | 32002 |
The pretraining was made with these hyperparameters and frameworks:
Hyperparameter | Value |
---|---|
learning_rate | 5e-05 |
train_batch_size | 1 |
eval_batch_size | 1 |
seed | 42 |
gradient_accumulation_steps | 8 |
total_train_batch_size | 8 |
optimizer | Adam with betas=(0.9,0.999) and epsilon=1e-08 |
lr_scheduler_type | linear |
Framework | Version |
---|---|
Transformers | 4.38.0.dev0 |
Pytorch | 2.1.2 |
Datasets | 2.16.1 |
Tokenizers | 0.15.1 |
Usage
This is just a base model. For your task, you will likely want to perform application-specific fine-tuning as recommended above.
Also note that this model was trained on internet text data, which may contain biases, offensive or inappropriate content, and may produce incorrect or irrelevant responses. No evaluation has been conducted, so use with care.
Having that said, here's how you can run it:
from transformers import pipeline
generate = pipeline("text-generation", "Felladrin/Minueza-32M-Base")
prompt = "The best way to improve your health is"
output = generate(
prompt,
max_new_tokens=256,
do_sample=True,
temperature=0.72,
top_p=0.73,
top_k=50,
repetition_penalty=1.176,
)
print(output[0]["generated_text"])
Open LLM Leaderboard Evaluation Results
Detailed results can be found here
Metric | Value |
---|---|
Avg. | 28.92 |
AI2 Reasoning Challenge (25-Shot) | 21.33 |
HellaSwag (10-Shot) | 26.39 |
MMLU (5-Shot) | 24.80 |
TruthfulQA (0-shot) | 47.45 |
Winogrande (5-shot) | 53.20 |
GSM8k (5-shot) | 0.38 |
License
This model is licensed under the Apache License 2.0.
- Downloads last month
- 868
Model tree for Felladrin/Minueza-32M-Base
Datasets used to train Felladrin/Minueza-32M-Base
Collection including Felladrin/Minueza-32M-Base
Evaluation results
- normalized accuracy on AI2 Reasoning Challenge (25-Shot)test set Open LLM Leaderboard21.330
- normalized accuracy on HellaSwag (10-Shot)validation set Open LLM Leaderboard26.390
- accuracy on MMLU (5-Shot)test set Open LLM Leaderboard24.800
- mc2 on TruthfulQA (0-shot)validation set Open LLM Leaderboard47.450
- accuracy on Winogrande (5-shot)validation set Open LLM Leaderboard53.200
- accuracy on GSM8k (5-shot)test set Open LLM Leaderboard0.380