|
--- |
|
language: |
|
- da |
|
license: llama2 |
|
library_name: transformers |
|
base_model: |
|
- meta-llama/Llama-2-7b-hf |
|
pipeline_tag: text-generation |
|
--- |
|
 |
|
|
|
## Model Details |
|
|
|
**SnakModel** is a 7B-parameter model specifically designed for the Danish language. This is the base variant: `SnakModel-7B (base)`. Our models build upon [Llama 2](https://huggingface.co/meta-llama/Llama-2-7b-hf), which we continuously pre-train on a diverse collection of Danish corpora comprising 350M documents and 13.6B words, before tuning it on 3.7M Danish instruction-answer pairs. |
|
|
|
**Model Developers** |
|
|
|
[NLPnorth research unit](https://nlpnorth.github.io) at the [IT University of Copenhagen](https://itu.dk), Denmark. |
|
|
|
**Variations** |
|
|
|
SnakModel comes as an instruction-tuned, and a base version. In addition, each model includes intermediate checkpoints (under model revisions). |
|
|
|
**Input** |
|
|
|
Text only. |
|
|
|
**Output** |
|
|
|
Text only. |
|
|
|
**Model Architecture** |
|
|
|
SnakModel is an auto-regressive, transformer-based language model. The `instruct` version uses supervised fine-tuning (SFT) to enable instruction following in Danish. |
|
|
|
**Model Dates** |
|
|
|
SnakModel was trained between January 2024 and September 2024. |
|
|
|
**License** |
|
|
|
This model follows the original [Llama 2 license agreement](https://huggingface.co/meta-llama/Llama-2-7b-hf/blob/main/LICENSE.txt). |
|
|
|
**Research Paper** |
|
|
|
[Released in Q1 2025] |
|
|
|
## Intended Use & Limitations |
|
|
|
**Intended Use Cases** |
|
|
|
SnakModel is intended for use in Danish. The instruction-tuned variant is intended for assistant-like chat. |
|
|
|
The `instruct` variant follows the Llama 2 (chat) instruction template, in which instructions are encapsulated in special tokens, i.e., `[INST] {instruction} [/INST]`. |
|
|
|
**Limitations** |
|
|
|
SnakModel variants are fine-tuned on Danish data. As such, the use in other languages falls out-of-scope. While we found SnakModel to be more proficient in Danish, than other Llama 2-based models, it still frequently generates factually incorrect output. Make sure to carefully evaluate and weigh these factors before deploying the model. In addition, make sure to adhere to the original [Llama 2 license agreement](https://huggingface.co/meta-llama/Llama-2-7b-hf/blob/main/LICENSE.txt). |
|
|
|
## Hardware and Software |
|
|
|
**Training Factors** |
|
|
|
SnakModel is trained on private infrastructure with one node, containing four NVIDIA A100-PCIe 40GB GPUs. The node has an AMD Epyc 7662 128 Core Processor and 1TB of RAM. |
|
|
|
**Carbon Footprint** |
|
|
|
Total training time accounted to 8,928 GPU hours, with an average carbon efficiency at 0.122kg CO2eq / kWh. This is equivalent to 272.3kg CO2eq emitted, based on the [Machine Learning Impact calculator](https://mlco2.github.io/impact). |
|
|
|
## Training Data |
|
|
|
**Overview** |
|
|
|
SnakModel was continuously pre-train on a diverse collection of Danish corpora comprising 350M documents and 13.6B words. The `instruct` version is further tuned on 3.7M Danish instruction-answer pairs. |
|
|
|
[Details to follow in Q1 2025] |
|
|
|
**Data Freshness** |
|
|
|
The pre-training data has a cutoff of January 2024. |
|
|
|
## Evaluation Results |
|
|
|
[Released in Q1 2025] |
|
|
|
## Citation |
|
|
|
[Released in Q1 2025] |
