---
datasets:
- kz-transformers/multidomain-kazakh-dataset
language:
- kk
library_name: transformers
pipeline_tag: text-generation
license: afl-3.0
---
# LLaMA 1.9B - Kazakh Causal Language Model

<div style="display: flex; justify-content: center;">
    <img src="https://github.com/Nurgali-Kadyrbek/assets/blob/main/llama-kz-logo.PNG?raw=true" alt="Llama Model Logo" width="300" height="300"/>
</div>

<div style="position:relative; text-align: center; padding: 20px; border-radius: 10px; box-shadow: 0 4px 10px rgba(0, 0, 0, 0.1); margin-top: 20px;">
    <h1 style="font-size: 2.5em; margin: 0; color: #005b96; text-shadow: 1px 1px #fff; font-family: 'Arial', sans-serif; border-bottom: 5px solid #ffc72c; padding-bottom: 10px;">
        <span style="border-bottom: 4px double #ffc72c; padding-bottom: 5px;">LLaMA 1.9B</span>
    </h1>
    <p style="font-size: 1.2em; margin: 10px 0 0; color: #005b96; font-family: 'Arial', sans-serif;">
        <span style="background: linear-gradient(to right, #005b96 0%, #ffc72c 100%); color: white; padding: 2px 8px; border-radius: 5px;">Kazakh Causal Language Model</span>
    </p>
  
</div>

  

## Model Description
This model is a Kazakh language version of the LLaMA model with 1.9 billion parameters, trained for causal language modeling. The train dataset includes some mixed Russian texts, which occasionally cause the model to generate Russian text. Despite this, the model shows promising results. Future steps may include retraining with a purer dataset, fine-tuning, or using the model for various NLP tasks with additional fine-tuning.

## Training Setup
- Training Examples: Over 5.3 million examples
- Training Hardware: Two NVIDIA A100 GPUs (80GB each)
- Training Status: Ongoing, currently partway through the first epoch
- Optimizer: Cosine with restarts scheduler
- Parallelism: Distributed Data Parallel (DDP)
- Number of Warmup Steps: 8000

## Model Authors

**Name:** Kadyrbek Nurgali
- **Email:** nurgaliqadyrbek@gmail.com
- **LinkedIn:** [Kadyrbek Nurgali](https://www.linkedin.com/in/nurgali-kadyrbek-504260231/)

<pre>
  @misc{
  nurgali_kadyrbek_2024,
  author       = {NURGALI, Kadyrbek},
  title        = {llama-1.9B-kaz (Revision 299ebbb)},
  year         = 2024,
  url          = {https://huggingface.co/nur-dev/llama-1.9B-kaz},
  doi          = {10.57967/hf/3043},
  publisher    = {Hugging Face}
}</pre>