Edit model card

Trained on Discord chatlogs from this dataset.

Uses Llama 3.1 formatting.

LoRA: mpasila/Llama-3.1-Discord-Short-LoRA-8B

Trained with regular LoRA (not quantized/QLoRA) and LoRA rank was 128 and Alpha set to 32. Trained for 1 epoch using A40 for about 5,5 hours.

    args = UnslothTrainingArguments(
        per_device_train_batch_size = 1,
        gradient_accumulation_steps = 8,

        warmup_ratio = 0.1,
        num_train_epochs = 1,

        learning_rate = 5e-5,
        embedding_learning_rate = 5e-6,

        fp16 = not is_bfloat16_supported(),
        bf16 = is_bfloat16_supported(),
        logging_steps = 1,
        optim = "adamw_8bit",
        weight_decay = 0.00,
        lr_scheduler_type = "cosine",
        seed = 3407,
        output_dir = "outputs",
    ),

Uploaded model

  • Developed by: mpasila
  • License: Llama 3.1 Community License Agreement
  • Finetuned from model : unsloth/Meta-Llama-3.1-8B

This llama model was trained 2x faster with Unsloth and Huggingface's TRL library.

Downloads last month
8
Safetensors
Model size
8.03B params
Tensor type
BF16
·
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Model tree for mpasila/Llama-3.1-Discord-Short-8B

Finetuned
(89)
this model
Quantizations
2 models

Dataset used to train mpasila/Llama-3.1-Discord-Short-8B