---
library_name: transformers
tags:
- PEFT
- mistral
- sft
- 'TensorBoard '
- Safetensors
- ' trl'
- generated_from_trainer 4-bit
- ' precision'
license: mit
datasets:
- yahma/alpaca-cleaned
language:
- en
pipeline_tag: question-answering
---

# Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. -->
This Model is Finetuned for Document Question and Answering purpose Trained on the yahma/alpaca-cleaned(https://huggingface.co/TheBloke/zephyr-7B-beta-GPTQ) dataset.


## Model Details

### Training hyperparameters

The following hyperparameters were used during training:

-gradient_accumulation_steps=1,

-warmup_steps=5,

-max_steps=20,

-learning_rate=2e-4,

-fp16=not torch.cuda.is_bf16_supported(),

-bf16=torch.cuda.is_bf16_supported(),

-logging_steps=1,

-optim="adamw_8bit",

-weight_decay=0.01,

-lr_scheduler_type="linear",

-seed=3407,

- ### Framework versions

- PEFT 0.7.1
- Transformers 4.36.0
- Pytorch 2.0.0
- Datasets 2.16.1
- Tokenizers 0.15.0