File size: 2,931 Bytes
1a89142 07693ae 1a89142 07693ae 1a89142 07693ae 1a89142 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 |
---
license: other
datasets:
- blip-solutions/SlovAlpaca
language:
- sk
---
# SlovAlpaca
This repository contains the LORA weights finetuned on the translated version of the original Alpaca dataset (more info on the dataset card)
## Training procedure
The training was done on the 7B LLaMA model (decapoda-research/llama-7b-hf) quantized to 8bits with the following Hyperparameters:
```
MICRO_BATCH_SIZE = 3
BATCH_SIZE = 128
GRADIENT_ACCUMULATION_STEPS = BATCH_SIZE // MICRO_BATCH_SIZE
EPOCHS = 2 # paper uses 3
LEARNING_RATE = 2e-5 # from the original paper
CUTOFF_LEN = 256
LORA_R = 4
LORA_ALPHA = 16
LORA_DROPOUT = 0.05
```
The sole goal of this project is to explore the effects of single-language finetuning using the same dataset and methods as the original paper did and comapre the results
@misc{alpaca,
author = {Rohan Taori and Ishaan Gulrajani and Tianyi Zhang and Yann Dubois and Xuechen Li and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto },
title = {Stanford Alpaca: An Instruction-following LLaMA model},
year = {2023},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/tatsu-lab/stanford_alpaca}},
}
## How to use:
### Prerequisites
```
!pip install datasets loralib sentencepiece
!pip uninstall -y transformers
!pip install git+https://github.com/zphang/transformers@c3dc391#egg=transformers
!pip install git+https://github.com/huggingface/peft.git
!pip install bitsandbytes
```
### Load model:
```
from peft import PeftModel
from transformers import LLaMATokenizer, LLaMAForCausalLM, GenerationConfig
tokenizer = LLaMATokenizer.from_pretrained("decapoda-research/llama-7b-hf")
model = LLaMAForCausalLM.from_pretrained(
"decapoda-research/llama-7b-hf",
load_in_8bit=True,
device_map="auto",
)
model = PeftModel.from_pretrained(model, "blip-solutions/SlovAlpaca")
```
### Generation
Here is a colab notebook for inference: https://colab.research.google.com/drive/1z4aMG7tGjchLBlg_iXDuqt3sH6bQRuQk?usp=sharing
```
PROMPT = """Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
Kde žijú lamy?
### Response:"""
inputs = tokenizer(
PROMPT,
return_tensors="pt",
)
input_ids = inputs["input_ids"].cuda()
generation_config = GenerationConfig(
temperature=0.6,
top_p=0.95,
repetition_penalty=1.15,
)
print("Generating...")
generation_output = model.generate(
input_ids=input_ids,
generation_config=generation_config,
return_dict_in_generate=True,
output_scores=True,
max_new_tokens=128,
)
for s in generation_output.sequences:
print(tokenizer.decode(s))
```
### Response:
```
Generating...
Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
Kde žijú lamy?
### Response:
Lamy žiju v horách, na poli, alebo v lesoch.
```
|