File size: 5,872 Bytes
3083de1 9095279 316a7a6 cd85a0e 7a8b0c4 fcf0a55 3083de1 316a7a6 da52493 316a7a6 da52493 e265dad 316a7a6 9095279 316a7a6 20565e6 316a7a6 f62b8bf 316a7a6 cd85a0e |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 |
---
library_name: transformers
base_model: codellama/CodeLlama-7b-Instruct-hf
license: apache-2.0
datasets:
- semantixai/Test-Dataset-Lloro
language:
- pt
tags:
- code
- analytics
- analise-dados
- portugues-BR
---
**Lloro 7B**
<img src="https://cdn-uploads.huggingface.co/production/uploads/653176dc69fffcfe1543860a/h0kNd9OTEu1QdGNjHKXoq.png" width="300" alt="Lloro-7b Logo"/>
Lloro, developed by Semantix Research Labs , is a language Model that was trained to effectively perform Portuguese Data Analysis. It is a fine-tuned version of codellama/CodeLlama-7b-Instruct-hf, that was trained on synthetic datasets . The fine-tuning process was performed using the QLORA metodology on a GPU V100 with 16 GB of RAM.
**Model description**
Model type: A 7B parameter fine-tuned on synthetic datasets.
Language(s) (NLP): Primarily Portuguese, but the model is capable to understand English as well
Finetuned from model: codellama/CodeLlama-7b-Instruct-hf
**What is Lloro's intended use(s)?**
Lloro is built for data analysis in Portuguese contexts .
Input : Text
Output : Text (Code)
**Usage**
Using Transformers
```python
#Import required libraries
import torch
from transformers import (
AutoModelForCausalLM,
AutoTokenizer
)
#Load Model
model_name = "semantixai/LloroV2"
base_model = AutoModelForCausalLM.from_pretrained(
model_name,
return_dict=True,
torch_dtype=torch.float16,
device_map="auto",
)
#Load Tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
#Define Prompt
user_prompt = "Desenvolva um algoritmo em Python para calcular a média e a mediana dos preços de vendas por tipo de material do produto."
system = "Provide answers in Python without explanations, only the code"
prompt_template = f"[INST] <<SYS>>\\n{system}\\n<</SYS>>\\n\\n{user_prompt}[/INST]"
#Call the model
input_ids = tokenizer([prompt_template], return_tensors="pt")["input_ids"].to("cuda")
outputs = base_model.generate(
input_ids,
do_sample=True,
top_p=0.95,
max_new_tokens=1024,
temperature=0.1,
)
#Decode and retrieve Output
output_text = tokenizer.batch_decode(outputs, skip_prompt=True, skip_special_tokens=False)
display(output_text)
```
Using an OpenAI compatible inference server (like [vLLM](https://docs.vllm.ai/en/latest/index.html))
```python
from openai import OpenAI
client = OpenAI(
api_key="EMPTY",
base_url="http://localhost:8000/v1",
)
user_prompt = "Desenvolva um algoritmo em Python para calcular a média e a mediana dos preços de vendas por tipo de material do produto."
completion = client.chat.completions.create(temperature=0.1,frequency_penalty=0.1,model="semantixai/LloroV2",messages=[{"role":"system","content":"Provide answers in Python without explanations, only the code"},{"role":"user","content":user_prompt}])
```
**Params**
Training Parameters
| Params | Training Data | Examples | Tokens | LR |
|----------------------------------|---------------------------------|---------------------------------|----------|--------|
| 7B | Pairs synthetic instructions/code | 28907 | 3 031 188 | 1e-5 |
**Model Sources**
Test Dataset Repository: https://huggingface.co/datasets/semantixai/Test-Dataset-Lloro
Model Dates Lloro was trained between November 2023 and January 2024.
**Performance**
| Modelo | LLM as Judge | Code Bleu Score | Rouge-L | CodeBert- Precision | CodeBert-Recall | CodeBert-F1 | CodeBert-F3 |
|----------------|--------------|------------------|---------|----------------------|-----------------|-------------|-------------|
| GPT 3.5 | 91.22% | 0.2745 | 0.2189 | 0.7502 | 0.7146 | 0.7303 | 0.7175 |
| Instruct -Base | 97.40% | 0.2487 | 0.1146 | 0.6997 | 0.6473 | 0.6713 | 0.6518 |
| Instruct -FT | 97.76% | 0.3264 | 0.3602 | 0.7942 | 0.8178 | 0.8042 | 0.8147 |
**Training Infos:**
The following hyperparameters were used during training:
| Parameter | Value |
|---------------------------|----------------------|
| learning_rate | 1e-5 |
| weight_decay | 0.0001 |
| train_batch_size | 1 |
| eval_batch_size | 1 |
| seed | 42 |
| optimizer | Adam - paged_adamw_32bit |
| lr_scheduler_type | cosine |
| lr_scheduler_warmup_ratio | 0.03 |
| num_epochs | 5.0 |
**QLoRA hyperparameters**
The following parameters related with the Quantized Low-Rank Adaptation and Quantization were used during training:
| Parameter | Value |
|------------------|---------|
| lora_r | 16 |
| lora_alpha | 64 |
| lora_dropout | 0.1 |
| storage_dtype | "nf4" |
| compute_dtype | "float16"|
**Experiments**
| Model | Epochs | Overfitting | Final Epochs | Training Hours | CO2 Emission (Kg) |
|-----------------------|--------|-------------|--------------|-----------------|--------------------|
| Code Llama Instruct | 1 | No | 1 | 8.1 | 1.337 |
| Code Llama Instruct | 5 | Yes | 3 | 45.6 | 9.12 |
**Framework versions**
| Library | Version |
|---------------|-----------|
| bitsandbytes | 0.40.2 |
| Datasets | 2.14.3 |
| Pytorch | 2.0.1 |
| Tokenizers | 0.14.1 |
| Transformers | 4.34.0 | |