--- language: - pt - en library_name: adapter-transformers datasets: - dominguesm/alpaca-data-pt-br pipeline_tag: text-generation thumbnail: https://blog.cobasi.com.br/wp-content/uploads/2022/08/AdobeStock_461738919.webp model-index: - name: Caramelo_7B results: - task: type: text-generation name: Text Generation dataset: name: ENEM Challenge (No Images) type: eduagarcia/enem_challenge split: train args: num_few_shot: 3 metrics: - type: acc value: 19.8 name: accuracy source: url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=Bruno/Caramelo_7B name: Open Portuguese LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: BLUEX (No Images) type: eduagarcia-temp/BLUEX_without_images split: train args: num_few_shot: 3 metrics: - type: acc value: 24.48 name: accuracy source: url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=Bruno/Caramelo_7B name: Open Portuguese LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: OAB Exams type: eduagarcia/oab_exams split: train args: num_few_shot: 3 metrics: - type: acc value: 25.28 name: accuracy source: url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=Bruno/Caramelo_7B name: Open Portuguese LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: Assin2 RTE type: assin2 split: test args: num_few_shot: 15 metrics: - type: f1_macro value: 54.27 name: f1-macro source: url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=Bruno/Caramelo_7B name: Open Portuguese LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: Assin2 STS type: eduagarcia/portuguese_benchmark split: test args: num_few_shot: 15 metrics: - type: pearson value: 7.47 name: pearson source: url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=Bruno/Caramelo_7B name: Open Portuguese LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: FaQuAD NLI type: ruanchaves/faquad-nli split: test args: num_few_shot: 15 metrics: - type: f1_macro value: 43.97 name: f1-macro source: url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=Bruno/Caramelo_7B name: Open Portuguese LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: HateBR Binary type: ruanchaves/hatebr split: test args: num_few_shot: 25 metrics: - type: f1_macro value: 33.65 name: f1-macro source: url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=Bruno/Caramelo_7B name: Open Portuguese LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: PT Hate Speech Binary type: hate_speech_portuguese split: test args: num_few_shot: 25 metrics: - type: f1_macro value: 41.23 name: f1-macro source: url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=Bruno/Caramelo_7B name: Open Portuguese LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: tweetSentBR type: eduagarcia-temp/tweetsentbr split: test args: num_few_shot: 25 metrics: - type: f1_macro value: 35.37 name: f1-macro source: url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=Bruno/Caramelo_7B name: Open Portuguese LLM Leaderboard ---
Caramelo
# CARAMELO ## Adapter Description This adapter was created with the [PEFT](https://github.com/huggingface/peft) library and allowed the base model **Falcon-7b** to be fine-tuned on the **https://huggingface.co/datasets/dominguesm/alpaca-data-pt-br** by using the method **QLoRA**. ## Model description [Falcon 7B](https://huggingface.co/tiiuae/falcon-7b) ## Intended uses & limitations TBA ## Training and evaluation data TBA ### Training results ### How to use ```py from peft import PeftModel, PeftConfig from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, GenerationConfig peft_model_id = "Bruno/Caramelo_7B" config = PeftConfig.from_pretrained(peft_model_id) bnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.float16, ) tokenizer = AutoTokenizer.from_pretrained(peft_model_id) model = AutoModelForCausalLM.from_pretrained(config.base_model_name_or_path, return_dict=True, quantization_config=bnb_config, trust_remote_code=True, device_map={"": 0}) prompt_input = "Abaixo está uma declaração que descreve uma tarefa, juntamente com uma entrada que fornece mais contexto. Escreva uma resposta que conclua corretamente a solicitação.\n\n ### Instrução:\n{instruction}\n\n### Entrada:\n{input}\n\n### Resposta:\n" prompt_no_input = "Abaixo está uma instrução que descreve uma tarefa. Escreva uma resposta que conclua corretamente a solicitação.\n\n### Instrução:\n{instruction}\n\n### Resposta:\n" def create_prompt(instruction, input=None): if input: return prompt_input.format(instruction=instruction, input=input) else: return prompt_no_input.format(instruction=instruction) def generate( instruction, input=None, max_new_tokens=128, temperature=0.1, top_p=0.75, top_k=40, num_beams=4, repetition_penalty=1.5, max_length=512 ): prompt = create_prompt(instruction, input) inputs = tokenizer.encode_plus(prompt, return_tensors="pt", truncation=True, max_length=max_length, padding="longest") input_ids = inputs["input_ids"].to("cuda") attention_mask = inputs["attention_mask"].to("cuda") generation_output = model.generate( input_ids=input_ids, attention_mask=attention_mask, max_length=max_length, pad_token_id=tokenizer.pad_token_id, eos_token_id=tokenizer.eos_token_id, temperature=temperature, top_p=top_p, top_k=top_k, num_beams=num_beams, repetition_penalty=repetition_penalty, length_penalty=0.8, early_stopping=True, output_scores=True, return_dict_in_generate=True ) output = tokenizer.decode(generation_output.sequences[0], skip_special_tokens=True) return output.split("### Resposta:")[1] instruction = "como faço um bolo de cenoura?" print(Instrução:", instruction) print("Resposta:", generate(instruction)) ### Saída Instrução: como faço um bolo de cenoura? Resposta: 1. Pegue uma cenoura e corte-a em pedaços pequenos. 2. Coloque os pedaços de cenoura em uma panela e cozinhe por 10 minutos. 3. Retire a cenoura da panela e deixe-a esfriar. 4. Coloque a cenoura em uma bolsa de plástico e congele. 5. Quando precisar, coloque a cenoura congelada na máquina de bolo. ### Framework versions - Transformers 4.30.0.dev0 - Pytorch 2.0.1+cu118 - Datasets 2.12.0 - Tokenizers 0.13.3 # [Open Portuguese LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard) Detailed results can be found [here](https://huggingface.co/datasets/eduagarcia-temp/llm_pt_leaderboard_raw_results/tree/main/Bruno/Caramelo_7B) | Metric | Value | |--------------------------|---------| |Average |**31.73**| |ENEM Challenge (No Images)| 19.80| |BLUEX (No Images) | 24.48| |OAB Exams | 25.28| |Assin2 RTE | 54.27| |Assin2 STS | 7.47| |FaQuAD NLI | 43.97| |HateBR Binary | 33.65| |PT Hate Speech Binary | 41.23| |tweetSentBR | 35.37|