language:
- it
license: apache-2.0
library_name: transformers
tags:
- text-generation-inference
- unsloth
- llama
- llama3.1
- trl
- word-game
- rebus
- italian
- word-puzzle
- crossword
datasets:
- gsarti/eureka-rebus
base_model: unsloth/Meta-Llama-3.1-8B-bnb-4bit
model-index:
- name: gsarti/llama-3.1-8b-rebus-solver-fp16
results:
- task:
type: verbalized-rebus-solving
name: Verbalized Rebus Solving
dataset:
type: gsarti/eureka-rebus
name: EurekaRebus
config: llm_sft
split: test
revision: 0f24ebc3b66cd2f8968077a5eb058be1d5af2f05
metrics:
- type: exact_match
value: 0.59
name: First Pass Exact Match
- type: exact_match
value: 0.56
name: Solution Exact Match
LLaMA-3.1 8B Verbalized Rebus Solver - GGUF Q8_0 🇮🇹
This model is a parameter-efficient fine-tuned version of LLaMA-3.1 8B trained for verbalized rebus solving in Italian, as part of the release for our paper Non Verbis, Sed Rebus: Large Language Models are Weak Solvers of Italian Rebuses. The task of verbalized rebus solving consists of converting an encrypted sequence of letters and crossword definitions into a solution phrase matching the word lengths specified in the solution key. An example is provided below.
The model was trained in 4-bit precision for 5070 steps on the verbalized subset of the EurekaRebus using QLora via Unsloth and TRL.This repository contains the GGUF exported checkpoint of the model in Q8_0 format, and the Modelfile for usage with Ollama (see below).
We also provide adapter checkpoints through training and FP16 merged versions of this model for analysis and local execution.
Local usage with Ollama
A ready-to-use local version of this model is hosted on the Ollama Hub and can be used as follows:
ollama run gsarti/llama3.1-8b-rebus-solver "Rebus: [Materiale espulso dai vulcani] R O [Strumento del calzolaio] [Si trovano ai lati del bacino] C I [Si ingrassano con la polenta] E I N [Contiene scorte di cibi] B [Isola in francese]\nChiave risolutiva: 1 ' 5 6 5 3 3 1 14"
# EXAMPLE GENERATION:
#
# Procediamo alla risoluzione del rebus passo per passo:
# - [Materiale espulso dai vulcani] = lava
# - R O = R O
# - [Strumento del calzolaio] = lesina
# - [Si trovano ai lati del bacino] = anche
# - C I = C I
# - [Si ingrassano con la polenta] = oche
# - E I N = E I N
# - [Contiene scorte di cibi] = dispensa
# - B = B
# - [Isola in francese] = ile
#
# Prima lettura: lava R O lesina anche C I oche E I N dispensa B ile
#
# Ora componiamo la soluzione seguendo la chiave risolutiva:
# 1 = L
# ' = '
# 5 = avaro
# 6 = lesina
# 5 = anche
# 3 = ciò
# 3 = che
# 1 = è
# 14 = indispensabile
#
# Soluzione: L'avaro lesina anche ciò che è indispensabile
Limitations
Lexical overfitting: As remarked in the related publication, the model overfitted the set of definitions/answers for first pass words. As a result, words that were explicitly witheld from the training set cause significant performance degradation when used as solutions for verbalized rebuses' definitions. You can compare model performances between in-domain and out-of-domain test examples to verify this limitation.
Model curators
For problems or updates on this model, please contact [email protected].
Citation Information
If you use this model in your work, please cite our paper as follows:
@article{sarti-etal-2024-rebus,
title = "Non Verbis, Sed Rebus: Large Language Models are Weak Solvers of Italian Rebuses",
author = "Sarti, Gabriele and Caselli, Tommaso and Nissim, Malvina and Bisazza, Arianna",
journal = "ArXiv",
month = jul,
year = "2024",
volume = {abs/2408.00584},
url = {https://arxiv.org/abs/2408.00584},
}
Acknowledgements
We are grateful to the Associazione Culturale "Biblioteca Enigmistica Italiana - G. Panini" for making its rebus collection freely accessible on the Eureka5 platform.
