Edit model card

poem-gen-spanish-t5-small

This model is a fine-tuned version of flax-community/spanish-t5-small on the Spanish Poetry Dataset dataset.

The model was created during the First Spanish Hackathon organized by Somos NLP.

The team who participated was composed by:

It achieves the following results on the evaluation set:

  • Loss: 2.8707
  • Perplexity: 17.65

Model description

The model was trained to generate spanish poems attending to some parameters like style, sentiment, words to include and starting phrase.

Example:

poema:
  estilo: Pablo Neruda &&
  sentimiento: positivo &&
  palabras: cielo, luna, mar &&
  texto: Todos fueron a verle pasar

How to use

You can use this model directly with a pipeline for masked language modeling:

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
model_name = 'hackathon-pln-es/poem-gen-spanish-t5-small'
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)

author, sentiment, word, start_text = 'Pablo Neruda', 'positivo', 'cielo', 'Todos fueron a la plaza'
input_text = f"""poema: estilo: {author} && sentimiento: {sentiment} && palabras: {word} && texto: {start_text} """
inputs = tokenizer(input_text, return_tensors="pt")

outputs = model.generate(inputs["input_ids"],
                         do_sample = True,
                         max_length = 30,
                         repetition_penalty = 20.0,
                         top_k = 50,
                         top_p = 0.92)
detok_outputs = [tokenizer.decode(x, skip_special_tokens=True) for x in outputs]
res = detok_outputs[0]

Training and evaluation data

The original dataset has the columns author, content and title. For each poem we generate new examples:

  • content: line_i , generated: line_i+1
  • content: concatenate(line_i, line_i+1) , generated: line_i+2
  • content: concatenate(line_i, line_i+1, line_i+2) , generated: line_i+3

The resulting dataset has the columns author, content, title and generated.

For each example we compute the sentiment of the generated column and the nouns. In the case of sentiment, we used the model mrm8488/electricidad-small-finetuned-restaurant-sentiment-analysis and for nouns extraction we used spaCy.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 2e-05
  • train_batch_size: 6
  • eval_batch_size: 6
  • seed: 42
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • num_epochs: 6

Training results

Training Loss Epoch Step Validation Loss
2.7082 0.73 30000 2.8878
2.6251 1.46 60000 2.8940
2.5796 2.19 90000 2.8853
2.5556 2.93 120000 2.8749
2.527 3.66 150000 2.8850
2.5024 4.39 180000 2.8760
2.4887 5.12 210000 2.8749
2.4808 5.85 240000 2.8707

Framework versions

  • Transformers 4.17.0
  • Pytorch 1.10.0+cu111
  • Datasets 2.0.0
  • Tokenizers 0.11.6
Downloads last month
42
Safetensors
Model size
60.5M params
Tensor type
F32
·
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Model tree for somosnlp-hackathon-2022/poem-gen-spanish-t5-small

Finetuned
(5)
this model

Spaces using somosnlp-hackathon-2022/poem-gen-spanish-t5-small 2

Collection including somosnlp-hackathon-2022/poem-gen-spanish-t5-small