README.md

---
license: apache-2.0
language:
- en
tags:
  - conversation
  - dialogue
  - commonsense
---

# Model Card for ConvoSenseGenerator

ConvoSenseGenerator is a generative model that produces commonsense inferences for dialogue contexts, covering 10 common social commonsense types such as emotional reactions, motivations, causes, subsequent events, and more! 

It is trained on the large-scale dataset, ConvoSense, that is collected synthetically using ChatGPT3.5. 

ConvoSenseGenerator produces inferences that humans judge to achieve high reasonability, high rates of novel information for the corresponding dialogue contexts, and high degree of detail, outperforming models trained on previous datasets that are human-written.

## Model Description
- **Repository:** [Code](https://github.com/emorynlp/convosense)
- **Paper:** [ConvoSense: Overcoming Monotonous Commonsense Inferences for Conversational AI](https://doi.org/10.1162/tacl\_a\_00659)
- **Point of Contact:** [Sarah E. Finch](mailto:[email protected])

## Model Training

ConvoSenseGenerator is trained on our recent dataset: ConvoSense.
The backbone model of ConvoSenseGenerator is [T5-3b](https://huggingface.co/t5-3b).

### How to use

ConvoSenseGenerator covers the following commonsense types, using the provided questions:

```python
commonsense_questions = {
    "cause": 'What could have caused the last thing said to happen?', 
    "prerequisities": 'What prerequisites are required for the last thing said to occur?', 
    "motivation": 'What is an emotion or human drive that motivates Speaker based on what they just said?', 
    "subsequent": 'What might happen after what Speaker just said?', 
    "desire": 'What does Speaker want to do next?',
    "desire_o": 'What will Listener want to do next based on what Speaker just said?',
    "react": 'How is Speaker feeling after what they just said?',
    "react_o": 'How does Listener feel because of what Speaker just said?',
    "attribute": 'What is a likely characteristic of Speaker based on what they just said?',
    "constituents": 'What is a breakdown of the last thing said into a series of required subevents?' 
}
```

The best-performing configuration of ConvoSenseGenerator according to the experiments in the paper uses the following generation hyperparameters:

```python
generation_config = {
    "repetition_penalty": 1.0,
    "num_beams": 10,
    "num_beam_groups": 10,
    "diversity_penalty": 0.5
}
```

Below is a simple code snippet to get ConvoSenseGenerator running:

```python
import torch
from transformers import AutoTokenizer, T5ForConditionalGeneration

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
tokenizer = AutoTokenizer.from_pretrained("sefinch/ConvoSenseGenerator")
model = T5ForConditionalGeneration.from_pretrained("sefinch/ConvoSenseGenerator").to(device)

# ConvoSenseGenerator covers these commonsense types, using the provided questions
commonsense_questions = {
    "cause": 'What could have caused the last thing said to happen?', 
    "prerequisities": 'What prerequisites are required for the last thing said to occur?', 
    "motivation": 'What is an emotion or human drive that motivates Speaker based on what they just said?', 
    "subsequent": 'What might happen after what Speaker just said?', 
    "desire": 'What does Speaker want to do next?',
    "desire_o": 'What will Listener want to do next based on what Speaker just said?',
    "react": 'How is Speaker feeling after what they just said?',
    "react_o": 'How does Listener feel because of what Speaker just said?',
    "attribute": 'What is a likely characteristic of Speaker based on what they just said?',
    "constituents": 'What is a breakdown of the last thing said into a series of required subevents?' 
}

def format_input(conversation_history, commonsense_type):

    # prefix last turn with Speaker, and alternately prefix each previous turn with either Listener or Speaker
    prefixed_turns = list(
        reversed(
            [
                f"{'Speaker' if i % 2 == 0 else 'Listener'}: {u}"
                for i, u in enumerate(reversed(conversation_history))
            ]
        )
    )

    # model expects a maximum of 7 total conversation turns to be given
    truncated_turns = prefixed_turns[-7:]

    # conversation representation separates the turns with newlines
    conversation_string = '\n'.join(truncated_turns)

    # format the full input including the commonsense question
    input_text = f"provide a reasonable answer to the question based on the dialogue:\n{conversation_string}\n\n[Question] {commonsense_questions[commonsense_type]}\n[Answer]"

    return input_text

def generate(conversation_history, commonsense_type):
    # convert the input into the expected format to run the model
    input_text = format_input(conversation_history, commonsense_type) 

    # tokenize the input_text
    inputs = tokenizer([input_text], return_tensors="pt").to(device)

    # get multiple model generations using the best-performing generation configuration (based on experiments detailed in paper)
    outputs = model.generate(
        inputs["input_ids"],
        repetition_penalty=1.0,
        num_beams=10,
        num_beam_groups=10,
        diversity_penalty=0.5,
        num_return_sequences=5,
        max_new_tokens=400
    )

    # decode the generated inferences
    inferences = tokenizer.batch_decode(outputs, skip_special_tokens=True, clean_up_tokenization_spaces=False)

    return inferences

conversation = [
    "Hey, I'm trying to convince my parents to get a dog, but they say it's too much work.",
    "Well, you could offer to do everything for taking care of it. Have you tried that?",
    "But I don't want to have to take the dog out for walks when it is the winter!"
]

inferences = generate(conversation, "cause")
print('\n'.join(inferences))

# Outputs:
# the speaker's fear of the cold and the inconvenience of having to take the dog out in the winter.
# the speaker's preference for indoor activities during winter, such as watching movies or playing video games.
# the speaker's fear of getting sick from taking the dog out in the cold.
# a previous negative experience with taking dogs for walks in the winter.
# the listener's suggestion to offer to help with taking care of the dog, which the speaker may have considered but was not willing to do.
```

### Citation

Please cite our work if you find the resources in this repository useful:
```
@article{convosense_finch:24,
    author = {Finch, Sarah E. and Choi, Jinho D.},
    title = "{ConvoSense: Overcoming Monotonous Commonsense Inferences for Conversational AI}",
    journal = {Transactions of the Association for Computational Linguistics},
    volume = {12},
    pages = {467-483},
    year = {2024},
    month = {05},
    issn = {2307-387X},
    doi = {10.1162/tacl_a_00659},
    url = {https://doi.org/10.1162/tacl\_a\_00659},
    eprint = {https://direct.mit.edu/tacl/article-pdf/doi/10.1162/tacl\_a\_00659/2369521/tacl\_a\_00659.pdf},
}
```