Update README.md

README.md

@@ -7,4 +7,119 @@ language:
 base_model:
 - EleutherAI/gpt-j-6b
 - CarperAI/openai_summarize_tldr_sft
----
+---
+# ALT-RM model (reward model-based feedback)
+Fine-tuned **GPT-J (6B)** model on the **TL;DR Summarization** dataset to be better aligned with humans' preferences on summaries, i.e., accounting for axes such as accuracy, coverage, and coherence, following the alignment approach introduced in the [ALT paper](https://www.arxiv.org/abs/2407.16970). This corresponds to the official model checkpoint and the code can be found in [here](https://github.com/sauc-abadal/ALT/tree/main).
+
+# Model description
+The alignment process departs from a [SFT checkpoint](https://huggingface.co/CarperAI/openai_summarize_tldr_sft) released by CarperAI and trained using their [trlx](https://github.com/CarperAI/trlx/tree/main/examples/summarize_rlhf) library. 
+
+In a nutshell, the ALT method consists on providing textual feedback to on-policy sampled generations to learn the conditional probability distribution of a generation given both the prompt and the feedback. This logic is implemented in a three-stage decoupled pipeline, namely *sampling*, *feedback*, and *training*, where training is based on a language modelling objective by preppending the feedback tokens before the prompt. 
+In this way, the model learns to discriminate between different generations associated with various feedback types: it learns from both positive and negative examples that encompass the entire feedback spectrum, overcoming one of the main limitations of supervised fine-tuning, which typically learns only from positive demonstrations.
+
+For extensive coverage on the ALT method, please refer to the paper.
+
+In particular, the **ALT-RM** checkpoint collects the feedback by leveraging a [Reward Model](https://huggingface.co/CarperAI/openai_summarize_tldr_rm_checkpoint) to score the generations, and then maps reward quantiles computed for several generations under the same prompt to pre-defined textual feedbacks. For the summarization task on the TL;DR dataset, the mapping from quantiles to feedback employed was: 
+```python
+{'QUANTILE 0': 'Excellent.',
+ 'QUANTILE 1': 'Good.',
+ 'QUANTILE 2': 'Mediocre.',
+ 'QUANTILE 3': 'Bad.',
+ 'QUANTILE 4': 'Horrible.'}
+```
+Thus, at inference time, the expected aligned behavior can be attained by conditioning the input with the *'Excellent.'* feedback.
+
+**Related Models:** [ALT-Quark]().
+
+# Intended uses & limitations
+This model originates from a research project focused on alignment and is intended primarily for research purposes. Commercial use as an off-the-shelf model is discouraged, as it was not designed with such applications in mind. The model is tailored specifically for the summarization task, having been trained on the TL;DR dataset, though some out-of-distribution generalization may be possible for related datasets.
+
+# How to use
+
+You should format the input by preppending the feedback as follows: `Excellent. input: {prompt}`
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig
+
+checkpoint_path = "sauc-abadal-lloret/gpt-j-6b-ALT-RM-tldr"
+
+tokenizer = AutoTokenizer.from_pretrained(checkpoint_path) 
+tokenizer.pad_token = tokenizer.eos_token
+
+model = AutoModelForCausalLM.from_pretrained(checkpoint_path)
+model.eval()
+
+prompt = "Excellent. input: SUBREDDIT: r/relationship_advice\nTITLE: I'm [18M] going to a party where an old middle \
+school crush [17F] is also going.\nPOST: Story time! Back in the summer after 8th grade, I hung out with my group of \
+friends everyday for the whole summer. There was this girl in the group and I really liked her. Like I had the biggest \
+and dumbest crush on her. I was only 13 so I didn't know shit, but I was thinking she's perfect for me, I gotta marry \
+her and all this dumb stuff. The puppy love was so strong I wanted to be a part of her life and I wanted her to be a \
+part of my life. I never had the courage to ask her out, and we went to different high schools. Eventually we stopped \
+talking but during high school I never really liked anyone else. Every other girl felt dull compared to her. I still \
+get nostalgic thinking about her and what would've been different if I had the balls to ask her out. Anyway I'm going \
+to a party this Friday and I heard she's coming. I honestly don't know what to do to so this goes great and eventually \
+ends up in a relationship.\nTL;DR:"
+
+inputs = tokenizer([prompt], padding=True, truncation=True, return_tensors="pt")
+input_seq_len = inputs["input_ids"].shape[1]
+
+generation_config = GenerationConfig(
+    max_length = 2048,
+    max_new_tokens = 64,
+    do_sample = False,
+    num_beams = 1,
+    bad_words_ids = None,
+    num_return_sequences = 1,   
+    return_dict_in_generate = True,
+    pad_token_id = tokenizer.pad_token_id,
+)
+
+outputs = model.generate(**inputs, generation_config=generation_config)
+generated_input_ids = outputs["sequences"][:, input_seq_len:]
+generated_text = tokenizer.batch_decode(
+    generated_input_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True
+)
+generated_text
+```
+
+```
+[" I have a huge crush on a girl who I never asked out and we went to different high schools. I'm going to a party this Friday and I heard she's coming. I honestly don't know what to do to so this goes great and eventually ends up in a relationship."]
+```
+
+## Training data
+The model was trained on the TL;DR summarization dataset introduced in the Stiennon et al.'s, ["Learning to Summarize from human feedback"](https://arxiv.org/abs/2009.01325) paper. We employed the dataset version from CarperAI, which can be found in the HuggingFace Hub in [here](CarperAI/openai_summarize_tldr).
+
+## Training procedure
+The exact training procedure and hyper-parameters configuration can be found in our paper. 
+
+## Variable and metrics
+As an evaluation metric, we compute GPT-4 win-rates over PPO on a 1k random subset of the test set. We use the prompt provided in the DPO paper and we ask GPT-4 to compare generations between ALT-RM and Quark and PPO. Furthermore, we report the following metrics computed on the whole test set: average reward model score, perplexity measured by the SFT reference policy as a proxy for fluency, and average length of the generations. In addition, we conduct an out-of-domain evaluation and compute GPT-4 win-rates on 100 articles from the test split of the CNN/DailyMail dataset.
+
+| **Model**      | **TL;DR** (In-domain) | **CNN/DailyMail** (Out-of-domain) |
+|:---------------:|:---------------------:|:----------------------------------:|
+| Quark vs PPO   |         0.36          |                0.40                |
+| ALT-RM vs PPO  |         0.50          |                0.48                |
+
+*Win-rates with GPT-4. TL;DR on 1000 randomly chosen test prompts and CNN/daily mail on 100 randomly chosen test prompts.*
+
+| **Model**      | **RM** | **PPL** | **Avg. len** | **# Train** |
+|:---------------:|:---------------------:|:----------------------------------:|:----------------------------------:|:----------------------------------:|
+| SFT   | 2.89 | 1.96 | 31.25 | - |
+| Refrences | 2.89 | 11.84 | 32.60 | - |
+| PPO  | 3.38 | 2.29 | 67.52 | 116k |
+| Quark  | 3.52 | 1.82 | 49.42 | 19k |
+| ALT-RM  | 3.58 | 2.20 | 46.14 | 19k |
+
+*TL;DR metrics on the whole test set, including avg. reward model score, perplexity, avg. generations’ length, and number of training prompts.*
+
+## BibTeX entry and citation info
+```
+@misc{lloret2024aligninglanguagemodelstextual,
+      title={Towards Aligning Language Models with Textual Feedback}, 
+      author={Saüc Abadal Lloret and Shehzaad Dhuliawala and Keerthiram Murugesan and Mrinmaya Sachan},
+      year={2024},
+      eprint={2407.16970},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/2407.16970}, 
+}
+```