@inproceedings{
chaudhary2024riskaverse,
title={Risk-Averse Fine-tuning of Large Language Models},
author={Sapana Chaudhary and Ujwal Dinesha and Dileep Kalathil and Srinivas Shakkottai},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=1BZKqZphsW}
}
We consider the challenge of mitigating the generation of negative or toxic content by the Large Language Models (LLMs) in response to certain prompts.  We propose integrating risk-averse principles into LLM fine-tuning to minimize the occurrence of harmful outputs, particularly rare but significant events.  By optimizing the risk measure of Conditional Value at Risk (CVaR), our methodology trains LLMs to exhibit superior performance in avoiding toxic outputs while maintaining effectiveness in generative tasks. Empirical evaluations on sentiment modification and toxicity mitigation tasks demonstrate the efficacy of risk-averse reinforcement learning with human feedback (RLHF) in promoting a safer and more constructive online discourse environment.
Risk-Averse Fine-tuning of Large Language Models
Sapana Chaudhary Ujwal Dinesha Dileep Kalathil Srinivas Shakkottai