README.md

metadata

title: GPU Poor LLM Arena
emoji: 🏆
colorFrom: blue
colorTo: purple
sdk: gradio
sdk_version: 5.5.0
app_file: app.py
pinned: false
license: mit
short_description: 'Compact LLM Battle Arena: Frugal AI Face-Off!'

🏆 GPU-Poor LLM Gladiator Arena 🏆

Welcome to the GPU-Poor LLM Gladiator Arena, where frugal meets fabulous in the world of AI! This project pits compact language models (maxing out at 9B parameters) against each other in a battle of wits and words.

🤔 Starting from "Why?"

In the recent months, we've seen a lot of these "Tiny" models released, and some of them are really impressive.

• Gradio Exploration: This project serves me as a playground for experimenting with Gradio app development; I am learning how to create interactive AI interfaces with it.

• Tiny Model Evaluation: I wanted to develop a personal (and now public) stats system for evaluating tiny language models. It's not too serious, but it provides valuable insights into the capabilities of these compact powerhouses.

• Accessibility: Built on Ollama, this arena allows pretty much anyone to experiment with these models themselves. No need for expensive GPUs or cloud services!

• Pure Fun: At its core, this project is about having fun with AI. It's a lighthearted way to explore and compare different models. So, haters, feel free to chill – we're just here for a good time!

🌟 Features

• Battle Arena: Pit two mystery models against each other and decide which pint-sized powerhouse reigns supreme.
• Leaderboard: Track the performance of different models over time using an improved scoring system.
• Performance Chart: Visualize model performance with interactive charts.
• Privacy-Focused: Uses local Ollama API, avoiding pricey commercial APIs and keeping data close to home.
• Customizable: Easy to add new models and prompts.

🚀 Getting Started

Prerequisites

• Python 3.7+
• Gradio
• Plotly
• Ollama (running locally)

Installation

1. Clone the repository:

   git clone https://huggingface.co/spaces/k-mktr/gpu-poor-llm-arena.git
   cd gpu-poor-llm-arena
   

2. Install the required packages:

   pip install gradio plotly requests
   

3. Ensure Ollama is running locally or via a remote server.

4. Run the application:

   python app.py
   

🎮 How to Use

1. Open the application in your web browser (typically at http://localhost:7860).
2. In the "Battle Arena" tab:
   • Enter a prompt or use the random prompt generator (🎲 button).
   • Click "Generate Responses" to see outputs from two random models.
   • Vote for the better response.
3. Check the "Leaderboard" tab to see overall model performance.
4. View the "Performance Chart" tab for a visual representation of model wins and losses.

🛠 Configuration

You can customize the arena by modifying the arena_config.py file:

• Add or remove models from the APPROVED_MODELS list.
• Adjust the API_URL and API_KEY if needed.
• Customize example_prompts for more variety in random prompts.

📊 Leaderboard

The leaderboard data is stored in leaderboard.json. This file is automatically updated after each battle.

Main Leaderboard Scoring System

We use a scoring system to rank the models fairly. The score for each model is calculated using the following formula:

Score = Win Rate * (1 - 1 / (Total Battles + 1))

Let's break down this formula:

1. Win Rate: This is the number of wins divided by the total number of battles. It ranges from 0 (no wins) to 1 (all wins).

2. 1 - 1 / (Total Battles + 1): This factor adjusts the win rate based on the number of battles:

   • We add 1 to the total battles to avoid division by zero and to ensure that even with just one battle, the score isn't discounted too heavily.
   • As the number of battles increases, this factor approaches 1.
   • For example:
     • With 1 battle: 1 - 1/2 = 0.5
     • With 10 battles: 1 - 1/11 ≈ 0.91
     • With 100 battles: 1 - 1/101 ≈ 0.99

3. Purpose of this adjustment:

   • It gives more weight to models that have participated in more battles.
   • A model with a high win rate but few battles will have a lower score than a model with the same win rate but more battles.
   • This encourages models to participate in more battles to improve their score.

4. How it works in practice:

   • For a new model with just one battle, its score will be at most 50% of its win rate.
   • As the model participates in more battles, its score will approach its actual win rate.
   • This prevents models with very few battles from dominating the leaderboard based on lucky wins.

In essence, this formula balances two factors:

1. How well a model performs (win rate)
2. How much experience it has (total battles)

It ensures that the leaderboard favors models that consistently perform well over a larger number of battles, rather than those that might have a high win rate from just a few lucky encounters.

We sort the results primarily by this calculated score, and secondarily by the total number of battles. This ensures that models with similar scores are ranked by their experience (number of battles).

The leaderboard displays this calculated score alongside wins, losses, and other statistics.

ELO Leaderboard

In addition to the main leaderboard, we also maintain an ELO-based leaderboard:

• Models start with an initial ELO rating based on their size.
• ELO ratings are updated after each battle, with adjustments made based on the size difference between models.
• The ELO leaderboard provides an alternative perspective on model performance, taking into account the relative strengths of opponents.

🤖 Models

The arena currently supports the following compact models:

• LLaMA 3.2 (1B, 3B, 8-bit)
• LLaMA 3.1 (8B, 4-bit)
• Gemma 2 (2B, 4-bit; 2B, 8-bit; 9B, 4-bit)
• Qwen 2.5 (0.5B, 8-bit; 1.5B, 8-bit; 3B, 4-bit; 7B, 4-bit)
• Mistral 0.3 (7B, 4-bit)
• Phi 3.5 (3.8B, 4-bit)
• Mistral Nemo (12B, 4-bit)
• GLM4 (9B, 4-bit)
• InternLM2 v2.5 (7B, 4-bit)
• Falcon2 (11B, 4-bit)
• StableLM2 (1.6B, 8-bit; 12B, 4-bit)
• Yi v1.5 (6B, 4-bit; 9B, 4-bit)
• Ministral (8B, 4-bit)
• Dolphin 2.9.4 (8B, 4-bit)
• Granite 3 Dense (2B, 8-bit; 8B, 4-bit)
• Granite 3 MoE (1B, 8-bit; 3B, 4-bit)

🤝 Contributing

Contributions are welcome! Please feel free to suggest a model that Ollama supports. Some results are already quite surprising.

📜 License

This project is open-source and available under the MIT License

🙏 Acknowledgements

• Thanks to the Ollama team for providing that amazing tool.
• Shoutout to all the AI researchers and compact language models teams for making this frugal AI arena possible!

Enjoy the battles in the GPU-Poor LLM Gladiator Arena! May the best compact model win! 🏆