Upload 69 files

.gitattributes

@@ -33,3 +33,9 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+data_dir/test_images/aircraft1.png filter=lfs diff=lfs merge=lfs -text
+data_dir/test_images/car1.png filter=lfs diff=lfs merge=lfs -text
+data_dir/test_images/food1.png filter=lfs diff=lfs merge=lfs -text
+data_dir/test_images/fungi1.png filter=lfs diff=lfs merge=lfs -text
+data_dir/test_images/shoe1.png filter=lfs diff=lfs merge=lfs -text
+data_dir/test_images/StacksofJossPaper.jpg filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,13 +1,15 @@
 ---
-title: Lumos Data Demo
-emoji: 🐠
-colorFrom: yellow
-colorTo: red
+title: AI2 WildBench Leaderboard
+emoji: 🦁 
+colorFrom: blue
+colorTo: yellow
 sdk: gradio
-sdk_version: 4.21.0
+sdk_version: 4.19.2
 app_file: app.py
-pinned: false
-license: apache-2.0
+pinned: true
+fullWidth: true
+hf_oauth: true
+api: false
 ---
 
 Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference

_about_us.md

@@ -0,0 +1,14 @@
+## About Us
+
+### Team 
+
+We are from [AllenAI](https://allenai.org/) (AI2) Mosaic Group. Team members include:
+
+[Da Yin](https://wadeyin9712.github.io/), [Faeze Brahman](https://fabrahman.github.io/), [Abhilasha Ravichander](https://www.cs.cmu.edu/~aravicha/), [Khyathi Chandu](https://khyathiraghavi.github.io/), [Kai-Wei Chang](https://web.cs.ucla.edu/~kwchang/), [Yejin Choi](https://homes.cs.washington.edu/~yejin/), and [Bill Yuchen Lin](https://yuchenlin.xyz/).
+ 
+### Contact
+
+Please contact us in the following ways:
+- Github Issues: [https://github.com/allenai/lumos](https://github.com/allenai/lumos)
+- Other questions: Please contact Da Yin with email: da.yin[at]cs[dot]ucla[dot]edu
+ 

_header.md

@@ -0,0 +1,5 @@
+<br/>
+
+# 🪄 Agent Lumos: Unified and Modular Training for Open-Source Language Agents
+[⚙️ GitHub](https://github.com/allenai/lumos) | [🤗 HuggingFace](https://huggingface.co/ai2lumos)
+

_intro.md

@@ -0,0 +1,131 @@
+
+<details open><summary style="font-size: 1.8em; font-weight: bold;">1. What is WildBench? Why should I use it?</summary>
+<div style="font-size: 1.4em; margin-top: 30px;">
+    🦁 <b>WildBench</b> is a benchmark for evaluating large language models (LLMs) on challenging tasks that are more representative of real-world applications. The examples are collected from real users by the <a href="https://wildchat.allen.ai/"><b>AI2 WildChat</b></a> project.</li>
+    <br>
+    <b>🆕 Motivation</b>: We aim to provide a more <strong>realistic</strong> and <strong>challenging</strong> benchmark for evaluating LLMs, as opposed to existing benchmarks that do not capture the <em>diversity</em> and <em>complexity</em> of <em>real-world</em> tasks.    
+    <h2 style="color: purple">🌠 Key Features:</h2>
+    <ul>
+        <!-- <li><b style="color: purple">🌟 Fine-grained:</b> 
+        We provide a fine-grained annotation for each example, including task types and <b>checklists</b> for evaluating the quality of responses. In addition, we use <b>length-penalized</b> Elo ratings to ensure that the quality of responses is not biased towards longer outputs.</li>
+        <li><b style="color: purple">🌟 Transparent & Fair: </b> We test all LLMs on the SAME set of examples, ensuring a fair  evaluation. You can explore the data and see the difference between two models to analyze the concrete gap between any pair of LLMs. </li>
+        <li><b style="color: purple">🌟 Easy & Fast:</b> WildBench (v1.0) contains 1024 examples, and it is extremely easy to add your own LLMs to our leaderboard! 1️⃣ Let us know your model ID and suggested inference configs; 2️⃣ We'll run inference and evaluation for you; 3️⃣ Voilà! We'll notify you when your results are ready on the leaderboard.</li>
+        <li><b style="color: purple">🌟 Dynamic:</b> WildBench will not be a static dataset. We will continue adding new examples and updating evaluation methods. Our goal is to include new challenging examples from real users over time and provide fast yet reliable evaluations.</li>
+        <li><b style="color: purple">🌟 Human Verification (ongoing):</b> Although we currently use GPT-4 as the automatic evaluator, we are also collecting human preferences here (see the 🔍 🆚 Tab). We plan to update the leaderboard by incorporating human evaluations in the near future.</li>
+        <li><b style="color: purple">🌟 Community-driven:</b> In addition to collecting human preferences for improving our evaluation, we also welcome community users to contribute new examples they find challenging to top LLMs like GPT-4/Claude3. Any feedback and suggestions are welcome, and we'll do our best to upgrade our data and evaluation methods accordingly.    </li> -->
+        <li><b style="color: purple">🌟 Challenging & Real:</b> We carefully curate a collection of 1024 hard tasks from real users, which cover common use cases such as code debugging, creative writing, and data analysis.</li>
+        <li><b style="color: purple">🌟 Reliable AutoEval w/ Checklists:</b> Instead of merely asking GPT-4 to choose between A and B, we provide an instance-specific Checklist (i.e., a list of evaluation questions) for it to reason before making a judgment. It’s similar to CoT. Thus, our eval is highly interpretable and easy-to-verify.</li>
+        <li><b style="color: purple">🌟 Length Penalty:</b> GPT-4 judges tend to prefer longer outputs (although humans do too); to avoid this, we devise a simple method to add length penalty on Elo. You can even slide it on our leaderboard UI!</li>
+        <li><b style="color: purple">🌟 Task Categorization:</b> We tag each example with 12 task types, so we can analyze task-specific performance of LLMs, in addition to their overall ranking.</li>
+        <li><b style="color: purple">🌟 Fair Comparisons:</b> WildBench tests all examples on all LLMs. This is different from arena-style evaluation, where one example is only tested on a single pair of models and never seen again.</li>
+        <li><b style="color: purple">🌟 Easy & Fast:</b> WildBench (v1.0) contains 1024 examples now, and it is extremely easy to add your own LLMs to our leaderboard! We will do the work for you!</li>
+        <li><b style="color: purple">🌟 Dynamic:</b> WildBench will not be a static dataset. We will continue adding new examples and updating evaluation methods based on community feedback.</li>
+        <li><b style="color: purple">🌟 Human Evaluation (ongoing):</b> We are collecting human preferences via our Leaderboard UI (check the 🔍 🆚 tab). Please help us vote! (We’re planning to recruit domain experts too.)</li>
+        <li><b style="color: purple">🌟 Community driven:</b> We welcome everyone to contribute to human evaluation and create challenging examples. We also value your feedback and suggestions, and will continue enhancing our benchmark leaderboard accordingly.</li>
+    </ul>
+</div>
+</details>
+
+
+--- 
+
+<details>
+    <summary style="font-size: 1.8em; font-weight: bold;">2. Where are the examples of WildBench from? </summary>
+    <div style="font-size: 1.4em; margin-top: 30px;">
+        <p>
+            <b>WildBench</b> was designed with a focus on capturing the real-world complexity and diversity of tasks that large language models (LLMs) encounter. The design process involved several key steps:
+        </p>
+        <h2>2.1. Task Collection from WildChat</h2>
+        <p>
+            <b>WildChat</b>, a dataset akin to ShareGPT but larger and with user consent, was utilized to gather human-GPT conversations. We filtered the data for English, non-toxic responses and used various popular LLMs to generate responses, which were then scored using reward models such as StarlingRM and PairRM. The examples with the highest score variance were shortlisted, from which 1024 were chosen for curating the <b>WildBench v1.0</b>, ensuring a mix of diversity and quality.
+        </p>
+        <h2>2.2. Task Categories</h2>
+        <img src="https://huggingface.co/spaces/WildEval/WildBench-Leaderboard/resolve/main/task_dist.png" width="80%" />
+        <p>
+            The tasks are classified into 12 categories to cover a broad spectrum of real-user scenarios. This categorization helps in maintaining a balanced task distribution, mirroring the task variety in WildChat and differing significantly from traditional benchmarks.
+        </p>
+        <h2>2.3. Additional Annotations</h2>
+        <p>
+            WildBench includes further annotations like secondary task types, conversation turn counts, user intents, moderation tags, and evaluation checklists, providing deeper insights into the tasks and enhancing response assessments. These annotations are generated by GPT-4.
+        </p>
+    </div>
+</details>
+
+<!-- ---
+
+<details>
+    <summary style="font-size: 1.8em; font-weight: bold;">3. How is WildBench different from other benchmarks?</summary>
+    <div style="font-size: 1.4em; margin-top: 30px;">
+        <h2>3.1. WildBench vs AlpacaEval</h2>
+        <p>
+            Unlike AlpacaEval's simpler, single-turn prompts, WildBench employs over 1024 multi-turn prompts from genuine user interactions, focusing on challenging and varied tasks. This represents a significant shift towards realism and complexity, aiming to reflect authentic LLM usage.
+        </p>
+        <h2>3.2. WildBench vs MT-Bench</h2>
+        <p>
+            MT-Bench offers two-turn instruction-following tasks, while WildBench provides a broader and more challenging array of multi-turn scenarios, ensuring a comprehensive evaluation across different dimensions.
+        </p>
+        <h2>3.3. WildBench vs Chatbot Arena</h2>
+        <p>
+            Though both benchmarks use real-user data, WildBench is distinct in its focus on challenging content, task diversity, and a structured, transparent evaluation methodology that offers more detailed insights into LLM performance.
+        </p>
+    </div>
+</details> 
+
+-->
+
+---
+
+<details>
+    <summary style="font-size: 1.8em; font-weight: bold;">3. How do you evaluate the performance of LLMs on WildBench?</summary>
+    <div style="font-size: 1.4em; margin-top: 30px;">
+        <h2>3.1. Elo Rating</h2>
+        <p>We show two Elo rating for each model in our Main table. The "Overall" Elo rating is the standard method of using bootstrap method to compute the Elo scores for each model. The "Task-Avg" Elo is computed by first computing standard Elo on subsets of our data for each task type and then perform the average of them. </p>
+        <h2>3.2. Length Penalty</h2>
+        <p>We know that GPT-based evaluation tends to prefer longer responses, which is also the case for human evaluation. To mitigate this, we use a length penalty to normalize the Elo rating of the responses. Specifically, we compute two versions of Elo ratings for each model: one is based on win rates, and the other is based on "longer rates". The <code>WinElo</code> is the standard Elo rating, and the LongElo is the Elo rating considering longer outputs are always better than shorter outputs.
+Then, we present the final adjusted Elo by taking the difference between <code>WinElo</code> and <code>LongElo</code>, i.e.,
+        <code>AdjustedElo = WinElo - LengthPenalty * LongElo</code>.
+        </p>
+        <h2>3.3. Checklist-based Evaluation</h2>
+        <p>In our automatic evaluation, we use a checklist (a list of 5~10 questions) for prompting GPT-4 to judge which model output is better. This checklist is example-specific. You can find real examples in "🔍 Explore | 🆚 Evaluate". The checklists help us ensure that GPT-4 uses a rather fixed standard to compare different model pairs on the same examples. Also, they facilitate us to better explain how GPT-4 make the decisions.  </p>
+        <h2>3.4. Estimated Win Rates</h2>
+        <p>We estimate the win rates of each model winning GPT-4 by the differences of their Elo Rating versus GPT-4's. The formula can be found in  <a href="https://www.hexwiki.net/index.php/Elo_rating#Definition"> this page</a>. </p>
+        <h2>3.5. Human-Verified Auto Evaluation</h2>
+        <p>Although the current version of our WildBench is purely based on automatic evaluators, we aim to collect human preferences from our demo here ("🔍 Explore | 🆚 Evaluate") and then incorporate these human evaluation for mitigating the bias of GPT-4 based evaluation. We also plan to recruit domain experts for further improving the fairness of our evaluation. Please stay tuned! </p>
+    </div>
+</details>
+
+---
+
+<details>
+    <summary style="font-size: 1.8em; font-weight: bold;">4. How can I test my model on WildBench?</summary>
+    <div style="font-size: 1.4em; margin-top: 30px;">
+        <p>Please refer to our Github <a href="https://github.com/allenai/WildBench">here</a> and create a PR or issue to tell us the information about your model. </p>
+    </div>
+</details>
+
+---
+
+<details>
+    <summary style="font-size: 1.8em; font-weight: bold;">5. How do I know why a particular model is weaker than others?</summary>
+    <div style="font-size: 1.4em; margin-top: 30px;">
+        <p>Please click the tab for "🔍 Explore | 🆚 Evaluate" and select the models and task types that you're interested in. We'll sample an example with two model outputs for you to compare and you can see the model ids after you submit your feedback. </p>
+    </div>
+</details>
+
+---
+
+<details>
+    <summary style="font-size: 1.8em; font-weight: bold;">6. Any future plans for WildBench?</summary>
+    <div style="font-size: 1.4em; margin-top: 30px;">
+        <p>We have many todo items! The most important one is to collect human preferences for improving our evaluation. We are also going to recruit domain experts for further improving the fairness of our evaluation. As for auto-evaluation, we will add multiple auto evaluators for mitigating the bias of GPT-4 based evaluation. For example, we aim to use Claude 3 as evaluator to check if the ranking would be different. We're also developing our open-source evaluation models for supporting faster local evaluation. </p>
+    </div>
+</details>
+
+--- 
+
+<details>
+    <summary style="font-size: 1.8em; font-weight: bold;">7. How do I contact you?</summary>
+    <div style="font-size: 1.4em; margin-top: 30px;">
+        <p>Please use the community discussion board <a href="https://huggingface.co/spaces/allenai/WildBench/discussions">here</a> or the Github issues. Also, please feel free to email us at [email protected] and mention "WildBench" in the title. </p>
+    </div>
+</details>

app.py

@@ -0,0 +1,190 @@
+"""A gradio app that renders a static leaderboard. This is used for Hugging Face Space."""
+import ast
+import argparse
+import glob
+import pickle
+import plotly
+import gradio as gr
+import numpy as np
+import pandas as pd
+import gradio as gr
+import pandas as pd
+from pathlib import Path
+import json
+from constants import BANNER, CITATION_TEXT, WINRATE_HEATMAP, css, js_code, all_task_types, DEFAULT_LP, TASK_TYPE_STR, js_light
+from datetime import datetime, timezone
+from data_utils import load_eval_results, sample_an_eval_result, apply_length_penalty, post_processing, add_winrates, add_winrates_tasks
+# from gradio.themes.utils import colors, fonts, sizes
+from themes import Seafoam
+from huggingface_hub import HfApi
+# from datasets import Dataset, load_dataset, concatenate_datasets
+import os, uuid 
+from utils_display import model_info
+
+# get the last updated time from the elo_ranks.all.jsonl file
+LAST_UPDATED = None 
+with open("_intro.md", "r") as f:
+    INTRO_MD = f.read()
+
+with open("_about_us.md", "r") as f:
+    ABOUT_MD = f.read()
+
+with open("_header.md", "r") as f:
+    HEADER_MD = f.read()
+
+original_df, ablation_df = None, None
+eval_results = load_eval_results() 
+ 
+available_models = [] # to be filled in later
+
+  
+def display_chat_history(model_selections):
+    eval_item = sample_an_eval_result(eval_results, model_selections)
+    task = eval_item["task"]
+    if eval_item["image"]:
+        image_path = eval_item["image"]
+    else:
+        image_path = ""
+    chats_plan = []
+    for item_user, item_asst in zip(eval_item["plan_history"]["user"], eval_item["plan_history"]["assistant"]):
+        chats_plan += [item_user, item_asst]
+    chats_ground = []
+    for item_user, item_asst in zip(eval_item["ground_history"]["user"], eval_item["ground_history"]["assistant"]):
+        chats_ground += [item_user, item_asst]
+    chats_plan = [(chats_plan[i], chats_plan[i+1]) for i in range(0, len(chats_plan), 2)]
+    chats_ground = [(chats_ground[i], chats_ground[i+1]) for i in range(0, len(chats_ground), 2)]
+    
+    if image_path != "":
+        image = f'<div style="text-align: center;"> <img src="{image_path}" style="height: 250px;"> </div>'
+        return task, chats_plan, chats_ground, image
+    else:
+        return task, chats_plan, chats_ground, f'<div style="text-align: center;"> </div>'
+
+
+
+
+def slider_change_main(length_penalty):
+    global original_df, ablation_df
+    adjusted_df = apply_length_penalty(original_df, ablation_df, length_penalty) 
+    adjusted_df = adjusted_df[["Model", "Overall Elo", "Task-Avg Elo", "# battles", "Length"]]
+    adjusted_df = adjusted_df.sort_values(by="Overall Elo", ascending=False)
+    adjusted_df = add_winrates(adjusted_df) 
+    adjusted_df = adjusted_df.drop(columns=["Length"])
+    return adjusted_df
+
+def slider_change_full(length_penalty, show_winrate):
+    global original_df, ablation_df
+    adjusted_df = apply_length_penalty(original_df, ablation_df, length_penalty)
+    # sort the model by the "Task-Avg Elo" column
+    adjusted_df = adjusted_df.sort_values(by="Task-Avg Elo", ascending=False)
+    adjusted_df.drop(columns=["Overall Elo", "Task-Avg Elo", "# battles", "Length"], inplace=True)
+    if show_winrate == "none":
+        return adjusted_df
+    elif show_winrate == "gpt-3.5":
+        adjusted_df = add_winrates_tasks(adjusted_df, ref="gpt-3.5")
+    elif show_winrate == "gpt-4":
+        adjusted_df = add_winrates_tasks(adjusted_df, ref="gpt-4")
+    return adjusted_df
+
+seafoam = Seafoam()
+def build_demo(TYPES):
+    global original_df, ablation_df, skip_empty_original_df, skip_empty_ablation_df, available_models
+    with gr.Blocks(theme=gr.themes.Soft(), css=css, js=js_light) as demo:
+        gr.Markdown(HEADER_MD, elem_classes="markdown-text")
+
+        with gr.Tabs(elem_classes="tab-buttons") as tabs:
+            with gr.TabItem("🔍 Explore", elem_id="od-benchmark-tab-table", id=2): 
+
+                with gr.Row(): 
+                    btn_show_history = gr.Button("🎲  Click here to sample an example of 🪄 Lumos outputs! ", elem_classes="sample_button")
+
+                with gr.Row():
+                    with gr.Column():
+                        with gr.Accordion("Choose models to sample from", open=False, elem_classes="accordion-label"):
+                            model_options = available_models
+                            selected_models = gr.CheckboxGroup(model_options, info="", value=model_options, show_label=False, elem_id="select-models") 
+                            clear_button = gr.Button("Clear", elem_classes="btn_boderline_gray", scale=1)
+                            # clear the selected_models
+                            clear_button.click(lambda: {selected_models: {"value": [], "__type__": "update"}}, inputs=[], outputs=[selected_models])
+                
+                with gr.Row():
+                    with gr.Column(scale=1.5):
+                        with gr.Accordion("📝 Task Description", open=True, elem_classes="accordion-label"):
+                            task = gr.Markdown("", elem_classes="markdown-text-tiny")
+                            task.change(lambda x: x, inputs=[], outputs=[], scroll_to_output=False, js=js_code)
+                    
+                    with gr.Column(scale=1):
+                        with gr.Accordion("Input Image (optional)", open=True, elem_classes="accordion-label"):
+                            image = gr.HTML("", elem_id="input_image")
+                            image.change(lambda x: x, inputs=[], outputs=[], scroll_to_output=False, js=js_code)
+
+                with gr.Row():
+                    with gr.Column():
+                        gr.Markdown("## 📢 Plan Module Process History", elem_classes="markdown-text")
+                        Chatbot_Common_Plan = gr.Chatbot(avatar_images=["human_icon.jpeg", "ai_icon.png"], height="auto", container=False, label="Common Plan History", likeable=False, show_share_button=False, show_label=True, elem_classes="chat-common", layout="bubble")
+                        Chatbot_Common_Plan.change(lambda x: x, inputs=[], outputs=[], scroll_to_output=False, js=js_code)
+                    with gr.Column():
+                        gr.Markdown("## 📢 Ground Module Process History", elem_classes="markdown-text")
+                        Chatbot_Common_Ground = gr.Chatbot(avatar_images=["human_icon.jpeg", "ai_icon.png"], height="auto", container=False, label="Common Ground History", likeable=False, show_share_button=False, show_label=True, elem_classes="chat-common", layout="bubble")
+                        Chatbot_Common_Ground.change(lambda x: x, inputs=[], outputs=[], scroll_to_output=False, js=js_code)
+                
+                # Display chat history when button is clicked
+                btn_show_history.click(fn=display_chat_history, inputs=[selected_models], outputs=[task, Chatbot_Common_Plan, Chatbot_Common_Ground, image])
+
+            with gr.TabItem("📮 About Us", elem_id="od-benchmark-tab-table", id=3):
+                gr.Markdown(ABOUT_MD, elem_classes="markdown-text")
+        gr.Markdown(f"Last updated on **{LAST_UPDATED}**", elem_classes="markdown-text-small")
+        
+        with gr.Row():
+            with gr.Accordion("📙 Citation", open=False, elem_classes="accordion-label"):
+                gr.Textbox(
+                    value=CITATION_TEXT, 
+                    lines=7,
+                    label="Copy the BibTeX snippet to cite this source",
+                    elem_id="citation-button",
+                    show_copy_button=True)
+                # ).style(show_copy_button=True)
+
+    return demo
+
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--share", action="store_true")
+    parser.add_argument("--result_file", help="Path to results table", default="data_dir/elo_ranks.all.jsonl")
+    parser.add_argument("--length_balation_file", help="Path to results table", default="data_dir/elo_ranks.length_ablation.all.jsonl")
+    parser.add_argument("--skip_empty_result_file", help="Path to results table", default="data_dir/elo_ranks.skip_empty.all.jsonl")
+    parser.add_argument("--skip_empty_length_balation_file", help="Path to results table", default="data_dir/elo_ranks.skip_empty.length_ablation.all.jsonl")
+    args = parser.parse_args()
+
+    LAST_UPDATED = datetime.fromtimestamp(Path(args.result_file).stat().st_mtime, tz=timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
+ 
+    original_df = pd.read_json(args.result_file , lines=True)
+    ablation_df = pd.read_json(args.length_balation_file, lines=True)
+    skip_empty_original_df = pd.read_json(args.skip_empty_result_file , lines=True)
+    skip_empty_ablation_df = pd.read_json(args.skip_empty_length_balation_file, lines=True)
+   
+    
+    # available_models = sorted(list(set(list(original_df["model name "])))) 
+    available_models = list(model_info.keys())
+    # remove the rows where the model name is not in the available_models
+    original_df = original_df[original_df["model name "].isin(available_models)]
+    ablation_df = ablation_df[ablation_df["model name "].isin(available_models)]
+    skip_empty_ablation_df = skip_empty_ablation_df[skip_empty_ablation_df["model name "].isin(available_models)]
+    skip_empty_original_df = skip_empty_original_df[skip_empty_original_df["model name "].isin(available_models)]
+
+    model_len_info = json.load(open("model_len_info.json", "r"))
+
+    original_df = post_processing(original_df, model_len_info)
+    ablation_df = post_processing(ablation_df, model_len_info)
+    skip_empty_original_df = post_processing(skip_empty_original_df, model_len_info)
+    skip_empty_ablation_df = post_processing(skip_empty_ablation_df, model_len_info)
+    
+
+    
+
+    TYPES = ["markdown", "number"]
+
+    demo = build_demo(TYPES)
+    demo.launch(share=args.share, allowed_paths=["."], height=1000)

compute_model_output_len.py

@@ -0,0 +1,24 @@
+from utils_display import model_info
+from data_utils import load_infer_results
+from tqdm import tqdm 
+import json 
+
+length_info = {}
+
+for model_name in tqdm(list(model_info.keys())): 
+    result = load_infer_results(model_name)
+    lens = []
+    for item in result:
+        o = item["output"]
+        if type(o) == list:
+            L = len(o[0].strip())
+        else:
+            L = len(o.strip())
+        if L > 0:
+            lens.append(L)
+    avg_len = sum(lens) / len(lens)
+    print(f"{model_name}: {avg_len}")
+    length_info[model_name] = avg_len
+
+with open("model_len_info.json", "w") as f:
+    json.dump(length_info, f, indent=2)

constants.py

@@ -0,0 +1,205 @@
+from pathlib import Path
+
+DEFAULT_LP = 0.3
+
+banner_url = "https://allenai.github.io/WildBench/gray_banner.png" # the same repo here.
+BANNER = f'<div style="display: flex; justify-content: flex-start;"><img src="{banner_url}" alt="Banner" style="width: 40vw; min-width: 300px; max-width: 800px;"> </div>'
+
+TITLE = "<html> <head> <style> h1 {text-align: center;} </style> </head> <body> <h1> 🦁 AI2 WildBench Leaderboard </b> </body> </html>"
+ 
+WINRATE_HEATMAP = "<div><img src='https://github.com/WildEval/WildBench-Leaderboard/blob/main/gradio/pairwise_win_fractions.png?raw=true' style='width:100%;'></div>"
+
+CITATION_TEXT = """@article{yin2023lumos,
+  title={Agent Lumos: Unified and Modular Training for Open-Source Language Agents},
+  author={Yin, Da and Brahman, Faeze and Ravichander, Abhilasha and Chandu, Khyathi and Chang, Kai-Wei and Choi, Yejin and Lin, Bill Yuchen},
+  journal={arXiv preprint arXiv:2311.05657},
+  year={2023}
+}
+"""
+
+
+column_names = {
+    "model name ": "Model",
+    "elo overall": "Overall Elo",
+    'Information seeking': 'InfoSek',
+    'Creative Writing': 'CrtWrt',
+    'Coding & Debugging': 'Code',
+    'Reasoning': 'Reason',
+    'Editing': 'Edit',
+    'Math': 'Math',
+    'Planning': 'Plan',
+    'Brainstorming': 'Brnstrm',
+    'Role playing': 'RolPly',
+    'Advice seeking': 'AdvSek',
+    'Data Analysis': 'DataAna',
+    'Others': 'Misc',
+    "average": "Task-Avg Elo",
+}
+
+all_task_types = [
+    'Information seeking',
+    'Creative Writing',
+    'Coding & Debugging',
+    'Reasoning',
+    'Editing',
+    'Math',
+    'Planning',
+    'Brainstorming',
+    'Role playing',
+    'Advice seeking',
+    'Data Analysis',
+    'Others'
+]
+
+
+
+js_light = """
+function refresh() {
+    const url = new URL(window.location);
+
+    if (url.searchParams.get('__theme') !== 'light') {
+        url.searchParams.set('__theme', 'light');
+        window.location.href = url.href;
+    }
+}
+"""
+
+js_code = """
+function scroll_top() {
+    console.log("Hello from Gradio!");  
+    const bubbles = document.querySelectorAll('.bubble-wrap');
+    bubbles.forEach((bubble, index) => {
+        setTimeout(() => {
+            bubble.scrollTop = 0;
+        }, index * 100); // Delay of 100ms between each iteration
+    });
+} 
+"""
+
+
+TASK_TYPE_STR = "**Tasks**: Info seeking (**InfoSek**), Creative Writing (**CrtWrt**), Coding&Debugging (**Code**), Reasoning (**Reason**), Editing (**Edit**), **Math**, Planning (**Plan**), Brainstorming (**Brnstrm**), Role playing (**RolPly**), Advice seeking (**AdvSek**), Data Analysis (**DataAna**)"
+
+css = """
+code {
+    font-size: large;
+}
+footer {visibility: hidden}
+.top-left-LP{
+    margin-top: 6px;
+    margin-left: 5px;
+}
+.markdown-text{font-size: 14pt}
+.markdown-text-small{font-size: 13pt}
+.markdown-text-tiny{font-size: 12pt}
+.markdown-text-tiny-red{
+    font-size: 12pt;
+    color: red;
+    background-color: yellow;
+    font-color: red;
+    font-weight: bold;
+}
+th {
+  text-align: center;
+  font-size: 17px; /* Adjust the font size as needed */
+}
+td {
+  font-size: 15px; /* Adjust the font size as needed */
+  text-align: center;
+}
+
+.sample_button{
+    border: 1px solid #000000;
+    border-radius: 5px;
+    padding: 5px;
+    font-size: 15pt;
+    font-weight: bold;
+    margin: 5px;
+}
+
+.chat-common{
+    height: auto;
+    max-height: 400px;
+    min-height: 100px; 
+}
+.chat-specific{
+    height: auto;
+    max-height: 600px;
+    min-height: 200px; 
+}
+#od-benchmark-tab-table-button{
+    font-size: 15pt;
+    font-weight: bold;
+} 
+
+.btn_boderline{
+    border: 1px solid #000000;
+    border-radius: 5px;
+    padding: 5px;
+    margin: 5px;
+    font-size: 15pt;
+    font-weight: bold; 
+}
+
+.btn_boderline_next{
+    border: 0.1px solid #000000;
+    border-radius: 5px;
+    padding: 5px;
+    margin: 5px;
+    font-size: 15pt;
+    font-weight: bold; 
+}
+
+.btn_boderline_gray{
+    border: 0.5px solid gray;
+    border-radius: 5px;
+    padding: 5px;
+    margin: 5px;
+    font-size: 15pt;
+    font-weight: italic; 
+}
+.btn_boderline_selected{
+    border: 2px solid purple;
+    background-color: #f2f2f2;
+    border-radius: 5px;
+    padding: 5px;
+    margin: 5px;
+    font-size: 15pt;
+    font-weight: bold;  
+}
+.accordion-label button span{
+    font-size: 14pt;
+    font-weight: bold;
+} 
+
+#select-models span{
+    font-size: 10pt;
+}
+
+#select-tasks span{
+    font-size: 10pt;
+}
+
+
+.markdown-text-details{
+    margin: 10px;
+    padding: 10px;
+}
+
+
+button.selected[role="tab"][aria-selected="true"] {
+    font-size: 18px; /* or any other size you prefer */
+    font-weight: bold;
+}
+
+#od-benchmark-tab-table-ablation-button {
+    font-size: larger; /* Adjust the font size as needed */
+}
+
+
+.plotly-plot{
+    height: auto;
+    max-height: 600px;
+    min-height: 600px; 
+}
+"""
+ 

data_dir/elo_ranks.all.jsonl

@@ -0,0 +1,22 @@
+{"model name ": "gpt-4-0125-preview", "elo overall": 1309, "Information seeking": 1282, "Creative Writing": 1267, "Coding & Debugging": 1376, "Reasoning": 1366, "Editing": 1211, "Math": 1276, "Planning": 1299, "Brainstorming": 1322, "Role playing": 1240, "Advice seeking": 1295, "Data Analysis": 1286, "Others": 1068, "average": 1274.0, "# battles": 5429}
+{"model name ": "claude-3-opus-20240229", "elo overall": 1155, "Information seeking": 1131, "Creative Writing": 1066, "Coding & Debugging": 1282, "Reasoning": 1204, "Editing": 1146, "Math": 1160, "Planning": 1159, "Brainstorming": 1111, "Role playing": 964, "Advice seeking": 1114, "Data Analysis": 1151, "Others": 1031, "average": 1126.5833333333333, "# battles": 2366}
+{"model name ": "claude-3-sonnet-20240229", "elo overall": 1140, "Information seeking": 1115, "Creative Writing": 1081, "Coding & Debugging": 1247, "Reasoning": 1170, "Editing": 1152, "Math": 1142, "Planning": 1163, "Brainstorming": 1092, "Role playing": 1005, "Advice seeking": 1081, "Data Analysis": 1170, "Others": 1014, "average": 1119.3333333333333, "# battles": 2608}
+{"model name ": "mistral-large-2402", "elo overall": 1120, "Information seeking": 1090, "Creative Writing": 1107, "Coding & Debugging": 1155, "Reasoning": 1136, "Editing": 1097, "Math": 1092, "Planning": 1081, "Brainstorming": 1078, "Role playing": 1067, "Advice seeking": 1060, "Data Analysis": 1097, "Others": 1021, "average": 1090.0833333333333, "# battles": 1881}
+{"model name ": "Mistral-7B-Instruct-v0.2", "elo overall": 1102, "Information seeking": 1084, "Creative Writing": 1138, "Coding & Debugging": 1140, "Reasoning": 1047, "Editing": 1105, "Math": 1021, "Planning": 1090, "Brainstorming": 1076, "Role playing": 1118, "Advice seeking": 1067, "Data Analysis": 1058, "Others": 1002, "average": 1078.8333333333333, "# battles": 2371}
+{"model name ": "gemini-1.0-pro", "elo overall": 1073, "Information seeking": 1089, "Creative Writing": 1040, "Coding & Debugging": 1083, "Reasoning": 1067, "Editing": 1021, "Math": 1063, "Planning": 1067, "Brainstorming": 1023, "Role playing": 1032, "Advice seeking": 1072, "Data Analysis": 1090, "Others": 1007, "average": 1054.5, "# battles": 1815}
+{"model name ": "Mixtral-8x7B-Instruct-v0.1", "elo overall": 1069, "Information seeking": 1079, "Creative Writing": 1050, "Coding & Debugging": 1125, "Reasoning": 1063, "Editing": 1018, "Math": 1061, "Planning": 1004, "Brainstorming": 1026, "Role playing": 1080, "Advice seeking": 1005, "Data Analysis": 1060, "Others": 993, "average": 1047.0, "# battles": 3417}
+{"model name ": "Yi-34B-Chat", "elo overall": 1058, "Information seeking": 1072, "Creative Writing": 1105, "Coding & Debugging": 939, "Reasoning": 1073, "Editing": 1042, "Math": 1011, "Planning": 1094, "Brainstorming": 1140, "Role playing": 1071, "Advice seeking": 1087, "Data Analysis": 943, "Others": 997, "average": 1047.8333333333333, "# battles": 2428}
+{"model name ": "Llama-2-70b-chat-hf.nosp", "elo overall": 1031, "Information seeking": 1092, "Creative Writing": 1065, "Coding & Debugging": 883, "Reasoning": 1022, "Editing": 987, "Math": 975, "Planning": 1003, "Brainstorming": 1049, "Role playing": 1070, "Advice seeking": 1036, "Data Analysis": 959, "Others": 1020, "average": 1013.4166666666666, "# battles": 1942}
+{"model name ": "Llama-2-13b-chat-hf.nosp", "elo overall": 1007, "Information seeking": 1056, "Creative Writing": 1058, "Coding & Debugging": 850, "Reasoning": 975, "Editing": 986, "Math": 947, "Planning": 984, "Brainstorming": 1041, "Role playing": 1053, "Advice seeking": 1008, "Data Analysis": 912, "Others": 1014, "average": 990.3333333333334, "# battles": 1888}
+{"model name ": "tulu-2-dpo-70b", "elo overall": 997, "Information seeking": 1003, "Creative Writing": 1024, "Coding & Debugging": 939, "Reasoning": 993, "Editing": 980, "Math": 975, "Planning": 965, "Brainstorming": 941, "Role playing": 1065, "Advice seeking": 976, "Data Analysis": 932, "Others": 991, "average": 982.0, "# battles": 3435}
+{"model name ": "zephyr-7b-beta", "elo overall": 986, "Information seeking": 1002, "Creative Writing": 991, "Coding & Debugging": 991, "Reasoning": 962, "Editing": 981, "Math": 941, "Planning": 967, "Brainstorming": 936, "Role playing": 1022, "Advice seeking": 938, "Data Analysis": 1003, "Others": 962, "average": 974.6666666666666, "# battles": 3367}
+{"model name ": "Llama-2-7b-chat-hf.nosp", "elo overall": 984, "Information seeking": 1033, "Creative Writing": 1026, "Coding & Debugging": 843, "Reasoning": 962, "Editing": 963, "Math": 908, "Planning": 987, "Brainstorming": 1003, "Role playing": 1048, "Advice seeking": 1020, "Data Analysis": 934, "Others": 1011, "average": 978.1666666666666, "# battles": 1898}
+{"model name ": "gpt-3.5-turbo-0125", "elo overall": 983, "Information seeking": 935, "Creative Writing": 934, "Coding & Debugging": 1155, "Reasoning": 1018, "Editing": 994, "Math": 1148, "Planning": 986, "Brainstorming": 929, "Role playing": 924, "Advice seeking": 961, "Data Analysis": 1071, "Others": 977, "average": 1002.6666666666666, "# battles": 14003}
+{"model name ": "gemma-7b-it", "elo overall": 951, "Information seeking": 931, "Creative Writing": 938, "Coding & Debugging": 993, "Reasoning": 957, "Editing": 928, "Math": 985, "Planning": 981, "Brainstorming": 984, "Role playing": 954, "Advice seeking": 973, "Data Analysis": 979, "Others": 992, "average": 966.25, "# battles": 2518}
+{"model name ": "command", "elo overall": 948, "Information seeking": 939, "Creative Writing": 942, "Coding & Debugging": 969, "Reasoning": 925, "Editing": 943, "Math": 940, "Planning": 979, "Brainstorming": 988, "Role playing": 948, "Advice seeking": 964, "Data Analysis": 966, "Others": 979, "average": 956.8333333333334, "# battles": 1733}
+{"model name ": "Llama-2-70b-chat-hf", "elo overall": 890, "Information seeking": 902, "Creative Writing": 902, "Coding & Debugging": 824, "Reasoning": 903, "Editing": 949, "Math": 896, "Planning": 907, "Brainstorming": 946, "Role playing": 905, "Advice seeking": 946, "Data Analysis": 927, "Others": 999, "average": 917.1666666666666, "# battles": 2391}
+{"model name ": "gemma-2b-it", "elo overall": 874, "Information seeking": 843, "Creative Writing": 898, "Coding & Debugging": 894, "Reasoning": 829, "Editing": 888, "Math": 896, "Planning": 906, "Brainstorming": 919, "Role playing": 906, "Advice seeking": 891, "Data Analysis": 914, "Others": 977, "average": 896.75, "# battles": 2484}
+{"model name ": "Llama-2-13b-chat-hf", "elo overall": 860, "Information seeking": 878, "Creative Writing": 866, "Coding & Debugging": 802, "Reasoning": 883, "Editing": 955, "Math": 915, "Planning": 897, "Brainstorming": 891, "Role playing": 865, "Advice seeking": 938, "Data Analysis": 885, "Others": 1007, "average": 898.5, "# battles": 2350}
+{"model name ": "vicuna-13b-v1.5", "elo overall": 847, "Information seeking": 849, "Creative Writing": 862, "Coding & Debugging": 817, "Reasoning": 847, "Editing": 870, "Math": 897, "Planning": 830, "Brainstorming": 873, "Role playing": 911, "Advice seeking": 852, "Data Analysis": 879, "Others": 973, "average": 871.6666666666666, "# battles": 2478}
+{"model name ": "Mistral-7B-Instruct-v0.1", "elo overall": 821, "Information seeking": 790, "Creative Writing": 800, "Coding & Debugging": 947, "Reasoning": 798, "Editing": 871, "Math": 892, "Planning": 806, "Brainstorming": 783, "Role playing": 892, "Advice seeking": 823, "Data Analysis": 922, "Others": 968, "average": 857.6666666666666, "# battles": 2479}
+{"model name ": "Llama-2-7b-chat-hf", "elo overall": 806, "Information seeking": 805, "Creative Writing": 834, "Coding & Debugging": 754, "Reasoning": 806, "Editing": 917, "Math": 856, "Planning": 836, "Brainstorming": 856, "Role playing": 853, "Advice seeking": 883, "Data Analysis": 861, "Others": 997, "average": 854.8333333333334, "# battles": 2297}

data_dir/elo_ranks.jsonl

@@ -0,0 +1,22 @@
+{"model name ": "gpt-4-0125-preview", "elo overall": 1309, "# battles": 5429}
+{"model name ": "claude-3-opus-20240229", "elo overall": 1155, "# battles": 2366}
+{"model name ": "claude-3-sonnet-20240229", "elo overall": 1140, "# battles": 2608}
+{"model name ": "mistral-large-2402", "elo overall": 1120, "# battles": 1881}
+{"model name ": "Mistral-7B-Instruct-v0.2", "elo overall": 1102, "# battles": 2371}
+{"model name ": "gemini-1.0-pro", "elo overall": 1073, "# battles": 1815}
+{"model name ": "Mixtral-8x7B-Instruct-v0.1", "elo overall": 1069, "# battles": 3417}
+{"model name ": "Yi-34B-Chat", "elo overall": 1058, "# battles": 2428}
+{"model name ": "Llama-2-70b-chat-hf.nosp", "elo overall": 1031, "# battles": 1942}
+{"model name ": "Llama-2-13b-chat-hf.nosp", "elo overall": 1007, "# battles": 1888}
+{"model name ": "tulu-2-dpo-70b", "elo overall": 997, "# battles": 3435}
+{"model name ": "zephyr-7b-beta", "elo overall": 986, "# battles": 3367}
+{"model name ": "Llama-2-7b-chat-hf.nosp", "elo overall": 984, "# battles": 1898}
+{"model name ": "gpt-3.5-turbo-0125", "elo overall": 983, "# battles": 14003}
+{"model name ": "gemma-7b-it", "elo overall": 951, "# battles": 2518}
+{"model name ": "command", "elo overall": 948, "# battles": 1733}
+{"model name ": "Llama-2-70b-chat-hf", "elo overall": 890, "# battles": 2391}
+{"model name ": "gemma-2b-it", "elo overall": 874, "# battles": 2484}
+{"model name ": "Llama-2-13b-chat-hf", "elo overall": 860, "# battles": 2350}
+{"model name ": "vicuna-13b-v1.5", "elo overall": 847, "# battles": 2478}
+{"model name ": "Mistral-7B-Instruct-v0.1", "elo overall": 821, "# battles": 2479}
+{"model name ": "Llama-2-7b-chat-hf", "elo overall": 806, "# battles": 2297}

data_dir/elo_ranks.length_ablation.all.jsonl

@@ -0,0 +1,22 @@
+{"model name ": "gpt-4-0125-preview", "elo overall": 1228, "Information seeking": 1274, "Creative Writing": 1202, "Coding & Debugging": 1224, "Reasoning": 1285, "Editing": 1132, "Math": 1202, "Planning": 1198, "Brainstorming": 1181, "Role playing": 1225, "Advice seeking": 1274, "Data Analysis": 1198, "Others": 1033, "average": 1202.3333333333333, "# battles": 5430}
+{"model name ": "Mistral-7B-Instruct-v0.2", "elo overall": 1112, "Information seeking": 1102, "Creative Writing": 1136, "Coding & Debugging": 1119, "Reasoning": 1071, "Editing": 1080, "Math": 1094, "Planning": 1057, "Brainstorming": 1038, "Role playing": 1126, "Advice seeking": 1050, "Data Analysis": 1083, "Others": 1010, "average": 1080.5, "# battles": 2371}
+{"model name ": "Llama-2-70b-chat-hf.nosp", "elo overall": 1112, "Information seeking": 1206, "Creative Writing": 1171, "Coding & Debugging": 938, "Reasoning": 1091, "Editing": 1057, "Math": 993, "Planning": 1105, "Brainstorming": 1136, "Role playing": 1128, "Advice seeking": 1110, "Data Analysis": 965, "Others": 1017, "average": 1076.4166666666667, "# battles": 1942}
+{"model name ": "Llama-2-7b-chat-hf.nosp", "elo overall": 1107, "Information seeking": 1184, "Creative Writing": 1146, "Coding & Debugging": 937, "Reasoning": 1095, "Editing": 1017, "Math": 993, "Planning": 1113, "Brainstorming": 1140, "Role playing": 1141, "Advice seeking": 1136, "Data Analysis": 996, "Others": 1008, "average": 1075.5, "# battles": 1898}
+{"model name ": "Llama-2-13b-chat-hf.nosp", "elo overall": 1096, "Information seeking": 1182, "Creative Writing": 1135, "Coding & Debugging": 924, "Reasoning": 1075, "Editing": 1038, "Math": 1004, "Planning": 1093, "Brainstorming": 1141, "Role playing": 1103, "Advice seeking": 1116, "Data Analysis": 969, "Others": 1032, "average": 1067.6666666666667, "# battles": 1888}
+{"model name ": "zephyr-7b-beta", "elo overall": 1079, "Information seeking": 1027, "Creative Writing": 1094, "Coding & Debugging": 1134, "Reasoning": 1047, "Editing": 1091, "Math": 1051, "Planning": 1064, "Brainstorming": 1075, "Role playing": 1038, "Advice seeking": 985, "Data Analysis": 1092, "Others": 1017, "average": 1059.5833333333333, "# battles": 3367}
+{"model name ": "Yi-34B-Chat", "elo overall": 1047, "Information seeking": 1082, "Creative Writing": 1068, "Coding & Debugging": 952, "Reasoning": 1086, "Editing": 987, "Math": 1064, "Planning": 1090, "Brainstorming": 1119, "Role playing": 1055, "Advice seeking": 1117, "Data Analysis": 970, "Others": 1004, "average": 1049.5, "# battles": 2428}
+{"model name ": "tulu-2-dpo-70b", "elo overall": 1037, "Information seeking": 1012, "Creative Writing": 1068, "Coding & Debugging": 1040, "Reasoning": 1023, "Editing": 1066, "Math": 1044, "Planning": 1027, "Brainstorming": 1007, "Role playing": 1060, "Advice seeking": 1016, "Data Analysis": 1030, "Others": 1026, "average": 1034.9166666666667, "# battles": 3435}
+{"model name ": "claude-3-sonnet-20240229", "elo overall": 1014, "Information seeking": 979, "Creative Writing": 944, "Coding & Debugging": 1162, "Reasoning": 1056, "Editing": 1056, "Math": 1076, "Planning": 1037, "Brainstorming": 953, "Role playing": 899, "Advice seeking": 985, "Data Analysis": 1080, "Others": 988, "average": 1017.9166666666666, "# battles": 2608}
+{"model name ": "Mixtral-8x7B-Instruct-v0.1", "elo overall": 1011, "Information seeking": 1002, "Creative Writing": 1001, "Coding & Debugging": 1055, "Reasoning": 1004, "Editing": 1001, "Math": 1008, "Planning": 975, "Brainstorming": 967, "Role playing": 1029, "Advice seeking": 965, "Data Analysis": 1028, "Others": 1021, "average": 1004.6666666666666, "# battles": 3418}
+{"model name ": "claude-3-opus-20240229", "elo overall": 1008, "Information seeking": 965, "Creative Writing": 941, "Coding & Debugging": 1191, "Reasoning": 1044, "Editing": 1082, "Math": 1045, "Planning": 1028, "Brainstorming": 945, "Role playing": 886, "Advice seeking": 1014, "Data Analysis": 1089, "Others": 999, "average": 1019.0833333333334, "# battles": 2367}
+{"model name ": "command", "elo overall": 1000, "Information seeking": 969, "Creative Writing": 982, "Coding & Debugging": 1071, "Reasoning": 996, "Editing": 1024, "Math": 1000, "Planning": 1059, "Brainstorming": 1033, "Role playing": 951, "Advice seeking": 978, "Data Analysis": 1018, "Others": 994, "average": 1006.25, "# battles": 1733}
+{"model name ": "mistral-large-2402", "elo overall": 986, "Information seeking": 944, "Creative Writing": 993, "Coding & Debugging": 1011, "Reasoning": 978, "Editing": 993, "Math": 996, "Planning": 930, "Brainstorming": 987, "Role playing": 985, "Advice seeking": 951, "Data Analysis": 1015, "Others": 1004, "average": 982.25, "# battles": 1881}
+{"model name ": "Llama-2-70b-chat-hf", "elo overall": 955, "Information seeking": 978, "Creative Writing": 968, "Coding & Debugging": 861, "Reasoning": 988, "Editing": 987, "Math": 950, "Planning": 994, "Brainstorming": 1017, "Role playing": 974, "Advice seeking": 1034, "Data Analysis": 923, "Others": 1025, "average": 974.9166666666666, "# battles": 2391}
+{"model name ": "gemini-1.0-pro", "elo overall": 950, "Information seeking": 940, "Creative Writing": 942, "Coding & Debugging": 998, "Reasoning": 964, "Editing": 958, "Math": 972, "Planning": 912, "Brainstorming": 946, "Role playing": 983, "Advice seeking": 923, "Data Analysis": 1004, "Others": 998, "average": 961.6666666666666, "# battles": 1816}
+{"model name ": "Llama-2-13b-chat-hf", "elo overall": 934, "Information seeking": 956, "Creative Writing": 959, "Coding & Debugging": 838, "Reasoning": 940, "Editing": 983, "Math": 991, "Planning": 967, "Brainstorming": 1001, "Role playing": 978, "Advice seeking": 1013, "Data Analysis": 926, "Others": 1001, "average": 962.75, "# battles": 2350}
+{"model name ": "Llama-2-7b-chat-hf", "elo overall": 924, "Information seeking": 943, "Creative Writing": 951, "Coding & Debugging": 817, "Reasoning": 943, "Editing": 961, "Math": 938, "Planning": 979, "Brainstorming": 982, "Role playing": 953, "Advice seeking": 985, "Data Analysis": 913, "Others": 1007, "average": 947.6666666666666, "# battles": 2297}
+{"model name ": "Mistral-7B-Instruct-v0.1", "elo overall": 917, "Information seeking": 867, "Creative Writing": 917, "Coding & Debugging": 1006, "Reasoning": 894, "Editing": 963, "Math": 969, "Planning": 883, "Brainstorming": 870, "Role playing": 958, "Advice seeking": 857, "Data Analysis": 1011, "Others": 988, "average": 931.9166666666666, "# battles": 2480}
+{"model name ": "gemma-7b-it", "elo overall": 915, "Information seeking": 914, "Creative Writing": 864, "Coding & Debugging": 1028, "Reasoning": 894, "Editing": 880, "Math": 939, "Planning": 887, "Brainstorming": 875, "Role playing": 954, "Advice seeking": 906, "Data Analysis": 988, "Others": 1002, "average": 927.5833333333334, "# battles": 2519}
+{"model name ": "gemma-2b-it", "elo overall": 884, "Information seeking": 885, "Creative Writing": 890, "Coding & Debugging": 925, "Reasoning": 837, "Editing": 895, "Math": 836, "Planning": 923, "Brainstorming": 902, "Role playing": 910, "Advice seeking": 897, "Data Analysis": 911, "Others": 987, "average": 899.8333333333334, "# battles": 2484}
+{"model name ": "vicuna-13b-v1.5", "elo overall": 846, "Information seeking": 837, "Creative Writing": 837, "Coding & Debugging": 830, "Reasoning": 851, "Editing": 874, "Math": 931, "Planning": 852, "Brainstorming": 879, "Role playing": 891, "Advice seeking": 883, "Data Analysis": 882, "Others": 993, "average": 878.3333333333334, "# battles": 2478}
+{"model name ": "gpt-3.5-turbo-0125", "elo overall": 822, "Information seeking": 754, "Creative Writing": 789, "Coding & Debugging": 955, "Reasoning": 844, "Editing": 880, "Math": 902, "Planning": 826, "Brainstorming": 815, "Role playing": 777, "Advice seeking": 803, "Data Analysis": 918, "Others": 844, "average": 842.25, "# battles": 14003}

data_dir/elo_ranks.length_ablation.jsonl

@@ -0,0 +1,22 @@
+{"model name ": "gpt-4-0125-preview", "elo overall": 1228, "# battles": 5430}
+{"model name ": "Mistral-7B-Instruct-v0.2", "elo overall": 1112, "# battles": 2371}
+{"model name ": "Llama-2-70b-chat-hf.nosp", "elo overall": 1112, "# battles": 1942}
+{"model name ": "Llama-2-7b-chat-hf.nosp", "elo overall": 1107, "# battles": 1898}
+{"model name ": "Llama-2-13b-chat-hf.nosp", "elo overall": 1096, "# battles": 1888}
+{"model name ": "zephyr-7b-beta", "elo overall": 1079, "# battles": 3367}
+{"model name ": "Yi-34B-Chat", "elo overall": 1047, "# battles": 2428}
+{"model name ": "tulu-2-dpo-70b", "elo overall": 1037, "# battles": 3435}
+{"model name ": "claude-3-sonnet-20240229", "elo overall": 1014, "# battles": 2608}
+{"model name ": "Mixtral-8x7B-Instruct-v0.1", "elo overall": 1011, "# battles": 3418}
+{"model name ": "claude-3-opus-20240229", "elo overall": 1008, "# battles": 2367}
+{"model name ": "command", "elo overall": 1000, "# battles": 1733}
+{"model name ": "mistral-large-2402", "elo overall": 986, "# battles": 1881}
+{"model name ": "Llama-2-70b-chat-hf", "elo overall": 955, "# battles": 2391}
+{"model name ": "gemini-1.0-pro", "elo overall": 950, "# battles": 1816}
+{"model name ": "Llama-2-13b-chat-hf", "elo overall": 934, "# battles": 2350}
+{"model name ": "Llama-2-7b-chat-hf", "elo overall": 924, "# battles": 2297}
+{"model name ": "Mistral-7B-Instruct-v0.1", "elo overall": 917, "# battles": 2480}
+{"model name ": "gemma-7b-it", "elo overall": 915, "# battles": 2519}
+{"model name ": "gemma-2b-it", "elo overall": 884, "# battles": 2484}
+{"model name ": "vicuna-13b-v1.5", "elo overall": 846, "# battles": 2478}
+{"model name ": "gpt-3.5-turbo-0125", "elo overall": 822, "# battles": 14003}

data_dir/elo_ranks.skip_empty.all.jsonl

@@ -0,0 +1,22 @@
+{"model name ": "gpt-4-0125-preview", "elo overall": 1296, "Information seeking": 1276, "Creative Writing": 1265, "Coding & Debugging": 1327, "Reasoning": 1359, "Editing": 1200, "Math": 1269, "Planning": 1286, "Brainstorming": 1320, "Role playing": 1237, "Advice seeking": 1298, "Data Analysis": 1250, "Others": 1068, "average": 1262.9166666666667, "# battles": 5362}
+{"model name ": "claude-3-opus-20240229", "elo overall": 1139, "Information seeking": 1124, "Creative Writing": 1065, "Coding & Debugging": 1236, "Reasoning": 1191, "Editing": 1132, "Math": 1148, "Planning": 1149, "Brainstorming": 1109, "Role playing": 965, "Advice seeking": 1110, "Data Analysis": 1117, "Others": 1030, "average": 1114.6666666666667, "# battles": 2258}
+{"model name ": "claude-3-sonnet-20240229", "elo overall": 1127, "Information seeking": 1112, "Creative Writing": 1081, "Coding & Debugging": 1197, "Reasoning": 1168, "Editing": 1140, "Math": 1134, "Planning": 1157, "Brainstorming": 1091, "Role playing": 1003, "Advice seeking": 1079, "Data Analysis": 1137, "Others": 1014, "average": 1109.4166666666667, "# battles": 2483}
+{"model name ": "mistral-large-2402", "elo overall": 1109, "Information seeking": 1087, "Creative Writing": 1111, "Coding & Debugging": 1105, "Reasoning": 1126, "Editing": 1088, "Math": 1085, "Planning": 1076, "Brainstorming": 1078, "Role playing": 1069, "Advice seeking": 1060, "Data Analysis": 1061, "Others": 1021, "average": 1080.5833333333333, "# battles": 1794}
+{"model name ": "Yi-34B-Chat", "elo overall": 1094, "Information seeking": 1078, "Creative Writing": 1100, "Coding & Debugging": 1059, "Reasoning": 1091, "Editing": 1067, "Math": 1032, "Planning": 1127, "Brainstorming": 1138, "Role playing": 1070, "Advice seeking": 1099, "Data Analysis": 1019, "Others": 997, "average": 1073.0833333333333, "# battles": 2134}
+{"model name ": "Mistral-7B-Instruct-v0.2", "elo overall": 1089, "Information seeking": 1076, "Creative Writing": 1136, "Coding & Debugging": 1089, "Reasoning": 1041, "Editing": 1094, "Math": 1011, "Planning": 1077, "Brainstorming": 1078, "Role playing": 1115, "Advice seeking": 1062, "Data Analysis": 1029, "Others": 1003, "average": 1067.5833333333333, "# battles": 2313}
+{"model name ": "gemini-1.0-pro", "elo overall": 1082, "Information seeking": 1113, "Creative Writing": 1061, "Coding & Debugging": 1056, "Reasoning": 1062, "Editing": 1016, "Math": 1062, "Planning": 1071, "Brainstorming": 1039, "Role playing": 1053, "Advice seeking": 1078, "Data Analysis": 1064, "Others": 1008, "average": 1056.9166666666667, "# battles": 1643}
+{"model name ": "Llama-2-70b-chat-hf.nosp", "elo overall": 1069, "Information seeking": 1109, "Creative Writing": 1068, "Coding & Debugging": 996, "Reasoning": 1056, "Editing": 1011, "Math": 994, "Planning": 1025, "Brainstorming": 1050, "Role playing": 1074, "Advice seeking": 1042, "Data Analysis": 1029, "Others": 1018, "average": 1039.3333333333333, "# battles": 1715}
+{"model name ": "Mixtral-8x7B-Instruct-v0.1", "elo overall": 1057, "Information seeking": 1076, "Creative Writing": 1046, "Coding & Debugging": 1073, "Reasoning": 1050, "Editing": 1003, "Math": 1048, "Planning": 996, "Brainstorming": 1025, "Role playing": 1080, "Advice seeking": 1003, "Data Analysis": 1023, "Others": 993, "average": 1034.6666666666667, "# battles": 3349}
+{"model name ": "Llama-2-13b-chat-hf.nosp", "elo overall": 1035, "Information seeking": 1061, "Creative Writing": 1061, "Coding & Debugging": 943, "Reasoning": 997, "Editing": 1008, "Math": 967, "Planning": 1006, "Brainstorming": 1037, "Role playing": 1048, "Advice seeking": 1014, "Data Analysis": 977, "Others": 1016, "average": 1011.25, "# battles": 1657}
+{"model name ": "Llama-2-7b-chat-hf.nosp", "elo overall": 1015, "Information seeking": 1043, "Creative Writing": 1025, "Coding & Debugging": 927, "Reasoning": 980, "Editing": 986, "Math": 918, "Planning": 1006, "Brainstorming": 1004, "Role playing": 1046, "Advice seeking": 1026, "Data Analysis": 983, "Others": 1011, "average": 996.25, "# battles": 1667}
+{"model name ": "tulu-2-dpo-70b", "elo overall": 983, "Information seeking": 996, "Creative Writing": 1022, "Coding & Debugging": 884, "Reasoning": 971, "Editing": 966, "Math": 965, "Planning": 949, "Brainstorming": 940, "Role playing": 1059, "Advice seeking": 973, "Data Analysis": 902, "Others": 992, "average": 968.25, "# battles": 3342}
+{"model name ": "gpt-3.5-turbo-0125", "elo overall": 968, "Information seeking": 930, "Creative Writing": 934, "Coding & Debugging": 1101, "Reasoning": 1009, "Editing": 976, "Math": 1142, "Planning": 970, "Brainstorming": 929, "Role playing": 923, "Advice seeking": 954, "Data Analysis": 1021, "Others": 978, "average": 988.9166666666666, "# battles": 13333}
+{"model name ": "zephyr-7b-beta", "elo overall": 968, "Information seeking": 993, "Creative Writing": 999, "Coding & Debugging": 927, "Reasoning": 949, "Editing": 966, "Math": 924, "Planning": 954, "Brainstorming": 939, "Role playing": 1021, "Advice seeking": 931, "Data Analysis": 965, "Others": 960, "average": 960.6666666666666, "# battles": 3310}
+{"model name ": "gemma-7b-it", "elo overall": 928, "Information seeking": 928, "Creative Writing": 935, "Coding & Debugging": 938, "Reasoning": 951, "Editing": 905, "Math": 982, "Planning": 966, "Brainstorming": 979, "Role playing": 953, "Advice seeking": 968, "Data Analysis": 950, "Others": 994, "average": 954.0833333333334, "# battles": 2429}
+{"model name ": "command", "elo overall": 926, "Information seeking": 937, "Creative Writing": 941, "Coding & Debugging": 906, "Reasoning": 914, "Editing": 930, "Math": 927, "Planning": 974, "Brainstorming": 984, "Role playing": 942, "Advice seeking": 961, "Data Analysis": 915, "Others": 980, "average": 942.5833333333334, "# battles": 1655}
+{"model name ": "Llama-2-70b-chat-hf", "elo overall": 912, "Information seeking": 904, "Creative Writing": 901, "Coding & Debugging": 923, "Reasoning": 922, "Editing": 974, "Math": 911, "Planning": 927, "Brainstorming": 943, "Role playing": 904, "Advice seeking": 954, "Data Analysis": 1005, "Others": 999, "average": 938.9166666666666, "# battles": 2090}
+{"model name ": "Llama-2-13b-chat-hf", "elo overall": 877, "Information seeking": 877, "Creative Writing": 865, "Coding & Debugging": 893, "Reasoning": 900, "Editing": 985, "Math": 929, "Planning": 917, "Brainstorming": 891, "Role playing": 865, "Advice seeking": 954, "Data Analysis": 944, "Others": 1006, "average": 918.8333333333334, "# battles": 2045}
+{"model name ": "vicuna-13b-v1.5", "elo overall": 856, "Information seeking": 849, "Creative Writing": 862, "Coding & Debugging": 902, "Reasoning": 861, "Editing": 881, "Math": 916, "Planning": 841, "Brainstorming": 867, "Role playing": 911, "Advice seeking": 857, "Data Analysis": 938, "Others": 969, "average": 887.8333333333334, "# battles": 2170}
+{"model name ": "gemma-2b-it", "elo overall": 849, "Information seeking": 836, "Creative Writing": 893, "Coding & Debugging": 822, "Reasoning": 807, "Editing": 875, "Math": 886, "Planning": 895, "Brainstorming": 919, "Role playing": 909, "Advice seeking": 877, "Data Analysis": 870, "Others": 979, "average": 880.6666666666666, "# battles": 2410}
+{"model name ": "Llama-2-7b-chat-hf", "elo overall": 821, "Information seeking": 807, "Creative Writing": 834, "Coding & Debugging": 820, "Reasoning": 819, "Editing": 938, "Math": 868, "Planning": 848, "Brainstorming": 854, "Role playing": 856, "Advice seeking": 890, "Data Analysis": 912, "Others": 996, "average": 870.1666666666666, "# battles": 2036}
+{"model name ": "Mistral-7B-Instruct-v0.1", "elo overall": 798, "Information seeking": 786, "Creative Writing": 797, "Coding & Debugging": 881, "Reasoning": 782, "Editing": 857, "Math": 880, "Planning": 789, "Brainstorming": 782, "Role playing": 891, "Advice seeking": 812, "Data Analysis": 883, "Others": 970, "average": 842.5, "# battles": 2415}

data_dir/elo_ranks.skip_empty.jsonl

@@ -0,0 +1,22 @@
+{"model name ": "gpt-4-0125-preview", "elo overall": 1296, "# battles": 5362}
+{"model name ": "claude-3-opus-20240229", "elo overall": 1139, "# battles": 2258}
+{"model name ": "claude-3-sonnet-20240229", "elo overall": 1127, "# battles": 2483}
+{"model name ": "mistral-large-2402", "elo overall": 1109, "# battles": 1794}
+{"model name ": "Yi-34B-Chat", "elo overall": 1094, "# battles": 2134}
+{"model name ": "Mistral-7B-Instruct-v0.2", "elo overall": 1089, "# battles": 2313}
+{"model name ": "gemini-1.0-pro", "elo overall": 1082, "# battles": 1643}
+{"model name ": "Llama-2-70b-chat-hf.nosp", "elo overall": 1069, "# battles": 1715}
+{"model name ": "Mixtral-8x7B-Instruct-v0.1", "elo overall": 1057, "# battles": 3349}
+{"model name ": "Llama-2-13b-chat-hf.nosp", "elo overall": 1035, "# battles": 1657}
+{"model name ": "Llama-2-7b-chat-hf.nosp", "elo overall": 1015, "# battles": 1667}
+{"model name ": "tulu-2-dpo-70b", "elo overall": 983, "# battles": 3342}
+{"model name ": "gpt-3.5-turbo-0125", "elo overall": 968, "# battles": 13333}
+{"model name ": "zephyr-7b-beta", "elo overall": 968, "# battles": 3310}
+{"model name ": "gemma-7b-it", "elo overall": 928, "# battles": 2429}
+{"model name ": "command", "elo overall": 926, "# battles": 1655}
+{"model name ": "Llama-2-70b-chat-hf", "elo overall": 912, "# battles": 2090}
+{"model name ": "Llama-2-13b-chat-hf", "elo overall": 877, "# battles": 2045}
+{"model name ": "vicuna-13b-v1.5", "elo overall": 856, "# battles": 2170}
+{"model name ": "gemma-2b-it", "elo overall": 849, "# battles": 2410}
+{"model name ": "Llama-2-7b-chat-hf", "elo overall": 821, "# battles": 2036}
+{"model name ": "Mistral-7B-Instruct-v0.1", "elo overall": 798, "# battles": 2415}

data_dir/elo_ranks.skip_empty.length_ablation.all.jsonl

@@ -0,0 +1,22 @@
+{"model name ": "gpt-4-0125-preview", "elo overall": 1219, "Information seeking": 1266, "Creative Writing": 1199, "Coding & Debugging": 1154, "Reasoning": 1275, "Editing": 1120, "Math": 1192, "Planning": 1188, "Brainstorming": 1179, "Role playing": 1218, "Advice seeking": 1277, "Data Analysis": 1161, "Others": 1033, "average": 1188.5, "# battles": 5363}
+{"model name ": "Llama-2-70b-chat-hf.nosp", "elo overall": 1166, "Information seeking": 1225, "Creative Writing": 1172, "Coding & Debugging": 1063, "Reasoning": 1130, "Editing": 1078, "Math": 1008, "Planning": 1135, "Brainstorming": 1139, "Role playing": 1130, "Advice seeking": 1115, "Data Analysis": 1043, "Others": 1016, "average": 1104.5, "# battles": 1715}
+{"model name ": "Llama-2-7b-chat-hf.nosp", "elo overall": 1151, "Information seeking": 1195, "Creative Writing": 1149, "Coding & Debugging": 1051, "Reasoning": 1120, "Editing": 1042, "Math": 1009, "Planning": 1141, "Brainstorming": 1142, "Role playing": 1139, "Advice seeking": 1149, "Data Analysis": 1059, "Others": 1008, "average": 1100.3333333333333, "# battles": 1667}
+{"model name ": "Llama-2-13b-chat-hf.nosp", "elo overall": 1148, "Information seeking": 1192, "Creative Writing": 1137, "Coding & Debugging": 1032, "Reasoning": 1102, "Editing": 1064, "Math": 1021, "Planning": 1121, "Brainstorming": 1140, "Role playing": 1102, "Advice seeking": 1126, "Data Analysis": 1040, "Others": 1032, "average": 1092.4166666666667, "# battles": 1657}
+{"model name ": "Yi-34B-Chat", "elo overall": 1106, "Information seeking": 1096, "Creative Writing": 1066, "Coding & Debugging": 1126, "Reasoning": 1118, "Editing": 1013, "Math": 1095, "Planning": 1137, "Brainstorming": 1118, "Role playing": 1052, "Advice seeking": 1133, "Data Analysis": 1072, "Others": 1003, "average": 1085.75, "# battles": 2134}
+{"model name ": "Mistral-7B-Instruct-v0.2", "elo overall": 1088, "Information seeking": 1092, "Creative Writing": 1132, "Coding & Debugging": 1053, "Reasoning": 1057, "Editing": 1068, "Math": 1086, "Planning": 1042, "Brainstorming": 1037, "Role playing": 1123, "Advice seeking": 1041, "Data Analysis": 1048, "Others": 1010, "average": 1065.75, "# battles": 2313}
+{"model name ": "zephyr-7b-beta", "elo overall": 1060, "Information seeking": 1022, "Creative Writing": 1093, "Coding & Debugging": 1065, "Reasoning": 1036, "Editing": 1074, "Math": 1040, "Planning": 1051, "Brainstorming": 1072, "Role playing": 1035, "Advice seeking": 976, "Data Analysis": 1047, "Others": 1014, "average": 1043.75, "# battles": 3310}
+{"model name ": "tulu-2-dpo-70b", "elo overall": 1020, "Information seeking": 1004, "Creative Writing": 1065, "Coding & Debugging": 970, "Reasoning": 998, "Editing": 1052, "Math": 1030, "Planning": 1005, "Brainstorming": 1008, "Role playing": 1061, "Advice seeking": 1007, "Data Analysis": 998, "Others": 1026, "average": 1018.6666666666666, "# battles": 3342}
+{"model name ": "claude-3-sonnet-20240229", "elo overall": 994, "Information seeking": 975, "Creative Writing": 944, "Coding & Debugging": 1099, "Reasoning": 1039, "Editing": 1045, "Math": 1069, "Planning": 1017, "Brainstorming": 953, "Role playing": 896, "Advice seeking": 976, "Data Analysis": 1041, "Others": 988, "average": 1003.5, "# battles": 2483}
+{"model name ": "Mixtral-8x7B-Instruct-v0.1", "elo overall": 991, "Information seeking": 993, "Creative Writing": 1003, "Coding & Debugging": 980, "Reasoning": 987, "Editing": 991, "Math": 994, "Planning": 960, "Brainstorming": 965, "Role playing": 1024, "Advice seeking": 959, "Data Analysis": 983, "Others": 1020, "average": 988.25, "# battles": 3350}
+{"model name ": "claude-3-opus-20240229", "elo overall": 986, "Information seeking": 951, "Creative Writing": 940, "Coding & Debugging": 1137, "Reasoning": 1034, "Editing": 1067, "Math": 1028, "Planning": 1010, "Brainstorming": 945, "Role playing": 885, "Advice seeking": 1009, "Data Analysis": 1055, "Others": 1000, "average": 1005.0833333333334, "# battles": 2259}
+{"model name ": "Llama-2-70b-chat-hf", "elo overall": 985, "Information seeking": 988, "Creative Writing": 969, "Coding & Debugging": 998, "Reasoning": 1017, "Editing": 1021, "Math": 960, "Planning": 1026, "Brainstorming": 1018, "Role playing": 972, "Advice seeking": 1049, "Data Analysis": 1014, "Others": 1025, "average": 1004.75, "# battles": 2090}
+{"model name ": "command", "elo overall": 982, "Information seeking": 965, "Creative Writing": 983, "Coding & Debugging": 1004, "Reasoning": 980, "Editing": 1005, "Math": 992, "Planning": 1052, "Brainstorming": 1030, "Role playing": 954, "Advice seeking": 974, "Data Analysis": 967, "Others": 995, "average": 991.75, "# battles": 1655}
+{"model name ": "Llama-2-13b-chat-hf", "elo overall": 966, "Information seeking": 968, "Creative Writing": 956, "Coding & Debugging": 956, "Reasoning": 965, "Editing": 1019, "Math": 1011, "Planning": 999, "Brainstorming": 999, "Role playing": 976, "Advice seeking": 1027, "Data Analysis": 1012, "Others": 1000, "average": 990.6666666666666, "# battles": 2045}
+{"model name ": "mistral-large-2402", "elo overall": 958, "Information seeking": 939, "Creative Writing": 995, "Coding & Debugging": 941, "Reasoning": 969, "Editing": 985, "Math": 985, "Planning": 919, "Brainstorming": 986, "Role playing": 988, "Advice seeking": 952, "Data Analysis": 968, "Others": 1005, "average": 969.3333333333334, "# battles": 1794}
+{"model name ": "gemini-1.0-pro", "elo overall": 951, "Information seeking": 957, "Creative Writing": 960, "Coding & Debugging": 942, "Reasoning": 955, "Editing": 947, "Math": 969, "Planning": 904, "Brainstorming": 953, "Role playing": 1000, "Advice seeking": 926, "Data Analysis": 964, "Others": 998, "average": 956.25, "# battles": 1644}
+{"model name ": "Llama-2-7b-chat-hf", "elo overall": 948, "Information seeking": 952, "Creative Writing": 957, "Coding & Debugging": 918, "Reasoning": 962, "Editing": 990, "Math": 962, "Planning": 995, "Brainstorming": 982, "Role playing": 958, "Advice seeking": 998, "Data Analysis": 993, "Others": 1006, "average": 972.75, "# battles": 2036}
+{"model name ": "gemma-7b-it", "elo overall": 889, "Information seeking": 907, "Creative Writing": 858, "Coding & Debugging": 953, "Reasoning": 881, "Editing": 854, "Math": 929, "Planning": 870, "Brainstorming": 872, "Role playing": 949, "Advice seeking": 899, "Data Analysis": 943, "Others": 1003, "average": 909.8333333333334, "# battles": 2430}
+{"model name ": "Mistral-7B-Instruct-v0.1", "elo overall": 887, "Information seeking": 862, "Creative Writing": 913, "Coding & Debugging": 930, "Reasoning": 875, "Editing": 948, "Math": 964, "Planning": 862, "Brainstorming": 870, "Role playing": 956, "Advice seeking": 845, "Data Analysis": 964, "Others": 987, "average": 914.6666666666666, "# battles": 2416}
+{"model name ": "gemma-2b-it", "elo overall": 854, "Information seeking": 876, "Creative Writing": 883, "Coding & Debugging": 838, "Reasoning": 813, "Editing": 878, "Math": 820, "Planning": 904, "Brainstorming": 902, "Role playing": 909, "Advice seeking": 881, "Data Analysis": 861, "Others": 985, "average": 879.1666666666666, "# battles": 2410}
+{"model name ": "vicuna-13b-v1.5", "elo overall": 854, "Information seeking": 830, "Creative Writing": 837, "Coding & Debugging": 929, "Reasoning": 864, "Editing": 889, "Math": 953, "Planning": 863, "Brainstorming": 874, "Role playing": 894, "Advice seeking": 883, "Data Analysis": 954, "Others": 993, "average": 896.9166666666666, "# battles": 2170}
+{"model name ": "gpt-3.5-turbo-0125", "elo overall": 789, "Information seeking": 744, "Creative Writing": 788, "Coding & Debugging": 860, "Reasoning": 822, "Editing": 857, "Math": 874, "Planning": 797, "Brainstorming": 815, "Role playing": 775, "Advice seeking": 788, "Data Analysis": 832, "Others": 845, "average": 816.4166666666666, "# battles": 13333}

data_dir/elo_ranks.skip_empty.length_ablation.jsonl

@@ -0,0 +1,22 @@
+{"model name ": "gpt-4-0125-preview", "elo overall": 1219, "# battles": 5363}
+{"model name ": "Llama-2-70b-chat-hf.nosp", "elo overall": 1166, "# battles": 1715}
+{"model name ": "Llama-2-7b-chat-hf.nosp", "elo overall": 1151, "# battles": 1667}
+{"model name ": "Llama-2-13b-chat-hf.nosp", "elo overall": 1148, "# battles": 1657}
+{"model name ": "Yi-34B-Chat", "elo overall": 1106, "# battles": 2134}
+{"model name ": "Mistral-7B-Instruct-v0.2", "elo overall": 1088, "# battles": 2313}
+{"model name ": "zephyr-7b-beta", "elo overall": 1060, "# battles": 3310}
+{"model name ": "tulu-2-dpo-70b", "elo overall": 1020, "# battles": 3342}
+{"model name ": "claude-3-sonnet-20240229", "elo overall": 994, "# battles": 2483}
+{"model name ": "Mixtral-8x7B-Instruct-v0.1", "elo overall": 991, "# battles": 3350}
+{"model name ": "claude-3-opus-20240229", "elo overall": 986, "# battles": 2259}
+{"model name ": "Llama-2-70b-chat-hf", "elo overall": 985, "# battles": 2090}
+{"model name ": "command", "elo overall": 982, "# battles": 1655}
+{"model name ": "Llama-2-13b-chat-hf", "elo overall": 966, "# battles": 2045}
+{"model name ": "mistral-large-2402", "elo overall": 958, "# battles": 1794}
+{"model name ": "gemini-1.0-pro", "elo overall": 951, "# battles": 1644}
+{"model name ": "Llama-2-7b-chat-hf", "elo overall": 948, "# battles": 2036}
+{"model name ": "gemma-7b-it", "elo overall": 889, "# battles": 2430}
+{"model name ": "Mistral-7B-Instruct-v0.1", "elo overall": 887, "# battles": 2416}
+{"model name ": "gemma-2b-it", "elo overall": 854, "# battles": 2410}
+{"model name ": "vicuna-13b-v1.5", "elo overall": 854, "# battles": 2170}
+{"model name ": "gpt-3.5-turbo-0125", "elo overall": 789, "# battles": 13333}

data_dir/leaderboard_data.jsonl

@@ -0,0 +1,13 @@
+{"model name": "gpt-4-0125-preview", "win rate": 0.833984375, "win+tie rate": 0.9423828125, "num votes": 1024, "longer rate": 0.9296875, "charlen": 3190.716796875, "elo": 0}
+{"model name": "Mixtral-8x7B-Instruct-v0.1", "win rate": 0.568359375, "win+tie rate": 0.6767578125, "num votes": 1024, "longer rate": 0.7724609375, "charlen": 2483.9638671875, "elo": 0}
+{"model name": "Yi-34B-Chat", "win rate": 0.5322265625, "win+tie rate": 0.658203125, "num votes": 1024, "longer rate": 0.7412109375, "charlen": 2567.92578125, "elo": 0}
+{"model name": "gpt-3.5-turbo-0125", "win rate": 0.5, "win+tie rate": 0.5, "num votes": 12288, "longer rate": 0.3017578125, "charlen": 1725.7216796875, "elo": 0}
+{"model name": "tulu-2-dpo-70b", "win rate": 0.44921875, "win+tie rate": 0.5771484375, "num votes": 1024, "longer rate": 0.7919921875, "charlen": 2614.822265625, "elo": 0}
+{"model name": "zephyr-7b-beta", "win rate": 0.4404296875, "win+tie rate": 0.5478515625, "num votes": 1024, "longer rate": 0.822265625, "charlen": 3011.2529296875, "elo": 0}
+{"model name": "gemma-7b-it", "win rate": 0.4150390625, "win+tie rate": 0.49609375, "num votes": 1024, "longer rate": 0.6064453125, "charlen": 1951.2548828125, "elo": 0}
+{"model name": "Llama-2-70b-chat-hf", "win rate": 0.3125, "win+tie rate": 0.4462890625, "num votes": 1024, "longer rate": 0.6845703125, "charlen": 2007.029296875, "elo": 0}
+{"model name": "gemma-2b-it", "win rate": 0.296875, "win+tie rate": 0.375, "num votes": 1024, "longer rate": 0.560546875, "charlen": 1862.732421875, "elo": 0}
+{"model name": "Llama-2-13b-chat-hf", "win rate": 0.2666015625, "win+tie rate": 0.4130859375, "num votes": 1024, "longer rate": 0.6826171875, "charlen": 1945.42578125, "elo": 0}
+{"model name": "vicuna-13b-v1.5", "win rate": 0.2548828125, "win+tie rate": 0.3759765625, "num votes": 1024, "longer rate": 0.509765625, "charlen": 1642.1640625, "elo": 0}
+{"model name": "Mistral-7B-Instruct-v0.1", "win rate": 0.23046875, "win+tie rate": 0.3232421875, "num votes": 1024, "longer rate": 0.6171875, "charlen": 2208.8115234375, "elo": 0}
+{"model name": "Llama-2-7b-chat-hf", "win rate": 0.2216796875, "win+tie rate": 0.345703125, "num votes": 1024, "longer rate": 0.65234375, "charlen": 1878.521484375, "elo": 0}

data_dir/pairwise_win_fractions.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:53b8c101b7473e39e483771317b1bd34c9feda733d4b5cdc67b3508b2ca968b1
+size 11827