New data model

The new model is constructed by taking individual json files in data/new_eval, combining them together into a simple format, and from the combined df, we create individual files for each models.

For the new eval runs which has to be appended, we first analyze the model associated with the json file produced from eval harness, select the corresponding model file to append, find the unique rows (unique configuration of model name, language, task group and few shot) in the json file, append if unique rows are not 0.

title: Leaderboard emoji: 👁 colorFrom: blue colorTo: blue sdk: gradio sdk_version: 4.19.2 app_file: app.py pinned: false license: unknown

Introduction

This is the OpenGPT-X mutlilingual leaderboard source code repository. The leaderboard aims to provied an overview of LLM performance over various languages. The basic task set consists of MMLU, ARC, HellaSwag, GSM8k, TruthfulQA and belebele. To make the results comparable to the Open LLM leaderboard (https://huggingface.co/open-llm-leaderboard) we selected the former five tasks based on our internal machine translations of the English base tasks, in addition to the high-quality multilingual benchmark belebele by Meta.

Usage

The actually hosted leaderboard can be found under https://huggingface.co/spaces/openGPT-X/leaderboard. In order to extend its functionality please create a PR.

Adding new tasks

In order to add new evaluation tasks proceed as follows:

Add task information to TASK_INFO in src/data.py. It should be a dict mapping the task display name to the metric to be shown, as well as a dict containing mappings from two-letter language codes to the corresponding lm-eval-harness task selection string. See existing task information for reference.
Add evaluation results as detailed below.

Adding new models

It is possible to change the display name of a particular model. Simply add an entry to _MODEL_NAMES in src/data.py.

Adding evaluation results

Copy the .json-output generated by the lm-eval-harness into data.