Useful Tools

Prompt Viewer

This tool allows you to directly view the generated prompt without starting the full training process. If the passed configuration is only the dataset configuration (such as configs/datasets/nq/nq_gen.py), it will display the original prompt defined in the dataset configuration. If it is a complete evaluation configuration (including the model and the dataset), it will display the prompt received by the selected model during operation.

Running method:

python tools/prompt_viewer.py CONFIG_PATH [-n] [-a] [-p PATTERN]

-n: Do not enter interactive mode, select the first model (if any) and dataset by default.
-a: View the prompts received by all models and all dataset combinations in the configuration.
-p PATTERN: Do not enter interactive mode, select all datasets that match the input regular expression.

Case Analyzer (To be updated)

Based on existing evaluation results, this tool produces inference error samples and full samples with annotation information.

Running method:

python tools/case_analyzer.py CONFIG_PATH [-w WORK_DIR]

-w: Work path, default is './outputs/default'.

Lark Bot

Users can configure the Lark bot to implement real-time monitoring of task status. Please refer to this document for setting up the Lark bot.

Configuration method:

Open the configs/secrets.py file, and add the following line to the file:

lark_bot_url = 'YOUR_WEBHOOK_URL'

Normally, the Webhook URL format is like https://open.feishu.cn/open-apis/bot/v2/hook/xxxxxxxxxxxxxxxxx .
Inherit this file in the complete evaluation configuration
To avoid the bot sending messages frequently and causing disturbance, the running status will not be reported automatically by default. If necessary, you can start status reporting through -l or --lark:

python run.py configs/eval_demo.py -l

API Model Tester

This tool can quickly test whether the functionality of the API model is normal.

Running method:

python tools/test_api_model.py [CONFIG_PATH] -n

Prediction Merger

This tool can merge patitioned predictions.

Running method:

python tools/prediction_merger.py CONFIG_PATH [-w WORK_DIR]

-w: Work path, default is './outputs/default'.

List Configs

This tool can list or search all available model and dataset configurations. It supports fuzzy search, making it convenient for use in conjunction with run.py.

Usage:

python tools/list_configs.py [PATTERN1] [PATTERN2] [...]

If executed without any parameters, it will list all model configurations in the configs/models and configs/dataset directories by default.

Users can also pass any number of parameters. The script will list all configurations related to the provided strings, supporting fuzzy search and the use of the * wildcard. For example, the following command will list all configurations related to mmlu and llama:

python tools/list_configs.py mmlu llama

Its output could be:

+-----------------+-----------------------------------+
| Model           | Config Path                       |
|-----------------+-----------------------------------|
| hf_llama2_13b   | configs/models/hf_llama2_13b.py   |
| hf_llama2_70b   | configs/models/hf_llama2_70b.py   |
| hf_llama2_7b    | configs/models/hf_llama2_7b.py    |
| hf_llama_13b    | configs/models/hf_llama_13b.py    |
| hf_llama_30b    | configs/models/hf_llama_30b.py    |
| hf_llama_65b    | configs/models/hf_llama_65b.py    |
| hf_llama_7b     | configs/models/hf_llama_7b.py     |
| llama2_13b_chat | configs/models/llama2_13b_chat.py |
| llama2_70b_chat | configs/models/llama2_70b_chat.py |
| llama2_7b_chat  | configs/models/llama2_7b_chat.py  |
+-----------------+-----------------------------------+
+-------------------+---------------------------------------------------+
| Dataset           | Config Path                                       |
|-------------------+---------------------------------------------------|
| cmmlu_gen         | configs/datasets/cmmlu/cmmlu_gen.py               |
| cmmlu_gen_ffe7c0  | configs/datasets/cmmlu/cmmlu_gen_ffe7c0.py        |
| cmmlu_ppl         | configs/datasets/cmmlu/cmmlu_ppl.py               |
| cmmlu_ppl_fd1f2f  | configs/datasets/cmmlu/cmmlu_ppl_fd1f2f.py        |
| mmlu_gen          | configs/datasets/mmlu/mmlu_gen.py                 |
| mmlu_gen_23a9a9   | configs/datasets/mmlu/mmlu_gen_23a9a9.py          |
| mmlu_gen_5d1409   | configs/datasets/mmlu/mmlu_gen_5d1409.py          |
| mmlu_gen_79e572   | configs/datasets/mmlu/mmlu_gen_79e572.py          |
| mmlu_gen_a484b3   | configs/datasets/mmlu/mmlu_gen_a484b3.py          |
| mmlu_ppl          | configs/datasets/mmlu/mmlu_ppl.py                 |
| mmlu_ppl_ac766d   | configs/datasets/mmlu/mmlu_ppl_ac766d.py          |
+-------------------+---------------------------------------------------+

Dataset Suffix Updater

This tool can quickly modify the suffixes of configuration files located under the configs/dataset directory, aligning them with the naming conventions based on prompt hash.

How to run:

python tools/update_dataset_suffix.py