TITLE: Beyond Static Datasets: A Deep Interaction Approach to LLM Evaluation 

ABSTRACT: Large Language Models (LLMs) have made progress in various real-world tasks,
which stimulates requirements for the evaluation of LLMs. Existing LLM
evaluation methods are mainly supervised signal-based which depends on static
datasets and cannot evaluate the ability of LLMs in dynamic real-world
scenarios where deep interaction widely exists. Other LLM evaluation methods
are human-based which are costly and time-consuming and are incapable of
large-scale evaluation of LLMs. To address the issues above, we propose a novel
Deep Interaction-based LLM-evaluation framework. In our proposed framework,
LLMs' performances in real-world domains can be evaluated from their deep
interaction with other LLMs in elaborately designed evaluation tasks.
Furthermore, our proposed framework is a general evaluation method that can be
applied to a host of real-world tasks such as machine translation and code
generation. We demonstrate the effectiveness of our proposed method through
extensive experiments on four elaborately designed evaluation tasks.
Large Language Models (LLMs) have made progress in various real-world tasks,
which stimulates requirements for the evaluation of LLMs. Existing LLM
evaluation methods are mainly supervised signal-based which depends on static
datasets and cannot evaluate the ability of LLMs in dynamic real-world
scenarios where deep interaction widely exists. Other LLM evaluation methods
are human-based which are costly and time-consuming and are incapable of
large-scale evaluation of LLMs. To address the issues above, we propose a novel
Deep Interaction-based LLM-evaluation framework. In our proposed framework,
LLMs' performances in real-world domains can be evaluated from their deep
interaction with other LLMs in elaborately designed evaluation tasks.
Furthermore, our proposed framework is a general evaluation method that can be
applied to a host of real-world tasks such as machine translation and code
generation. We demonstrate the effectiveness of our proposed method through
extensive experiments on four elaborately designed evaluation tasks.
Beyond Static Datasets: A Deep Interaction Approach to LLM Evaluation
http://arxiv.org/abs/2309.04369v1