The rapid development of Large Language Models (LLMs)  has led to great strides in model capabilities like reasoning and long-context understanding.
However, as LLMs are able to process longer contexts, it becomes more challenging to evaluate whether they have acquired certain capabilities, since the length of text (e.g., 100K tokens)  they can process far exceeds what humans can reliably assess in a reasonable duration.
In this paper, we propose using complex synthetic tasks as a proxy evaluation method, and presentS3Eval, aSynthetic,Scalable,Systematic evaluation suite for LLMs evaluation.
As a synthetic benchmark,S3Evalenables the creation of any number of evaluation examples that are theoretically invisible to LLMs, mitigating the test set contamination issue.
The synthetic nature ofS3Evalprovides users full control over the dataset, allowing them to systematically probe LLM capabilities by scaling text length and varying task difficulty across diverse scenarios.
The strong correlation betweenS3Evalperformance and scores of real-world benchmarks like Big-Bench Hard (BBH)  demonstrates the soundness of usingS3Evalfor evaluation of LLMs.
The in-depth analysis also uncover additional insights, including performance drop when the answer is sparsely distributed or located in the middle context, as well as some counter-intuitive trends of model performance. Our code is available athttps://github.com/lfy79001/SQLEval.