As Large Language Models (LLMs)  are integrated into critical real-world applications, their strategic and logical reasoning abilities are increasingly crucial.
This paper evaluates LLMs’ reasoning abilities in competitive environments through game-theoretic tasks, e.g., board and card games that require pure logic and strategic reasoning to compete with opponents.
We first proposeGTBench, a language-driven environment composing 10 widely-recognized tasks, across a comprehensive game taxonomy: complete versus incomplete information, dynamic versus static, and probabilistic versus deterministic scenarios.
Then, we investigate two key problems: ➊ Characterizing game-theoretic reasoning of LLMs; ➋ LLM-vs-LLM competitions as reasoning evaluation.
We observe that ➊ LLMs have distinct behaviors regarding various gaming scenarios; for example, LLMs fail in complete and deterministic games yet they are competitive in probabilistic gaming scenarios; ➋ Open-source LLMs, e.g., CodeLlama-34b-Instruct, are less competitive than commercial LLMs, e.g., GPT-4, in complex games.
In addition, code-pretraining greatly benefits strategic reasoning, while advanced reasoning methods such as Chain-of-Thought (CoT)  and Tree-of-Thought (ToT)  do not always help.
Detailed error profiles are also provided for a better understanding of LLMs’ behavior.