@inproceedings{
ma2024large,
title={Large Language Models Play StarCraft {II}:Benchmarks and A Chain of Summarization Approach},
author={Weiyu Ma and Qirui Mi and Yongcheng Zeng and Xue Yan and Runji Lin and Yuqiao Wu and Jun Wang and Haifeng Zhang},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=kEPpD7yETM}
}
With the continued advancement of Large Language Models (LLMs) Agents in reasoning, planning, and decision-making, benchmarks have become crucial in evaluating these skills. However, there is a notable gap in benchmarks for real-time strategic decision-making. StarCraft II (SC2), with its complex and dynamic nature, serves as an ideal setting for such evaluations. To this end, we have developed TextStarCraft II, a specialized environment for assessing LLMs in real-time strategic scenarios within SC2. Addressing the limitations of traditional Chain of Thought (CoT) methods, we introduce the Chain of Summarization (CoS) method, enhancing LLMs' capabilities in rapid and effective decision-making. Our key experiments included:
1. LLM Evaluation: Tested 10 LLMs in TextStarCraft II, most of them defeating LV5 build-in AI, showcasing effective strategy skills.
2. Commercial Model Knowledge: Evaluated four commercial  models on SC2 knowledge; GPT-4 ranked highest by Grandmaster-level experts.
3. Human-AI Matches: Experimental results showed that fine-tuned LLMs performed on par with Gold-level players in real-time matches, demonstrating comparable strategic abilities.

All code and data from this
study have been made pulicly available at https://github.com/histmeisah/Large-Language-Models-play-StarCraftII
Large Language Models Play StarCraft II:Benchmarks and A Chain of Summarization Approach
Weiyu Ma Qirui Mi Yongcheng Zeng Xue Yan Runji Lin Yuqiao Wu Jun Wang Haifeng Zhang