**External Feedback.** In addition to LLMs, external objects can also provide feedback signals. For example, tools like code interpreters are widely used in programming tasks to provide real-time error messages [450], models like stable diffusion [532] can be used in multimodal tasks to provide visual perception [446], and virtual worlds like Minecraft can provide immersive experiences [528]. Besides, some work (_e.g.,_ Generative Agents [533]) explores multi-agent collaboration in simulated environments, where each agent receives feedback not only from interaction with the environment but also from communication with other agents.