Large language models (LLMs)  exhibit in-context learning abilities which enable the same model to perform several tasks without any task-specific training. In contrast, traditional adaptation approaches, such as fine-tuning, modify the underlying models foreachspecific task. In-context learning, however, consistently underperforms task-specific tuning approachesevenwhen presented with the same examples. While most existing approaches (e.g., prompt engineering)  focus on the LLM’s learned representations to patch this performance gap, our analysis actually reveal that LLM representations contain sufficient information to make good predictions. As such, we focus on the LLM’s reasoning abilities and demonstrate that this performance gap exists due to their inability to perform simple probabilistic reasoning tasks. This raises an intriguing question: Are LLMs actually capable of learning how to reason in a task-agnostic manner? We answer this in the affirmative and proposeTartwhich generically improves an LLM’s reasoning abilities using a synthetically trained Transformer-based reasoning module.Tarttrains this reasoning module in a task-agnostic manner usingonly syntheticlogistic regression tasks and composes it with an arbitrary real-world pre-trained model without any additional training. With a single inference module,Tartimproves performance across different model families (GPT-Neo,Pythia,Bloom) , model sizes (100M - 6B) , tasks (14 NLP binary classification tasks) , and even across different modalities (audio and vision) . Additionally, on the RAFT Benchmark,TartimprovesGPT-Neo (125M) ’s performance such that it outperformsBloom (176B) , and is withinofGPT-3 (175B) .111Our code and model is available athttps://github.com/HazyResearch/TART