1. We curate the largest, most recent dataset of real-world forecasting questions to date, for evaluating and optimizing automated forecasting systems. 2. We build a retrieval-augmented LM system that significantly improves upon the baseline and approaches the human crowd performance on competitive forecasting platforms. 3. We propose and apply a self-supervised fine-tuning method to improve LM's capability in reasoning about forecasting tasks.