Forecasting is the task of predicting the future. LLMs have recently been applied to forecasting, with some claims of performing as well as good human forecasters. In our new paper, we argue that evaluating LLM forecasters is easy to get wrong, and that confident estimates of forecasting performance require much more careful evaluation.