How well to LLMs forecast company KPIs?

Benchmarking forecasting models on the hedge fund use case

There’s a new trend of AI forecasting arenas like Metaculus AIB and FutureX, where LLM engineers compete to forecast a broad range of future events.

We analyzed how the the top forecasting models on a discretionary hedge fund workflow.

History of forecasting

The forecasting community was first popularized two decades ago by Prof Philip Tetlock’s popular book Superforecasting. Competitions, like the Good Judgement Open and Metaculus, are months long contests where forecasting professionals predict the outcomes of various events that will resolve in the coming months such as elections, sports championships, weather and wars.

Recently, the community has started to accelerate as forecasters use LLMs to automate some of the manual research and calculation steps. Estimates that used to take hours or days can now be done in seconds. Now forecasting expertise is being channeled into specialty LLM system design so the marginal forecast can be automated.

This explains why we’re seeing these new LLM forecasting arenas thousands of individuals and teams competing in them. Historically, hedge funds haven’t had much overlap with the forecasting community - too many questions requiring too much specialized knowledge, with answers needed in too short of a time horizon.

We think this is about to change as LLM forecasting systems can provide answers in real time.