Amazingly, 2k forecasts from FutureSearch users have already resolved. First ever such dataset, if I'm not mistaken. (The questions people find useful are very different from the questions you see on prediction markets / in tournaments.)
This is all preliminary, and accuracy is not a great indication of usefulness. Also, we can only resolve what's observable, which doesn't work on decision forecasts.
Still, it's exciting to do forecast research on questions that people actually care about and use! It's the demand side of forecasting. I'm tired of studying the supply side, e.g people trading or forecasting in tournaments, where the forecasts themselves are not actually used or valued by anyone.
One thing we looked at right away is: does the FutureSearch world model, the latent view of the entire future, improve accuracy on questions users care about? The sample size is low, but at p=~0.1, it improves accuracy by about the difference between the #1 and #2 forecasters in evals.
Here's the accuracy by category. Note that this is mostly a function of how 50/50 the questions are, not how superhuman the forecaster is. In sports, it's hard to do better than chance at all. Still, it's interesting to see FutureSearch accuracy so much higher on business, (geo)politics, and AI compared to macro, finance, sports, and culture.
