New working paper: Can AI forecast treatment effects in economics from pre-registration documents?
Answer: Yes. 1/
Conversation
Post your reply
You've seen the stunning improvements in other fields like math. I ask LLMs to forecast effects of studies registered on the AEA RCT Registry and the Open Science Framework using public pre-registration documents.
I restrict attention to papers that first came out in 2026. I
The models are pretty good: for example, the median absolute forecast error of Astra with a simple prompt is 0.05 SD, with a MSE of 0.02.
For comparison, the leave-one-paper-out mean of the other papers in the same sample yield a median absolute forecast error of 0.09 SD, with a
The models also seem to know when they do not know: when they give a larger 10th - 90th percentile interval, indicating more uncertainty, the forecast error is indeed larger. 4/
One implication is that forecasts can be quite useful for making decisions when you do not (yet) have RCT "gold standard" evidence. 5/
Personally, I also wonder a little at what we are doing in economics if treatment effects can be forecast so well.
Part of it is surely that many treatments have very small effects.
But can we not shift to working on those projects that have particularly high uncertainty and
Yes, an error of 0.05 SD (the median absolute error of AI forecasts) could be meaningful for some treatments and outcomes, but most of the time do we really care about such small differences?
Maybe sometimes - or maybe experiments yield more generalizable knowledge which
Maybe it's worth adding: this one is backwards-looking, but I'm working on a forwards-looking version with others.
And, in case anyone is interested, there is going to be an AI forecasting tournament in the near future with prizes available! Stay tuned.
This is so interesting, and I find it really confusing that the LLMs seem to mildly underestimate real effects. Given the stuff we tested, that’s really weird! Though I suppose the pre-reg is only available conditional on the study being published?
So, a couple of things could be going on here.
First, LLMs used to systematically overestimate treatment effects the same way that humans overestimate them (we saw it for humans here: nber.org/papers/w34493. Using some older models, AI was similar). One might think either
Very cool, it would also be interesting to understand why the forecasts are so good. Is it mostly about being better at base rates. What do we make of cases when the forecast was very off?
For the cases that were way off - they are few and far between and generally seem to be when a treatment had an unexpectedly large effect.
E.g., for Astra, six questions with results purported to be above 0.5 SD account for roughly half the MSE, including two with results above
It's forecasting the reported estimated treatment effects, right? Not the "true" treatment effects?
Said differently, is it internalizing researcher degrees of freedom, publication bias and typical effect sizes from different research designs?
It seems like this might be more a commentary on the norms of our profession more than genuine out of sample prediction.
beats the 0.112 sd a leave-one-out meta-analysis got in your 2020 paper too, and that one got to pool real results
In addition to the points made, folks are almost certainly piloting, seeing promising results, and then writing Pre-registered plans. AI might not do so well when trying to predict effects among full universe of proposals, including those that don’t make it past pilot stage.
