Australasian Microeconomic Theory Seminar - Eduardo Azevedo (Wharton - UPenn)
A/B Testing with Fat Tails
(with Alex Deng, Jose Montiel Olea, Justin Rao and Glen Weyl)
Abstract: Large and statistically powerful A/B tests are increasingly popular to screen new business and policy ideas. We study how to use scarce experimental resources to screen multiple potential innovations by proposing a new framework for optimal experimentation, that we term the A/B testing problem. The main departure from the literature is that the model allows for fat tails. The key insight is that the optimal experimentation strategy depends on whether most gains accrue from typical innovations or from rare and unpredictable large successes that can be detected using tests with small samples. We show that, if the tails of the unobserved distribution of innova- tion quality are not too fat, the standard approach of using a few high-powered “big” experiments is optimal. However, when this distribution is very fat tailed, a “lean” experimentation strategy consisting of trying more ideas, each with possibly smaller sample sizes, is preferred. We measure the relevant tail parameter using experiments from Microsoft Bing’s EXP platform and find extremely fat tails. Our theoretical re- sults and empirical analysis suggest that even simple changes to business practices within Bing could increase innovation productivity.