Affordable Access

Access to the full text

Are Multi-Armed Bandits Susceptible to Peeking?

Authors
  • Loecher, Markus
Type
Published Article
Journal
Zagreb International Review of Economics and Business
Publisher
Sciendo
Publication Date
May 01, 2018
Volume
21
Issue
1
Pages
95–104
Identifiers
DOI: 10.2478/zireb-2018-0004
Source
De Gruyter
Keywords
License
Green

Abstract

A standard method to evaluate new features and changes to e.g. Web sites is A/B testing. A common pitfall in performing A/B testing is the habit of looking at a test while it’s running, then stopping early. Due to the implicit multiple testing, the p-value is no longer trustworthy and usually too small. We investigate the claim that Bayesian methods, unlike frequentist tests, are immune to this “peeking” problem. We demonstrate that two regularly used measures, namely posterior probability and value remaining are severely affected by repeated testing. We further show a strong dependence on the prior probability of the parameters of interest.

Report this publication

Statistics

Seen <100 times