Affordable Access

Equations of States in Singular Statistical Estimation

Authors
  • Watanabe, Sumio
Type
Preprint
Publication Date
May 11, 2009
Submission Date
Dec 05, 2007
Identifiers
arXiv ID: 0712.0653
Source
arXiv
License
Yellow
External links

Abstract

Learning machines which have hierarchical structures or hidden variables are singular statistical models because they are nonidentifiable and their Fisher information matrices are singular. In singular statistical models, neither the Bayes a posteriori distribution converges to the normal distribution nor the maximum likelihood estimator satisfies asymptotic normality. This is the main reason why it has been difficult to predict their generalization performances from trained states. In this paper, we study four errors, (1) Bayes generalization error, (2) Bayes training error, (3) Gibbs generalization error, and (4) Gibbs training error, and prove that there are mathematical relations among these errors. The formulas proved in this paper are equations of states in statistical estimation because they hold for any true distribution, any parametric model, and any a priori distribution. Also we show that Bayes and Gibbs generalization errors are estimated by Bayes and Gibbs training errors, and propose widely applicable information criteria which can be applied to both regular and singular statistical models.

Report this publication

Statistics

Seen <100 times