Affordable Access

Publisher Website

A simple probabilistic neural network for machine understanding

Authors
  • Xie, Rongrong
  • Marsili, Matteo
Type
Published Article
Journal
Journal of Statistical Mechanics: Theory and Experiment
Publisher
IOP Publishing
Publication Date
Feb 20, 2024
Volume
2024
Issue
2
Identifiers
DOI: 10.1088/1742-5468/ad0a8c
Source
ioppublishing
Keywords
Disciplines
  • Paper section: Interdisciplinary statistical mechanics
License
Unknown

Abstract

We discuss the concept of probabilistic neural networks with a fixed internal representation being models for machine understanding. Here, ‘understanding’ is interpretted as the ability to map data to an already existing representation which encodes an a priori organisation of the feature space. We derive the internal representation by requiring that it satisfies the principles of maximal relevance and of maximal ignorance about how different features are combined. We show that, when hidden units are binary variables, these two principles identify a unique model—the hierarchical feature model—which is fully solvable and provides a natural interpretation in terms of features. We argue that learning machines with this architecture possess a number of interesting properties, such as the continuity of the representation with respect to changes in parameters and data, the possibility of controlling the level of compression and the ability to support functions that go beyond generalisation. We explore the behaviour of the model with extensive numerical experiments and argue that models in which the internal representation is fixed reproduce a learning modality which is qualitatively different from that of traditional models, such as restricted Boltzmann machines.

Report this publication

Statistics

Seen <100 times