We investigated a psychophysical similarity measure for selection of images similar to those of unknown masses on mammograms, which may assist radiologists in the distinction between benign and malignant masses. Sixty pairs of masses were selected from 1445 mass images prepared for this study, which were obtained from the Digital Database for Screening Mammography by the University of South Florida. Five radiologists provided subjective similarity ratings for these 60 pairs of masses based on the overall impression for diagnosis. Radiologists' subjective ratings were marked on a continuous rating scale and quantified between 0 and 1, which correspond to pairs not similar at all and pairs almost identical, respectively. By use of the subjective ratings as "gold standard," similarity measures based on the Euclidean distance between pairs in feature space and the psychophysical measure were determined. For determination of the psychophysical similarity measure, an artificial neural network (ANN) was employed to learn the relationship between radiologists' average subjective similarity ratings and computer-extracted image features. To evaluate the usefulness of the similarity measures, the agreement with the radiologists' subjective similarity ratings was assessed in terms of correlation coefficients between the average subjective ratings and the similarity measures. A commonly used similarity measure based on the Euclidean distance was moderately correlated (r=0.644) with the radiologists' average subjective ratings, whereas the psychophysical measure by use of the ANN was highly correlated (r=0.798). The preliminary result indicates that a psychophysical similarity measure would be useful in the selection of images similar to those of unknown masses on mammograms. © 2005 American Association of Physicists in Medicine.