Abstract A prolonged exposure (i.e., perceptual adaptation) to a male or a female face can produce changes (i.e., aftereffects) in the subsequent gender attribution of a neutral or average face, so that it appears respectively more female or more male. Studies using imagery adaptation and its aftereffects have yielded conflicting results. In the present study we used an adaptation paradigm with both imagined and perceived faces as adaptors, and assessed the aftereffects in judged masculinity/femininity when viewing an androgynous test face. We monitored eye movements and pupillary responses as a way to confirm whether participants did actively engage in visual imagery. The results indicated that both perceptual and imagery adaptation produce aftereffects, but that they run in opposite directions: a contrast effect with perception (e.g., after visual exposure to a female face, the androgynous appears as more male) and an assimilation effect with imagery (e.g., after imaginative exposure to a female face, the androgynous face appears as more female). The pupillary responses revealed dilations consistent with increased cognitive effort during the imagery phase, suggesting that the assimilation aftereffect occurred in the presence of an active and effortful mental imagery process, as also witnessed by the pattern of eye movements recorded during the imagery adaptation phase.