Summary Decline of cognitive function with age may be due, in part, to hormonal changes and it has been hypothesized that higher levels of endogenous sex hormones preserve brain function. The aim of this prospective cohort study was to determine the relative contribution of endogenous sex hormones to cognitive decline in a population-based sample of 242 elderly men aged 73–91 at baseline. Endogenous sex hormone levels were measured at baseline and participants underwent a cognitive assessment at baseline and at follow-up after 4 years. Higher estradiol (total and bioavailable) and estrone levels were associated with an increased risk of cognitive decline in elderly men independent of age, cardiovascular risk factors, atherosclerosis, and APOE genotype. These findings do not support the hypotheses that higher levels of endogenous sex hormones preserve brain function.