Background Brain size is associated with cognitive ability in adulthood (correlation ~ .3), but few studies have investigated the relationship in normal ageing, particularly beyond age 75 years. With age both brain size and fluid-type intelligence decline, and regional atrophy is often suggested as causing decline in specific cognitive abilities. However, an association between brain size and intelligence may be due to the persistence of this relationship from earlier life. Methods We recruited 107 community-dwelling volunteers (29% male) aged 75–81 years for cognitive testing and neuroimaging. We used principal components analysis to derived a 'general cognitive factor' (g) from tests of fluid-type ability. Using semi-automated analysis, we measured whole brain volume, intracranial area (ICA) (an estimate of maximal brain volume), and volume of frontal and temporal lobes, amygdalo-hippocampal complex, and ventricles. Brain atrophy was estimated by correcting WBV for ICA. Results Whole brain volume (WBV) correlated with general cognitive ability (g) (r = .21, P < .05). Statistically significant associations between brain areas and specific cognitive abilities became non-significant when corrected for maximal brain volume (estimated using ICA), i.e. there were no statistically significant associations between atrophy and cognitive ability. The association between WBV and g was largely attenuated (from .21 to .03: i.e. attenuating the variance by 98%) by correcting for ICA. ICA accounted for 6.2% of the variance in g in old age, whereas atrophy accounted for < 1%. Conclusion The association between brain regions and specific cognitive abilities in community dwelling people of older age is due to the life-long association between whole brain size and general cognitive ability, rather than atrophy of specific regions. Researchers and clinicians should therefore be cautious of interpreting global or regional brain atrophy on neuroimaging as contributing to cognitive status in older age without taking into account prior mental ability and brain size.