Abstract Background Increased physical activity may have beneficial effects on cognitive outcomes; a role of brain-derived neurotrophic factor (BDNF) has been suggested in animal models but not yet tested in humans. This study investigated modification by BDNF val66met polymorphism of the association between physical activity, incident dementia and other cognitive outcomes. Methods Of 732 community elders, 107 had dementia at baseline, and 518 (83%) of the remainder were followed over 2.4 years. Cognitive impairment and decline were defined from Mini-Mental State Examination scores. Self-reported level of physical activity was recorded on a 4-point scale. BDNF val66met and apolipoprotein E genotypes were ascertained. Covariates included age, sex, education, depression, vascular risk factors, and instrumental activities of daily living. Results Baseline lower physical activity was significantly associated with incident dementia as well as with baseline dementia and cognitive impairment and incident cognitive decline. BDNF val66met polymorphism itself was not associated with any cognitive outcome. However, the strength of association between lower activity and all cognitive outcomes increased incrementally with the number of met alleles, and was strongest in those with the met/met genotype. BDNF × activity interaction terms were stronger for prospective outcomes (incident dementia, cognitive decline) compared to cross-sectional outcomes (prevalent dementia, cognitive impairment no dementia). Conclusions This study supports a previously suggested neurobiological basis for the effects of physical activity on dementia involving the BDNF system since the met allele is recognised to be associated with lower activity-dependent secretion of BDNF.