Abstract Apolipoprotein E (APOE) ε4 is a major risk factor for Alzheimer's disease (AD) and dementia, but not all ε4 carriers develop dementia. We sought to identify factors that may play a role in modifying the risk of dementia due to ε4. A cognitively intact cohort (n = 932, age ≥ 75) was followed for 9 years to detect incident dementia cases. At baseline, information on education, leisure activities, and vascular risk factors was collected, and APOE was genotyped. During the follow-up, 324 subjects developed dementia, including 247 AD cases. The hazard ratio (HR, 95% confidence interval [95% CI]) of dementia related to the ε4 was 1.39 (1.11–1.76), while the risk was reduced when ε4 carriers had high education, no vascular risk factors, or high score of leisure activities. Among ε4 carriers, the multiadjusted HRs of dementia that were associated with high education, high level of leisure activities, and absence of vascular risk factors were 0.59 (0.40–0.87), 0.49 (0.29–0.85), and 0.61 (0.41–0.90), respectively. The ε4 carriers with these factors had about 1.2 years delayed time to dementia onset compared with those without these factors. High education, active leisure activities, or maintaining vascular health seems to reduce the risk of dementia related to APOE ε4. The ε4 carriers with these characteristics appear to have similar dementia-free survival time to non-ε4 carriers.