Purpose Prostate specific antigen (PSA) is a predictor of prostate growth in men with lower urinary tract symptoms. The long-term risk of prostate enlargement as a function of PSA among community dwelling volunteers is unknown. Materials and Methods A Cox proportional hazards regression model was used to study the relationship between baseline PSA level at ages 40 to 49.9 years in 194 men, 50 to 59.9 in 191 and 60 to 69.9 in 144, and prostate enlargement, defined as a prostate volume larger than the 75th percentile for age decade, as measured by magnetic resonance imaging in a longitudinal study of aging (Baltimore Longitudinal Study of Aging, National Institute on Aging). Kaplan-Meier survival analysis was performed to estimate the probability of freedom from prostate enlargement with time as a function of baseline PSA level. Results The relative risk of prostate enlargement was 3- to 6-fold higher for men 40 to 49.9 years old with a baseline PSA of 0.31 ng./ml. or more compared to men with PSA levels of 0.30 ng./ml. or less at baseline. The relative risk was increased 5- to 9-fold in men 50 to 59.9 years old and 11-fold in those 60 to 69.9 years old when comparing men with PSA greater than 0.80 ng./ml. and greater than 1.70 ng./ml. with those with PSA 0.50 or less. The cumulative probability of freedom from prostate enlargement at 20 years was 0.89 (95% confidence interval ([CI] 0.79–0.99) and 0.63 (0.52–0.74) for men 40 to 49.9 years old with PSA levels below and above 0.30 ng./ml., respectively. For men 50 to 59.9 years old the 10-year probability of freedom from prostate enlargement was 0.90 (95% CI 0.84–0.96) and 0.59 (0.43–0.74) when PSA levels were below and above 0.80 ng./ml., respectively. At age 60 to 69.9 years the 10-year probability of freedom from prostate enlargement was 0.83 (95% CI 0.72–0.93) and 0.27 (0.09 to 0.48) when PSA levels were below and above 1.70 ng./ml., respectively. Conclusions These data demonstrate the long-term risk of prostate enlargement by PSA level. Risk stratification based on PSA level may be useful to identify men at greatest risk for adverse events due to prostate enlargement and selection of men for future benign prostatic hyperplasia studies.