Background Impaired physical fitness, a contributor to obesity and cardiovascular disease, has been associated with both an individual's socioeconomic status (SES) and with residence in disadvantaged neighborhoods. The aim of the study was to examine the extent to which neighborhood socioeconomic status (SES) is associated with impaired fitness, independent of clinical characteristics and individual-level SES. Methods Two thousand five hundred five participants 25 to 42 years old examined in the CARDIA study from 1992 to 1993 underwent symptom-limited exercise stress testing. Physical fitness was considered impaired if metabolic equivalents were in the lowest sex-specific quintile. Neighborhood SES was determined for each census tract using 1990 census data. Generalized estimating equations assessed the association between neighborhood SES and physical fitness, before and after adjustments for individual SES, sociodemographic, and clinical characteristics, and accounted for clustering within census tracts. Results Individuals in disadvantaged neighborhoods had lower educational attainment and income, and were more likely unemployed, black, and uninsured. The odds ratio (95% CI) for impaired physical fitness in the lowest vs highest tertile of neighborhood SES was 5.8 (3.7-7.3). These became 3.9 (2.7-5.7) after adjusting for individuals' educational attainment, personal income, employment status, and ability to pay for basic needs; and 1.9 (1.2-2.9) after additional adjustment for other sociodemographic and clinical factors. Conclusions Features of one's neighborhood of residence are relevant to cardiovascular health. A health policy perspective that looks beyond an individual's characteristics may therefore be useful in identifying more effective interventions to reduce the prevalence of low physical fitness and its consequences in young adults.