Emerging research has found that both perceptions of the environment and leisure time physical activity (LTPA) behavior can contribute to sense of community (SOC) in the neighborhood. The strength of these associations, however, may vary according to age. This research examined the influence of LTPA and environmental perceptions on SOC among adults in four age groups to determine whether age-related differences existed among the causal pathways. Quota sampling was utilized to survey residents (N = 597) in Phoenix, Arizona. A multi-group structural equation modeling analysis indicated that all structural paths were significantly associated, yet distinct patterns were revealed depending on age. Notably, the positive relationship between perceptions of the neighborhood environment and SOC was strongest among older adults. Furthermore, LTPA was found to mediate the relationship, suggesting that engaging in physical activity during leisure time may be a promising strategy to promote neighborhood cohesion, particularly among adults aged 65 and older.