Previous studies have shown that exposure to early life stress (ELS) is associated with reduced volume of brain regions critical for information processing, memory and emotional function. Further, recent studies from our lab utilizing diffusion tensor imaging (DTI) have found alterations in the microstructural integrity of white matter pathways among adults exposed to ELS. However, it is not clear if these relationships extend to children and adolescents, and it is also unclear if these DTI abnormalities are associated with cognitive performance. The present study examined the relationship between ELS and the microstructural integrity of the corpus callosum among a sample of otherwise healthy controls between the ages of 8 and 73. The participants were subdivided into four age groups (8–12, 13–18, 19–50, 51–73). Individuals with three or more ELS events were compared to individuals with fewer than 3 ELS events on fractional anisotropy (FA) in the genu of the corpus callosum. Separate analyses examined the two groups on tests of verbal memory, information processing speed, psychomotor speed and cognitive flexibility. Results revealed that the youngest group and the oldest group of individuals with ELS exhibited significantly lower FA in the genu compared to individuals without ELS. However, there were no group differences on any of the cognitive tasks. Our results indicate that ELS is related to subtle alterations in brain structure, but not function. The effects found with regard to DTI occurred during periods of critical age-related developmental windows.