Objectives To examine brain volumes and cortical surface area and thickness and to relate these brain measures to cognitive function in young adults born small for gestational age (SGA) at term compared with non-SGA control patients. Study design This population-based follow-up study at age 20 years included 58 term-born SGA (birth weight <10th percentile, mean: 2915 g) and 81 non-SGA controls (birth weight ≥10th percentile, mean: 3707 g). Brain volumes and cortical surface area and thickness were investigated with magnetic resonance imaging, which was successfully obtained in 47 SGA patients and 61 control patients. Cognitive function was assessed using the Wechsler Adult Intelligence Scale, 3rd edition. A subgroup analysis was performed in the SGA group among subjects diagnosed with fetal growth restriction (FGR) based on repeated fetal ultrasound measurements. Results The SGA group showed regional reductions in cortical surface area, particularly in the frontal, parietal, and temporal lobes. Total brain volume, cortical gray matter, cerebral white matter, and putamen volumes were reduced in the SGA group compared with control patients, but there were no differences in specific subcortical brain structure volumes when correcting for intracranial volume. Reductions were most pronounced among SGA subjects with FGR. No associations were found between brain measures and IQ measures in either group. Conclusion Young adults born SGA at term show a global reduction in brain volume as well as regional reductions in cortical surface area. We speculate whether these reductions may be confined to those exposed to FGR. None of the brain measures correlated with cognition.