There are numerous reports of sexual dimorphism in brain structure in children and adults, but data on sex differences in infancy are extremely limited. Our primary goal was to identify sex differences in neonatal brain structure. Our secondary goal was to explore whether brain structure was related to androgen exposure or sensitivity. Two hundred and ninety-three neonates (149 males) received high-resolution structural magnetic resonance imaging scans. Sensitivity to androgen was measured using the number of cytosine, adenine, guanine (CAG) triplets in the androgen receptor gene and the ratio of the second to fourth digit, provided a proxy measure of prenatal androgen exposure. There was a significant sex difference in intracranial volume of 5.87%, which was not related to CAG triplets or digit ratios. Tensor-based morphometry identified extensive areas of local sexual dimorphism. Males had larger volumes in medial temporal cortex and rolandic operculum, and females had larger volumes in dorsolateral prefrontal, motor, and visual cortices. Androgen exposure and sensitivity had minor sex-specific effects on local gray matter volume, but did not appear to be the primary determinant of sexual dimorphism at this age. Comparing our study with the existing literature suggests that sex differences in cortical structure vary in a complex and highly dynamic way across the human lifespan.