Although we know the transition from childhood to adulthood is marked by important social and neural development, little is known about how social network size might affect neurocognitive development or vice versa. Neuroimaging research has identified several brain regions, such as the amygdala, as key to this affiliative behavior. However, white matter connectivity among these regions, and its behavioral correlates, remain unclear. Here we tested two hypotheses: that an amygdalocentric structural white matter network governs social affiliative behavior and that this network changes during adolescence and young adulthood. We measured social network size behaviorally, and white matter microstructure using probabilistic diffusion tensor imaging in a sample of neurologically normal adolescents and young adults. Our results suggest amygdala white matter microstructure is key to understanding individual differences in social network size, with connectivity to other social brain regions such as the orbitofrontal cortex and anterior temporal lobe predicting much variation. In addition, participant age correlated with both network size and white matter variation in this network. These findings suggest the transition to adulthood may constitute a critical period for the optimization of structural brain networks underlying affiliative behavior.