Early-life adversity is associated with increased risk for substance abuse in later life, with women more likely to report past and current stress as a mediating factor in their substance use and relapse as compared to men. Preclinical models of neonatal and peri-adolescent (early through late adolescence) stress all support a direct relationship between experiences of early-life adversity and adult substance-related behaviors, and provide valuable information regarding the underlying neurobiology. This review will provide an overview of these animal models and how these paradigms alter drug and alcohol consumption and/or seeking in male and female adults. An introduction to the corticotropin-releasing factor (CRF) and serotonin systems, their development and their interactions at the level of the dorsal raphe will be provided, illustrating how this particular stress system is sexually dimorphic, and is well positioned to be affected by stressors early in development and throughout maturation. A model for CRF-serotonin interactions in the dorsal raphe and how these influence dopaminergic activity within the nucleus accumbens and subsequent reward-associated behaviors will be provided, and alterations to the activity of this system following early-life adversity will be identified. Overall, converging findings suggest that early-life adversity has long-term effects on the functioning of the CRF-serotonin system, highlighting a potentially important and targetable mediator linking stress to addiction. Future work should focus on identifying the exact mechanisms that promote long-term changes to the expression and activity of CRF receptors in the dorsal raphe. Moreover, it is important to clarify whether similar neurobiological mechanisms exist for males and females, given the sexual dimorphism both in CRF receptors and serotonin indices in the dorsal raphe and in the behavioral outcomes of early-life adversity.