Terminal duct lobular units (TDLUs) are the predominant anatomical structures where breast cancers originate. Having lesser degrees of age-related TDLU involution, measured as higher TDLUs counts or more epithelial TDLU substructures (acini), is related to increased breast cancer risk among women with benign breast disease (BBD). We evaluated whether a recently developed polygenic risk score (PRS) based on 313-common variants for breast cancer prediction is related to TDLU involution in the background, normal breast tissue, as this could provide mechanistic clues on the genetic predisposition to breast cancer. Among 1398 women without breast cancer, higher values of the PRS were significantly associated with higher TDLU counts (P = 0.004), but not with acini counts (P = 0.808), in histologically normal tissue samples from donors and diagnostic BBD biopsies. Mediation analysis indicated that TDLU counts may explain a modest proportion (≤10%) of the association of the 313-variant PRS with breast cancer risk. These findings suggest that TDLU involution might be an intermediate step in the association between common genetic variation and breast cancer risk.