Obesity and related disorders are a burgeoning public health epidemic, particularly in the U.S. Currently 34% of the U.S. population is clinically obese (BMI > 30) and 68% are overweight (BMI > 25), more than double the worldwide average and 10-fold higher than Japan and South Korea. Obesity occurs when energy intake exceeds energy expenditure; however, individuals vary widely in their propensity to gain weight and accrue fat mass, even at identical levels of excess caloric input. Clinical, epidemiological, and biological studies show that obesity is largely programmed during early life, including the intrauterine period. The environmental obesogen hypothesis holds that prenatal or early life exposure to certain endocrine disrupting chemicals can predispose exposed individuals to increased fat mass and obesity. Obesogen exposure can alter the epigenome of multipotent stromal stem cells, biasing them toward the adipocyte lineage at the expense of bone. Hence, humans exposed to obesogens during early life might have an altered stem cell compartment, which is preprogrammed toward an adipogenic fate. This results in a higher steady state number of adipocytes and potentially a lifelong struggle to maintain a healthy weight, which can be exacerbated by societal influences that promote poor diet and inadequate exercise. This review focuses on the developmental origins of the adipocyte, the relationship between adipocyte number and obesity, and how obesogenic chemicals may interfere with the highly efficient homeostatic mechanisms regulating adipocyte number and energy balance.