The age- and sex-related differences on the impacts of body composition on diabetes mellitus (DM) remain uncertain. The fourth and fifth Korea National Health and Nutrition Examination Survey included 15,586 subjects over 30 years of age who completed dual-energy X-ray absorptiometry. We conducted a cross-sectional study to investigate whether muscle mass index (MMI), defined as appendicular skeletal muscle divided by body mass index (BMI), and fat mass index (FMI), defined as trunk fat mass divided by BMI, were differently associated with DM according to age and sex. In multivariate logistic regression, the risk for DM significantly increased across quartiles of FMI in men aged ≥70. Meanwhile, MMI showed a protective association with DM in men of the same age. The odds ratios (ORs) for the highest quartile versus the lowest quartile of FMI and MMI were 3.116 (95% confidence interval [CI], 1.405 to 6.914) and 0.295 (95% CI, 0.157 to 0.554), respectively. In women, the ORs of DM was significantly different across FMI quartiles in those over age 50. The highest quartile of FMI exhibited increased ORs of DM in subjects aged 50 to 69 (OR, 1.891; 95% CI, 1.229 to 2.908) and ≥70 (OR, 2.275; 95% CI, 1.103 to 4.69) compared to lowest quartile. However, MMI was not significantly associated with DM in women of all age groups. Both FMI and MMI were independent risk factors for DM in men aged 70 years or more. In women over 50 years, FMI was independently associated with DM. There was no significant association between MMI and DM in women.