Understanding the mechanisms underlying the relationship between biodiversity and ecosystem function (BEF) is critical for the implementation of productive and resilient ecosystem management. However, the differences in BEF relationships along altitudinal gradients between forests and shrublands are poorly understood, impeding the ability to manage terrestrial ecosystems and promote their carbon sinks. Using data from 37962 trees of 115 temperate forest and 134 shrubland plots of Taihang Mountains Priority Reserve, we analyzed the effects of species diversity, structural diversity, climate factors and soil moisture on carbon storage along altitudinal gradients in temperate forests and shrublands. We found that: (1) Structural diversity, rather than species diversity, mainly promoted carbon storage in forests. While species diversity had greater positive effect on carbon storage in shrublands. (2) Mean annual temperature (MAT) had a direct negative effect on forest carbon storage, and indirectly affected forest carbon storage by inhibiting structural diversity. In contrast, MAT promoted shrubland carbon storage directly and indirectly through the positive mediating effect of species diversity. (3) Increasing altitudinal gradients enhanced the structural diversity-carbon relationship in forests, but weakened the species diversity-carbon relationship in shrublands. Niche and architectural complementarity and different life strategies of forests and shrubs mainly explain these findings. These differential characteristics are critical for our comprehensive understanding of the BEF relationship and could help guide the differentiated management of forests and shrublands in reaction to environmental changes.