In many bird and mammal species, males are significantly larger than females. The prevailing explanation for larger-sized males is that sexual selection drives increased male size. In addition, researchers commonly assume that the extent of dimorphism indicates the strength of selection for increased size in males. Here, through reconstruction of ancestral morphology for males and females of one large avian clade we present data that contradict this assumption and illustrate that selection for decreased female size explains 'male-biased' dimorphism ca. 50% of the time. Our findings are also inconsistent with ecological niche partitioning between the sexes and increased breeding benefits from reduced female size as general explanations for the evolution of size dimorphism within the clade. We conclude that it is incorrect to assume sexual dimorphism results from a single selective factor, such as directional sexual selection on increased male size. Rather, we suggest that the selective forces leading to sexual dimorphism may vary between species and should be tested on a case-by-case basis using a phylogenetic approach.