Chromosomal polymorphism has been reported to be associated with infertility, but its effect on IVF/ICSI-ET outcome is still controversial. To evaluate whether or not chromosomal polymorphism in men plays a role in spermatogenesis and the outcome of IVF/ICSI-ET, we retrospectively analysed 281 infertile couples. Measures included fertilization rate, implantation rate, pregnancy rate, clinical pregnancy rate, ongoing pregnancy rate, early miscarriage rate and preterm rate. Men with chromosomal polymorphism had significantly higher frequencies of severe oligozoospermia and azoospermia than those without (37.12% vs. 16.11%, p < 0.001; 27.27% vs. 10.74%, p < 0.001; respectively). Significantly, lower fertilization rate (68.02% vs. 78.00%, p < 0.001) and clinical pregnancy rate (45.00% vs. 66.67%, p = 0.031) were observed in polymorphism-carrying men with severe oligozoospermia compared with non-carriers with severe oligozoospermia. This suggests that chromosomal polymorphism has adverse effects on spermatogenesis, negatively influencing the outcome of IVF/ICSI-ET treatment. Polymorphic variations on the Y chromosome have been found to be the most prevalent polymorphism in infertile men, most frequently occurring in patients with severe oligozoospermia.