Abstract Machining stability analysis is important for chatter avoidance and machining efficiency improvement. To accurately predict the stability, the chatter mechanism must be recognized. Chatter is a kind of self-excited vibrations and the two most widely used theories explaining chatter in milling are the regenerative effect and the mode coupling effect. However, these two mechanisms are always separately considered in the previous stability researches, and none of them can explain the great difference between the stability prediction results with the classical model and the experimental results in many cases. This paper investigates the structural mode coupling effect in the regenerative milling stability analysis. Based on lots of experimental data, we found that these two mechanisms actually co-exist during the practical milling process, and the usually neglected structural mode coupling effect has a great effect on the stability lobe diagram in many practical milling cases. The theoretical prediction taking the cross coupled terms into account alters the stability boundary and such prediction is verified by the chatter experimental results.