Planetary gear set is the critical component in helicopter transmission train, and an important problem in condition monitoring and health management of planetary gear set is quantitative damage detection. In order to resolve this problem, an approach based on physical models is presented to detect damage quantitatively in planetary gear set. A particular emphasis is put on a feature generation and selection method, which is used for sun gear tooth breakage damage detection quantitatively in planetary gear box of helicopter transmission system. In this feature generation procedure, the pure torsional dynamical models of 2K-H planetary gear set is established for healthy case and sun gear tooth-breakage case. Then, a feature based on the spectrum of simulation signals of the dynamical models is generated. Aiming at selecting the best feature suitable for quantitative damage detection, a two-sample Z-test procedure is used to analyze the performance of features on damage evolution tracing. A feature named SR, which had better performance in tracking damage, is proposed to detect damage in planetary gear set. Meanwhile, the sun gear tooth-chipped seeded experiments with different severity are designed to validate the method above, and then the test vibration signal is picked up and used for damage detection. With the results of several experiments for quantitative damage detection, the feasibility and the effect of this approach are verified. The proposed method can supply an effective tool for degradation state identification in condition monitoring and health management of helicopter transmission system.