Abstract Tooth wedging, also known as tight mesh, occurs when a gear tooth comes into contact on the drive-side and back-side simultaneously. Tooth wedging risks bearing failures from elevated forces. This work studies the nonlinear tooth wedging behavior and its correlation with planet bearing forces by analyzing the dynamic response of an example planetary gear. This planetary gear is representative of a wind turbine geartrain. A two-dimensional lumped-parameter model is extended to include tooth separation, back-side contact, tooth wedging, and bearing clearances. The results show significant impact of tooth wedging on planet bearing forces for a wide range of operating speeds. To develop a physical understanding of the tooth wedging mechanism, connections between planet bearing forces and tooth forces are studied by investigating physical forces and displacements acting throughout the planetary gear. A method to predict tooth wedging based on geometric interactions is developed and verified. The major causes of tooth wedging relate directly to translational vibrations caused by gravity forces and the presence of clearance-type nonlinearities in the form of backlash and bearing clearance.