Abstract This paper presents the dynamic responses of the coupled textile/rotor system by finite element analysis. When textile is wound either on or off the rotor, the system is non-conservative because mass, inertia and eccentricity of the unbalance of rotor change with time, and also the length of textile is time-dependent. Both the time-varying equations for textile and the whirling vibrations for rotor are derived by Hamilton's principle. It is a moving boundary problem since the unknown length of textile has to be determined as a part of the solutions. The special finite element formulations are developed by applying a complete linear polynomial approximation. The number of elements is fixed while the size of the element changes with time. The Runge-Kutta method is used to obtain numerical results. The effects of constant and non-constant angular rotating speeds, shaft stiffness and non-linear terms on the transient amplitudes of the textile and the whirling deflection of the shaft are investigated.