We introduce a topological gauge vector potential which influences spin wave excitations over arbitrary non-uniform, slowly moving magnetization distribution. The time-component of the gauge potential plays a principal role in magnetization dynamics, whereas its spatial components can be often neglected for typical magnetic nanostructures. As an example, we consider spin modes excited in the vortex state magnetic dots. It is shown that the vortex/ spin wave interaction can be described as a consequence of the gauge field arising due to non uniform moving vortex magnetization distribution. The coupled equations of motion of the vortex and spin waves are solved within small excitation amplitude approximation. The model yields a giant frequency splitting of the spin wave modes having non-zero overlapping with the vortex mode as well as a finite vortex mass of dynamical origin.