The low-energy vortex effective action is constructed in a wide class of systems in a color-flavor locked vacuum, which generalizes the results found earlier in the context of U(N) models. It describes the weak fluctuations of the non-Abelian orientational moduli on the vortex worldsheet. For instance, for the minimum vortex in SO(2N) x U(1) or USp(2N) x U(1) gauge theories, the effective action found is a two-dimensional sigma model living on the Hermitian symmetric spaces SO(2N)/U(N) or USp(2N)/U(N), respectively. The fluctuating moduli have the structure of that of a quantum particle state in spinor representations of the GNO dual of the color-flavor SO(2N) or USp(2N) symmetry, i.e. of SO(2N) or of SO(2N+1). Applied to the benchmark U(N) model our procedure reproduces the known CP(N-1) worldsheet action; our recipe allows us to obtain also the effective vortex action for some higher-winding vortices in U(N) and SO(2N) theories.