Abstract A normal mode instability study of a steady nondivergent flow on a rotating sphere is considered. A real-order derivative and family of the Hilbert spaces of smooth functions on the unit sphere are introduced, and some embedding theorems are given. It is shown that in a viscous fluid on a sphere, the operator linearized about a steady flow has a compact resolvent, that is, a discrete spectrum with the only possible accumulation point at infinity, and hence, the dimension of the unstable manifold of a steady flow is finite. Peculiarities of the operator spectrum in the case of an ideal flow on a rotating sphere are also considered. Finally, as examples, we consider the normal mode stability of polynomial (zonal) basic flows and discuss the role of the linear drag, turbulent diffusion and sphere rotation in the normal mode stability study.