Abstract This paper presents a study of the relationship between the homotopy analysis method (HAM) and harmonic balance (HB) method. The HAM is employed to obtain periodic solutions of conservative oscillators and limit cycles of self-excited systems, respectively. Different from the usual procedures in the existing literature, the HAM is modified by retaining a given number of harmonics in higher-order approximations. It is proved that as long as the solution given by the modified HAM is convergent, it converges to one HB solution. The Duffing equation, the van der Pol equation and the flutter equation of a two-dimensional airfoil are taken as illustrations to validate the attained results.