Abstract The two-dimensional spin one half XY model may undergo an unusual type of phase transition. A formulation of the renormalization group approach of Niemeijer and van Leeuwen suitable for quantum-mechanical systems is presented. The method is applied to the spin one half XY model on the triangular lattice using a second order cumulant expansion based on division of the triangular lattice into seven spin cells. The resulting free energy curve is in excellent agreement with that obtained from the high temperature series expansion. An unstable fixed point is found but the corresponding critical temperature is not in good agreement with the series expansion value. Pathological values are found for the critical exponents.