Abstract In this work, a hybrid method in frequency domain for design of a linear-phase quadrature mirror filter bank is proposed. The nonlinear least square hybrid method based on Levenberg–Marquardt (LM) and Quasi-Newton (QN) is developed for the design of a low-pass prototype filter whose responses in the passband and stopband are ideal, and the filter bank response at quadrature frequency is 0.707. The proposed method is employed to optimize the quadratic measure of ideal characteristics of the prototype filter and filter bank at quadrature frequency. Performance and effectiveness of the proposed method in terms of peak reconstruction error (PRE), mean squares error in passband and stopband regions, and error in transition band at quadrature frequency are shown through numerical examples and comparison of the results with other existing method. It was found that the proposed method is very simple and easy to implement for QMF bank design problem.