Abstract A numerical scheme for solving the three dimensional interior acoustic problem governed by the Helmholtz equation has been developed. The main feature of this formulation is that the singularity can be removed analytically. The boundary integral equation is solved for a specific geometry by using second-order quadrilateral surface elements and does not have the difficulty of non-uniqueness associated with the boundary integral equation formulation in an exterior problem. The transmission loss of a muffler is computed by using the derived four-pole parameters. The effects of higher-order modes due to the area discontinuity as well as the various inlet/outlet alignment on the acoustic performance of a muffler are also studied. The case of a muffler with elliptic cross-section, which is troublesome in analytical solution, can be treated easily by the present scheme. Finally, the transmission loss characteristics of a muffler with extended inlet/outlet are investigated. All the numerical results are compared with experimental measurement and agreement is good.