Abstract The effect of particle–wake interaction on the coupling between the longitudinal and transverse dust lattice waves in two-dimensional hexagonal lattice in an external magnetic field is investigated. Meanwhile the electrostatic interactions between a particle charge and all other particle charges together with the corresponding effective wake charges are taken into account. Explicit formulations of dispersion relations of dust lattice waves are given when both the primary wave vector and the external magnetic field are parallel to axes. The dispersion relations are calculated numerically and hybrid phonon mode and hybrid optical mode are obtained. Furthermore, the mode mixing rates of the hybrid phonon mode and the hybrid optical mode are calculated and the mode changing points are obtained. Finally, the mode coupling is very intense in the region where the dispersion curves are close together and the mode coupling intensity increases when the magnitude of the external magnetic field increases.