5% $B_2H_6/Ar$ and 5% $PH_3/Ar$ were introduced to the plasma enhanced chemical vapor deposition process of silicon oxynitride. The bond configurations and refractive indexes of the boron/phosphorous (B/P) doped layers were characterized by Fourier transform infrared spectroscopy and the prism coupling method, respectively. The effect of annealing on the N-H bonds, which mainly contribute to undesired optical losses around 1500 nm wavelength, has been studied. Compared to undoped samples, significant reduction of N-H bonds was observed in the as-deposited B/P doped layers. The reduction of N-H bonds during annealing was found to depend primarily on the applied temperatures rather than on the duration of the annealing process. Complete elimination of N-H bonds has been achieved by temperature treatment of samples at 70C for 3 hours.