We study the influence of Born impurity scattering on the zero-energy Andreev bound states near the surface of a d-wave superconductor with and without an externally applied magnetic field. Without an external magnetic field we show that the effect of Born impurity scattering is stronger at the surface than in the bulk. In the presence of an external magnetic field the splitting of the zero-energy Andreev bound states is shown to have a nonmonotonous temperature dependence. Born impurity scattering does not wash out the peak splitting, but instead the peak splitting is shown to be quite robust against impurities. We also show that a nonzero gap renormalization appears near the surface.