Grating interferometry has been successfully adapted at standard X-ray tubes and is a promising candidate for a broad use of phase-contrast imaging in medical diagnostics or industrial testing. The achievable image quality using this technique is mainly dependent on the interferometer performance with the interferometric visibility as crucial parameter. The presented study deals with experimental investigations of the spectral dependence of the visibility in order to understand the interaction between the single contributing energies. Especially for the choice which type of setup has to be preferred using a polychromatic source, this knowledge is highly relevant. Our results affirm previous findings from theoretical investigations but also show that measurements of the spectral contributions to the visibility are necessary to fully characterize and optimize a grating interferometer and cannot be replaced by only relying on simulated data up to now.