Electronic structure calculations are used to investigate the binding energies of defect pairs composed of lattice vacancies and phosphorus or arsenic atoms (E centers) in silicon-germanium alloys. To describe the local environment surrounding the E center we have generated special quasirandom structures that represent random silicon-germanium alloys. It is predicted that the stability of E centers does not vary linearly with the composition of the silicon-germanium alloy. Interestingly, we predict that the nonlinear behavior does not depend on the donor atom of the E center but only on the host lattice. The impact on diffusion properties is discussed in view of recent experimental and theoretical results.