The polarizability α determines the absorption, extinction, and scattering by small particles. Beyond being purely set by scatterer size and material, in fact polarizability can be affected by backaction: the influence of the photonic environment on the scatterer. As such, controlling the strength of backaction provides a tool to tailor the (radiative) properties of nanoparticles. Here, we control the backaction between broadband scatterers and a single mode of a high-quality cavity. We demonstrate that backaction from a microtoroid ring resonator significantly alters the polarizability of an array of nanorods: the polarizability is renormalized as fields scattered from-and returning to-the nanorods via the ring resonator depolarize the rods. Moreover, we show that it is possible to control the strength of the backaction by exploiting the diffractive properties of the array. This perturbation of a strong scatterer by a nearby cavity has important implications for hybrid plasmonic-photonic resonators and the understanding of coupled optical resonators in general.