Heavy-ion collisions create deformed quark-gluon plasma (QGP) fireballs which explode anisotropically. The viscosity of the fireball matter determines its ability to convert the initial spatial deformation into momentum anisotropies that can be measured in the final hadron spectra. A quantitatively precise empirical extraction of the QGP viscosity thus requires a good understanding of the initial fireball deformation. This deformation fluctuates from event to event, and so does the finally observed momentum anisotropy. We present a harmonic decomposition of the initial fluctuations in shape and orientation of the fireball and perform event-by-event ideal fluid dynamical simulations to extract the resulting fluctuations in the magnitude and direction of the corresponding harmonic components of the final anisotropic flow at midrapidity. The final harmonic flow coefficients are found to depend non-linearly on the initial harmonic eccentricity coefficients. We show that, on average, initial density fluctuations suppress the buildup of elliptic flow relative to what one obtains from a smooth initial profile of the same eccentricity, and discuss implications for the phenomenological extraction of the QGP shear viscosity from experimental elliptic flow data.