The fourth harmonic of the azimuthal distribution of particles $v_4$ has been measured for Au-Au collisions at the Relativistic Heavy Ion Collider (RHIC). The centrality dependence of $v_4$ does not agree with the prediction from hydrodynamics. In particular, the ratio $v_4/(v_2)^2$, where $v_2$ denotes the second harmonic of the azimuthal distribution of particles, is significantly larger than predicted by hydrodynamics. We argue that this discrepancy is mostly due to elliptic flow ($v_2$) fluctuations. We evaluate these fluctuations on the basis of a Monte Carlo Glauber calculation. The effect of deviations from local thermal equilibrium is also studied, but appears to be only a small correction. Combining these two effects allows us to reproduce experimental data for peripheral and midcentral collisions. However, we are unable to explain the large magnitude of $v_4/(v_2)^2$ observed for the most central collisions.