Inhibition of Return (IOR) is a delay in initiating attentional shifts to previously attended locations. It is believed to facilitate attentional exploration of a scene. Computational models of attention have implemented IOR as a simple mechanism for driving attention through a scene. However, evidence for IOR during scene viewing is inconclusive. In this study IOR during scene memorization and in response to sudden onsets at the last (1-back) and penultimate (2-back) fixation location was measured. The results indicate that there is a tendency for saccades to continue the trajectory of the last saccade (Saccadic Momentum), but contrary to the “foraging facilitator” hypothesis of IOR, there is also a distinct population of saccades directed back to the last fixation location, especially in response to onsets. Voluntary return saccades to the 1-back location experience temporal delay but this does not affect their likelihood of occurrence. No localized temporal delay is exhibited at 2-back. These results suggest that IOR exists at the last fixation location during scene memorization but that this temporal delay is overridden by Facilitation of Return. Computational models of attention will fail to capture the pattern of saccadic eye movements during scene viewing unless they model the dynamics of visual encoding and can account for the interaction between Facilitation of Return, Saccadic Momentum, and Inhibition of Return.