It is well known that pain attracts attention and interferes with cognition. Given that the mechanisms behind this phenomenon are largely unknown, we used functional magnetic resonance imaging and presented visual objects with or without concomitant pain stimuli. To test for the specificity of pain, we compared this modulatory effect with a previously established modulatory effect of working memory on visual object processing. Our data showed a comparable behavioral effect of both types of modulation and identified the lateral occipital complex (LOC) as the site of modulation in the ventral visual stream, for both pain and working memory. However, the sources of these modulatory effects differed for the two processes. Whereas the source of modulation for working memory could be attributed to the parietal cortex, the modulatory effect of pain was observed in the rostral anterior cingulate cortex (rACC), an area ideally suited to link pain perception and attentional control.