Even dissimilar tasks interfere with one another when done together. We used visual search to examine the underlying cause of such interference. In many models, visual search is a process of biased competition controlled by a template describing the target to be sought. When the display is processed, matching against this template guides attention to the target. We show that increasing template complexity increased interference with a dissimilar concurrent task, story memory. This result was independent of reaction time: Increases in template complexity were associated with no increase in search time in Experiment 1 and with a decrease in search time in Experiment 2. The results show that the dual-task demands of visual search reflect the complexity of the template used in task control, and that this factor can be isolated from other sources of difficulty.