Males of many swordtail species possess vertical bar pigment patterns that are used both in courtship and agonistic interactions. Expression of the bars may function as a conventional threat signal during conflicts with rival males; bars intensify at the onset of aggression and fade in the subordinate male at contest's end. We used mirror image stimulation and bar manipulations to compare the aggressive responses of the males of four swordtail species to their barred and barless images. We found that having a response to the bars is tightly linked to having genes for bars, while the nature of the response the bars evoked varied across species. Specifically, we report the first known instance where closely related species exhibited differing and contradictory responses to a signal of aggressive motivation. Demonstrating that a signal conveys the same information across species (aggressive intent) while the response to that information has changed among species suggests that the nature of the responses are more evolutionarily labile than the signal.