The extinct human relatives known as Neanderthals (Homo neanderthalensis) have long been described as brutish and dumb. This conception is often traced to paleontologist Marcellin Boule (1861-1942), who published a detailed analysis on a Neanderthal skeleton in the early twentieth century. The conventional historical narrative claims that Boule made an error in his analysis, causing the Neanderthals to be considered brutish. This essay challenges the narrative of "Boule's error," arguing instead that the brutish Neanderthal concept originated much earlier in the history of Neanderthal research and was, in fact, an invention of the earliest analyses of the first specimen recognized as a Neanderthal in the mid-nineteenth century. I argue that temporally relocating this conception of Neanderthals allows for a better understanding of the interconnected nature of the study of fossil humans and the science of living human races during the nineteenth century. This new view of the brutish Neanderthal sheds light on the earliest phases of the science that became paleoanthropology, while examining the racial, cultural, and political attitudes about race and extinction that accompanied the science at that time. By inspecting the ways in which the Neanderthals' image was a product of a particular time and place, we gain a perspective that provides a new basis for thinking about the conceptions of hominin fossil species.