Alexander Bird and Darrell Rowbottom have argued for two competing accounts of the concept of scientific progress. For Bird, progress consists in the accumulation of scientific knowledge. For Rowbottom, progress consists in the accumulation of true scientific beliefs. Both appeal to intuitions elicited by thought experiments in support of their views, and it seems fair to say that the debate has reached an impasse. In an attempt to avoid this stalemate, we conduct a systematic study of the factors that underlie judgments about scientific progress. Our results suggest that (internal) justification plays an important role in intuitive judgments about progress, questioning the intuitive support for the claim that the concept of scientific progress is best explained in terms of the accumulation of true scientific belief.