Abstract Consumer product acceptability trials can be carried out at the company premises, at a central location (hall) test, or by home test. This paper addresses some of the issues associated with data collected by different types of consumer panels. It looks at consistency of results over the test methods, considering also repeatability of individual consumer judgements. Results indicated similarities and differences between the three data collection procedures. It was found that consumers across the panels were more sure of what they did not like, than what they did. Using the coefficient of variation to measure the repeatability of consumer judgements revealed that, overall, consumers were reliable over a short period of time, with likelihood of purchase proving more variable than the appearance, flavour and texture acceptability measures. The sample that provided the most consistent judgements was the same for both the home and hall test, and was one of the most liked samples. The least consistent samples were also the least liked samples overall. As expected, the likelihood of purchase measurements were less consistent overall than the other acceptability attributes. The preference maps derived from each method, while showing some similarities, would lead to different conclusions, and potentially different decisions being taken in a commercial environment.