Affordable Access

Seeing the whole picture: evaluating automated assessment systems

Publication Date


This paper argues that automated assessment systems can be useful for both students and educators provided that its results correspond well with human markers. Thus, evaluating such a system is crucial. We present an evaluation framework and show why it can be useful for both producers and consumers of automated assessment. The framework builds on previous work to analyse Latent Semantic Analysis- (LSA) based systems, a particular type of automated assessment, that produced a research taxonomy that could help developers publish their results in a format that is comprehensive, relatively compact, and useful to other researchers. The paper contends that, in order to see a complete picture of an automated assessment system, certain pieces must be emphasised. It presents the framework as a jigsaw puzzle whose pieces join together to form the whole picture and provides an example of the utility of the framework by presenting some empirical results from our assessment system that marks questions about html. Finally, the paper suggests that the framework is not limited to LSA-based systems. With slight modifications, it can be applied to any automated assessment system.

There are no comments yet on this publication. Be the first to share your thoughts.


Seen <100 times

More articles like this

Seeing the whole picture.

on Current Biology August 1991

Quantitative microscopy and systems biology: seein...

on Histochemistry and cell biolog... November 2008

Seeing the whole picture. A gift from the sea.

on Nursing New Zealand (Wellingto... February 1998
More articles like this..