We propose and test a tool to evaluate and compare EMG signal decomposition algorithms. A model for the generation of synthetic intra-muscular EMG signals, previously described, has been used to obtain reference decomposition results. In order to evaluate the performance of decomposition algorithms it is necessary to define indexes which give a compact but complete indication about the quality of the decomposition. The indexes given by traditional detection theory are in this paper adapted to the multi-class EMG problem. Moreover, indexes related to model parameters are also introduced. It is possible in this way to compare the sensitivity of an algorithm to different signal features. An example application of the technique is presented by comparing the results obtained from a set of synthetic signals decomposed by expert operators having no information about the signal features using two different algorithms. The technique seems to be appropriate for evaluating decomposition performance and constitutes a useful tool for EMG signal researchers to identify the algorithm most appropriate for their needs.