A multi-level procedure is described in order to develop a total quality management survey tool in the field of engineering academia. As a first step a review of available evaluation tools for universities is conducted, resulting in over 150 items used for evaluation purposes. Secondly all dimensions of educational evaluation used in previous research are summarized, resulting in 15 dimensions. In a third step, items are assigned to the dimensions, overlapping items were combined or removed, and item content and dimensions were adjusted to the specific conditions of the target faculty. Fourthly, the resulting twelve dimensions were used in first, investigative interviews in the target population. Results indicate that eleven dimensions sufficiently mapped all aspects of evaluation. After revising the items to improve understanding in a fifth step cognitive pretests were conducted. The final revision resulted in 83 items assigned to eleven dimensions.