Abstract The paper describes the concept of the Responsive Workbench (RW). This virtual environment was designed to support end users working on desks, workbenches, and tables as physicians, architects and scientists with an adequate human-machine interface. We attempt to construct a task-driven interface for this class of users by working in an interdisciplinary team from the beginning. The system is explained and evaluated along three medical applications: medical education, a cardiological tutorial with a simulation system for ultrasonographic examinations of the heart and surgery planning. Virtual objects are located on a real “workbench”. The objects, displayed as computer generated stereoscopic images are projected onto the surface of a table. The participants operate within a non-immersive virtual environment. A “guide” uses the virtual environment while several observers can also watch events by using shutter glasses. Depending on the application, various input and output modules can be integrated, such as motion, gesture and speech recognition systems which characterize the general trend away from the classical multimedia desktop interface. The RW is compared with other common virtual reality systems as head mounted displays. First experiences of the collaborators are drawn and future enhancements are proposed.