We demonstrate quantitative phase delay measurements with a spiral phase contrast microscope working in confocal mode. Such a confocal configuration is sensitive to weak phase objects due to background rejection but does not give direct access to the phase delay introduced by the sample. We develop a theory showing that shifting the illumination spot relative to the detector gives access to the local phase gradient in the first-order approximation. Subsequently, we present an iterative integration algorithm for phase delay measurements. This approach is validated on simulated and calibrated experimental images. Finally, the algorithm is applied to measure the phase profile of a cell, in which phase delays of 10 mrad are observed.