Abstract This paper presents a novel algorithm for evaluating the circularity of a mechanical part by using measurement points obtained with a coordinate measuring machine (CMM). Following the minimum zone criterion set forth in the current ANSI and ISO standards, evaluation of circularity is formulated as a non-differentiable unconstrained optimization problem, and based on the geometric representation of the necessary and sufficient condition for the optimal solution, an efficient steepest descent optimization procedure is proposed to find the circularity value. The steepest descent direction is determined by the method of calculating the minimum translational distance between two convex polygons, which is initially introduced in the field of robot path planning, and the length of the moving step is exactly determined by a presented geometrical method. A computational geometry-based method for pre-processing the measured data is also proposed. In comparison with existing methods, this algorithm has the advantages of computational efficiency and high precision. Simulations and practical example confirm the validity of the presented algorithm.