Abstract Background The accuracy of self-assessments has not been well-supported in the literature. This study was undertaken to examine the validity of medical students’ ratings of their proficiency during encounters with simulated patients and simulation devices. Methods Confidential self-assessments for 10 skills were collected from 195 students during a formal clinical skills assessment related to 3 cases at the end of a surgery clerkship. The cases required students to gather data from simulated patients and perform procedures such as rectal examinations, naso-gastric tube insertions, and suturing on bench simulation models. The patients were trained to assess student performance. Results There were significant differences between student self-assessments and simulated patient scores for general clinical skills as opposed to procedural skills. Students’ mean self-assessments in the data gathering and interpersonal skills were 2 to 6 percentage points higher than ratings of their proficiency by simulated patients. However, self-assessments on procedures were 5 to 8 points lower than patient ratings. The median correlation between self-assessments and patient ratings for general clinical skills such as data gathering and interpersonal skills was 0.08 (NS), whereas the median correlation between student and patient ratings in procedures was 0.22 (p<.01). Conclusions Third year medical students’ self-assessments for specific procedures are more valid than self-assessments of general clinical skills. Students are less confident in their procedural skills compared to general clinical skills. While self-assessments should not be used as the sole measure of performance in clinical simulations, self-assessments for specific procedures can provide supplemental information on proficiency.