Background: Endobronchial ultrasound-guided transbronchial needle aspiration (EBUS-TBNA) is an accurate procedure to sample mediastinal tissue. Rapid on-site cytologic evaluation (ROSE) has been advocated to improve the performance of this procedure, but its benefit remains controversial. Our objective is to assess the utility of ROSE for EBUS-TBNA diagnostic accuracy among unselected patients. Methods: We prospectively collected data from all consecutive EBUS-TBNA procedures performed between 2008 and 2014. ROSE was introduced since 2011 in our daily practice. The accuracy of EBUS-TBNA with and without ROSE was compared in a univariate and multivariate model accounting for confounding factors. The impact of ROSE was then analyzed according to the etiology and size of the lesions. Results: Among 348 EBUS-TBNA procedures analyzed, 213 were performed with ROSE. The overall accuracy tended to be better with ROSE than without (90.6 vs. 84.4%; p = 0.082). After adjustment in a multivariate model, the benefit of ROSE still did not reach statistical significance (adjusted odds ratio 1.86; 95% confidence interval 0.79–4.41). Similar results were obtained in subgroups of patients with malignant disease or sarcoidosis. The size of the lesion did not influence the impact of ROSE on accuracy. Conclusions: ROSE was associated with a moderate increase in the accuracy of EBUS-TBNA, but the difference was not statistically significant. The same effect of ROSE was observed in malignant and nonmalignant lesions and this effect was not influenced by the lesion’s size.