For realizing the real-time monitoring of organic thin film preparation process in vacuum environment, the present paper proposes a high precision measurement approach based on differential reflectance spectroscopy (DRS). An optical system was constructed with off the shelf optical components, such as off-axis parabolic mirror, optical flat and optical fiber. A differential algorithm was employed to analyze the spectral signals. Based on the homebuilt setup, instability induced by variation of temperature was investigated. It was concluded that with the good control of temperature and air flow, the measurement repeatability of this system is better than 2 per thousand for a long-term period. Furthermore, an initial stage of organic thin film growth of pentacene molecules on the surface of Au was studied. As compared with the data of film thickness gauge and atomic force microscope, DR spectra accurately recorded the fine optical evolution with sub-monolayer resolution, which is related to the growth of the thin film. As a result, the DR optical system exhibits characteristics of broad spectrum (range from 300 to 820 nm), high stability (repeatability better than 2X 10(-3)), and high precision (sub-monolayer resolution) after efforts were done to decrease the influences on the spectral quality produced by misalignments of the optical components, the defects of the optics, and the disturbances of the environmental conditions. It is indicated that the proposed DR method is suitable for real-time online monitoring of thin film growth with high precision.