As the optical communication technology advances, vortex beam with orbital angular momentum (OAM) has gained wide attention due to its potential to significantly increase the channel capacity. Under the influence of atmospheric turbulence, there are still challenging problems in the OAM multiplexing system. To the best of our knowledge, in this paper one multiple-input-multiple-output (MIMO) detection technology named K-best detection, is first applied to the OAM multiplexing system. Numerical simulation results indicate the proposed solution enhances the performance of the optical communication system compared with data-aided least mean square (DA-LMS) and minimum mean squared error (MMSE) detection. Furthermore, with MMSE sorted QR decomposition (MMSE-SQRD) preprocessing, the performance of K-best detection can be further improved. When Cn2 = 1×10−14, about 4.4 dB signal-to-noise ratio (SNR) gain can be obtained by preprocessing at k = 2 while the complexity is not significantly increased. Computational complexity is also analyzed in this paper, results show that K-best detection with winner path extension (WPE) algorithm can achieve 43% system complexity reduction, achieving a compromise between performance and complexity in K-best detection.