Abstract In order to decrease the Bit Error Rate (BER) and increase coupling efficiency of free space optical (FSO) communication system, we compute and compensate the wavefront aberrations of the laser signal with the wavefront measurement method based on the focal plane proposed in this paper. The aberrations of laser signal are calculated based on the two images which are respectively generated from focal plane (PF) channel and the defocused plane (DF) channel. Assuming the On–Off Keying (OOK) modulation is used under the condition of weak turbulence, the fundamental of wavefront measurement method based on focal plane is introduced. By numerical simulation, we analyze the variation trend of coupling efficiency and BER, using the method we proposed to measure and compensate the wavefront aberrations for the FSO communication system. The experiment results show that the wavefront measurement error can reach nearly 0.004 wavelengths, the average coupling efficiency increases from 10.32% to 63.82%, and the average BER of FSO communication system decreases from 10−5 to 10−13.