Abstract This paper presents a kind of novel binary code compression algorithm to reduce the size of the signal processing circuit and to realize real time compensation for the modulation error in fiber optic gyroscope (FOG). The disadvantage of general compensation method is figured out firstly. The modulation error is modeled and analyzed with taking the closed-loop characteristics into consideration. The analysis indicates the condition in which the error compensation is effective and real time. Then the compensation algorithm based on binary code compression operation is described. The functional design of the algorithm is realized in a single field programmable gate array (FPGA) by using modular design method, substituting some harewares in signal processing circuit, and system speed is very fast. The simulation and experiment results suggest that the algorithm and implementation is valid. The modulation error is real time compensated in every error detection cycle, and the error compensation resolution is enhanced from 2 −10 to 2 −16 rad/V.