This paper proposes an optimum radio-over-fiber (RoF) system design to extend the coverage of the third-generation partnership program (3GPP) long-term evolution (LTE) base station, i.e., eNodeB. The system is theoretically and experimentally demonstrated as the high-speed interface between eNodeB and a relay node. The LTE signals under test comprise three different modulation schemes, namely, quaternary phase-shift keying (QPSK), 16-quadratic-amplitude modulation (QAM), and 64-QAM, which are modulated onto orthogonal frequency-division multiplexing (OFDM) at 2.6 GHz. The RoF system design is based on the distributed feedback (DFB) laser direct modulation and direct detection receiver. The spurious-free dynamic range (SFDR) considering the third-order intermodulation analysis of the DFB laser achieved 1.93-dB dynamic range gain to improve the modulation efficiency. The practical investigation reveals three distinctive optical power transmission regions, namely, linear, intermixing, and nonlinear regions. The QPSK, 16-QAM, and 64-QAM systems in the intermixing region achieved error vector magnitudes (EVMs) of $sim$1.144%, $sim$1.2%, and $sim$ 1.21%, respectively, for 10-km transmission, whereas at 60 km, the achieved EVMs are $sim$5.86%, $sim$ 5.96%, and $sim$6.01%, respectively. The intermixing region for the 10–60-km transmission span achieved the most optimized EVM and within the 3GPP LTE limit of 8%. Additionally, we also demonstrate that nonlinear distortion proportionally increases with linear distortion as the transmission span increases.