Orientation selectivity (OS) of visual cortical neurons is progressively sharpened during development. However, synaptic circuit mechanisms underlying the OS sharpening remain unclear. In the current study, in vivo whole-cell voltage-clamp recordings from layer 4 excitatory neurons in the developing mouse primary visual cortex revealed changes of orientation tuning profiles of their excitatory and inhibitory inputs during a post-eye-opening period when OS of their spiking responses becomes sharpened. In addition to a parallel strengthening of excitation and inhibition during this developmental period, the orientation tuning of excitatory inputs keeps relatively constant, whereas the tuning of inhibitory inputs is broadened, and becomes significantly broader than that of excitatory inputs. Neuron modeling and dynamic-clamp recording demonstrated that this developmental broadening of the inhibitory tuning is sufficient for sharpening OS. Depriving visual experience by dark rearing impedes the normal developmental strengthening of excitation, but a similar broadening of inhibitory tuning, likely caused by a nonselective strengthening of inhibitory connections, results in the apparently normal OS sharpening in excitatory neurons. Our results thus provide the first demonstration that an inhibitory synaptic mechanism can primarily mediate the functional refinement of cortical neurons.