Abstract Subsolidus phase equilibrium of CoO–In2O3–B2O3 system has been investigated mainly by solid-state reaction and powder X-ray diffraction method. A novel compound Co2InBO5 was found and synthesized for the first time. There are five definite three-phase regions and one solid solution Co3−XInXBO5 (0<X⩽1) in this system under present experimental conditions. This new compound Co2InBO5 and high temperature metastable phase Co3BO5 can form continuous solid solution at all relative concentrations. Crystal structure of three representative Co3−XInXBO5 solid solution compounds (X=0.4, 0.7, and 1) were refined and determined from X-ray powder diffraction data using Rietveld method. The results showed that the Co3−XInXBO5 (0<X⩽1) solid solution crystallized in the orthorhombic space group Pbam (No. 55) with the ludwigite structure type. The lattice parameters (a, b and c) of these solid solution compounds increase linearly when In replace Co atom gradually (i.e. with the increase of the value of X), which obeys the Vegard’s rule. The trigonal BO3 group in these Co3−XInXBO5 solid solution compounds was confirmed by their infrared absorption spectra.