Abstract The structure of first- and second-stage combustion is investigated in a heavy-duty, single-cylinder optical engine using chemiluminescence imaging, Mie-scatter imaging of liquid-fuel, and OH planar laser-induced fluorescence (OH-PLIF) along with calculations of fluorescence quenching. Three different diesel combustion modes are studied: conventional non-diluted high-temperature combustion (HTC) with either (1) short or (2) long ignition delay, and (3) highly diluted low-temperature combustion (LTC) with early fuel injection. For the short ignition delay HTC condition, the OH fluorescence images show that second-stage combustion occurs mainly on the fuel jet periphery in a thickness of about 1 mm. For the long ignition delay HTC condition, the second-stage combustion zone on the jet periphery is thicker (5–6 mm). For the early-injection LTC condition, the second-stage combustion is even thicker (20–25 mm) and occurs only in the down-stream regions of the jet. The relationship between OH concentration and OH-PLIF intensity over a range of equivalence ratios is estimated from quenching calculations using collider species concentrations predicted by chemical kinetics simulations of combustion. The calculations show that both OH concentration and OH-PLIF intensity peak near stoichiometric mixtures and fall by an order of magnitude or more for equivalence ratios less than 0.2–0.4 and greater than 1.4–1.6. Using the OH fluorescence quenching predictions together with OH-PLIF images, quantitative boundaries for mixing are established for the three engine combustion modes.