Abstract A hybrid ray-tracing method is developed for the solution to the radiative transfer in a plane-parallel participating medium having one specular surface and another diffuse surface. By this method, radiative transfer coefficients (RTCs) for specular–diffuse (S–D) surfaces are deduced. The medium surfaces are considered to be semitransparent. The effects of convection–radiation parameter, conduction–radiation parameter and refractive index on transient coupled heat transfer are investigated. Results show that the temperature curves of the medium having S–D surfaces is higher than those of the medium having S–S surfaces (two specular surfaces); the total heat flux at steady state for the S–D surfaces is lower than that for the S–S surfaces.