Abstract Numerical investigations were carried out for natural and mixed convection within domains with stationary and rotating cylinder by using an immersed-boundary method. The method was first validated with flows induced by natural convection in the annulus between concentric circular cylinder and square enclosure. Steady mixed convection in a square enclosure with an active rotating cylinder was further investigated for different rotating speeds. The parameters investigated in the study included Rayleigh number, Prandtl number and the aspect ratio between inner cylinder and outer enclosure. The heat transfer quantities of the system were obtained for different Rayleigh numbers (Ra) within the range of 104–106. The influence of rotation on the instability of the flow at different Prandtl numbers within the range of 0.07–7.0 was also investigated. Local and average heat transfer characteristics were fully studied around the surfaces of both inner cylinder and outer enclosure.