Abstract Oxy-fuel combustion is considered as one of the most promising technologies for carbon capture and storage (CCS). In this study, a commercial computational fluid dynamics (CFD) code has been employed for the simulation of an air-fired coal combustion and an oxy-fired coal combustion with recycled flue gas in a 1 MW th combustion test facility. Reynolds–averaged Navier–Stokes (RANS) solutions have been obtained for both cases. Results indicate that the CFD code with existing physical sub-models can provide a reasonable prediction for the air-fired combustion. However, the prediction for the oxy-fired case has not been as satisfactory as expected. In order to assess the impact of the turbulence treatment in CFD on the predictions, large eddy simulation (LES) has been performed for oxy-fired case and compared with the results from the RANS simulation and the available experimental data. Although the results suggest that LES can provide a more realistic prediction of the shape and the physical properties of the flame, there has not been significant improvement in the prediction of the temperature. In addition, the complexity of the problem requires more detailed experimental data for the validation of the LES. In order to improve the validity of numerical simulations for design purposes, further modelling improvements for oxy-coal combustion that are necessary for more accurate predictions are addressed. Based on this study, it is envisaged that the complexity in the oxy-coal combustion process requires more detailed analyses of the available physical sub-models.