Abstract Molecular weight, cross-flow velocity, and transmembrane pressure were considered among the main factors affecting the rejection of target compounds and natural organic matter (NOM) by the nanofiltration (NF) process. The mass transfer coefficient was derived from MW and used to interpret the rejection of the target compounds so as to elucidate the rejection contributed by size exclusion of NOM during the NF process. Results showed that increasing the cross-flow velocity or the transmembrane pressure results in an increase in the Sherwood and Peclect numbers, thereby enhancing solute and water transport through the membrane pores. The experimental results showed that increasing the cross-flow velocity or transmembrane pressure could improve rejection, but the error between experimental result and the estimated value increased for larger target compounds due to the neglect of adsorption effect. For the removal of NOM under high cross-flow velocity, rejection had already reached its maximum; further increase in transmembrane pressure did not increase the rejection.