Abstract This paper focused on the investigation of colloidal fouling of nanofiltration (NF) membranes during the application of such a NF process to the treatment of a new type of semiconductor indium phosphide (InP) wastewater. The permeate flux behaviors of two types of NF membranes indicated that the densely structured ES10 membrane had a higher permeate flux than that of the loosely structured NTR7450 membrane. Investigation of their membrane surface morphology using atomic force microscopy (AFM) revealed that it could be due to the different surface roughness of the two membranes. The rough membrane surface had a larger effective membrane area than that of the smooth membrane surface. Permeate flux reduction due to deposition of InP particles at the membrane surface was found mainly reversible; however, the dissolved indium solids may interact with the NF membrane surface and cause irreversible fouling. The scanning electron microscope (SEM) images confirmed that there existed two different fouling layers at the membrane surface. Increased total suspended solids (TSS) concentration of InP particles in the wastewater had a negative impact on permeate flux, while particle size distribution (PSD) changes showed an insignificant impact on initial membrane fouling. The understanding of the fouling mechanisms in this NF process provides a better insight of engineering controls of the wastewater recycling system.