Trace element contamination caused by mining is a serious environmental problem. The potential effects of exploiting the Yunfu pyrite mine (southern China) on soil were investigated by determining trace elements in 56 surface soil samples from the vicinity of the Yunfu pyrite mine. The samples were acid dissolved and measured by an inductively coupled plasma mass spectrometry (ICP-MS). Principal component analysis and hierarchical cluster analysis were used to identify factors influencing the trace element contents and possible sources of the trace elements. The degree of trace element pollution was determined using the geological accumulation index Igeo. Monte Carlo simulations were used to assess the health risks posed. The results show that (1) six factors (parent material, mining activities, ore composition, rainfall, terrain, and other inputs) strongly affected the trace element contents of the soil samples. (2) There were three groups of trace elements, according to their possible sources. One group (Cs, Ga, Ge, Hf, Nb, Rb, Ta, Th, Ti, U, and Zr) mainly originated in parent rocks. Another group (Cr, Ni, Sr, and V) was mainly supplied by industrial plants and traffic emissions. The third group (Ba, Co, Cu, Mn, Pb, and Zn) was mainly supplied through pyrite ore exploitation processes. (3) Some samples were slightly to moderately polluted with Cs, Ga, Ge, Nb, Rb, Ta, and Ti. Most samples were moderately to highly polluted with Ba, Co, Cu, Mn, Pb, and Zn. (4) Trace elements in soil pose strong non-carcinogenic and carcinogenic health risks to people (particularly children) living near the Yunfu pyrite mine.