As one of the major methods for location positioning, angle-of-arrival (AOA) estimation is a significant technology in radar, sonar, radio astronomy, and mobile communications. AOA measurements can be exploited to locate mobile units, enhance communication efficiency and network capacity, and support location-aided routing, dynamic network management, and many location-based services. In this paper, we propose an algorithm for AOA estimation in colored noise fields and harsh application scenarios. By modeling the unknown noise covariance as a linear combination of known weighting matrices, a maximum likelihood (ML) criterion is established, and a particle swarm optimization (PSO) paradigm is designed to optimize the cost function. Simulation results demonstrate that the paired estimator PSO-ML significantly outperforms other popular techniques and produces superior AOA estimates.