We propose a vision-based localization algorithm with multiswarm particle swarm optimization for driving an autonomous vehicle. With stereo vision, the vehicle can be localized within a 3D point cloud map using the particle swarm optimization. For vehicle localization, the GPS (global positioning system)-based algorithms are often affected by the certain conditions resulting in intermittent missing signal. We address this issue in vehicle localization by using stereo vision in addition to the GPS information. The depth-based localization is formulated as an optimization-based tracking problem. Virtual depth images generated from the point cloud are matched with the online stereo depth images using the particle swarm optimization. The virtual depth images are generated from the point cloud using a series of coordinate transforms. We propose a novel computationally efficient tracker, i.e., a multiswarm particle swarm optimization-based algorithm. The tracker is initialized with GPS information and employs a Kalman filter in the bootstrapping phase. The Kalman filter stabilizes the GPS information in this phase and, subsequently, initializes the online tracker. The proposed localization algorithm is validated with acquired datasets from driving tests. A detailed comparative and parametric analysis is conducted in the experiments. The experimental results demonstrate the effectiveness and robustness of the proposed algorithm for vehicle localization, which advances the state of the art for autonomous driving.