Fogged surgical field visualization that is a common and potentially harmful problem can lead to inappropriate device use and incorrectly targeted tissue and increase surgical risks in endoscopic surgery. This paper aims to remove fog or smoke on endoscopic video sequences to augment and maintain a direct and clear visualization of the operating field. A new visibility-driven fusion defogging framework is proposed for surgical endoscopic video processing. This framework first recovers the visibility and enhances the contrast of hazy images. To address the color infidelity problem introduced by the visibility recovery, the luminances of the recovered and enhanced images are fused in the gradient domain, and the fused luminance is reconstructed by solving the Poisson equation in the frequency domain. The proposed method is evaluated on clinical videos that were collected from prostate cancer surgery. The experimental results demonstrate that the proposed framework defogs endoscopic images more robustly than currently available methods. Additionally, our method also provides an effective way to improve the visual quality of medical or high-dynamic range images.