Abstract The robust trajectory tracking problem for an eye-in-hand system is addressed in this paper. A novel visual feedback control model is proposed. It considers not only the uncertainties and disturbances in the robot model, but also the unknown camera parameters. By using sliding mode control, filter method and adaptive technique, the controller is designed such that the robot can track the desired trajectory well by using information provided by camera. Finally, stability and robustness are rigorously proved by using Lyapunov method. Computer simulations are presented to show the effectiveness of the proposed visual feedback controller.