Computer vision is forever expanding into new areas, especially in the field of inspection. The work presented here shows how computer vision has the potential to improve the exterior inspection of large buildings. A robot is being developed at City University, London that will perform a number of tests and the initial use of computer vision is to locate the robot thus enabling the position of building defects to be recorded more reliably than at present. A system has been developed that allows the inspector to tag just four points on the building, seen through a camera, to points on a CAD diagram of that building. With that information, a mapping can be found, so that for any new point in the image, and in particular that of the robot, its position on the building can be found. Work has progressed to automate the process such that now the computer uses a number of image processing techniques to extract key features of the building scene. These features are compared and matched to a CAD diagram, from which the mapping between the image and diagram is produced. This then allows the true position of the robot to be found. Results are given for a model building and robot demonstrating the accuracy of the algorithm, together with some results from residential flats, concluding that computer vision can indeed help building inspection.