Perspective taking plays an important role in different areas of psychological and neuroscientific research. Visual perspective taking is an especially prominent approach generally using one of two experimental tasks: in the own-body-transformation task observers are asked to judge the laterality of a salient feature of a human figure (e.g., is the glove on the left or right hand?) from the figure’s perspective. In the avatar-in-scene task they decide about the laterality of objects in a scene (e.g., is the flower on the left or right?) from the avatar’s point of view. Increases in latencies and/or errors are interpreted as originating from additional cognitive processes predominately described as observer-based perspective transformations. A closer look reveals that such an account is disputable on grounds related to the use of laterality judgments. Other transformation accounts, i.e., object or array transformations, as well as non-transformational accounts, i.e., extra processing due to spatial conflicts, have not been adequately considered, tested, or ruled out by existing research. Our review examines visual perspective tasks in detail, identifies problems and makes recommendations for future research.