A model of the AFM (atomic force microscope) with certain tip-nanostructure interaction, cantilever elasticity and damping of its oscillations is proposed. Stable and unstable motion of the AFM tip interacting with the graphene sheet is investigated by the Lyapunov exponent computation. In our approximation, a hundred Si atoms (top of the AFM tip) interact with C atoms of the nanostructure. Тhis interaction is described by Lennard-Jones potential, and the distance between the top and the center of the cantilever mass is a constant. Complex influence of the initial tip-nanostructure distance and nanostructure size on stability has been examined. We discuss a possible new mode of the AFM operation based on the Lyapunov exponent computation. Maxima and minima of the Lyapunov exponent show where certain parts of the elementary cells are placed.