Abstract In this paper, an efficient self-adaptive model for chaotic image encryption algorithm is proposed. With the help of the classical structure of permutation-diffusion and double simple two-dimensional chaotic systems, an efficient and fast encryption algorithm is designed. However, different from most of the existing methods which are found insecure upon chosen-plaintext or known-plaintext attack in the process of permutation or diffusion, the keystream generated in both operations of our method is dependent on the plain-image. Therefore, different plain-images will have different keystreams in both processes even just only a bit is changed in the plain-image. This design can solve the problem of fixed chaotic sequence produced by the same initial conditions but for different images. Moreover, the operation speed is high because complex mathematical methods, such as Runge–Kutta method, of solving the high-dimensional partial differential equations are avoided. Numerical experiments show that the proposed self-adaptive method can well resist against chosen-plaintext and known-plaintext attacks, and has high security and efficiency.