Abstract The paper studies a recently developed evolutionary-based image encryption algorithm. A novel image encryption algorithm based on a hybrid model of deoxyribonucleic acid (DNA) masking, a genetic algorithm (GA) and a logistic map is proposed. This study uses DNA and logistic map functions to create the number of initial DNA masks and applies GA to determine the best mask for encryption. The significant advantage of this approach is improving the quality of DNA masks to obtain the best mask that is compatible with plain images. The experimental results and computer simulations both confirm that the proposed scheme not only demonstrates excellent encryption but also resists various typical attacks.