We theoretically study a one-dimensional quasi-periodic Fermi system with topological $p$-wave superfluidity, which can be deduced from a topologically non-trivial tight-binding model on the square lattice in a uniform magnetic field and subject to a non-Abelian gauge field. The system may be regarded a non-Abelian generalization of the well-known Aubry-Andr\'e-Harper model. We investigate its phase diagram as functions of the strength of the quasi-disorder and the amplitude of the $p$-wave order parameter, through a number of numerical investigations, including a multifractal analysis. There are four distinct phases separated by three critical lines, i.e., two phases with all extended wave-functions (I and IV), a topologically trivial phase (II) with all localized wave-functions and a critical phase (III) with all multifractal wave-functions. The phase I is related to the phase IV by duality. It also seems to be related to the phase II by duality. Our proposed phase diagram may be observable in current cold-atom experiments, in view of simulating non-Abelian gauge fields and topological insulators/superfluids with ultracold atoms.