The instanton-noninstanton (I-NI) transition in the tunneling process, which has been numerically observed in classically nonintegrable quantum maps, can be described by a perturbation theory based on an integrable Hamiltonian renormalized so as to incorporate the integrable part of the map. The renormalized perturbation theory is successfully applied to the two quantum maps, the H\'enon and standard maps. In spite of different nature of tunneling in the two systems, the I-NI transition exhibits very common characteristics. In particular, the manifestation of I-NI transition is obviously explained by a remarkable quenching of the renormalized transition matrix element. The enhancement of tunneling probability after the transition can be understood as a sudden change of the tunneling mechanism from the instanton to quite a different mechanism supported by classical flows just outside of the stable-unstable manifolds of the saddle on the top of the potential barrier.