We study the gravitational self-force using the effective field theory formalism. We show that in the ultra-relativistic limit γ → ∞, with γ the boost factor, many simplifications arise. Drawing parallels with the large N limit in quantum field theory, we introduce the parameter 1/N ≡ 1/γ^2 and show that the effective action admits a well defined expansion in powers of λ ≡ N ϵ at each order in 1/N , where ϵ ≡ E_m/M and E_m = γm is the (kinetic) energy of the small mass. Moreover, we show that diagrams with nonlinear bulk interactions first enter at O (λ^2/N^2) and only diagrams with nonlinearities in the worldline couplings, which are significantly easier to compute, survive in the large N /ultra-relativistic limit. Finally, we derive the self-force to O (λ^4/N) and provide expressions for some conservative quantities for circular orbits.