We study almost bi-paracontact structures on contact manifolds. We prove that if an almost bi-paracontact structure is defined on a contact manifold $(M,\eta)$, then under some natural assumptions of integrability, $M$ carries two transverse bi-Legendrian structures. Conversely, if two transverse bi-Legendrian structures are defined on a contact manifold, then $M$ admits an almost bi-paracontact structure. We define a canonical connection on an almost bi-paracontact manifold and we study its curvature properties, which resemble those of the Obata connection of an anti-hypercomplex (or complex-product) manifold. Further, we prove that any contact metric manifold whose Reeb vector field belongs to the $(\kappa,\mu)$-nullity distribution canonically carries an almost bi-paracontact structure and we apply the previous results to the theory of contact metric $(\kappa,\mu)$-spaces.