The fluxonium qubit has arisen as one of the most promising candidate devices for implementing quantum information in superconducting devices, since it is both insensitive to charge noise (like flux qubits) and insensitive to flux noise (like charge qubits). Here, we investigate the stability of the quantum information to quasiparticle tunneling through a Josephson junction. Microscopically, this dephasing is due to the dependence of the quasiparticle transmission probability on the qubit state. We argue that on a phenomenological level the dephasing mechanism can be understood as originating from heat currents, which are flowing in the device due to possible effective temperature gradients, and their sensitivity to the qubit state. The emerging dephasing time is found to be insensitive to the number of junctions with which the superinductance of the fluxonium qubit is realised. Furthermore, we find that the dephasing time increases quadratically with the shunt-inductance of the circuit which highlights the stability of the device to this dephasing mechanism.