We study the layered video transmission optimization problem in the cellular networks, in which all User Equipments (UEs) require the same video content simultaneously via the cellular downlink transmission from the Base Station (BS) and Device-to-Device (D2D) transmission from other UEs. First, we propose a probability-based framework for each video layer to measure the video playback quality in terms of the outage probability. Next, the layered video transmission optimization problem is formulated as a Peak Signal-to-Noise Ratio (PSNR) loss minimization problem, which is then converted to an unmatured probability minimization problem by adding a differentiation weight on each layer based on its importance to the playback quality. In addition, we prove the formulated problem to be a log-convex optimization problem. Finally, we conduct extensive simulations to show that the proposed approach achieves the optimal allocation of the broadcast duration for each layer under various network conditions.