Abstract This paper presents a rate-distortion optimized MPEG-4 FGS enhancement-layer truncation scheme. Our objective is to minimize the quality variation within each frame when the last transmitted enhancement layer is truncated according to the available network bandwidth. By properly redistributing the bit-budget to the enhancement layer that will be truncated, we can raise the quality of the whole frame uniformly. To achieve this goal, we use a trellis-search approach to decide which “1” bits will be truncated from the enhancement layer for each block. An operational rate-distortion optimization scheme based on the Lagrange Multiplier algorithm is adopted to decide the truncation criteria. Our proposed method can improve the visual quality and reduce the intra-frame quality variation both subjectively and objectively. Compared to previously reported straightforward truncation methods, our approach reduces the intra-frame quality variation significantly, and the decoded visual quality in terms of PSNR is also improved.