Abstract Background and purpose To derive planning organ at risk volume (PRV) margins for the rectum and to analyse the impact of such margins on rectum dose volume histograms (DVHs). Patients and methods Weekly repeat computer tomography (CT) scans of 19 bladder cancer patients acquired during a conformal radiotherapy course were registered with the corresponding planning CT scans. From these scans, the internal rectal motion was quantified, and the margins that had to be added to the rectum contour in the planning scan to encompass the observed span of rectum motion were determined. These margins were compared to the margins derived using a recent PRV margin recipe. To illustrate the impact of margins on rectum DVHs, the margins were applied in treatment plans of six prostate cancer patients. Results Altogether 141 CT scans were analysed. On average 24% of the repeat scan rectum volume was displaced outside the planning scan contours, and wall movements of up to 30 mm were observed. Margins of 16 mm anterior and 11 mm posterior encompassed all rectal motion except for the two most displaced rectum walls in each of these directions, in 89% of the patients. Using a recently published statistics-based recipe, margins of 6 mm anterior and 5 mm posterior accounted for the systematic rectum variation, i.e. the average wall position, in 90% of the patients. Adding anterior margin only caused consistent increases (up to 20%) in the fraction of the volume inside the high-dose region (40–70 Gy) compared to the DVH of rectum only. When using both anterior and posterior margins only small shifts (<5%) in the volume fractions were observed. Conclusions Rectum PRV margins of 5–6 mm will encompass the systematic component of rectum motion, while margins up to 16 mm are required to also account for most of the random variation. Use of anterior margins only caused large shifts in the DVHs in the clinically significant dose range, while only minor shifts were seen when using both anterior and posterior margins.