Purpose To evaluate the accuracy of Multi-detector row CT (MDCT) for the prediction of tumor invasion of the mesorectal fascia (MRF). Materials and methods A total of 35 patients with primary rectal cancer underwent preoperative staging magnetic resonance imaging (MRI) and MDCT. The tumor relationship to the MRF, expressed in 3 categories (1—tumor free MRF = tumor distance ≥ 1 mm; 2—threatened = distance < 1 mm; 3—invasion = distance 0 mm) was determined on CT by two observers at patient level and at different anatomical locations. A third expert reader evaluated the MRF tumor relationship on MRI, which served as reference standard. Receiver operating characteristic curves (ROC-curves) and areas under these curves (AUC) were calculated. The inter-observer agreement of CT was determined by using linear weighted kappa statistics. Results The AUC of CT for MRF invasion was 0.71 for observer 1 and 0.62 for observer 2. The inter-observer agreement was kappa = 0.34. The performance of CT at mid-high rectal levels was statistically significant better compared to low anterior (obs.1: AUC = 0.88 vs. 0.50; obs 2: AUC = 0.84 vs. 0.31; P ≤ 0.040). Conclusion Multi-detector row CT has a poor accuracy for predicting MRF invasion in low-anterior located tumors.The accuracy of CT significantly improves for tumors in the mid-high rectum. There is a high inconsistency among readers.