Background Multiple studies have shown that the exercise electrocardiogram (ECG) is less accurate for predicting ischemia, especially in women, and there is additional evidence to suggest that heart size may affect its diagnostic accuracy. Hypothesis The purpose of this investigation was to assess the diagnostic accuracy of the exercise ECG based on heart size. Methods We evaluated 1,011 consecutive patients who were referred for an exercise nuclear stress test. Patients were divided into two groups: small heart size defined as left ventricular end diastolic volume (LVEDV) <65 mL (Group A) and normal heart size defined as LVEDV ≥65 mL (Group B) and associations between ECG outcome (false positive vs. no false positive) and heart size (small vs. normal) were analyzed using the Chi square test for independence, with a Yates continuity correction. LVEDV calculations were performed via a computer-processing algorithm. SPECT myocardial perfusion imaging was used as the gold standard for the presence of coronary artery disease (CAD). Results Small heart size was found in 142 patients, 123 female and 19 male patients. There was a significant association between ECG outcome and heart size (χ2 = 4.7, p = 0.03), where smaller hearts were associated with a significantly greater number of false positives. Conclusions This study suggests a possible explanation for the poor diagnostic accuracy of exercise stress testing, especially in women, as the overwhelming majority of patients with small heart size were women.