Tumor-induced immune suppression has mainly been studied in patients with advanced cancer. Despite the fact that they are most likely to benefit from immunotherapy, patients with early stage cancers were under-represented in these studies. We analyzed blood and tumor-derived T cells from patients with stage 1 (n = 20), stage 2 (n = 23) or stage 3 (n = 1) breast cancer and found that, even early stage tumors induced T cell differentiation. Breast cancer patients had significantly more circulating CD8+ memory and fewer CD8+ naïve T cells than healthy controls (n = 10). Up-regulation of CD69 and PD1 on cancer patient T cells suggests previous activation, and increased expression of the chemokine receptors CCR5 and CXCR3 on CD8+ T cells indicates that their homing capacity differs from that of healthy individuals. Comparison of blood-derived and tumor-associated T cells from patients with different metastatic status and tumor grades revealed that tumor progression and aggressiveness seem to favor the expansion of memory T cells over naive T cells. We have previously shown that immunosuppression in this patient population is stronger in the tumor than in the blood. Here, we report signs of exhaustion, such as loss of CD28, on tumor-associated as compared to blood-derived CD8+ T cells, despite the fact that tumor-associated T cells are predominantly effector memory cells and express high levels of CD69. The finding that the presence of a tumor potentially induces immunosenescence early during tumorigenesis indicates that efficient immunotherapy might be difficult even in patients with early stage cancer due to T cell exhaustion and tolerance.