Fourier Transform InfraRed (FTIR) spectroscopy coupled to microscopy (IR imaging) has shown unique advantages in detecting morphological and molecular pathologic alterations in biological tissues. The aim of this study was to evaluate the potential of IR imaging as a diagnostic tool to identify characteristics of breast epithelial cells and the stroma. In this study a total of 19 breast tissue samples were obtained from 13 patients. For 6 of the patients, we also obtained Non-Adjacent Non-Tumor tissue samples. Infrared images were recorded on the main cell/tissue types identified in all breast tissue samples. Unsupervised Principal Component Analyses and supervised Partial Least Square Discriminant Analyses (PLS-DA) were used to discriminate spectra. Leave-one-out cross-validation was used to evaluate the performance of PLS-DA models. Our results show that IR imaging coupled with PLS-DA can efficiently identify the main cell types present in FFPE breast tissue sections, i.e. epithelial cells, lymphocytes, connective tissue, vascular tissue and erythrocytes. A second PLS-DA model could distinguish normal and tumor breast epithelial cells in the breast tissue sections. A patient-specific model reached particularly high sensitivity, specificity and MCC rates. Finally, we showed that the stroma located close or at distance from the tumor exhibits distinct spectral characteristics. In conclusion FTIR imaging combined with computational algorithms could be an accurate, rapid and objective tool to identify/quantify breast epithelial cells and differentiate tumor from normal breast tissue as well as normal from tumor-associated stroma, paving the way to the establishment of a potential complementary tool to ensure safe tumor margins.