Purpose: Temporal lobe epilepsy (TLE) has been considered to impair long-term memory, whilst not affecting working memory, but recent evidence suggests that working memory is compromised. Functional MRI (fMRI) studies demonstrate that working memory involves a bilateral frontoparietal network the activation of which is disrupted in hippocampal sclerosis (HS). A specific role of the hippocampus to deactivate during working memory has been proposed with this mechanism faulty in patients with HS. Structural correlates of disrupted working memory in HS have not been explored. Methods: We studied 54 individuals with medically refractory TLE and unilateral HS (29 left) and 28 healthy controls. Subjects underwent 3T structural MRI, a visuospatial n-back fMRI paradigm and diffusion tensor imaging (DTI). Working memory capacity assessed by three span tasks (digit span backwards, gesture span, motor sequences) was combined with performance in the visuospatial paradigm to give a global working memory measure. Gray and white matter changes were investigated using voxel-based morphometry and voxel-based analysis of DTI, respectively. Key Findings: Individuals with left or right HS performed less well than healthy controls on all measures of working memory. fMRI demonstrated a bilateral frontoparietal network during the working memory task with reduced activation of the right parietal lobe in both patient groups. In left HS, gray matter loss was seen in the ipsilateral hippocampus and parietal lobe, with maintenance of the gray matter volume of the contralateral parietal lobe associated with better performance. White matter integrity within the frontoparietal network, in particular the superior longitudinal fasciculus and cingulum, and the contralateral temporal lobe, was associated with working memory performance. In right HS, gray matter loss was also seen in the ipsilateral hippocampus and parietal lobe. Working memory performance correlated with the gray matter volume of both frontal lobes and white matter integrity within the frontoparietal network and contralateral temporal lobe. Significance: Our data provide further evidence that working memory is disrupted in HS and impaired integrity of both gray and white matter is seen in functionally relevant areas. We suggest this forms the structural basis of the impairment of working memory, indicating widespread and functionally significant structural changes in patients with apparently isolated HS.