Temporal lobe epilepsy (TLE) can be a progressive disorder, potentially resulting in structural damage and a decline of cognitive abilities over time. This is particularly evident in cases that are refractory to medication. This review examines the changes that occur in refractory TLE over time and the factors associated with these changes. Imaging and histologic studies on the brains of patients with TLE reveal anatomic and metabolic changes associated with continued seizures. These changes can impair cognitive and behavioral function. Seizure control may help minimize or prevent these changes.