We aimed to longitudinally assess the relationship between changing brain energy metabolism (glucose and acetoacetate) and cognition during healthy aging. Participants aged 71 ± 5 year underwent cognitive evaluation and quantitative positron emission tomography (PET) and magnetic resonance imaging (MRI) scans at baseline (N = 25) and two (N = 25) and four (N = 16) years later. During the follow-up, the rate constant for brain extraction of glucose (Kglc) declined by 6%–12% mainly in the temporo-parietal lobes and cingulate gyri (p ≤ 0.05), whereas brain acetoacetate extraction (Kacac) and utilization remained unchanged in all brain regions (p ≥ 0.06). Over the 4 years, cognitive results remained within the normal age range but an age-related decline was observed in processing speed. Kglc in the caudate was directly related to performance on several cognitive tests (r = +0.41 to +0.43, all p ≤ 0.04). Peripheral insulin resistance assessed by the homeostasis model assessment of insulin resistance (HOMA-IR) was significantly inversely related to Kglc in the thalamus (r = −0.44, p = 0.04) and in the caudate (r = −0.43, p = 0.05), and also inversely related to executive function, attention and processing speed (r = −0.45 to −0.53, all p ≤ 0.03). We confirm in a longitudinal setting that the age-related decline in Kglc is directly associated with declining performance on some tests of cognition but does not significantly affect Kacac.