Recent neuroimaging studies have shown the possibility of cognitive impairment after pontine stroke. In this study, we aimed to use voxel-mirrored homotopic connectivity (VMHC) to investigate changes in the cognitive function in chronic pontine stroke. Functional MRI (fMRI) and behavioral assessments of cognitive function were obtained from 56 patients with chronic pontine ischemic stroke [28 patients with left-sided pontine stroke (LP) and 28 patients with right-sided pontine stroke (RP)] and 35 matched healthy controls (HC). The one-way ANOVA test was performed for the three groups after the VMHC analysis. Results showed that there were significant decreases in the bilateral lingual gyrus (Lingual_L and Lingual_R) and the left precuneus (Precuneus_L) in patients with chronic pontine ischemic stroke compared to HCs. However, in a post-hoc multiple comparison test, this difference remained only between the HC and RP groups. Moreover, we explored the relationship between the decreased z-values in VMHC and the behavior-task scores using a Pearson's correlation test and found that both scores of short-term memory and long-term memory in the Rey Auditory Verbal Learning Test were positively correlated with z-values of the left lingual gyrus (Lingual_L), the right lingual gyrus (Lingual_R), and the left precuneus (Precuneus_L) in VMHC. Besides that, the z-values of Precuneus_L in VMHC were also negatively correlated with the reaction time for correct responses in the Flanker task and the spatial memory task. In conclusion, first, the lingual gyrus played an important role in verbal memory. Second, the precuneus influenced the working memory, both auditory-verbal memory and visual memory. Third, the right-sided stroke played a greater role in the results of this study. This study provides a basis for further elucidation of the characteristics and mechanisms of cognitive impairment after pontine stroke.