Abstract Background Patients with panic disorder with agoraphobia (PDA) have clinical symptoms such as the fear of being outside or of open spaces from which escape would be difficult. Although recent neurobiological studies have suggested that fear conditioning and extinction are associated with PDA, no study has examined the possible structural abnormalities in patients with PDA. Methods This preliminary study compares the gray matter volume among patients with PDA, those with panic disorder without agoraphobia (PDW), and healthy controls (HC) using high-resolution 3.0T magnetic resonance imaging (MRI) with voxel-based morphometry (VBM). Results Compared with HC, patients with PDA showed decreased gray matter volume in their left medial orbitofrontal gyrus. However, differences were not found in the gray matter volumes of patients with PDW and whole panic disorder compared with HC. Conclusions These findings suggest that the phobic avoidance found in patients with PDA arise from abnormalities in the medial orbitofrontal cortex, which plays an important role in fear extinction. Future studies should investigate the neuroanatomical substrates of PDA and distinguish them from those of PDW.