Subtropical soils are generally characterized by phosphorus (P), calcium (Ca) and magnesium (Mg) deficiency; extreme P-rich soils develop on phosphate rocks. How such contrasting geological-derived soils influence genetic diversity and structure of local plant populations have not been well documented, hindering our understanding of plant adaptive evolution in subtropical areas. In this study, we applied double digest restriction-site-associated DNA sequencing (ddRAD-seq) to investigate the genetic variation and differentiation of natural Quercus variabilis populations growing at geologically derived soil P-rich and P-deficient sites in subtropical China. Results showed that Q. variabilis populations had lower genetic diversity at P-rich sites than those at P-deficient sites, and genetic diversity was negatively correlated only with soil P content (p < 0.05) across these sites. The genetic variation mainly occurred within populations (95.76%) with a much smaller amount among populations (2.27%) and between the two P-type sites (1.97%). Moreover, significant genetic divergence (p < 0.0001) occurred between the two site-type populations. Among genes embedded in the selective sweep areas in genome for populations at P-rich sites and P-deficient sites, some were annotated as transcription factors, and some were involved in protein synthesis and degradation, involved in regulatory responses to P availability and other environmental stresses. These results suggest that the availability of soil P could be a key selective force driving adaptive genetic differentiation among Q. variabilis populations across variable P content soils in subtropical areas.