Abstract Since the high- T c superconductor (HTSC) was discovered in 1986, research in this field has always been a focus of condensed matter physics. Some physical characteristics of the HTSC, a type II anisotropic superconductor, are greatly determined by vortex line statistical mechanics. The three major factors for deciding the characteristics of vortex line systems are the important thermal fluctuation, the anisotropy, and the disorder. A currently predominant framework in probe of this problem is the XY spin model obtained from the “phase only” simplification of the complex superconductor order parameter. In our research, we used the frustrated XY (FXY) spin model for studying the HTSC, where the frustration is produced by an external transverse magnetic field. As the HTSC has a quasi two-dimensional (2D) anisotropic structure, we studied the 2D XY spin model. We adopted the filling factor f=1/3 which we regard as representative. Some extensive Monte Carlo simulations with finite-size scaling analysis strongly suggest that the critical exponents of the phase transition are those given by the pure 2D Ising model. The fluctuation of domain walls of vortex determines the nature of the phase transition.