We present the results of deep and high-resolution (FWHM ~ 0".35) JHK NIR observations with the Subaru telescope, to search for very low mass young stellar objects (YSOs) in the W3 Main star-forming region. The NIR survey covers an area of ~ 2.6 arcmin^2 with 10-sigma limiting magnitude exceeding 20 mag in the JHK bands. The survey is sensitive enough to provide unprecedented details in W3 IRS 5 region and reveals a census of the stellar population down to objects below the hydrogen-burning limit. We construct JHK color-color (CC) and J-H/J and H-K/K color-magnitude (CM) diagrams to identify very low luminosity YSOs and to estimate their masses. Based on these CC and CM diagrams, we identified a rich population of embedded YSO candidates with infrared excesses (Class I and Class II), associated with the W3 Main region. A large number of red sources (H-K > 2) have also been detected around W3 Main. We argue that these red stars are most probably pre-main-sequence (PMS) stars with intrinsic color excesses. Based on the comparison between theoretical evolutionary models of very low-mass PMS objects with the observed CM diagram, we find there exists a substantial substellar population in the observed region. The mass function (MF) does not show the presence of cutoff and sharp turnover around the substellar limit, at least at the hydrogen-burning limit. Furthermore, the MF slope indicates that the number ratio of young brown dwarfs and hydrogen-burning stars in the W3 Main is probably higher than those in Trapezium and IC 348. The presence of mass segregation, in the sense that relatively massive YSOs lie near the cluster center, is seen. The estimated dynamical evolution time indicates that the observed mass segregation in the W3 Main may be the imprint of the star formation process.