Phosphorus (P) crop fertilization requires optimal management to avoid the waste of a non-renewable resource and water pollution, but current methods for assessing soil phyto-available P and plant P requirements are not sufficiently precise to meet this goal. The objectives of the present study were to (1) evaluate the effect of long-term P fertilization on the grain yield of winter wheat, maize, and rapeseed, (2) validate or establish models of critical shoot P concentration (PC) based on relationships of shoot P concentration with either shoot biomass or shoot nitrogen (N) concentration, and (3) assess both plant-based and soil-based diagnostic tools for managing P fertilization. A long-term field experiment with contrasted P fertilizer treatments, established in 1971 by Agroscope in Changins (Switzerland), was used to measure the shoot biomass and P concentration of winter wheat in 2011, maize in 2012, and rapeseed in 2014 weekly during the growing period and the grain yield at harvest. Soil available P in the 0–0.20 m soil layer was assessed by three chemical extractions: ammonium acetate EDTA (P-AAE), sodium bicarbonate (P-NaHCO3), and CO2-saturated water (P-CO2). Long-term P fertilization increased soil available P extracted by P-CO2 (+ 24%), P-AAE (+ 200%), and P-NaHCO3 (+ 155%), shoot growth and grain yield by 8.4% and 26.2% for winter wheat and rapeseed respectively but had no effect on maize. The relationships between PC and shoot biomass or N concentration were described respectively by allometric and linear models (R2 > 0.85, n = 21, 28 and 32 for winter wheat, maize and rapeseed respectively; slope P values for linear models < 0.05). The PC–shoot N concentration model (slope: 0.083, intercept: 0.88) for winter wheat confirmed results from previous studies and can be used for calculating the P nutrition index. For the three soil available P indicators, threshold values needed to achieve 95% of the maximum yield for the three crops were less than those currently used in the official fertilization guidelines in Switzerland. Our results obtained after 44 years of contrasted P fertilization confirm the relationship between PC and shoot N concentration for grain crops and the need to revise P fertilizer recommendations based on currently used soil P tests.