For a large class of dark energy (DE) models, for which the effective gravitational constant is a constant and there is no direct exchange of energy between DE and dark matter (DM), knowledge of the expansion history suffices to reconstruct the growth factor of linearized density perturbations in the non-relativistic matter component on scales much smaller than the Hubble distance. In this paper we develop a non-parametric method for extracting information about the perturbative growth factor from data pertaining to the luminosity or angular size distances. A comparison of the reconstructed density contrast with observations of large scale structure and gravitational lensing can help distinguish DE models such as the cosmological constant and quintessence from models based on modified gravity theories as well as models in which DE and DM are either unified, or interact directly. We show that for current SNe data, the linear growth factor at z = 0.3 can be constrained to 5%, and the linear growth rate to 6%. With future SNe data, such as expected from the JDEM mission, we may be able to constrain the growth factor to 2-3% and the growth rate to 3-4% at z = 0.3 with this unbiased, model-independent reconstruction method. For future BAO data which would deliver measurements of both the angular diameter distance and Hubble parameter, it should be possible to constrain the growth factor at z = 2.5 to 9%. These constraints grow tighter with the errors on the datasets. With a large quantity of data expected in the next few years, this method can emerge as a competitive tool for distinguishing between different models of dark energy.