We consider inverse problems for the coupled Einstein equations and the matter field equations on a 4-dimensional globally hyperbolic Lorentzian manifold (M, g). We give a positive answer to the question: Do the active measurements, done in a neighborhood U ⊂ M of a freely falling observed µ = µ([s − , s + ]), determine the conformal structure of the spacetime in the minimal causal diamond-type set V g = J + g (µ(s −)) ∩ J − g (µ(s +)) ⊂ M containing µ? More precisely, we consider the Einstein equations coupled with the scalar field equations and study the system Ein(g) = T , T = T (g, φ) + F 1 , and g φ − V ′ (φ) = F 2 , where the sources F = (F 1 , F 2) correspond to perturbations of the physical fields which we control. The sources F need to be such that the fields (g, φ, F) are solutions of this system and satisfy the conservation law ∇ j T jk = 0. Let (g, φ) be the background fields corresponding to the vanishing source F . We prove that the obser-vation of the solutions (g, φ) in the set U corresponding to sufficiently small sources F supported in U determine V g as a differentiable mani-fold and the conformal structure of the metric g in the domain V g . The methods developed here have potential to be applied to a large class of inverse problems for non-linear hyperbolic equations encountered e.g. in various practical imaging problems.