Abstract We use ultrahigh vacuum transmission electron microscopy (UHV-TEM) to study the growth of Ge on Si(001) in real time at different temperatures and for coverages ranging from the initial monolayers to the development and relaxation of 3D islands. During growth of the first monolayers the surface gradually changes from a disordered missing-dimer structure to a rather well ordered (2 × 8) reconstruction, an evolution clearly resolved by the TEM. As the coverage is increased 3D islands starts to form. The growth and relaxation of these islands are shown to depend significantly on the temperature, e.g. with different dislocations formed at high and low temperatures. We interpret this difference in terms of the brittle-ductile transition in Ge, below which dislocation glide is frozen out. An interesting observation is that islands grown at low temperatures are more fully relaxed than those grown at higher temperatures. At high enough temperature the islands are initially, up to a specific size, coherent with the substrate and further growth occurs in a remarkably oscillatory fashion with the introduction of each (60°-type) dislocation, where the core of the island, of about 2000 Å in diameter, remains fully strained. However, in the low-temperature regime the islands grow relaxed from the outset with pure edge dislocations continuously being introduced in the moving edges. For temperatures less than 600°C the transition from 2D to 3D growth occurs via the formation of small and strained 3D islands, so-called “hut clusters”. We monitor the nucleation and characteristics of these clusters and discuss their possible role in the formation of relaxed 3D islands. The different growth mechanisms are discussed in terms of a simple model for the energetics of strain-relaxed islands, leading to a qualitative description of the temperature-dependent growth modes.