The extent to which verbal (VM) and visuospatial memory (VSM) tests measure the same or multiple constructs is unclear. Likewise the relationship between VM and VSM across development is not known. These questions are addressed using genetically informative data, studying two age cohorts (young adults and children) of twins and siblings. VM and VSM were measured in the working memory and short-term memory domain. Multivariate genetic analyses revealed that two highly correlated common genetic factors, one for VM and one for VSM, gave the best description of the covariance structure among the measures. Only in children, specific genetic factors were also present. This led to the following conclusions: In children, one genetic factor is responsible for linking VM and VSM. Specific genetic factors create differences between these two domains. During the course of development, the influence of genetic factors unique to each of these domains disappears and the genetic factor develops into two highly correlated factors, which are specific to VM and VSM respectively. At the environmental level, in both age cohorts, environmental factors create differences between these domains.