Structural brain changes that occur during development and ageing are related to mental health and general cognitive functioning. Individuals differ in the extent to which their brain volumes change over time, but whether these differences can be attributed to differences in their genotypes has not been widely studied. Here we estimate heritability (h(2) ) of changes in global and subcortical brain volumes in five longitudinal twin cohorts from across the world and in different stages of the lifespan (N = 861). Heritability estimates of brain changes were significant and ranged from 16% (caudate) to 42% (cerebellar gray matter) for all global and most subcortical volumes (with the exception of thalamus and pallidum). Heritability estimates of change rates were generally higher in adults than in children suggesting an increasing influence of genetic factors explaining individual differences in brain structural changes with age. In children, environmental influences in part explained individual differences in developmental changes in brain structure. Multivariate genetic modeling showed that genetic influences of change rates and baseline volume significantly overlapped for many structures. The genetic influences explaining individual differences in the change rate for cerebellum, cerebellar gray matter and lateral ventricles were independent of the genetic influences explaining differences in their baseline volumes. These results imply the existence of genetic variants that are specific for brain plasticity, rather than brain volume itself. Identifying these genes may increase our understanding of brain development and ageing and possibly have implications for diseases that are characterized by deviant developmental trajectories of brain structure. Hum Brain Mapp, 2017. © 2017 Wiley Periodicals, Inc.