A positive association between intelligence (IQ) and height has been reported previously. It is generally assumed that this association reflects the effect of childhood environment on IQ, but there is still little research supporting directly this hypothesis. We studied the association between height and IQ in 209 Dutch twin pairs at the ages of 5, 7, 10 and 12 years, 208 twin pairs at 16 and 18 years of age and 567 twin pairs and their siblings in adulthood. The heritability of height was high in all cohorts and across all ages (a 2 = 0.93 − 0.96). In adulthood, heritability was also high for full‐scale IQ (FSIQ: a 2 = 0.83–0.84) and somewhat lower for verbal IQ (VIQ: a 2 = 0.66–0.84). In early childhood, the heritability was lower, and common environmental factors had a substantial effect on FSIQ and VIQ. A positive association of height and IQ was found in early childhood and adolescence. In adulthood, a correlation was found between height and FSIQ in young adulthood and between height and VIQ in middle age. All correlations could be ascribed to genetic factors influencing both height and IQ. Thus, these results show that the association between height and IQ should not be directly regarded as evidence for childhood living conditions affecting IQ, but the effect of genetic factors affecting independently or interacting with environmental factors should be considered as well.