Background Energy restriction during childhood and adolescence is suggested to lower colorectal cancer (CRC) risk. We investigated this in the Netherlands Cohort Study. Methods Information on diet and other risk factors was collected by a baseline questionnaire in 1986 when cohort members were 55–69 years of age ( n = 120 852). Three indicators of early life exposure to energy restriction were assessed: father’s employment status during the Economic Depression (1932–40), place of residence during Second World War years (1940–44) and the ‘Hunger Winter’ (1944–45), a severe famine. Using the case–cohort approach, incidence rate ratios (RRs) and 95% confidence intervals (CIs) were calculated for total colorectal, proximal colon, distal colon, rectosigmoid and rectal cancers, according to the three time periods of energy restriction. After 16.3 years of follow-up, 2573 cases were available for multivariate analyses. Results Men who lived in a western city during the Hunger Winter and therefore exposed to the highest degree of energy restriction, had a lower risk of developing CRC (RR: 0.81, 95% CI: 0.68–0.98), and tumours of the proximal colon (RR: 0.72, 95% CI: 0.54–0.96) and rectum (RR: 0.71, 95% CI: 0.53–0.96). In women, non-statistically significant inverse associations were observed for tumours of the distal colon, rectosigmoid and rectum. Inverse associations were also observed between the other two exposure times and studied endpoints, though not statistically significant. Conclusions This unique observational evidence suggests that severe energy restriction during childhood and adolescence may lower CRC risk, especially in men, thus providing insight regarding the role of energy intake during early life in CRC development.