The energy restriction (ER)-colorectal cancer (CRC) association is inconsistent in literature. To strengthen the biological plausibility of the ER-CRC association, we investigated whether genetic variation in the insulin-like growth factor (IGF) pathway, a putative underlying mechanism, modulated this association in the Netherlands Cohort Study. Participants completed a questionnaire (n = 120,852) and provided toenail clippings for DNA (similar to 75%) at baseline. Individuals living in a Western city during the Hunger Winter (1944-45) or Western rural versus non-Western area were exposed to (severe) ER at young age. Genotyping was performed for 3,768 subcohort members and 2,580 CRC cases (case-cohort with 16.3 years follow-up). Cox hazard ratios for CRC were estimated across combined categories of ER and a genetic sum score of unfavorable alleles based on 18 single nucleotide polymorphisms in IGF-related genes and ER and an IGF1 19-CA repeat polymorphism. The reference included ER exposed individuals, so that increased hazard ratios were expected in higher combined categories for calculating relative excess risks due to interaction (additive interactions). Wald tests for multiplicative interactions were also performed. Multiplicative and additive interactions were nonsignificant. Combined ER-genetic sum score categories showed increasing CRC risks in men, but confidence intervals were wide. Women carrying two variant IGF1 19-CA repeat alleles versus those carrying two wild type IGF1 19-CA repeat alleles were at an similar to 50% decreased CRC risk, irrespective of ER exposure. In conclusion, data indicate that the IGF pathway might be involved in the ER-CRC association in men, but not women, although interactions were nonsignificant, hampering definite conclusions.