Objective: Specific survival estimates are needed for the increasing number of colorectal cancer (CRC) survivors. The aim of this population-based study was to determine conditional loss in expectation of life (LEL) due to CRC. Study Design and Setting: All surgically treated patients with CRC registered in the Netherlands Cancer Registry with stage I–III between 1990–2016, were included (N = 203,216). Estimates of conditional LEL were predicted using flexible parametric models and the total life years lost due to cancer were estimated. Results: LEL decreased with older age and patients with rectal cancer or higher disease stage had highest LEL. In 2010, LEL for sixty-year old male and female patients was 2 vs. 2, 4 vs. 4, and 7 vs. 8 years for colon cancer, and 2 vs. 2, 4 vs. 5 and 7 vs. 8 years for rectal cancer, respectively. Conditional LEL in patients with CRC decreased during follow-up. Patients with combined stage I–III colon and rectal cancer in 2010 lost an estimated 18,628 and 11,336 life years. Conclusion: This study quantified the impact of CRC on patient's life expectancy, both on individual and population level and demonstrated temporal improvements in CRC survival. These results provide meaningful information that can be used during follow-up.