Background: Physical activity has been shown to reduce side-effects of breast cancer and its treatment. As physical activity levels of patients with breast cancer are largely unknown, we investigated these levels and compared them to women from the general population. Methods: In this prospective cohort study, physical activity levels of women with breast cancer participating in the UMBRELLA cohort were assessed at radiotherapy intake and thereafter every 6 months up to 3 years with the SQUASH questionnaire, which was also used in a random sample of the Dutch population. We compared physical activity levels (no activity, low, moderate or high levels of sports, leisure time or total activity) between patients and the Dutch female population using multinomial logistic regression. Standardized Prevalence Ratios (SPR) were calculated to compare adherence to Dutch physical activity guidelines. Results: Women with breast cancer (n baseline = 1655, n 6 months = 1414, n 12 months = 1186, n 18 months = 957, n 24 months = 744, n 30 months = 555, and n 36 months = 377) were less likely to spend time in physical activity compared to the general population (n = 11,710) until 3 years post-diagnosis, especially after 6 months (OR high-vs.-no activity = 0.34, 95% CI: 0.28–0.41). From 12 months onwards, patients were more likely to perform sports compared to the general population, especially patients who underwent systemic therapy. Guideline adherence was significantly lower in patients at baseline and 6 months (SPR baseline = 89, 95% CI: 82–97; SPR 6 months = 88, 95% CI: 81–96), and comparable to the general population at 12–36 months, especially in older women. Conclusions: Physical activity levels in women with breast cancer during and after treatment were lower compared to the Dutch female population. Three years post-treatment, they were still less physically active, although they spend more time in sport activities. As about half of the patients did not perform any sports, physical activity needs to be stimulated during ...