Purpose: This study aimed to investigate changes in physical work demands in association with self-rated health and musculoskeletal symptoms. Methods: Data from five waves over the period 2019–2021 of the Netherlands Working Conditions Survey COVID-19 were available for 7191 participants aged 19–64 years who worked (partly) on-site during at least two consecutive waves. Logistic generalized estimated equations (GEE) were used to estimate the odds ratios (OR) with 95% confidence interval (CI) for changes (increase or decrease compared to no change) in physical work demands between two waves and poor self-rated health and musculoskeletal symptoms in the following wave, adjusted for the health outcome at the first wave, age, educational level, working hours and hours worked from home. Results: In females, a statistically significant association was found between an increase in physical work demands compared to no change and musculoskeletal symptoms (OR 1.39, 95% CI 1.17–1.65). A decrease in physical work demands in females was not statistically significantly associated with musculoskeletal symptoms (OR 0.93, 95% CI 0.80–1.08). Similar trends were found for poor self-rated health, although non-statistically significant. For males, comparable but attenuated associations were found. Conclusion: While our study showed that increasing physical work demands are associated with adverse health (self-reported and musculoskeletal), it did not appear to benefit worker’s health to reduce work demands. Future research with multiple measurements in a shorter period and additionally using devices to measure physical work demands will be needed to confirm our study results.