Physical inactivity is a major public health problem, and there are concerns this might have increased during the COVID-19 pandemic. We aimed to identify distinct trajectories of physical activity over a 6-week period after the first restrictive measures and to explore determinants of these trajectories in a population-based cohort of middle-aged and elderly in the Netherlands (n = 5777). We observed that at least 59% of participants did not meet the World Health Organization recommendations for physical activity. Using latent class trajectory analyses over three time points, we identified five distinct trajectories, including four steady trajectories at different levels (very low, low, medium and high) and one increasing trajectory. Using multinomial logistic regression analyses, we observed that, compared to the ‘steadily high’ trajectory, participants in the ‘steadily very low’ trajectory were more often older, lower educated, reporting poorer physical health, more depressive symptoms, consuming a less healthy diet, smoking, and lower alcohol use, and were less often retired. A similar pattern of determinants was seen for those in the increasing trajectory, albeit with smaller effect sizes. Concluding, we observed low levels of physical activity that generally remained during the pandemic. The determinants we described can help identify groups that require additional preventive interventions.