Abstract Background Physical distancing is an effective preventative measure during respiratory infectious disease outbreaks. Prior studies on distancing behaviors have largely ignored context characteristics (physical, social) and time. Purpose We investigated patterns in physical distancing over time and across situations, as well as sociodemographic variation herein. Methods We employed data from five rounds of a cohort study conducted throughout the pandemic by the Dutch public health institute (RIVM; N ≈ 50.000 per round). We conducted Latent Class Analyses to investigate patterns of physical distancing in a range of situations, followed by regression models to investigate associations between distancing behavior and sociodemographic and context characteristics. Results Participants differed in their general tendency to adhere to distancing guidelines across situations, but there were also substantial differences in distancing behavior between situations, particularly at work, with friends and family and outdoors. Distancing at work was strongly associated with work environment characteristics. Younger age groups reported less distancing behavior, particularly with close relations (friends or family) and at work. In periods when the pandemic situation was most severe, people adhered more strongly to distancing guidelines and age differences were most pronounced during these periods. Conclusions Physical and social context matters for physical distancing, highlighting the importance of developing strategies for pandemic preparedness that improve opportunities for physical distancing (e.g., reducing crowding, one-way traffic) and accommodate young people to safely meet even in times of high pandemic severity and lockdowns. Future studies should account for the physical and social context in which distancing behavior is observed.