There is increasing evidence that people’s walking behavior is linked to the physical environment to which they are exposed. However, whether this association is moderated through local weather conditions is largely unclear. Based on Dutch National Travel Survey data, we applied latent class analysis to determine co-occurring weather conditions and used Tobit regression model to examine whether and, if so, to what extent weather conditions moderated the associations between the physical environment and different walking patterns on weekdays and weekends. We found that in warm and dry weather conditions on weekdays, people living nearby bus stops were more likely to walk to public transit. However, the same correlation was not found on weekends. In inclement weather (i.e., warm, very windy, rainy), people were less likely to walk to public transit. In warm, windy, and slightly rainy conditions, address density was negatively related to recreational walking. On weekends, people were more likely to walk for leisure outside the residential neighborhood (e.g., large open spaces, parks) in warm and dry conditions. Our results indicate that the combined impact of weather and the physical environment needs to be taken into account in walking infrastructure planning.