Numerous neighborhood environments have been recognized to affect mental health, but only a few longitudinal studies investigated these associations jointly and whether different population groups are affected differently. We used three-wave panel data of 2699 adults between 2010 and 2016 in the Netherlands to assess the associations between changes in neighborhood physical and social environments and mental health changes. Further, we assessed possible effect modification of gender and income. Mental health was measured using the Mental Health Inventory. Time-varying exposure to green space, blue space, population density, air pollution, socioeconomic deprivation, and social fragmentation were assigned based on individuals' neighborhood histories. Fixed-effect regressions were conducted to assess within-person associations between single and multiple exposures on mental health for the entire sample and stratified by gender and income. Our single-exposure models showed that increases in blue space were significantly associated with mental health improvements, while increases in fine particulate matter (PM2.5) resulted in declines in mental health. These associations were not attenuated in the multi-exposure model. We observed no significant associations for the remaining environments. Stratification analyses showed that females’ mental health further declined as PM2.5 concentrations increased compared to males. Increasing levels of socioeconomic deprivation were associated with further declines in mental health among the less well-off compared with higher-income earners. Our longitudinal findings suggested that neighborhood physical and social environment changes were associated with mental health changes. Future research is required to establish the underlying pathways.