This study examined associations between neighborhood social factors and physical activity (PA) and sedentary behavior (SB) in older adults. Furthermore, possible moderating effects of neighborhood walkability were explored. Data from 431 community-dwelling Belgian older adults (≥65 years) were analyzed. Neighborhood social factors included measures of neighboring, social trust and cohesion and social diversity. Neighborhood walkability was measured objectively. Outcome measures were self-reported weekly minutes of domain-specific walking and TV viewing, and accelerometer-assessed weekly minutes of moderate-to-vigorous physical activity (MVPA) and overall SB. A higher frequency of talking to neighbors was associated with higher levels of self-reported walking for transport and for recreation. Moderation analyses showed that only in highly-walkable neighborhoods, higher social diversity of the neighborhood environment was associated with more transport walking; and talking to neighbors and social interactions among neighbors were negatively associated with overall SB and television viewing, respectively. Findings suggest that a combination of a favorable neighborhood social and physical environment are important to promote older adults’ PA and limit SB.