When COVID-19 vaccines were implemented, nursing home residents (NHRs) and staff (NHS) in Belgium were prioritized for vaccination. To characterize the vaccine response over time in this population and to identify poorly responding groups, we assessed antibody concentrations two (T1), four (T2) and six months (T3) after primary course BNT162b2 vaccination in six groups of infection-naive/infection-primed NHRs/NHS, with/without comorbidity (NHRs only). Participant groups (N = 125 per group) were defined within a national serosurveillance study in nursing homes, based on questionnaire data. Dried blood spots were analyzed using ELISA for the quantification of SARS-CoV-2 S1RBD IgG antibodies. Among all groups, antibody concentrations significantly decreased between T1 and T2/T3, all with a ≥70% decrease at T3, except for infection-primed staff (−32%). Antibody concentrations among infection-naive NHRs were 11.96 times lower than those among infection-primed NHR, while the latter were comparable (x1.05) to infection-primed NHS. The largest proportion [13% (95% CI: 11–24%)] of vaccine non-responders was observed in the group of infection-naive NHRs with comorbidities. A longer interval between infection and vaccination (≥3 months) elicited higher antibody responses. Our data retrospectively show the necessity of timely COVID-19 booster vaccination. Infection-naive NHRs require special attention regarding immune monitoring in future epidemics or pandemics.