Vaccine-induced protection against severe COVID-19, hospitalization, and death is of the utmost importance, especially in the elderly. However, limited data are available on humoral immune responses following COVID-19 vaccination in the general population across a broad age range. We performed an integrated analysis of the effect of age, sex, and prior SARS-CoV-2 infection on Spike S1-specific (S1) IgG concentrations up to three months post-BNT162b2 (Pfizer/BioNTech; Comirnaty) vaccination. In total, 1735 persons, eligible for COVID-19 vaccination through the national program, were recruited from the general population (12 to 92 years old). Sixty percent were female, and the median vaccination interval was 35 days (interquartile range, IQR: 35–35). All participants had seroconverted to S1 one month after two vaccine doses. S1 IgG was higher in participants with a history of SARS-CoV-2 infection (median: 4535 BAU/mL, IQR: 2341–7205) compared to infection-naive persons (1842 BAU/mL, 1019–3116), p < 0.001. In infection-naive persons, linear mixed effects regression showed a strong negative association between age and S1 IgG (p < 0.001) across the entire age range. Females had higher S1 IgG than males (p < 0.001). In persons with an infection history, age nor sex was associated with S1 IgG concentrations. The lower magnitude of S1 antibodies in older persons following COVID-19 vaccination will affect long-term protection.