This research investigated whether and to what extent Dutch society is polarized in its attitudes towards refugees and migrants. We further investigated what factors were linked to polarization by testing an integrated three-dimensional model. A latent profile analysis of a representative Dutch sample (N = 1897) suggested that society is indeed polarized, with two substantial groups—anti (16.5%) and pro (18.7%)—at opposite ends. The middle was also divided into people with critical (31.8%) and lenient (33%) attitudes. The three dimensions of our model, (1) the individual and social self (education, political orientation, relative deprivation), (2) perceptions and experience regarding refugees and migrants (perceived cultural distance and contact), and (3) the societal context (societal discontent), were all significantly associated with polarized attitudes. Interestingly, perceived cultural distance from Islam was most strongly linked to polarized attitudes. Organizing different factors into an integrated model offers new insights into the complexity of polarization in society.