The Actively Open-minded Thinking scale (AOT; Stanovich & West, 2007) is a questionnaire that is used to measure the disposition towards rational thinking as a single psychological trait. Yet, despite its frequent use, also in abbreviated form, it is still unclear whether sumscores of the AOT can actually be used to order individuals on their disposition towards actively open-minded thinking and whether the questionnaire can be validly shortened. The present study aimed to obtain a valid and shorter AOT. We conducted Mokken scale analyses on the (Dutch) AOT using two samples of higher education students (N = 930; N = 509). Our analyses showed that none of the 41 items could discriminate sufficiently between respondents with varying latent trait levels. Furthermore, no item-set of the AOT could be obtained to validly order individuals on the assumed latent trait, which is a crucial assumption when using it in research. Consequently, it is questionable whether scores on the AOT provide insights into the concept it aims to measure.