We conceptualize objects based on sensorimotor information gleaned from real-world experience. To what extent is conceptual information structured according to higher-level linguistic features? We investigate whether classifiers, a grammatical category, shape the conceptual representations of objects. In three experiments native Mandarin speakers (a classifier language) and native Dutch speakers (a language without classifiers) judged the similarity of a target object with four objects (presented as words or pictures). One object shared a classifier with the target, the other objects did not. Overall, the target object was judged as more similar to the object with the shared classifier than distractor objects in both Dutch and Mandarin speakers, with no difference between the two languages. Thus, even speakers of a non-classifier language are sensitive to object similarities underlying classifier systems, and using a classifier system does not exaggerate these similarities. This suggests that classifier systems reflect, rather than affect, conceptual structure.