This study explores the difficulties in distinguishing different lexical tone contrasts at both sub-lexical and lexical levels for beginning and advanced Dutch learners of Mandarin, using a sequence-recall task and an auditory lexical decision task. In both tasks, the Tone 2-Tone 3 contrast is most prone to errors for both groups of learners. A significant improvement in the advanced group was found for this tone contrast in the sub-lexical sequence recall task, but not in the lexical decision task. This is taken as evidence that utilizing tones in on-line spoken word recognition is more complex and demanding for L2 learners than in a memory-based task. The results of the lexical decision task also revealed that advanced learners have developed a stronger sensitivity to Tone 1 compared to the other three tones, with Tone 4 showing the least sensitivity. These findings suggest different levels of robustness and distinctiveness for the representation of different lexical tones in L2 learners’ lexicon and consequently different levels of proficiency in integrating tones for lexical processing. The observed patterns of difficulty are potentially related to the acoustic characteristics of different lexical tone contrasts as well as to the interference of the suprasegmental features of learner’s native language (i.e., the tonal contrasts of Dutch intonation) on the acquisition of the Mandarin lexical tone contrasts.