Previous studies have shown that listeners of different language backgrounds attend to different pitch dimensions when perceiving non-native tones by acoustic approaches. The present study investigated which dimensions of pitch, namely, pitch contour, pitch level, together with the position where it occurs in a word, influenced non-native tone perception by tone language (Mandarin) and non-tone language (Dutch) listeners at the phonological level. A sequence-recall task with memory load and high phonetic variability was applied in the study. Language specific perceptual patterns were found for the two groups. Mandarin listeners outperformed Dutch listeners on encoding non-native pitch contour and pitch level contrasts on each position. Overall Mandarin listeners‟ perception of non-native tones was independent on the position. However, they needed contextual tonal references when encoding non-native pitch level contrasts. Dutch listeners showed perceptual difficulties in encoding pitch contrasts phonologically and were found partially “tone deaf” due to the lack of representations of contrastive tonal categories in their native language. They showed an overall preference for the word final position than word initial and word middle position when perceiving non-native tones.