What makes some novels literary? There is little agreement within literary studies on this question. The two main approaches focus either on text-intrinsic factors (e.g., aesthetic, stylistic), or text-extrinsic social factors (e.g., author prestige, critics). Until now, there has not been a comprehensive study taking both text-intrinsic and social factors into account. The project The Riddle of Literary Quality examines both factors by connecting literary texts to the appreciation of those texts: can we identify textual characteristics that are connected to readers’ literary appraisal of texts? In this paper, we describe the development of The National Reader Survey and present some results. The National Reader Survey is a large online survey of Dutch readers with about 14,000 respondents; its purpose is to collect readers’ literary appraisal of texts. We asked readers to rate both read and unread novels on a scale of 1–7 for their literary and overall quality. The agreement amongst respondents on which recent Dutch language novels are of high literary quality and which are not was greater than expected. Motivations respondents gave for their ratings show that the notion of literary quality is a familiar one and respondents most commonly relate it to two elements: the first is the text itself—style, structure, plot and layers; and the second is genre—if a novel is considered a ‘genre’ novel (e.g., suspense, romantic, fantasy), its chances of obtaining a high rating on literary quality are small. These results indicate how entwined social and text-intrinsic factors are. We also touch upon project results which make use of the survey data