Mathematics skills are the fundament of modern societies, especially those based on a knowledge-economy. The age of digitalization renders mathematics education even more crucial since it builds the starting point for all STEM-related fields. Consequently, mathematics is at the core of numerous educational Large-Scale Assessments on international (e.g. PISA, TIMSS) or national level (e.g. NAEP, NEPS, SNSA). Although the underlying test development frameworks are most often multi-dimensional or hierarchical, psychometric analyses usually focus on a single latent factor that represents a rather vague general mathematical ability. How and to what extent this simplification affects educational studies that rely on these data remains unclear. The present study takes Luxembourg’s national school monitoring program ÉpStan as example to tackle this question and clarify the consequences. ÉpStan’s mathematics test is conducted annually in elementary school Grades 1, 3, and 5 and is comprised of around 50 to 70 items. Since ÉpStan captures competencies of all students biyearly, each analysis will be based on the full cohort (n > 5000). First, we will investigate whether the curriculum-based test framework for mathematics can psychometrically be represented in a related (multi-dimensional) confirmatory factor model including the domains numbers & operations and space & form. This will be done in Grades 1, 3, and 5. Second, we will study the factor model’s cross-sectional stability within each Grade (over three consecutive years) and longitudinal stability between Grades. Finally, we will study the factors’ relations to students’ cognitive and sociodemographic characteristics and compare the results with correlations found using the most widely used one-dimensional model of mathematical abilities. Based on the results, we will discuss implications not only for educational studies that often uncritically make use of large-scale assessment data, but also highlight the consequences for group-level feedback that is ...