Abstract Background In out-of-hours primary care (OHS-PC), semi-automatic decision support tools are often used during telephone triage. In the Netherlands, the Netherlands Triage Standard (NTS) is used. The NTS is mainly expert-based and evidence on the diagnostic accuracy of the NTS’ urgency allocation against clinically relevant outcomes for patients calling with shortness of breath (SOB) is lacking. Methods We included data from adults (≥18 years) who contacted two large Dutch OHS-PC centres for SOB between 1 September 2020 and 31 August 2021 and whose follow-up data about final diagnosis could be retrieved from their own general practitioner (GP). The diagnostic accuracy (sensitivity and specificity with corresponding 95% confidence intervals (CI)) of the NTS’ urgency levels (high (U1/U2) versus low (U3/U4/U5) and ‘final’ urgency levels (including overruling of the urgency by triage nurses or supervising general practitioners (GPs)) was determined with life-threatening events (LTEs) as the reference. LTEs included, amongst others, acute coronary syndrome, pulmonary embolism, acute heart failure and severe pneumonia. Results Out of 2012 eligible triage calls, we could include 1833 adults with SOB who called the OHS-PC, mean age 53.3 (SD 21.5) years, 55.5% female, and 16.6% showed to have had a LTE. Most often severe COVID-19 infection (6.0%), acute heart failure (2.6%), severe COPD exacerbation (2.1%) or severe pneumonia (1.9%). The NTS urgency level had a sensitivity of 0.56 (95% CI 0.50–0.61) and specificity of 0.61 (95% CI 0.58–0.63). Overruling of the NTS’ urgency allocation by triage nurses and/or supervising GPs did not impact sensitivity (0.56 vs. 0.54, p = 0.458) but slightly improved specificity (0.61 vs. 0.65, p < 0.001). Conclusions The semi-automatic decision support tool NTS performs poorly with respect to safety (sensitivity) and efficiency (specificity) of urgency allocation in adults calling Dutch OHS-PC with SOB. There is room for improvement of telephone triage in patients calling OHS-PC ...