Abstract Background Early pre-hospital identification of critically ill patients reduces morbidity and mortality. To identify critically ill non-traumatic and non-cardiac arrest patients, a pre-hospital risk stratification tool was previously developed in the United States. The aim of this study was to investigate the accuracy of this tool in a Dutch Emergency Department. Methods This retrospective study included all patients of 18 years and older transported by ambulance to the Emergency Department of a tertiary referral hospital between January 1st 2017 and December 31st 2017. Documentation of pre-hospital vital parameters had to be available. The tool included a full set of vital parameters, which were categorized by predetermined thresholds. Study outcome was the accuracy of the tool in predicting critical illness, defined as admittance to the Intensive Care Unit for delivery of vital organ support or death within 28 days. Accuracy of the risk stratification tool was measured with the Area Under the Receiver Operating Characteristics (AUROC) curve. Results Nearly 3000 patients were included in the study, of whom 356 patients (12.2%) developed critical illness. We observed moderate discrimination of the pre-hospital risk score with an AUROC of 0.74 (95%-CI 0.71–0.77). Using a threshold of 3 to identify critical illness, we observed a sensitivity of 45.0% (95%-CI 44.8–45.2) and a specificity of 86.0% (95%-CI 85.9–86.0). Conclusion These data show that this pre-hospital risk stratification tool is a moderately effective tool to predict which patients are likely to become critically ill in a Dutch non-trauma and non-cardiac arrest population.