Abstract Background Clinimetric data for the fire fighting simulation test (FFST), a new test proposed for the Workers' Health Surveillance (WHS) of Dutch fire fighters, were evaluated. Methods Twenty-one fire fighters took the FFST three times with one and three weeks between testing. Clinimetric quality was determined by means of reliability, agreement and validity. For reliability and agreement, the intraclass correlation coefficient (ICC), and standard error of measurement (SEM), were analysed. For construct validity, the tests from 45 fire fighters were correlated with their own and their supervisors' rated work ability. Results The ICCs were 0.56 and 0.79 at the one-week and three-week test-retest periods, respectively. Testing times ranged from 9 to 17 minutes; the SEMs were 70 s at the one-week and 40 s at the three-week test-retest periods. The construct validity was moderate (-0.47 ≤ r ≤ -0.33; p < 0.05). Conclusions The FFST was reliable with acceptable agreement after three weeks. Construct validity was moderate. We recommend using FFST as a part of the WHS for Dutch fire fighters. It is advised that fire fighters should perform the FFST once as a trial before judging their performance in testing time during the second performance.