'Urban planning by experiment' can be seen as an approach that uses experimentation to innovate and improve urban planning instruments, approaches, and outcomes. Nowadays, urban experiments - interventions in the city with the aim to innovate, learn, or gain experience - are increasingly taking place in the context of Urban Living Labs. In the Netherlands, a certain type of Urban Living Lab, called city labs, is flourishing, and it has been suggested that these labs could make an important contribution to 'urban planning by experiment.' However, previous studies have indicated that this will depend on how experimentation is conducted in these labs. To obtain a more comprehensive picture of the practice of experimentation, we conducted a survey among Dutch city labs, supplemented by individual and group interviews with practitioners from a small subset of the 17 responding labs. We conclude that there is a poor match between the practice of experimentation in Dutch city labs and the characteristics that are considered to support effective 'urban planning by experiment' (i.e., a structured approach to experimentation, co-creation of experiments, active and targeted dissemination of lessons learned, and experiments as linking pins between municipal policy goals and the needs of urban society). This suggests that the current contribution of Dutch city labs to 'urban planning by experiment' is probably quite limited. Further research is needed to determine whether the typical practice of experimentation encountered in the Dutch city labs, i.e., action-oriented, resource-limited, and largely driven by opportunities, is also found in Urban Living Labs elsewhere.