Climate services, and research on climate services, have mutually developed over the past 20 years, with quality assessment a central issue for orienting both practitioners and researchers. However, quality assessment is becoming more complex as the field evolves, the range and types of climate services expands, and there is an increasing appeal to co-production of climate services. Scholars describe climate services as emerging from complex knowledge systems, where information moves through institutions and actors attribute various qualities to these services. Seeing climate services' qualities as derived from and activated in knowledge systems, we argue for comprehensive assessment conducted with an extended peer community of actors from the system; co-evaluation. Drawing inspiration from Knowledge Quality Assessment and post-normal science traditions, we develop the Co-QA assessment framework; a checklist-based framework for the co-creation of criteria to assess the quality of climate services. The Co-QA framework is a deliberation support tool for critical dialogue on the quality of climate services within a co-construction collective. It provides a novel, structured, and comprehensive way to engage an extended peer community in the process of quality assessment of climate services. We demonstrate how we tested the Co-QA—through interviews, focus groups and desktop research—in two co-production processes of innovative climate services; an ex post evaluation of the “Klimathon” in Bergen, Norway, and an ex ante evaluation for designing place-based climate services in Dordrecht, the Netherlands. These cases reveal the challenges of assessing climate services in complex knowledge systems, where many concerns cannot be captured in straight-forward metrics. And they show the utility of the Co-QA in facilitating co-evaluation.