Abstract Checklists for evaluation capacity builders include a wide range of building blocks for supporting evaluation activity. Yet, the relative importance of each building block is not clear. The purpose of this article is to identify the capacity related factors that are necessary, but not necessarily sufficient, for organisations that wish to institutionalise high quality policy evaluations. To retrieve these factors, we rely on the necessity function in qualitative comparative analysis. We present a study of twenty-seven organisations of the Flemish public sector (Belgium), in which the introduction of policy evaluations is relatively recent. Our case analysis thus sheds an interesting light upon how policy evaluation, and the underlying capacity to evaluate is given shape. Our findings point at evaluation demand as the most necessary prerequisite for fostering evaluation activity, more so than supply related factors.