peer reviewed ; Energy retrofitting of buildings is an unavoidable way to reduce energy use and greenhouse gases emissions. In this paper, we argue that the need to retrofit buildings in a view to energy efficiency cannot occult the intrinsic quality and specificities of patrimonial buildings. Understanding the relation between matter, structure and constructive reality is fundamental to adequately design the options of retrofitting without losing the soul of this heritage. A framework is presented to address this challenge. It deals with four main interrelated categories: reduction of energy consumption in the use phase, life-cycle impact of retrofitting material (including reuse of material, in situ or from previous dismantled buildings), architectural and heritage values (with a particular attention on matter) and return on investment. A particular focus is put on the following architectural issue: what to do when maintaining the original matter of buildings’ facades is not compatible with the necessary intervention to ensure energy reduction and necessary maintenance works? This research was developed during the energy refurbishment of several significant modernist buildings of the University of Liège in Belgium, which are presented as case studies. The retrofitting works are now completed, allowing a critical return on the results, outputs and the framework.