Historical censuses have an enormous potential for research. In order to fully use this potential harmonization of these censuses is essential. During the last decades enormous efforts have been undertaken in digitizing the published aggregated outcomes of the Dutch historical censuses (1795-1971). Although the accessibility has been improved enormously, we have to cope with hundreds of heterogeneous and disconnected Excel tables. As a result the census is still for the most part an untapped source of information. We describe the main harmonization challenges of the census and how we work towards one harmonized dataset. We propose a specific approach and model in creating an interlinked census dataset in the Semantic Web using the Resource Description Framework technology.