Social withdrawal is found across neuropsychiatric disorders and in numerous animal species under various conditions. It has substantial impact on the quality of life in patients suffering from neuropsychiatric disorders. Often it occurs prodromal to the disease, suggesting that it is either an early biomarker or central to its etiology. Healthy social functioning is supported by the social brain of which the building blocks go back millions of years, showing overlap between humans, rodents and insects. Thus, to elucidate social withdrawal, we have to approach its environmental triggers and its neural and molecular genetic determinants in an evolutionary context. Pathological social withdrawal may originate from a faulty regulation of specific neural circuits. As there is considerable heritability in social disorders, the genetic building blocks of the social decision making network might be our most relevant target to obtain an understanding of the transition of normal social interaction into social withdrawal.