Hair color is one of the most visible and heritable traits in humans. Here, we estimated heritability by structural equation modeling (N = 20,142), and performed a genome wide association (GWA) analysis (N = 7091) and a GCTA study (N = 3340) on hair color within a large cohort of twins, their parents and siblings from the Netherlands Twin Register (NTR). Self-reported hair color was analyzed as five binary phenotypes, namely “blond versus non-blond”, “red versus non-red”, “brown versus non-brown”, “black versus non-black”, and “light versus dark”. The broad-sense heritability of hair color was estimated between 73% and 99% and the genetic component included non-additive genetic variance. Assortative mating for hair color was significant, except for red and black hair color. From GCTA analyses, at most 24.6% of the additive genetic variance in hair color was explained by 1000G well-imputed SNPs. Genome-wide association analysis for each hair color showed that SNPs in the MC1R region were significantly associated with red, brown and black hair, and also with light versus dark hair color. Five other known genes (HERC2, TPCN2, SLC24A4, IRF4, and KITLG) gave genome-wide significant hits for blond, brown and light versus dark hair color. We did not find and replicate any new loci for hair color.