Health literacy (HL) encompasses someone’s knowledge and abilities to access and use health information in order to make appropriate health decisions in life. HL is particularly valuable in later life when health challenges grow. An individual’s HL is typically considered a fixed and skills-based characteristic, without taking into account how these are situated in the context of everyday life. Also, lay perspectives on health literacy are relatively scarce. Therefore, the aim of this article is to explore the context-specific perspectives of older adults and health professionals on HL in later life in Greece, Hungary, and the Netherlands. We adopted a qualitative methodology and conducted 12 focus groups: seven with 50 older adults and five with 30 health professionals to gain insight into individual perspectives on HL as situated in the health care and everyday life contexts. An informed grounded theory approach was used in analyzing the data. The results are structured in three themes: (1) interactions with health professionals, (2) perceived quality of the health care system, and (3) managing health in the context of everyday life. An overarching finding is that, for older adults, HL reflects the demands placed on them when managing their health. In the experience of older adults, these demands are placed upon them by healthcare professionals, the healthcare system, as well as their everyday lives. Our findings underscore the importance of Critical Health Literacy (CHL) as that concept foregrounds that HL is context specific. Also, CHL has been argued to be a community characteristic, which is why we call for community-based approaches to improve HL.