Adverse weather conditions greatly reduce crop yields, leading to economic losses and lower food availability. The characterization of adverse weather and the quantification of their potential impact on arable farming is necessary to advise farmers on feasible and effective adaptation strategies and to support decision making in the agriculture sector. This research aims to analyze the impact of adverse weather on the yield of winter wheat, grain maize and late potato using a yield gap approach. A time-series analysis was performed to identify the relationship between (agro-)meteorological indicators and crop yields and yield gaps in Flanders (northern Belgium) based on 10 years of field trial and weather data. Indicators were calculated for different crop growth stages and multiple soils. Indicators related to high temperature, water deficit and water excess were analyzed, as the occurrence frequency and intensity of these weather events will most likely increase by 2030–2050. The concept of “yield gap” was used to analyze the effects of adverse weather in relation to other yield-reducing factors such as suboptimal management practices. Winter wheat preferred higher temperatures during grain filling and was negatively affected by wet conditions throughout the growing season. Maize was especially vulnerable to drought throughout the growing season. Potato was more affected by heat and drought stress during tuber bulking and by waterlogging during the early growth stages. The impact of adverse weather on crop yield was influenced by soil type, and optimal management practices mitigated the impact of adverse weather.