COVID-19 vaccination is an effective intervention preventing individuals from contracting SARS-CoV-2 or transmitting the virus to others. However, in many countries, vaccine hesitancy has impeded the progress of mass vaccination to reach herd immunity. This study aimed to understand the similarities and differences in the determinants of COVID-19 vaccine hesitancy in Taiwan, the United States, the Netherlands, and Haiti. A qualitative study was conducted by face-to-face interviews with participants in Taiwan and remote online interviews with participants in the United States, the Netherlands, and Haiti. In total, 47 interviews were conducted. A reflective thematic analysis was employed to analyze the collected data. Distrust of COVID-19 vaccines was reported by the participants in all countries. A perception of a lack of necessity or urgency to be vaccinated was reported by the Taiwanese and Haitian participants. Lack of knowledge regarding COVID-19 vaccines was reported by the Taiwanese, U.S. and Haitian participants, contributing to hesitation or refusal to vaccination. Regarding misinformation and rumors, misinformation was found among a few Taiwanese and Dutch participants. Additionally, rumors concerning COVID-19 vaccines were mentioned by the Dutch and Haitian participants. Furthermore, a lack of verified information was reported by the participants in all four countries. Overall, the current study suggests that vaccine hesitancy exists among participants in Taiwan, the United States, the Netherlands, and Haiti. Building trust in the COVID-19 vaccine, cultivating vaccine literacy, clarifying misinformation and rumors concerning COVID-19 vaccines, and providing verified information are critical for increasing public acceptance of the COVID-19 vaccine.