Inflammatory bowel disease (IBD) and irritable bowel syndrome (IBS) share common culprit foods and potential pathophysiological factors. However, how diet may contribute to disease course and whether this differs between both entities is unclear. We therefore investigated the association of dietary indices with intestinal inflammation and gastrointestinal symptoms in both IBD and IBS patients. Food frequency questionnaires from 238 IBD, 261 IBS and 195 healthy controls (HC) were available to calculate the overall diet quality by the Dutch Healthy Diet-Index 2015 (DHD-2015) and its inflammatory potential by the Adapted Dietary Inflammatory Index (ADII). Intestinal inflammation and symptoms were evaluated by faecal calprotectin and the Gastrointestinal Symptom Rating Scale, respectively. The DHD-2015 was lower in IBD and IBS versus HC (p < 0.001), being associated with calprotectin levels in IBD (b = −4.009, p = 0.006), and with abdominal pain (b = −0.012, p = 0.023) and reflux syndrome (b = −0.016, p = 0.004) in IBS. ADII scores were comparable between groups and were only associated with abdominal pain in IBD (b = 0.194, p = 0.004). In this side-by-side comparison, we found a lower diet quality that was differentially associated with disease characteristics in IBD versus IBS patients. Longitudinal studies are needed to further investigate the role of dietary factors in the development of flares and predominant symptoms.