This longitudinal observational study evaluated the humoral immune response to SARS-CoV-2 vaccines in adults with immune-mediated rheumatic diseases (IMRDs) compared to healthy controls, 1–20 weeks after receiving a fourth vaccine dose. Neutralizing antibody titers against the Wuhan and Omicron variants were measured in 341 individuals (218 IMRD patients and 123 healthy controls). Results showed significantly lower antibody responses in IMRD patients, particularly against Omicron (p = 0.0015), and further decline when sample collection occurred more than 35 days post-vaccination. Vaccine type, dosage, timing, and immunosuppressive treatment were all influential factors. These findings underscore the variability in vaccine-induced immunity among IMRD patients.