Background Systemic rheumatic diseases are characterized by diverse symptoms that are exacerbated by stressors. Questions/Purposes Our goal was to identify COVID-19-related stressors that patients associated with worsening rheumatic disease symptoms. Methods With approval of their rheumatologists, patients at an academic medical center were interviewed with open-ended questions about the impact of COVID-19 on daily life. Responses were analyzed with qualitative methods using grounded theory and a comparative analytic approach to generate categories of stressors. Results Of 112 patients enrolled (mean age 50 years, 86% women, 34% non-white or Latino, 30% with lupus, 26% with rheumatoid arthritis), 2 patients had SARS-CoV-2 infection. Patients reported that coping with challenges due to the pandemic both directly and indirectly worsened their rheumatic disease symptoms. Categories associated with direct effects were increased fatigue (i.e., from multitasking, physical work, and taking precautions to avoid infection) and worsening musculoskeletal and cognitive function. Categories associated with indirect effects were psychological worry (i.e., about contracting SARS-COV-2, altering medications, impact on family, and impact on job and finances) and psychological stress (i.e., at work, at home, from non-routine family responsibilities, about uncertainty related to SARS-CoV-2, and from the media). Patients often reported several effects coalesced in causing more rheumatic disease symptoms. Conclusion Coping with the COVID-19 pandemic was associated with rheumatic disease–related physical and psychological effects, even among patients not infected with SARS-CoV-2. According to patients, these effects adversely impacted their rheumatic diseases. Clinicians will need to ascertain the long-term sequelae of these effects and determine what therapeutic and psychological interventions are indicated. Electronic supplementary material The online version of this article (10.1007/s11420-020-09798-w) contains supplementary material, which is available to authorized users.