Background: Uncertainty exists over the long-term prognostic significance of cerebral small vessel disease (CSVD) in primary intracerebral hemorrhage (ICH). Methods: We performed a longitudinal analysis of CSVD and clinical outcomes in consecutive patients with primary ICH who had MRI. Baseline CSVD load (including white matter hyperintensities [WMH], cerebral microbleeds [CMBs], lacunes, and enlarged perivascular spaces [EPVS]) was evaluated. The cumulative CSVD score was calculated by combining the presence of each CSVD marker (range 0–4). We followed participants for poor functional outcome [modified Rankin scale [mRS] ≥ 4], stroke recurrence, and time-varying survival during a median follow-up of 4.9 [interquartile range [IQR] 3.1–6.0] years. Parsimonious and fuller multivariable logistic regression analysis and Cox-regression analysis were performed to estimate the association of CSVD markers, individually and collectively, with each outcome. Results: A total of 153 patients were included in the analyses. CMBs ≥ 10 [adjusted OR [adOR] 3.252, 95% CI 1.181–8.956, p = 0.023] and periventricular WMH (PWMH) (adOR 2.053, 95% CI 1.220–3.456, p = 0.007) were significantly associated with poor functional outcome. PWMH (adOR 2.908, 95% CI 1.230–6.878, p = 0.015) and lobar CMB severity (adOR 1.811, 95% CI 1.039–3.157, p = 0.036) were associated with stroke recurrence. The cumulative CSVD score was associated with poor functional outcome (adOR 1.460, 95% CI 1.017–2.096) and stroke recurrence (adOR 2.258, 95% CI 1.080–4.723). Death occurred in 36.1% (13/36) of patients with CMBs ≥ 10 compared with 18.8% (22/117) in those with CMB < 10 (adjusted HR 2.669, 95% CI 1.248–5.707, p = 0.011). In addition, the cumulative CSVD score ≥ 2 was associated with a decreased survival rate (adjusted HR 3.140, 95% CI 1.066–9.250, p = 0.038). Conclusions: Severe PWMH, CMB, or cumulative CSVD burden exert important influences on the long-term outcome of ICH.