We assessed the contribution of histopathological features to systemic recurrence (SR) in patients with colorectal cancer, using a case-control design: 71 cases and 184 controls were included, with a mean time until SR of 1.4 ± 0.1 years and a mean follow-up of controls of 1.6 ± 0.06 years. Cases had significantly greater odds of rectal site (odds ratio [OR] = 1.82), stage ≥pT3 (OR = 2.11), suboptimal (<12) lymph node yield (OR = 4.6), stage ≥pN1 (OR = 2.46), KRAS mutation (OR = 2.76), and extramural venous invasion (OR = 1.97). By multiple regression analysis, rectal site, stage ≥pT3, suboptimal lymph node yield, and lymph node positivity independently predicted SR. Rectal cancers were more likely to have a suboptimal node yield than nonrectal cancers (relative risk = 1.6) among the entire cohort. We conclude that rectal cancers have greater risk of SR than colon cancers. A lower yield of lymph nodes in rectal cancer specimens may contribute to this.