Evidence independently links smoking, family history, and gender with increased risk of adenomatous polyps. Using data from the New Hampshire Colonoscopy Registry (2004–2006), we examined the relation of combined risk factors with adenoma occurrence in 5,395 individuals undergoing screening colonoscopy. Self-reported data on smoking, family history and other factors were linked to pathology reports identifying adenomatous polyps and modeled with multiple logistic regression. In adjusted models a >15 pack-year smoking history increased the likelihood of an adenoma (OR = 1.54, 95% CI 1.28–1.86), although ≤15 pack-years did not (OR = 1.07, 95% CI 0.87–1.32). Gender-stratified models showed a significantly increased risk of adenoma at lower smoking exposure even for men (OR = 1.32; 95% CI:1.00–1.76), but not for women (OR = 0.85; 95% CI:0.61–1.14). An ordered logistic regression model of adenoma occurrence showed a smoking history of ≥15 pack-years associated with 61% higher odds of adenoma at successively larger size categories (95% CI 1.34–1.93). For individuals with a family history of colorectal cancer, smoking does not further increase the risk of adenomas. Smoking duration is linked to occurrence and size of adenoma, especially for men.