This randomized phase II trial is studying how well erlotinib hydrochloride works in treating patients with stage I-III colorectal cancer or adenoma. Erlotinib hydrochloride may stop the growth of tumor cells by blocking some of the enzymes needed for cell growth and by blocking blood flow to the tumor. Erlotinib hydrochloride may also stop tumors from growing or coming back