This phase I trial studies the side effects and best dose of trametinib and TAS-102 in treating patients with colon or rectal cancer that has spread to other places in the body or cannot be removed by surgery. Trametinib may stop the growth of tumor cells by blocking some of the enzymes needed for cell growth. Drugs used in chemotherapy, such as TAS-102, work in different ways to stop the growth of tumor cells, either by killing the cells, by stopping them from dividing, or by stopping them from spreading. Giving trametinib and TAS-102 may prevent cancer cells from dividing and work better in treating patients with colon and rectal cancer.