This phase I trial studies the side effects and the best dose of sirolimus when given together with vismodegib in treating patients with solid tumors or pancreatic cancer that is metastatic or cannot be removed by surgery. Sirolimus and vismodegib may stop the growth of tumor cells by blocking some of the enzymes needed for cell growth