The purpose of this study is to assess the safety and efficacy of avelumab in patients with metastatic neuroendocrine-like prostate cancer. Eligible men will be started on avelumab every 2 weeks and will stay on therapy until progression or intolerable side effects. The central hypothesis is that PD-L1 inhibition with avelumab will be efficacious based on radiographic responses in a subset of men with metastatic neuroendocrine-like prostate cancer and be reasonably well tolerated, meeting criteria for further study in larger phase 2 and 3 trials based on meeting pre-specified efficacy rates and prolonged PFS in some men.