This randomized phase I trial studies the side effects and best dose of decitabine when given together with filgrastim, cladribine, cytarabine, and mitoxantrone hydrochloride in treating patients with acute myeloid leukemia or myelodysplastic syndrome that is newly diagnosed, has come back, or has not responded to treatment. Drugs used in chemotherapy, such as decitabine, cladribine, cytarabine, and mitoxantrone hydrochloride work in different ways to stop the growth of cancer cells, either by killing the cells, by stopping them from dividing, or by stopping them from spreading. Colony-stimulating factors, such as filgrastim, may increase the production of blood cells and may help the immune system recover from the side effects of chemotherapy. Decitabine, filgrastim, cladribine, cytarabine, and mitoxantrone hydrochloride may work better in treating patients with acute myeloid leukemia and myelodysplastic syndrome.