Albus didn’t like that Tom’s and his side were shrinking. He needed Voldemort to have a great force for a great battle, but now the force was so small that not even a rebellion could be made. With all the anti-creature laws gone, the forces of Voldemort shrank dramatically and even on his side, there were less creatures siding with his side apart from Remus.