I believe the sequence is designed for pedestrians that are already queued up and ready to cross.  If you are not in the queue than you have to wait your turn because the timing is designed to start at the light change....not 5 or 10 seconds later....that puts the late walker in peril.