it was at this point that the passengers , who had spent the previous days meandering round the decks like a restless swarm , retired , at first one by one , then in greater numbers , to their bunks .